# Transformers as memory: the mathematics, from matrices to a unified view Language models write code, translate, answer questions and learn from examples they are shown once. This monograph asks one question about the machine behind them, the transformer: *what is it computing, and why does that work?* Its answer is short. A transformer is two kinds of lookup, one in what it learned and one in what it is reading, and the rest of the note is the mathematics that makes that sentence precise, from a matrix times a vector to a single table that places the MLP, attention and their cheaper cousins in one frame. It is written to be studied in order, with a pencil. Every term is defined where it first appears or in an appendix; every derivation too long for the text is folded away or lives in Appendix D; every figure explains one fact and says in one line what it just showed; and every number is reproduced by a script. The framing of the MLP and of attention as associative memories with a capacity follows Alex Wang's post [*You could have invented transformers from memory*](https://alexwang.ai/posts/associative-memory/); the fast-weight, test-time-learning and kernel-regression threads follow the papers in the reading list. Where a derivation here is my own, it says so. ## Prologue: why transformers, and the one idea behind them ### The problem A language model reads a text and predicts what comes next. Doing that well takes three different things, and they are easy to run together. 1. **Knowledge that is not in the text.** To continue "The capital of France is" the model must know something about the world, and to continue any sentence it must know grammar, style and common sense. None of that is in the prompt; it was absorbed from a huge amount of text during training and has to live in the model's numbers. 2. **Information that is only in the text.** To continue "Alice gave Bob a book. Then Bob thanked" the model must use the names it was just given, which it never saw in training and which will be gone after this prompt. 3. **Reaching far back, cheaply.** The part of the text that matters may be thousands of tokens earlier, and training has to use modern hardware, which is fast at doing many things at once and slow at doing one thing after another. ### What came before recalled Recurrent networks read one token at a time and carry a fixed-size summary, a hidden state, from token to token. Everything that mattered about the past has to be squeezed into that one vector, so early details fade, and because each step needs the previous one the computation cannot be spread across the sequence during training. Convolutional sequence models look at a fixed window and cannot reach beyond it. *Attention* was first added to recurrent translation models so that, at each output step, the decoder could look back over all the encoder's states instead of only the final summary (Bahdanau et al., 2014). ### What the transformer changed cited Vaswani et al. (2017) proposed an architecture "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely". On English-to-German translation (WMT 2014) it reached 28.4 BLEU, more than 2 BLEU above the best earlier results including ensembles, and on English-to-French a single model reached 41.8 BLEU after 3.5 days of training on eight GPUs (abstract). The design lets every position look directly at every other, and computes all positions at once. recalled That is why it could be trained at the scale that followed, and why one architecture, with little change, now underlies language, code, speech and image models. ### What a transformer does, as a picture Each layer of a transformer makes two lookups, and they match two of the three needs above. - **Attention looks in the context.** It finds the earlier tokens that matter for the current position and brings their content forward. This is how the model uses the names, definitions and examples that are in front of it. - **The MLP looks in the weights.** It recalls what training stored: facts, patterns, syntax. This is how the model uses knowledge that is not in the prompt.
One prediction needs both lookups. Click an example: the copper token is where the prediction is made, the blue tokens are what attention reads from the context, and the violet box is what the weights supply. These are illustrations of the two kinds of lookup, not measurements of a particular model.
cited There is evidence for this division of labour in how the pieces are built and what they hold. Geva et al. (2021) find that the feed-forward layers, which contain about two thirds of a transformer's parameters, behave as key-value memories whose keys respond to recognisable text patterns, and Meng et al. (2022) edit a stored fact by changing one MLP matrix. Olsson et al. (2022) describe attention heads that find an earlier occurrence of the current token and copy what followed it, a lookup in the context. ### The idea in one equation Both lookups have the same form. Given a table of pairs $(k_i,v_i)$ of *keys* and *values*, and a *query* $q$ (the question being asked), the answer is a blend of the values, each weighted by how well its key matches the query: $$f(q)=\sum_i v_i\;\sigma(k_i^\top q).$$ If the pairs are the rows and columns of the layer's weight matrices, this is an MLP. If the pairs are made from the tokens of the prompt, it is attention (with a softmax for $\sigma$). The two differ in *who writes the table and how long it lives*: training writes the weights once and they last for the life of the model, while the forward pass writes the context table afresh for every prompt. That is the sense in which the weights are a **slow memory** and the context a **fast memory**, and the note builds the whole theory on this: how such a table stores and recalls (Chapters 1 to 5), how attention is the same thing with a softmax (6 to 8), what happens when the fast memory is made cheap (9 to 11), and one table that puts every variant in the same frame (12 to 15). ### What you gain from seeing it this way | Question | The answer the memory view gives | Chapter | | --- | --- | --- | | Why do the MLPs hold two thirds of the parameters? | They are where the facts are stored, and a memory needs many pairs | 5, 8 | | How many facts can a layer hold? | About its dimension for a plain matrix, far more with a sharper read | 3, 5 | | Why does a long prompt cost so much? | The context memory grows by one pair per token and is read in full | 9 | | Why do cheaper alternatives to attention forget? | A fixed-size memory has a fixed capacity | 3, 10 | | How can a model learn from examples in its prompt? | Reading examples with attention is one gradient step on them | 11 | | What does each attention head learn? | Which tokens to look up and what to carry forward | 7, 11 | | How can one fact be edited? | As a rank-one write into a weight memory | 14 | ### How to read the rest The note moves from the smallest object to the whole machine: a matrix is a memory (Part I and II), attention is a memory read through a softmax (Part III), weights and context are two memories on two clocks (Part IV), and everything fits one table (Part V). If you want the shortest route to the picture, read Chapters 3, 6, 9, 12 and 13 first and return for the proofs. ## Contents
**Start here** - [Prologue: why transformers, and the one idea behind them](#prologue-why-transformers-and-the-one-idea-behind-them): the problem, the history, two lookups, one equation - [0. How to use this note, and a curriculum](#0-how-to-use-this-note-and-a-curriculum): status tags, colour code, notation, reading paths, exit tests **Part I. Matrices** (the objects) - [1. A matrix times a vector](#1-a-matrix-times-a-vector): rows detect, columns carry content - [2. A matrix times a matrix: four readings](#2-a-matrix-times-a-matrix-four-readings): entry, row, column, outer; rank and the SVD **Part II. Memory** (a sum of outer products) - [3. A sum of outer products is a memory](#3-a-sum-of-outer-products-is-a-memory): write, read, interference, capacity - [4. Writing more carefully](#4-writing-more-carefully): least squares, the delta rule, recursive least squares - [5. The MLP is a nonlinear memory](#5-the-mlp-is-a-nonlinear-memory): score functions and their capacities **Part III. Attention** (a memory built from the tokens) - [6. Attention is a soft dictionary](#6-attention-is-a-soft-dictionary): temperature, scaling, kernel smoother, Hopfield - [7. Where the keys, values and queries come from](#7-where-the-keys-values-and-queries-come-from): projections and their gradients - [8. Many heads, and one block](#8-many-heads-and-one-block): heads, residual stream, parameter counts **Part IV. Two timescales** (weights versus context) - [9. Two memories: weights and context](#9-two-memories-weights-and-context): linear attention, cache cost - [10. Updating a fast memory: add, decay, correct](#10-updating-a-fast-memory-add-decay-correct): gated, delta, test-time training - [11. In-context learning as one gradient step](#11-in-context-learning-as-one-gradient-step) **Part V. Unification** - [12. The unified view](#12-the-unified-view): one table, two questions - [13. The right mental models](#13-the-right-mental-models): seven statements, five questions, failure modes - [14. An application: editing one fact in the slow memory](#14-an-application-editing-one-fact-in-the-slow-memory) - [15. A Rosetta stone: one idea under many names](#15-a-rosetta-stone-one-idea-under-many-names) **Back matter** - [16. Open questions](#16-open-questions) · [17. Exercises](#17-exercises) - Appendices: [A. Linear algebra](#appendix-a-linear-algebra-toolkit) · [B. Calculus and probability](#appendix-b-calculus-and-probability-toolkit) · [C. Machine learning](#appendix-c-machine-learning-toolkit) · [D. Derivations](#appendix-d-derivations) · [E. Glossary](#appendix-e-glossary) - [Further reading](#further-reading)
## 0. How to use this note, and a curriculum **Status tags.** A result carries one of three tags, and an untagged sentence is exposition. - derived I derived it in the text or in Appendix D, or checked it numerically; the check is in `scripts/check-attn-notes.cjs` or in the model's own invariants (`AttnMath.checks()`). - cited The statement matches the body of a source I read in full (a paper, or the post), or only its abstract where the text says "abstract". The link is in the reading list. - recalled From memory of the literature. Treat as a pointer to verify, not as a fact. A boxed **check** is a numerical confirmation. A boxed **question** is something I do not yet understand. A boxed **goal** opens each chapter, and a boxed **keep** closes its main idea. A boxed **definition** introduces a term. Folded blocks marked *Derivation* contain the long proofs. **Colour code.** Three roles run through the figures. Blue is a *key*, the thing a memory is addressed by. Violet is a *value*, the thing stored and returned. Copper is the *query*, the question being asked. Inside a grid of numbers a cell is teal when positive and rose when negative, with opacity for magnitude; that pair carries sign only and has nothing to do with the three roles. Every figure ends with a one-line reading of its current state. **Notation.** Vectors are columns. A matrix whose rows are $k_i^\top$ is called $K$ (one memory per row). | Symbol | Meaning | | --- | --- | | $d$ | model width, the size of a token vector $x\in\mathbb R^d$ | | $d_k,d_v$ | key and value dimension of a memory (in one attention head, $d_k=d_v=d/h$ with $h$ heads) | | $m$ | hidden width of an MLP (the number of hidden units) | | $k_i,v_i$ | key and value of memory $i=1,\dots,M$ (or token $i$ in a sequence of length $n$) | | $K,V$ | the $M\times d_k$ and $M\times d_v$ matrices with rows $k_i^\top$, $v_i^\top$ | | $W$ | a memory matrix in $\mathbb R^{d_v\times d_k}$; $W=\sum_i v_ik_i^\top=V^\top K$ | | $q$ | a query; $s_i=k_i^\top q$ the similarity of a key to it, and $\ell_i=\beta s_i$ the logit (Chapter 7) | | $\sigma$ | the score function that turns similarities into weights (ReLU, ReLU², softmax) | | $\beta$ | softmax inverse temperature; attention uses $\beta=1/\sqrt{d_k}$ on raw logits | | $a_i$ | attention weight on memory $i$, $a=\operatorname{softmax}(\beta Kq)$ | | $W_Q,W_K,W_V,W_O$ | projections of attention, $W_Q,W_K\in\mathbb R^{d\times d_k}$, so $q_i=W_Q^\top x_i$ | | $S_t$ | the state of a recurrent (fast) memory after $t$ tokens | | $\eta,\gamma,\beta_t$ | a write rate, a decay, and a data-dependent write strength; $\beta_t$ with a subscript is unrelated to the temperature $\beta$ | | $\mu_k$ | the $k$-th singular value of a matrix | | $\lVert\cdot\rVert_F$ | Frobenius norm of a matrix | **Map.** | Part | Chapters | Question | | --- | --- | --- | | I. Matrices | 1–2 | What do a matrix-vector and a matrix-matrix product mean, and which reading do we use? | | II. Memory | 3–5 | How does a sum of outer products store and recall, and how far does a nonlinearity extend it? | | III. Attention | 6–8 | What is softmax attention as a dictionary, and how are its keys, values and queries learned? | | IV. Two timescales | 9–11 | What is slow memory, what is fast memory, and what other update rules exist? | | V. Unification | 12–15 | One table, two questions, the mental models, an application, and the same idea under other names. | ### A curriculum The note is meant to be read in order, because each chapter's idea is the next one's tool. A suggested schedule, with the test to pass before moving on: | Session | Read | Exit test (answer without looking) | | --- | --- | --- | | 1 | Appendix A, Chapters 1–2 | Compute $Wx$ by rows and by columns; write $AB$ as a sum of outer products; say what rank counts. | | 2 | Appendix B, Chapter 3 | Why does interference grow like $\sqrt M$, and what is the capacity of an outer-product memory? | | 3 | Chapter 4, Derivations D3 and D5 | Derive the least-squares memory; say what the delta rule corrects that Hebbian writing does not. | | 4 | Appendix C, Chapter 5, D1–D2 | Why does squaring a ReLU raise capacity from order $d$ to order $d^2$? | | 5 | Chapter 6, D6–D8 | Draw softmax attention as a soft dictionary; say what $\beta$ trades; state the kernel-smoother view. | | 6 | Chapters 7–8, D9 | Derive $\partial L/\partial v_i$ and $\partial L/\partial\ell_i$; say what a head is and count a layer's parameters. | | 7 | Chapters 9–10, D10 | Derive linear attention as a recurrence; compare add, decay, delta; price the cache. | | 8 | Chapters 11–14 | Show one linear-attention layer is one gradient step; fill in the unified table for a layer you choose. | | 9 | Chapters 15–17, Appendix E | Recite the four questions (state, write, read, forget); do the exercises. | **Reading paths.** *Fast path* (the unified view in an afternoon): Chapters 3, 6, 9, 12, 13. *Mathematical path*: everything, with the folds opened. *Reference use*: the glossary (Appendix E) and the table in Chapter 12. ### The setting: what a transformer is, in one page The chapters that follow take a transformer apart. This page fixes the vocabulary of the whole machine once, so that nothing in them is used before it is defined. Appendix C repeats the definitions in dictionary form. - **Token and vocabulary.** Text is cut into *tokens*, pieces such as words or word fragments, drawn from a fixed list called the *vocabulary*. A *sequence* is $n$ tokens in order. - **Embedding.** A learned table assigns each vocabulary item a vector. The sequence becomes $n$ vectors $x_1,\dots,x_n\in\mathbb R^d$, where $d$ is the model's *width*. Stacked as rows they form $X\in\mathbb R^{n\times d}$. - **Language model, next-token prediction, loss.** A *language model* maps the tokens so far to a probability for every possible next token. It is trained by minimising the *loss* $L$, the average over positions of $-\log p(\text{true next token})$ (the *cross-entropy*). Everything in this note that says "the loss" means this quantity, or any scalar that measures error. - **Logits and softmax.** The last layer turns each position's vector into one raw score per vocabulary item, called a *logit* (it is multiplied by a matrix with one row per vocabulary item, the *unembedding*). The *softmax* turns scores into probabilities. The same two words are reused inside attention for the raw scores and the weights derived from them. - **Autoregressive generation.** To produce text the model predicts one token, appends it, and runs again. - **Layers and the residual stream.** The model is a stack of identical *layers* (also called *blocks*). Each position carries a running vector, the *residual stream*, which starts as the embedding. Each *sublayer* inside a block (an attention sublayer and an MLP sublayer) reads the stream, computes something, and *adds* it back. Chapter 8 draws this. - **Attention in one sentence.** The attention sublayer lets position $t$ read information from other positions: it compares a *query* made from position $t$ with a *key* made from every position, and returns a weighted combination of their *values*. Chapters 6 and 7 derive it. - **MLP in one sentence.** The *multilayer perceptron* sublayer is two matrices with a nonlinearity between them, applied to each position separately (Chapter 5). - **Context, prompt, causal mask.** The *context* (or *prompt*) is the tokens already in the sequence; its length is $n$. In a language model position $t$ must not read positions after it, because they are the answers: the *causal mask* forbids it, so position $t$ reads only positions $i\le t$. - **Position.** Attention by itself treats its inputs as an unordered set (Chapter 6 shows it is unchanged by reordering). Order is put back by *positional information*: position vectors added to the embeddings, or a rotation of queries and keys that depends on position (*rotary embeddings*). The memory picture of this note is unchanged by it, and where it matters the text says so. - **Parameters and training.** The numbers inside the matrices are the *parameters* or *weights*. *Training* adjusts them by gradient descent on the loss; *inference* or *test time* is using the trained model without changing it. The weights are fixed at test time, and that fact is what makes them a *slow* memory (Chapter 9). - **KV cache.** While generating, the keys and values computed for earlier tokens are kept so they are not recomputed. This store grows by one key and one value per token per layer, and it is the *fast* memory of Chapter 9. - **In-context learning.** A trained model improving on a task from examples placed in its prompt, with no change to its weights. ## 1. A matrix times a vector A matrix $W\in\mathbb R^{m\times n}$ applied to $x\in\mathbb R^n$ gives $y=Wx\in\mathbb R^m$. There are two ways to compute it, and the transformer uses both. **By rows.** Write $W$ as a stack of row vectors $w_i^\top$. Then $y_i=w_i^\top x$: each output is the inner product of one row with the input, a measure of how much $x$ *looks like* that row. Rows are **detectors** (templates, keys). **By columns.** Write $W$ as a row of column vectors $c_j$. Then $y=\sum_j x_j\,c_j$: the input's coordinates are *weights* and the output is a weighted sum of the columns. Columns are **contents** (things to be added into the output, values).
The same product, two readings. Drag the first coordinate of $x$: on the left every output (a row against $x$) can change, on the right only the first column is rescaled before the columns are added, and both end at the same $y$.
The two readings give a first picture of a memory. Suppose the rows of $W$ are *addresses* and the columns are *contents*. Feeding in $x$ asks "which addresses does this resemble?" (the row view, $w_i^\top x$) and returns "the contents of those addresses, weighted by the resemblance" (the column view). One matrix does both at once, which is why a matrix can be a memory. The rest of the note makes this precise and asks how many pairs one matrix can hold. A third reading, as a geometric map (stretching, rotating, projecting) is the one linear algebra courses start with. It matters less here: a transformer never needs the geometry of a single $W$, it needs what $W$ *stores* and what it *returns for a query*. ## 2. A matrix times a matrix: four readings For $A\in\mathbb R^{m\times r}$ and $B\in\mathbb R^{r\times n}$, the product $C=AB\in\mathbb R^{m\times n}$ can be read four ways. Write $a_i^\top$ for the rows of $A$, $\alpha_k$ for its columns, $\rho_k^\top$ for the rows of $B$ and $b_j$ for its columns. | Reading | Formula | What it says | | --- | --- | --- | | Entry | $C_{ij}=a_i^\top b_j$ | every entry is one row against one column | | Row | $C_{i\cdot}=\sum_k A_{ik}\,\rho_k^\top$ | row $i$ of $C$ mixes the rows of $B$, with row $i$ of $A$ as the weights | | Column | $C_{\cdot j}=\sum_k B_{kj}\,\alpha_k$ | column $j$ of $C$ mixes the columns of $A$, with column $j$ of $B$ as the weights | | Outer | $C=\sum_{k=1}^r \alpha_k\rho_k^\top$ | $C$ is a sum of $r$ rank-one matrices, one per shared index | derived All four are the same sum $C_{ij}=\sum_kA_{ik}B_{kj}$ grouped differently: fix $(i,j)$ (entry), fix $i$ (row), fix $j$ (column), or fix $k$ (outer). The outer reading is the one a memory needs, and it is the least familiar.
One product, four readings. Click a tab or use the arrow keys: the outlined cells are what each reading multiplies, and the outlined cells of $C$ are what it produces.
A product is a sum of outer products. Click the dots to add the first $m$ rank-one terms $\alpha_k\rho_k^\top$ (left): each is a single column pattern times a single row pattern, and after all four the sum equals $AB$ (right).
**Which reading does the transformer use where?** Attention uses all four. The score matrix $QK^\top$ is the *entry* reading: score $(i,j)$ is query $i$ against key $j$. The output $AV$ is the *row* reading: output $i$ is a mixture of the rows of $V$, weighted by row $i$ of the attention matrix. A memory $W=V^\top K$ is the *outer* reading: it is the sum of one outer product $v_ik_i^\top$ per stored pair. And the *column* reading is how a multi-head layer's output projection recombines heads (Chapter 8). ### Rank is the number of outer products A rank-one matrix $uv^\top$ has every row proportional to $v^\top$ and every column proportional to $u$. The **rank** of a matrix is the smallest number of such terms that sum to it, so a sum of $r$ outer products has rank at most $r$. The singular value decomposition (SVD) writes any matrix as an *ordered* sum $$\begin{gathered} W=\sum_k \mu_k\,u_kv_k^\top, \\ \mu_1\ge\mu_2\ge\dots\ge0, \end{gathered}$$ with orthonormal $u_k$ and $v_k$. Keeping only the first $r$ terms gives the best rank-$r$ approximation in Frobenius norm, and its squared error is exactly the discarded energy $\sum_{k>r}\mu_k^2$ (the Eckart–Young theorem). recalled The theorem is classical and I do not prove it here; the check below verifies the energy identity, and also that none of 300 random rank-3 competitors beats the truncated SVD, which is evidence for optimality and not a proof.
Derivation D4. Eckart–Young *Claim.* Among matrices $B$ of rank at most $r$, $\lVert A-B\rVert_F^2$ is smallest for the truncated SVD, with value $\sum_{k>r}\mu_k^2$. *Proof.* Weyl's inequality for singular values states $\mu_{i+j-1}(X+Y)\le\mu_i(X)+\mu_j(Y)$. Apply it to $X=A-B$, $Y=B$, $i=k$, $j=r+1$: since $\operatorname{rank}B\le r$, $\mu_{r+1}(B)=0$, so $\mu_{k+r}(A)\le\mu_k(A-B)$. Squaring and summing over $k$, $\lVert A-B\rVert_F^2=\sum_k\mu_k(A-B)^2\ge\sum_{k>r}\mu_k(A)^2$, and the truncated SVD attains it. $\square$ (Weyl's inequality is used without proof.)
Most of a structured matrix is in a few outer products. Click a singular-value bar to keep that many terms: three outer products already carry most of the energy of this $8\times8$ pattern.
This is the link to memory. A weight matrix that "knows" $r$ separate facts is a sum of about $r$ outer products, and a matrix of rank $r$ cannot hold more than $r$ independent associations. The next chapter makes "hold" precise. ## 3. A sum of outer products is a memory ### Writing and reading Take $M$ pairs $(k_i,v_i)$ with keys $k_i\in\mathbb R^{d_k}$ and values $v_i\in\mathbb R^{d_v}$. **Write** them by adding one outer product each, $$W=\sum_{i=1}^M v_i\,k_i^\top=V^\top K,$$ and **read** with a query $x$ by multiplying, $$f(x)=Wx=\sum_{i=1}^M v_i\,(k_i^\top x).$$ derived The read is a similarity-weighted sum of the stored values: the column reading of Chapter 1, with the row reading $k_i^\top x$ supplying the weights. Adding a new pair $(k,v)$ is $W\leftarrow W+vk^\top$, a rank-one update. If the keys are **orthonormal**, then $f(k_j)=\sum_iv_i(k_i^\top k_j)=v_j$ exactly, so the memory is perfect, but it can hold at most $d_k$ pairs because $\mathbb R^{d_k}$ has at most $d_k$ orthonormal vectors. With more pairs the keys cannot all be orthogonal, and each recall picks up the values of the others. ### Interference and capacity Read with $x=k_j$ and unit-norm keys: $$f(k_j)=\underbrace{v_j}_{\text{signal}}+\underbrace{\sum_{i\ne j}(k_i^\top k_j)\,v_i}_{\text{interference}}.$$ Now take the keys to be random unit vectors in $\mathbb R^{d_k}$ and the values independent with zero mean and unit norm. For two independent random unit vectors $E[(k_i^\top k_j)^2]=1/d_k$ (Derivation D1), so the interference is a sum of $M-1$ independent terms of size $\lvert k_i^\top k_j\rvert$ and $$\frac{\lVert\text{interference}\rVert}{\lVert\text{signal}\rVert}\approx\sqrt{\frac{M-1}{d_k}}.$$ derived The interference reaches the size of the signal at $M\approx d_k$. Call that the **capacity** of an outer-product memory. It is a heuristic (random keys, independent values, an interference-to-signal ratio of one as the threshold), and every capacity quoted in this note uses the same convention so that they compare.
Recall is the signal plus a random walk. Drag the number of stored memories: the copper bar (the memory asked for) stays at one while the other bars, the weights that leak other values into the answer, grow in number and in total size.
### Where this comes from in training A reminder of the training vocabulary (Appendix C): a *loss* $L$ measures error, *gradient descent* steps against its gradient, and the *error signal* of a layer is the gradient of the loss with respect to that layer's output. The outer-product write is not an exotic operation; gradient descent does it. (Derivation D14 gives the gradient formula.) For a linear layer $y=Wx$ with loss $L$ and error signal $\bar y=\partial L/\partial y$, the gradient is $\partial L/\partial W=\bar y\,x^\top$, so a step is $$W\leftarrow W-\eta\,\bar y\,x^\top,$$ which writes a memory with key $x$ and value $-\eta\bar y$. derived Plain SGD on a linear layer is a sequence of outer-product writes, with the training inputs as keys and the negative error signals as values (the model invariants verify the gradient formula by finite differences). The simplest write rule, adding $vk^\top$ for each pair with no correction, is called **Hebbian** after the classical idea that things seen together get wired together. It is gradient *ascent* on the correlation $v^\top Wk$, because $\partial(v^\top Wk)/\partial W=vk^\top$. derived The next chapter replaces the correlation with an error and gets a better rule. ## 4. Writing more carefully Interference has two causes: keys overlap, and the write ignores what is already stored. The Hebbian write is blind to both. Two better writes fix the second. ### Least squares: the best linear memory Ask for the matrix $W$ that recalls every stored pair as well as possible, $$\min_W\ \sum_{i=1}^M\lVert Wk_i-v_i\rVert^2 .$$ Setting the gradient to zero (the normal equations; Derivation D3) gives, for $M\ge d_k$ with independent keys, $W=V^\top K(K^\top K)^{-1}$, and for $M\le d_k$ with independent keys, $W=V^\top(KK^\top)^{-1}K$ (the minimum-norm solution, since it is not unique when $Mderived Compare the first with the Hebbian $V^\top K$: the optimal store is the outer-product store with the *query* whitened by $(K^\top K)^{-1}$, which undoes the overlap between keys. For $M\le d_k$ the second form gives *exact* recall, because $(KK^\top)^{-1}K$ stacks the **dual basis** $\tilde k_i$ with $\tilde k_i^\top k_j=\delta_{ij}$. For $M>d_k$ the store cannot be exact. Write the recall of key $j$ as $\sum_iP_{ji}v_i$ with $P=K(K^\top K)^{-1}K^\top$, the orthogonal projection onto the span of the columns of $K$. Because $P^2=P$, $\sum_{i\ne j}P_{ji}^2=P_{jj}-P_{jj}^2$, and because $\operatorname{tr}P=d_k$, a typical $P_{jj}\approx d_k/M$. So $$\begin{gathered} \frac{\lVert\text{interference}\rVert}{\lVert\text{signal}\rVert}\approx\sqrt{\frac{P_{jj}-P_{jj}^2}{P_{jj}^2}} \\ =\sqrt{\frac{M}{d_k}-1}. \end{gathered}$$ derived The ratio reaches one at $M=2d_k$: the optimal linear store holds twice what the Hebbian store holds, and the difference from the Hebbian $(M-1)/d_k$ is the interference of $d_k-1$ memories that whitening cancels.
Derivation D3. Least-squares memory and its projection Stack keys and values as $K\in\mathbb R^{M\times d_k}$ and $V\in\mathbb R^{M\times d_v}$. The loss $\sum_i\lVert Wk_i-v_i\rVert^2=\lVert KW^\top-V\rVert_F^2$ has gradient $2K^\top(KW^\top-V)$ with respect to $W^\top$, which vanishes when $K^\top KW^\top=K^\top V$. If $K$ has full column rank ($M\ge d_k$), $W^\top=(K^\top K)^{-1}K^\top V$, that is $W=V^\top K(K^\top K)^{-1}$. If $M\le d_k$ and $K$ has full row rank, the minimum-norm exact solution is $W^\top=K^\top(KK^\top)^{-1}V$, so $KW^\top=V$ and every pair is recalled exactly. For $M>d_k$ the recalled values are $KW^\top=PV$ with $P=K(K^\top K)^{-1}K^\top$, the orthogonal projection onto the column space of $K$. Row $j$ of $PV$ is $\sum_iP_{ji}v_i$, so the signal weight is $P_{jj}$ and the interference weights are $P_{ji}$, $i\ne j$. As $P$ is symmetric and idempotent, $\sum_iP_{ji}^2=(P^2)_{jj}=P_{jj}$, hence $\sum_{i\ne j}P_{ji}^2=P_{jj}-P_{jj}^2$; and $\operatorname{tr}P=d_k$ with $M$ equal diagonal entries on average gives $P_{jj}\approx d_k/M$. For independent unit-norm values the interference norm over the signal is therefore $\sqrt{(P_{jj}-P_{jj}^2)}/P_{jj}\approx\sqrt{M/d_k-1}$. $\square$
### The delta rule: write only what is missing Least squares needs every pair at once. The online version processes one pair $(k,v)$ at a time by gradient descent on $\tfrac12\lVert Wk-v\rVert^2$, whose gradient is $(Wk-v)k^\top$: $$W\leftarrow W+\eta\,(v-Wk)\,k^\top .$$ derived This is the **delta rule** (Widrow–Hoff). recalled Read it as a Hebbian write of the *prediction error* $v-Wk$ instead of the value $v$: what the memory already returns for $k$ is subtracted first, so with a unit-norm key and $\eta=1$ writing the same pair twice changes nothing after the first time. With that choice the pair is stored exactly: afterwards $Wk=v$ (model invariant). Where the old write could only add, this one can replace. **Recursive least squares** (RLS) goes one step further and matches the batch ridge optimum (with penalty $\lambda>0$) at every step. It is the delta rule with a *preconditioner* $R_t=(\lambda I+\sum_{i\le t}k_ik_i^\top)^{-1}$ replacing the raw key: $W\leftarrow W+(v-Wk)\,(R_{t-1}k)^\top/(1+k^\top R_{t-1}k)$, with $R_t$ updated by the Sherman–Morrison formula (Derivation D5; Appendix A states the formula). derived
Derivation D5. Recursive least squares from Sherman–Morrison Let $G_t=\sum_{i\le t}v_ik_i^\top$ and $R_t=(\lambda I+\sum_{i\le t}k_ik_i^\top)^{-1}$, so the ridge memory is $W_t=G_tR_t$. Write $R=R_{t-1}$, $k=k_t$, $v=v_t$. Sherman–Morrison gives $R_t=R-\dfrac{Rkk^\top R}{1+k^\top Rk}$. Then $$G_{t-1}R_t=W_{t-1}-\frac{W_{t-1}k\,k^\top R}{1+k^\top Rk},\qquad vk^\top R_t=\frac{v\,k^\top R}{1+k^\top Rk},$$ and adding them, $W_t=W_{t-1}+\dfrac{(v-W_{t-1}k)\,(Rk)^\top}{1+k^\top Rk}$. $\square$ This is the delta rule with the key replaced by the preconditioned key $Rk$ and the step size by $1/(1+k^\top Rk)$.
A careful write is exact up to $d_k$ memories and then degrades gracefully. Drag the number of stored memories ($d_k=16$, mean over three draws): the outer-product store is wrong from the start, the least-squares store is exact until $M=d_k$.
Three write rules, one ladder: **Hebbian** (add), **delta** (add the error), **RLS / least squares** (add the error in the geometry of the keys seen so far); only the last of these has a capacity derived here ($2d_k$), the delta rule being its online, unpreconditioned approximation. Chapter 10 returns to this ladder for the *fast* memory of a sequence model, where each rung has a name in the architecture literature. ## 5. The MLP is a nonlinear memory A two-layer MLP computes $f(x)=W_2\,\sigma(W_1x)$. Name the rows of $W_1\in\mathbb R^{m\times d}$ as keys $k_i^\top$ and the columns of $W_2\in\mathbb R^{d\times m}$ as values $v_i$. Then $$\begin{gathered} f(x)=\sum_{i=1}^{m} v_i\;\sigma(k_i^\top x) \\ =V^\top\sigma(Kx), \end{gathered}$$ derived which is the outer-product memory of Chapter 3 with a **score function** $\sigma$ applied to each similarity before the weighted sum. The linear memory is the special case $\sigma(s)=s$.
An MLP is a lookup in two matrices. Each hidden unit is a key (blue row) matched against the input, passed through the ReLU, and used as the weight on its value (violet row); the weighted values add up to the output.
The identification is old. cited Sukhbaatar et al. (2015) introduced a recurrent attention model over an external memory trained end to end: input memories are matched against a query by a softmax and output memories are summed with those weights. The post points out that a feed-forward layer has the same form. Geva et al. (2021) make the correspondence explicit, $\mathrm{FF}(x)=f(xK^\top)V$, a key-value memory whose only difference from a memory network is that the coefficients are not normalised, and note that feed-forward layers hold about two thirds of a transformer's parameters (Chapter 8 counts them). In one 16-layer language model they find keys that respond to human-recognisable text patterns (shallow patterns in the lower layers, semantic ones in the upper layers), values whose top token agrees with the token following the key's pattern in at most $3.5\%$ of cases in the upper layers (against $0.0004\%$ by chance), and layer outputs that are *compositions* of many active memories (10% to 50% of the 4096 memories fire on a typical example) rather than single lookups. The reason to apply $\sigma$ is selectivity. A good score function gives the memory asked for a weight near one and every other memory a weight near zero, even when many keys overlap with the query a little. ### Capacity of each score function Take a unit query equal to key 0, and random unit keys for the rest. Each other memory contributes $\sigma(s)\,v_i$ with $s=k_i^\top q$, and for unit vectors in $\mathbb R^{d}$ $$\begin{gathered} E[s^2]=\frac1d,\qquad E[s^4]=\frac{3}{d(d+2)}, \\ E[s^6]=\frac{15}{d(d+2)(d+4)} . \end{gathered}$$
Derivation: the moments of a unit-vector dot product (D2) Write a random unit vector as $k=g/\lVert g\rVert$ with $g$ a standard Gaussian vector. Then $s^2=g_1^2/(g_1^2+\dots+g_d^2)$ follows a Beta$(\tfrac12,\tfrac{d-1}2)$ law, whose $p$-th moment is $\prod_{j=0}^{p-1}\frac{1/2+j}{d/2+j}$. For $p=1,2,3$ this is $\tfrac1d$, $\tfrac{3}{d(d+2)}$, $\tfrac{15}{d(d+2)(d+4)}$. The Beta-moment formula is standard; the numbers are also confirmed by the Monte Carlo below.
derived For $\sigma=\mathrm{ReLU}^p$ the negative half of $s$ contributes nothing and the distribution of $s$ is symmetric, so $E[\sigma(s)^2]=\tfrac12E[s^{2p}]$. The target contributes weight $\sigma(1)=1$, so the interference-to-signal ratio is $\sqrt{(M-1)E[\sigma(s)^2]}$ and reaches one at $$\begin{gathered} M\approx\frac{2}{E[s^{2p}]}, \\ p=1:\ 2d,\qquad p=2:\ \tfrac23d^2, \\ p=3:\ \tfrac{2}{15}d^3 . \end{gathered}$$ cited These capacities and the scores' names (ReLU, squared ReLU, cubed ReLU) are those of the post, which also reports that a squared-ReLU MLP is a gated linear unit with shared gate and input weights; the derivation above is my own. The pattern is that suppressing small similarities (a power) raises capacity as a power of $d$: ReLU only removes negative scores and gains a factor 2 over the linear memory, while squaring turns a similarity of $0.1$ into $0.01$ and capacity grows from order $d$ to order $d^2$.
Sharper scores leak less. Drag the number of stored memories ($d=32$): the same keys feed four panels, each normalised so the target (copper) has height one; the number under each panel is its interference-to-signal ratio.
### Gating, biases and the standard MLP derived Because $\mathrm{ReLU}(s)^2=\mathrm{ReLU}(s)\cdot s$, a squared-ReLU memory is a **gated linear unit** whose gate and input share one weight matrix: the gate $\mathrm{ReLU}(Kx)$ decides how much of the linear read $Kx$ passes through. recalled The widely used SwiGLU MLP gives the gate its own weights and a smooth activation, $f(x)=V^\top(\operatorname{Swish}(Gx)\odot Kx)$, and cited Shazeer (2020) shrinks its hidden size by a factor $\tfrac23$ relative to a two-matrix MLP so that the parameter count stays the same, which for a $4d$ baseline is $\tfrac83d$ (derived the count: $3\cdot d\cdot\tfrac83d=8d^2$, checked in `check-attn-notes.cjs`). Llama 1 follows that rule; later models often use a larger multiplier (Llama 3 8B uses $3.5d$). *Superposition* means storing more features than dimensions by letting their directions overlap; it is exactly the interference of Chapter 3, and a *sparse* layer is one where few hidden units are active for a given input. A bias $b_i$ in $\sigma(k_i^\top x+b_i)$ moves the threshold of key $i$: a negative bias means the key only fires for queries that resemble it more than a baseline, which is how a memory makes each key respond to a narrower region of input space. ### What these numbers do and do not say The capacities count random keys and random values. cited The post notes that the estimate ignores the structure of real data and compares it with empirical estimates: about 3.6 bits per parameter for GPT-style models (Morris et al.) and about 2 bits of factual knowledge per parameter (Allen-Zhu and Li); I have not read those two papers. It also notes (its footnote 3) that $d_k$ is the capacity for recalling a clean value *vector*, and that a classification-style readout can hold about $d^2$ facts in a $d\times d$ matrix (Nichani et al.). Two cautions follow. First, trained MLPs do not use random keys. The post cites an explicit random-feature construction (Garcia et al.) in which a gated MLP stores far more facts than it has hidden units, and a report (Li et al.) that trained ReLU MLPs can activate as few as $1\%$ of their hidden units for a given input, so sparsity and superposition matter in practice; Geva et al. measured 10% to 50% of the memories active in a small language model, so the fraction is model-dependent. Second, capacity is about *recall of arbitrary pairs*. A network that learns a *rule* needs far fewer pairs than the rule has instances. ## 6. Attention is a soft dictionary ### From a dictionary to a soft one A **dictionary** (hash table) holds pairs $(k_i,v_i)$ and answers a query $q$ by returning $v_{i^\star}$ for the key that matches, $i^\star=\arg\max_ik_i^\top q$. That lookup is exact and discontinuous: a slightly different query can return an entirely different value, and the answer has no gradient. A **soft dictionary** returns a convex combination of all the values, with weights that favour the best match: $$\begin{gathered} o=\sum_{i=1}^{M}a_i\,v_i,\qquad a=\operatorname{softmax}(\beta Kq), \\ a_i=\frac{e^{\beta\,k_i^\top q}}{\sum_je^{\beta\,k_j^\top q}} . \end{gathered}$$ derived This is **self-attention** read as one query against $M=n$ stored tokens: the same outer-product memory of Chapter 3 with the score function $\sigma(s)=e^{\beta s}$ and a normalisation so the weights sum to one. Relative to the MLP there are two new ingredients: the exponential score, which the capacity argument below depends on, and the normalisation, which makes the weights compete.
Attention returns a weighted average of values. Drag the query around the circle: the dots near it grow, the answer $o$ moves toward their values, and the bars list each weight $a_i$ beside the value $v_i$ it multiplies.
### Temperature The inverse temperature $\beta$ sets how soft the dictionary is. derived At $\beta=0$ every weight is $1/M$ and the output is the average of all the values. As $\beta\to\infty$ the weights approach a one-hot on the largest similarity and the output is the exact hard lookup (model invariants verify both ends). In between, $\beta$ says how much one similarity point is worth: a weight ratio $a_i/a_j=e^{\beta(s_i-s_j)}$.
Temperature turns an average into a lookup. Drag $\beta$ from $0$ (every memory weighs $1/7$, dashed line) to $14$ (almost all weight on the best match).
Large $\beta$ is not free. A hard argmax has no gradient, and a saturated softmax passes almost none, so $\beta$ is a trade-off between capacity (large) and trainability (small). derived With unit-norm keys the interference of the softmax store is $\sqrt{M}\,e^{\beta^2/d-\beta}$ relative to the signal: the target has weight $e^\beta$, the other $M-1$ memories contribute independent terms with $E[e^{2\beta s}]\approx e^{2\beta^2/d}$ (Gaussian approximation of $s$). Setting this ratio to one gives (Derivation D7) $$M\approx e^{2\beta(1-\beta/d)} .$$ cited The post gives this form and the number $e^{14}\approx1.2\times10^{6}$ for $\beta=\sqrt d=8$ at $d=64$, and the derivation here reproduces it. Compare the polynomial capacities of Chapter 5: the exponential in the score turns capacity from $d^p$ into an exponential of $\beta$.
Derivation D7. Capacity of the softmax memory Query equal to key 0 (unit norm), the other $M-1$ keys random unit vectors with similarities $s_i=k_i^\top q$. The target has weight $e^{\beta}$ and memory $i$ has weight $e^{\beta s_i}$, all divided by the same $Z$ which cancels in the ratio. With independent unit-norm values the interference norm is $\sqrt{\sum_ie^{2\beta s_i}}$ and the signal is $e^\beta$, so the ratio is $\sqrt{M-1}\,\sqrt{E[e^{2\beta s}]}\,e^{-\beta}$. For large $d$, $s\approx\mathcal N(0,1/d)$ and $E[e^{2\beta s}]=e^{2\beta^2/d}$, hence ratio $\approx\sqrt M\,e^{\beta^2/d-\beta}$. It equals one when $\tfrac12\ln M=\beta-\beta^2/d$, i.e. $M=e^{2\beta(1-\beta/d)}$. $\square$ (The Gaussian approximation is asymptotic; Chapter 6 reports its error.)
Each sharper score buys orders of magnitude. Drag the dimension $d$: each curve crosses the dashed line (interference equals signal) at its capacity, marked with a dot and its value; softmax with $\beta=\sqrt d$ leaves the chart for large $d$.
### The standard scaling $1/\sqrt{d_k}$ In a trained transformer the logits are $q^\top k/\sqrt{d_k}$, not $\beta\,q^\top k$ with unit-norm vectors. cited Vaswani et al. (2017) define scaled dot-product attention as $\operatorname{softmax}(QK^\top/\sqrt{d_k})V$. Their footnote gives the variance argument: if the components of $q$ and $k$ are independent with mean 0 and variance 1, then $q^\top k$ has mean 0 and variance $d_k$ (checked here: over 4000 draws at $d_k=64$ the variance is $1.0\times d_k$ within $10\%$). The authors say they *suspect* that for large $d_k$ the dot products grow large and push the softmax into regions with extremely small gradients, and the scaling counteracts that; their evidence is that additive attention beat unscaled dot-product attention for large $d_k$. Dividing by $\sqrt{d_k}$ gives logits of unit variance. cited The post offers a second argument (its footnote 6): in trained transformers $q$ and $k$ have norm about $\sqrt{d_k}$, so a matching pair has $q^\top k\approx d_k$ while unit vectors have similarity $1$; hence the unit-vector memory needs $\beta=\sqrt{d_k}$ to match transformer logits. cited The post adds that a *fixed* $\beta$ is too small for very long contexts: the interference grows like $\sqrt M$ while the target weight is fixed, so $\beta$ must grow with $\log M$. As the post states them, the length-scaling rules used in practice are, with $M_0$ the training length: Su (2021) multiplies the logits by $\log M/\log M_0$ at every length; Scalable-Softmax by $s\log M$ with $s$ learned; Qwen's LogN scaling by $\log M/\log M_0$ only once $M>M_0$; and the YaRN attention scaling used by Qwen2.5-1M by $(0.1\log(M/M_0)+1)^2$. I have checked these forms against the post only, not against the original sources. (The capacity formula suggests why $\beta$ should grow like $\tfrac12\ln M$ for small $\beta/d$.) ### Attention as a matrix, and the three views of $A$ Stack $n$ queries as rows of $Q$. Then $$\begin{gathered} A=\operatorname{softmax}\!\big(\beta\,QK^\top\big)\ \text{(row-wise)}, \\ O=AV . \end{gathered}$$ derived $A$ is $n\times n$ with non-negative rows that sum to one. Its **row** $i$ is the distribution that query $i$ puts on the memories (the row reading of $AV$: output $i$ is a mixture of the rows of $V$). Its **column** $j$ lists how much every query reads memory $j$. Its **outer** reading is $$O=\sum_{j=1}^{n}A_{\cdot j}\,v_j^\top,$$ each token $j$ contributing a rank-one matrix: *who reads me* times *what I hold*. The score matrix $QK^\top$ has rank at most $d_k$, because it is a product of an $n\times d_k$ and a $d_k\times n$ matrix (Chapter 2). The softmax acts on each row and is nonlinear, so $A$ need not be low rank. derived On a random example with $n=10$ and $d_k=2$ the score matrix has rank $2$ and the attention matrix has rank $10$ (check). This is one way to see why the softmax is more than a normaliser: it lifts a rank-$d_k$ similarity into a full-rank mixing pattern. (Rank counts independent rows; the softmax mixes the entries of each row nonlinearly, so the row space is no longer confined to $d_k$ dimensions.)
The softmax lifts a low-rank matrix. Drag $\beta$: the score matrix $QK^\top$ (left) has rank 2, while the attention matrix (right) has full rank for most $\beta$ and sharpens toward one entry per row as $\beta$ grows.
### Softmax attention is a kernel smoother and a Hopfield network Two classical objects are the same computation. **Nadaraya–Watson.** A kernel smoother with kernel $\kappa$ estimates the value at a query as $\sum_i\kappa(q,k_i)v_i/\sum_j\kappa(q,k_j)$. Taking $\kappa(q,k)=e^{\beta q^\top k}$ is exactly softmax attention. derived With unit-norm keys, $e^{\beta q^\top k}\propto e^{-\beta\lVert q-k\rVert^2/2}$ up to factors that do not depend on $k$ and cancel in the normalisation, so attention is a *Gaussian-kernel smoother* with bandwidth $1/\sqrt\beta$ (both identities are in the model invariants; Derivation D19 shows the expansion).
Derivation D19. Softmax attention as a Gaussian-kernel smoother $\lVert q-k\rVert^2=\lVert q\rVert^2+\lVert k\rVert^2-2q^\top k$, so $e^{\beta q^\top k}=e^{\beta(\lVert q\rVert^2+\lVert k\rVert^2)/2}\,e^{-\beta\lVert q-k\rVert^2/2}$. If every key has $\lVert k\rVert=1$ the factor $e^{\beta\lVert k\rVert^2/2}$ is the same for all keys and the factor $e^{\beta\lVert q\rVert^2/2}$ does not depend on the key; both cancel between numerator and denominator of the smoother. What remains is the Gaussian kernel $e^{-\lVert q-k\rVert^2/(2\sigma^2)}$ with $\sigma^2=1/\beta$, bandwidth $\sigma=1/\sqrt\beta$. Without unit-norm keys the identity fails, and an attention head weights long keys more. $\square$
**Modern Hopfield network.** A *Hopfield network* stores patterns as stable states of an energy function and retrieves one by repeatedly updating a corrupted pattern until it stops changing. cited Ramsauer et al. (2020) define a continuous version with energy $E(\xi)=-\beta^{-1}\ln\sum_ie^{\beta x_i^\top\xi}+\tfrac12\xi^\top\xi+\text{const}$, whose update rule $\xi\leftarrow X^\top\operatorname{softmax}(\beta X\xi)$ (with the stored patterns as the rows of $X$) is exactly one attention readout with keys and values equal to the patterns; with $x_i=W_K^\top y_i$, $\xi=W_Q^\top r$ and $\beta=1/\sqrt{d_k}$ it is the transformer's. The update decreases the energy and converges to a fixed point (Appendix D8 shows the fixed-point identity). Three kinds of fixed point exist: a *global* one that averages all patterns, *metastable* ones that average a subset, and single-pattern ones; $\beta$ decides which, which is the Hopfield reading of temperature (Chapter 6). Their capacity result is that for random patterns on a sphere the number storable grows *exponentially in the dimension* (a bound of the form $\sqrt p\,c^{(d-1)/4}$ with probability $1-p$), and retrieval after *one* update is accurate when the patterns are well separated; this exponential in $d$ is a different statement from the post's $e^{2\beta(1-\beta/d)}$, which depends on $\beta$, and the two should not be conflated. In pre-trained BERT they report that first-layer heads tend toward global averaging and higher layers toward partial averaging.
Derivation D8. The Hopfield energy and attention as a fixed point Let $X\in\mathbb R^{M\times d}$ hold stored patterns as rows. Define the energy $E(\xi)=-\tfrac1\beta\ln\sum_ie^{\beta x_i^\top\xi}+\tfrac12\xi^\top\xi$. Its gradient is $\nabla E(\xi)=-X^\top\operatorname{softmax}(\beta X\xi)+\xi$, so a stationary point satisfies $\xi=X^\top\operatorname{softmax}(\beta X\xi)$: the output of one attention readout with keys and values both equal to the patterns and query $\xi$. Iterating that map is the update rule. cited Ramsauer et al. prove that this iteration decreases $E$ and give its retrieval and capacity properties; I give only the fixed-point identity.
The two views agree on the meaning of attention: it *denoises a query toward the stored patterns*. A key that is close to the query wins, and the more different the query is from every stored key, the more the answer is an average. ## 7. Where the keys, values and queries come from In Chapter 6 the keys and values were given. (Convention: the note writes $q_i=W_Q^\top x_i$ for column vectors $x_i$; the papers usually write $q_i=x_iW_Q$ for row vectors, with the same matrices.) In a transformer they are *computed*: from the token vectors $x_i\in\mathbb R^d$ by three learned projections, $$\begin{gathered} q_i=W_Q^\top x_i,\qquad k_i=W_K^\top x_i, \\ v_i=W_V^\top x_i, \end{gathered}$$ with $W_Q,W_K\in\mathbb R^{d\times d_k}$ and $W_V\in\mathbb R^{d\times d_v}$. A head's output is then mapped back to the model width by $W_O\in\mathbb R^{d_v\times d}$. Nothing here says which part of a token is an address and which is a content: the same $x_i$ is projected to a key (how to *find* me) and to a value (what to *give*), and training decides what each means. That is the whole answer to "how are $K$, $V$ and $Q$ learned": the projections are ordinary weights trained by gradient descent on the final loss, and this chapter says what the gradient asks of them. ### Only a product matters The score between a query token $i$ and a key token $j$ is $$\begin{gathered} \ell_{ij}=\beta\,q_i^\top k_j \\ =\beta\;x_i^\top\underbrace{W_QW_K^\top}_{B}\,x_j, \end{gathered}$$ a *bilinear form* in the two tokens. derived For plain dot-product attention (no rotary embeddings, query-key normalisation or biases), only the $d\times d$ matrix $B=W_QW_K^\top$, of rank at most $d_k$, affects the output: replacing $W_Q\to W_QG$ and $W_K\to W_KG^{-\top}$ for any invertible $G$ leaves every score unchanged (Derivation D9; checked numerically, to $10^{-9}$). So there is no canonical "query space" or "key space", only a learned similarity between two tokens. The same holds on the output side: $W_VW_O$ is the only thing that matters, again of rank at most $d_v$. A head is therefore two low-rank maps, one that decides *where to look* and one that decides *what to write*. ### The gradient of attention Consider one query $q$ and the output $o=\sum_ia_iv_i$ with $a=\operatorname{softmax}(\ell)$, $\ell_i=\beta\,q^\top k_i$, and let $g=\partial L/\partial o$ be the error signal arriving from the layers above. The four gradients are $$\begin{gathered} \frac{\partial L}{\partial v_i}=a_i\,g, \\ \frac{\partial L}{\partial \ell_i}=a_i\,(v_i-o)^\top g, \\ \frac{\partial L}{\partial k_i}=\beta\,\frac{\partial L}{\partial \ell_i}\,q, \\ \frac{\partial L}{\partial q}=\beta\sum_i\frac{\partial L}{\partial \ell_i}\,k_i . \end{gathered}$$ derived The first is immediate from $o=\sum a_iv_i$. The others follow from the softmax Jacobian (Derivation D6) $\partial a_i/\partial \ell_j=a_i(\delta_{ij}-a_j)$.
Derivation of $\partial L/\partial \ell_i$ With $L$ depending on $\ell$ only through $o=\sum_ja_jv_j$, $$\begin{gathered} \frac{\partial L}{\partial \ell_i}=g^\top\frac{\partial o}{\partial \ell_i}=g^\top\sum_jv_j\,a_j(\delta_{ij}-a_i) \\ =a_i\Big(g^\top v_i-g^\top\sum_ja_jv_j\Big)=a_i\,(v_i-o)^\top g, \end{gathered}$$ and $\ell_i=\beta q^\top k_i$ gives $\partial \ell_i/\partial k_i=\beta q$ and $\partial \ell_i/\partial q=\beta k_i$.
The gradient has one line per stage. Grey arrows run forward from the query and keys to the loss; copper arrows run back, and each carries the formula for the gradient at that node.
Three readings follow, and each is one of the memory ideas of the earlier chapters. **Values are written in proportion to attention.** The update $v_i\leftarrow v_i-\eta\,a_ig$ moves each value against the error by the weight it received. In terms of the weight matrix, $\partial L/\partial W_V=\sum_ix_i\,(a_ig)^\top$, a sum of outer products with key $x_i$ (the token) and value $a_ig$ (the weighted error): *the value projection is itself an outer-product memory written by the error signal*, the picture of Chapter 3. **Keys move toward the queries they help.** The sign of $\partial L/\partial \ell_i=a_i(v_i-o)^\top g$ says whether raising the logit of key $i$ helps. If $(v_i-o)^\top g<0$, memory $i$ points the output in a direction that reduces the loss relative to the current output $o$, so gradient descent *raises* $\ell_i$: the key $k_i\leftarrow k_i-\eta\beta\,(\partial L/\partial \ell_i)\,q$ rotates toward $q$. Items that are worse than the current answer are pushed away. The same step on the matrices is $\partial L/\partial W_K=\sum_i\beta\,(\partial L/\partial \ell_i)\,x_iq^\top$: again an outer product, whose key is the token features $x_i$ and whose value is the *query*. **It is a policy gradient.** (*Policy gradient* here means the gradient of an expected reward under a softmax distribution over choices, with the choice's reward minus the average reward, the *advantage*, as its weight; Derivation D16 gives the four-line version, and no reinforcement-learning background is needed.) The weights $a$ form a softmax distribution over memories, $-(v_i-o)^\top g$ is the "advantage" of choosing memory $i$ relative to the mean $o=\sum a_jv_j$ (a reward of $-v_i^\top g$ with the mean as baseline), and $\partial L/\partial \ell_i=-a_i\cdot\text{advantage}$ is the REINFORCE gradient of a softmax policy, computed exactly over all memories instead of from samples. derived This is the same mean-baseline algebra as the group-baseline gradient of GRPO in [the RL note](rl.html) (Chapter 18; GRPO also divides by the group's standard deviation and samples the group): attention learns by an advantage-weighted pull on the logits, with the group of memories as the sample.
Training moves the right key toward the query. Drag the number of gradient steps on the keys: the key holding the target value $1.0$ rotates toward the query $q$ and its weight $a_2$ grows while the output $o$ approaches the target. Dot size is attention weight; each key is kept at unit length so it can only rotate.
### What is actually learned The example above treats keys as free parameters; in a model they are $W_K^\top x_i$, so the update acts on $W_K$ and generalises across tokens that share features. What such learning produces in trained models, in particular the *induction heads* that copy earlier tokens, needs the residual stream (Chapter 8) and in-context learning (Chapter 11), and is described at the end of Chapter 11. ## 8. Many heads, and one block ### Heads are a low-rank decomposition With $h$ heads, each has its own $W_Q^{(m)},W_K^{(m)},W_V^{(m)}$ of width $d_k=d/h$, and the heads' outputs are concatenated and passed through one $W_O\in\mathbb R^{d\times d}$. Split $W_O$ into the row-blocks $W_O^{(m)}\in\mathbb R^{d_k\times d}$ that act on each head's slice. Then $$\begin{gathered} \text{output}=\big[\text{head}_1\ \cdots\ \text{head}_h\big]\,W_O \\ =\sum_{m=1}^{h}\text{head}_m\,W_O^{(m)}, \end{gathered}$$ derived a sum of per-head terms (a block-wise outer-product reading of Chapter 2; checked to $10^{-12}$ on a random two-head layer). Each head is an independent dictionary with its own addressing (its own $B_m=W_Q^{(m)}W_K^{(m)\top}$) and its own write (its own $W_V^{(m)}W_O^{(m)}$), and the heads add into the same output. The number of heads does not change the parameter count: $W_Q,W_K,W_V,W_O$ total $4d^2$ for any $h$. More heads means more independent addresses with a smaller key dimension $d/h$ each; fewer means one large address space. cited *Grouped-query attention* (Ainslie et al., 2023) is an interpolation between multi-head attention and multi-query attention: query heads are split into groups and each group shares one key head and one value head, which shrinks the cache (Chapter 9); Llama 3 uses 8 key-value heads against 32 to 128 query heads. ### The block A *layer* (or *block*) is one attention sublayer followed by one MLP sublayer; a *sublayer* is either of the two. A transformer layer adds two sublayers to the residual stream $x$ (the running vector for each position): $$\begin{gathered} x\leftarrow x+\operatorname{Attn}(\operatorname{LN}x), \\ x\leftarrow x+\operatorname{MLP}(\operatorname{LN}x). \end{gathered}$$ Here $\operatorname{LN}$ is layer normalisation (Appendix C), applied *before* each sublayer ("pre-norm"). cited The original transformer (Vaswani et al., 2017) normalises *after* the residual add, $\operatorname{LN}(x+\mathrm{sublayer}(x))$ ("post-norm"), with $d_{\mathrm{ff}}=2048=4\times512$; Xiong et al. (2020) analyse the difference, and many current models use pre-norm. The residual stream is the *bus*: each sublayer reads from it, computes, and *adds* its result back. cited Elhage et al. (2021) describe it as a communication channel that does no processing itself, with all layers communicating through it. In the memory picture, attention reads the fast memory (the other tokens) and the MLP reads the slow memory (its own weights), and each writes what it recalled into the stream, where later layers can read it.
Two lookups per layer, one bus. Each sublayer reads the residual stream (grey arrow), consults a memory, and adds its recall back (coloured arrow). The attention memory is built from the tokens of this sequence; the MLP memory is the layer's weights.
**Parameter count.** With width $d$ and an MLP hidden size $4d$, a layer has $4d^2$ in attention and $2\cdot d\cdot4d=8d^2$ in the MLP, $12d^2$ in all (embeddings, norms and biases ignored). derived The MLP therefore holds two thirds of the layer's parameters, which matches the "about two thirds" of feed-forward parameters that Geva et al. report (cited, abstract). Most of the slow memory is in the MLPs, and most of the machinery for reading the fast memory is in the four attention matrices. ## 9. Two memories: weights and context ### What differs between the two memories The MLP and the attention of one layer have the same read, $\sum_iv_i\,\sigma(k_i^\top q)$ with different $\sigma$, and the difference lies in where $(k_i,v_i)$ come from. | | MLP: slow memory | Attention: fast memory | | --- | --- | --- | | Keys | the rows of $W_1$ | $W_K^\top x_t$ for each token $t$ in context | | Values | the columns of $W_2$ | $W_V^\top x_t$ for each token $t$ in context | | Written by | gradient descent, during training | the forward pass, one pair per token | | Lives | as long as the model | as long as the sequence | | Size | fixed: $m$ pairs per layer | grows: $n$ pairs per layer and head | | Read cost per token | $O(dm)$ | $O(nd)$ | | Typically holds | facts and features of the world | the prompt, the document, the conversation | This is the *program and data* split. The slow weights, including $W_Q,W_K,W_V$, are a **program** that decides how to write the fast memory from the tokens; the context memory is the **data** that program produces. cited The term for this is *fast weight programmer*: Schlag, Irie and Schmidhuber (2021) describe a slow network that learns by gradient descent to program the fast weights of another network, and show that linearised self-attention is formally equivalent, apart from normalisation, to such a programmer that writes additive outer products of keys and values into a finite memory. They also state the capacity limit as an orthogonality bound: with keys in a $d_{\mathrm{dot}}$-dimensional space there cannot be more than $d_{\mathrm{dot}}$ orthogonal vectors, so storing more than $d_{\mathrm{dot}}$ associations causes retrieval errors (in their synthetic task linear attention with $d_{\mathrm{dot}}=64$ starts to err at about 60 associations). ### Linear attention: the fast memory as a recurrent state Two words first. A *recurrent* layer reads one token at a time and carries a fixed-size summary, its *hidden state*, from token to token. A *feature map* $\phi$ is a function applied to keys and queries so that an ordinary inner product $\phi(q)^\top\phi(k)$ approximates a similarity kernel (Chapter 12 develops this; the identity map is enough here). *Causal* attention lets token $t$ read tokens $i\le t$ only (Chapter 0). Drop the softmax: take $\sigma(s)=s$, so the weights are the raw similarities (with a feature map $\phi$ in place of the raw key, if one wants non-negativity). Causal attention for token $t$ then reads $$\begin{gathered} o_t=\sum_{i\le t}v_i\,(k_i^\top q_t) \\ =\Big(\sum_{i\le t}v_ik_i^\top\Big)q_t=S_t\,q_t, \\ S_t=S_{t-1}+v_tk_t^\top . \end{gathered}$$ derived The second equality is associativity of the matrix product: $(V^\top K)q$ versus $V^\top(Kq)$. The first form needs the whole history; the second needs only the $d_v\times d_k$ matrix $S_t$, a **recurrent state**, and each token costs one outer product and one matrix-vector product. $S_t$ is exactly the Hebbian outer-product memory of Chapter 3, written one token at a time. cited This is the observation of Katharopoulos et al. (2020), who write attention with a kernel feature map $\phi$ as $\sum_i\phi(q)^\top\phi(k_i)v_i\big/\sum_i\phi(q)^\top\phi(k_i)$, use associativity to move the sum inside, and show that causal masking gives a recurrence with *two* hidden states: the attention memory $S_t=S_{t-1}+\phi(k_t)v_t^\top$ and a normaliser $z_t=z_{t-1}+\phi(k_t)$. The note drops $\phi$ and the normaliser to keep the memory picture clean; the paper's feature map is $\phi(x)=\mathrm{elu}(x)+1$, chosen to be positive without zeroing gradients. Per-step inference cost and state are constant in the sequence length. Softmax attention cannot make this compression (Derivation D15 gives the reason in one line): the exponential of a dot product has no finite-dimensional rewriting $\sum_{i\le t}v_i\,e^{\beta k_i^\top q_t}=S\,\phi(q_t)$ with a fixed-size $S$ (the exponential kernel has infinite feature dimension). It must keep every $(k_i,v_i)$ pair. That is the cost of its capacity.
A cache grows; a state does not. Drag the number of tokens read: the cache on the left keeps one key and one value row per token, while the state on the right is always a $4\times4$ matrix. Under it, the error each makes when asked for the first value.
With $d_k=4$ and ten tokens the state is far past its capacity of order $d_k$, so its recall is poor; the cache with softmax recalls well because the exponential separates a matching key from the rest. ### What the fast memory costs Slow memory is paid for once, in parameters. Fast memory is paid for per sequence, in cache. With $L$ layers, width $d$ and a cache of keys and values at every layer, the cache holds $2dL$ numbers per token, against $12d^2L$ parameters. derived The two are equal at $$n^\star=\frac{12d^2L}{2dL}=6d\ \text{tokens}.$$ For $d=4096$ and $L=32$ in 16-bit numbers the parameters are $12\cdot4096^2\cdot32\cdot2$ bytes $\approx12.9$ GB ($6.4\times10^9$ parameters) and the cache is $0.52$ MB per token, so $n^\star=24\,576$ tokens. The estimate is for one sequence (with a batch of $B$ sequences sharing the weights the crossing is at $6d/B$) and ignores embeddings, grouped-query sharing, quantisation of the cache and mixture-of-experts; it is a scale, not a spec.
Past a few tens of thousands of tokens the context outweighs the weights. Drag the context length (powers of two, logarithmic axis, $d=4096$, 32 layers): the parameters stay at $13$ GB, the cache grows linearly and crosses them near $24\,576$ tokens (tick on the track).
## 10. Updating a fast memory: add, decay, correct Chapter 9 gave linear attention the update $S_t=S_{t-1}+v_tk_t^\top$, the Hebbian rule. It remembers everything and so interferes with everything: the matrix has a fixed size and every new pair adds to the same entries. Chapter 4's ladder suggests the repairs, and each has an architecture attached to it. *Test time* means using the trained model; a state-space model (SSM, Appendix C) is a layer built on a linear recurrence; a *self-supervised* loss is one whose target comes from the input itself, here the value $v_t$ built from $x_t$. Skim the named methods below on a first reading; the table is what the later chapters use. | Rule | Update of the state | What it changes | | --- | --- | --- | | Add (linear attention) | $S_t=S_{t-1}+v_tk_t^\top$ | nothing is ever removed | | Decay (gated) | $S_t=\gamma_t\,S_{t-1}+v_tk_t^\top$ | old memories fade at rate $\gamma_t$ | | Delta | $S_t=S_{t-1}+\beta_t\,(v_t-S_{t-1}k_t)\,k_t^\top$ | the old value at key $k_t$ is replaced, not added to | | Gated delta | $S_t=\alpha_t\,S_{t-1}\,(I-\beta_tk_tk_t^\top)+\beta_tv_tk_t^\top$ | both: forget everything a little, overwrite one slot | | Test-time training | $S$ = weights of a model; one gradient step on $\lVert f_S(k_t)-v_t\rVert^2$ | the memory is a small learner | derived The delta row is the delta rule of Chapter 4 applied to a *fast* matrix, and it can be written $S_{t-1}(I-\beta_tk_tk_t^\top)+\beta_tv_tk_t^\top$: first erase from $S$ the component along $k_t$ (what the memory currently returns for that key) and then write the new value. With a unit key and $\beta_t=1$ the new pair is stored exactly (model invariant). The test-time-training row is the same update with $f_S(k)=Sk$ and a gradient step of size $\beta_t$, since $\nabla_S\tfrac12\lVert Sk-v\rVert^2=(Sk-v)k^\top$. cited The literature has each row, and the papers say more than the table does. - **Delta rule in linear attention.** Schlag et al. (2021) replace the purely additive update by a delta-rule update so the memory can correct its current key-to-value mapping: with $\bar v=W\phi(k)$ and a write strength $\beta=\sigma(W_\beta x)\in[0,1]$ computed from the token, $W\leftarrow W+\beta\,(v-\bar v)\otimes\phi(k)$, so $\beta=1$ fully replaces the old value. Their keys are normalised by their *sum* (an $L_1$ normalisation); the exact-overwrite property in the table needs an $L_2$-unit key, which is the convention of the later DeltaNet. - **DeltaNet.** Yang et al. (2024) use $k_t=\mathrm{SiLU}(W_Kx_t)/\lVert\mathrm{SiLU}(W_Kx_t)\rVert_2$ and $\beta_t=\sigma(W_\beta x_t)$, note that the delta recurrence is one SGD step on $\tfrac12\lVert Sk_t-v_t\rVert^2$, and show that products of the matrices $I-\beta_tk_tk_t^\top$ (generalised Householder transforms) can be written in a compact "WY" form, which lets training parallelise over the sequence. At 1.3B parameters and 100B tokens it beats Mamba and GLA on perplexity and zero-shot tasks, and hybrids with attention beat strong transformer baselines; on recall-heavy tasks it trails GLA at 1.3B, and it lacks an explicit decay, which hurts length generalisation. - **Gated delta.** Gated DeltaNet (Yang, Kautz and Hatamizadeh) is the row $S_t=S_{t-1}\,\alpha_t(I-\beta_tk_tk_t^\top)+\beta_tv_tk_t^\top$ with a scalar decay $\alpha_t\in(0,1)$ and write strength $\beta_t\in(0,1)$, built to improve Mamba-2 with the delta rule. - **Decay rows.** Gated linear attention (GLA) uses $S_t=\mathrm{Diag}(\alpha_t)S_{t-1}+k_t^\top v_t$ with a vector gate over the key dimension; Mamba is a selective state-space model $h_t=\bar Ah_{t-1}+\bar Bx_t$ whose step $\Delta$, $B$ and $C$ depend on the input, so its decay $\exp(\Delta_tA)$ is per channel; Mamba-2's state-space duality restricts the decay to one input-dependent scalar per head and shows a duality with a masked form of linear attention (the paper notes this relates only to certain flavours of attention). - **Test-time training.** Sun et al. (2024) make the hidden state the weights $W$ of a model $f$ updated by gradient steps on a self-supervised reconstruction loss $\lVert f(\theta_Kx_t;W)-\theta_Vx_t\rVert^2$, read out with a third learned view $\theta_Q$. TTT-Linear and TTT-MLP use $f(x)=x+\mathrm{LN}(f_{\mathrm{res}}(x))$ with a linear and a two-layer model, a data-dependent learning rate, and mini-batches of 16 tokens for the gradient. The row of the table is the special case of per-token steps, a linear $f$ and no normalisation or residual; the paper says DeltaNet is equivalent to TTT-Linear with mini-batch size 1 and without those two, and proves that with batch gradient descent, $\eta=\tfrac12$ and $W_0=0$ a linear TTT layer equals linear attention. - **Titans.** Behrouz et al. (2025) update a neural memory $M$ (an MLP of one or more layers) by $M_t=(1-\alpha_t)M_{t-1}+S_t$ with $S_t=\eta_tS_{t-1}-\theta_t\nabla\ell(M_{t-1};x_t)$ and $\ell=\lVert M(k_t)-v_t\rVert^2$: gradient descent with momentum (the "surprise") and a forget gate $\alpha_t$. They combine this memory with attention in three ways (memory as context, as gate, as layer).
Adding never forgets, and that is the problem; the other two rules choose what to forget. Drag the decay $\gamma$ (blue curve): recall error of each memory in a stream of 24 pairs in $\mathbb R^8$ against how long ago it was written, averaged over 12 streams.
Notice what each rule *assumes about the stream*. Adding assumes every pair is equally valuable forever. Decay assumes recent pairs matter more. The delta rule assumes keys *repeat* and later values should overwrite earlier ones. Softmax attention, which is not a state update at all, assumes nothing and keeps everything, at the price of Chapter 9. ## 11. In-context learning as one gradient step *In-context learning* is a trained model improving on a new input from examples placed in its prompt, with no change to its weights. The memory picture also explains a striking behaviour of attention: given a few input-output examples in the prompt, a transformer can predict for a new input. Here is the cleanest case (Derivation D11, and D12 for several layers), derived in full. A prompt contains pairs $(x_i,y_i)_{i=1}^n$ from a linear map and a query $x_q$. A model with a weight matrix $W$ predicts $\hat y=Wx_q$. Gradient descent from $W_0=0$ on $L(W)=\tfrac12\sum_i\lVert Wx_i-y_i\rVert^2$ with step $\eta$ gives $$\begin{gathered} W_1=W_0-\eta\sum_i(W_0x_i-y_i)x_i^\top \\ =\eta\sum_iy_ix_i^\top, \\ \hat y=W_1x_q=\eta\sum_iy_i\,(x_i^\top x_q). \end{gathered}$$ This is a **linear-attention readout**: keys $x_i$, values $y_i$, query $x_q$, score function the identity, and weight $\eta$. In other words, *a linear attention layer over the prompt performs one step of gradient descent on the in-context regression loss*, and its memory is the Hebbian outer-product memory of Chapter 3 with the inputs as keys and the labels as values. A second layer reads the *residual* values $y_i-W_1x_i$ and performs the second step, and so on: $k$ layers are $k$ steps. (The residuals appear because the same layer also updates the example tokens, as the next paragraph describes.)
Each layer is a gradient step. Drag the number of layers: the copper line is the model after that many in-context gradient steps from zero, and it converges to the least-squares line (dashed).
cited von Oswald et al. (2022) prove by construction (their Proposition 1) that a linear self-attention layer can implement this: tokens are $(x_j,y_j)$, the keys and queries read only the $x$-part, the values are $(0,\,W_0x_i-y_i)$, and every token, including the examples, is updated, so the residuals needed by the next layer arise because each layer rewrites the $y$-slot of every token; the prediction is read out from the query token's $y$-slot with a sign flip and a $1/N$ factor, and attention is non-causal over the examples. For trained models they report that, in linear self-attention-only models on linear regression, one layer matches one gradient step and $K$ layers match a curvature-corrected variant ("GD++") better than $K$ plain steps; with softmax or LayerNorm the match is weaker, and a single softmax layer cannot match gradient descent. They argue that a first layer can build the $(x,y)$ token format by copying, a step like the one induction heads perform, but they say the gradient-descent mechanism differs from induction heads and derive no formal special-case result. Their own scope statement is that the findings are restricted to small transformers and simple regression problems. What the derivation above does *not* show is that softmax attention in a language model does the same. ### Induction heads: how copying is learned *Induction heads* are the best-understood example of what the gradient of Chapter 7 produces in a trained model. A *head* is one copy of $(W_Q,W_K,W_V)$ (Chapter 8), and a *previous-token head* is a head whose query attends to the position just before. cited Olsson et al. (2022) define an **induction head** operationally: a head with *prefix matching* (it attends back to earlier tokens that were followed by the current token) and *copying* (its output raises the logit of the attended token), measured on a repeated random token sequence. This implements $[A][B]\dots[A]\to[B]$: find the earlier occurrence of the current token and copy what followed it. They report that induction heads form at the same point in training as a sharp rise in an *in-context learning score* (the loss on the 500th token minus the loss on the 50th), roughly 2.5 to 5 billion tokens into training. The evidence is a three-tier picture: strong causal evidence in small attention-only models (ablation and reverse engineering), only suggestive causal evidence once MLPs are added, and correlational evidence plus analogy for large models, which the authors call preliminary and indirect. cited The mechanism is a *composition* of two heads: prefix matching is implemented by *K-composition*, in which the induction head's key reads from a subspace written by an earlier "previous-token" head, so that its query (the current token) matches positions whose previous token equals it; copying is done by the value-and-output path. The paper uses the *QK circuit* (where to look) and *OV circuit* (what to write) vocabulary, which comes from the transformer-circuits framework of Elhage et al. (2021), a source I did not open. Induction heads need at least two layers. ## 12. The unified view ### One object Every layer so far has the same three parts: a **state** (the memory), a **write** rule that updates the state from a key-value pair, and a **read** rule that returns a value for a query. cited The test-time-regression paper (Wang, Shi and Fox, 2025) states a version of this: associative recall is memorisation, a regression of values on keys, followed by retrieval; layers are special cases distinguished by three choices, the regression weights, the function class and the test-time optimisation algorithm. In its vignettes, equal weights, a linear class and an exact least-squares solution give $V^\top K(K^\top K)^{-1}$ (Chapter 4's ladder), and linear attention is the crude approximation $K^\top K\approx I$ that ignores the covariance between key dimensions; geometrically decaying weights give gated and state-space layers and a recursive-least-squares form; first-order gradient steps on the same loss give fast-weight, DeltaNet and test-time-training layers; and softmax attention is the Nadaraya–Watson (zeroth-order local) estimator, with query-key normalisation justified by making the distance kernel match the dot-product kernel. Sun et al. (2024) prove two precedents inside their own framework: a linear TTT layer equals linear attention, and a Nadaraya–Watson estimator with the exponential kernel equals softmax attention. | Layer | State | Write | Read | | --- | --- | --- | --- | | Linear layer | $W\in\mathbb R^{d_v\times d_k}$ | SGD in training: $W-\eta\bar yx^\top$ | $Wq$ | | MLP | rows $k_i$, columns $v_i$ of two matrices | SGD in training | $V^\top\sigma(Kq)$ | | Softmax attention | all pairs $(k_i,v_i)_{i\le t}$ | append | $\sum_iv_i\,\mathrm{softmax}_i(\beta k_i^\top q)$ | | Linear attention | $S\in\mathbb R^{d_v\times d_k}$ | $S+vk^\top$ | $Sq$ | | Decayed / gated | $S$ | $\gamma S+vk^\top$ | $Sq$ | | Delta rule | $S$ | $S+\beta(v-Sk)k^\top$ | $Sq$ | | Test-time training | weights of a model $f_S$ | gradient step | $f_S(q)$ | | Hopfield / modern | stored patterns | append | $X^\top\mathrm{softmax}(\beta Xq)$ | ### Two questions sort the table Every row answers two questions. *Is the state fixed or does it grow with the sequence?* *Is the read linear in the query or not?*
The corner that cannot exist. A fixed state with a linear read (top left) is linear attention and state-space models; fixed with a nonlinear read (top right) is the MLP; growing with a nonlinear read (bottom right) is softmax attention. A growing state with a *linear* read collapses into the top-left cell (arrow); a nonlinear read of low feature rank would too.
derived **The collapse** (Derivation D10). A linear read of a growing memory is $\sum_{i\le t}v_i(k_i^\top q)=\big(\sum_{i\le t}v_ik_i^\top\big)q$, and the sum in brackets is a fixed-size matrix (Chapter 9). So a growing state buys nothing when the read is linear in $q$. A nonlinear read does not by itself force growth: a degree-$p$ polynomial read $\sum_iv_i(k_i^\top q)^p$ is also exactly a fixed state, of size $d_v\times d_k^p$ (checked for $p=2$). What forces growth is a read whose kernel has a very large or infinite feature rank, and the exponential kernel of softmax is the infinite one. derived **Compression is a feature map.** If the kernel $e^{\beta q^\top k}$ is approximated by a finite-dimensional inner product $\phi(q)^\top\phi(k)$, with $\phi:\mathbb R^{d_k}\to\mathbb R^D$, the numerator becomes $\sum_iv_i\,\phi(k_i)^\top\phi(q)=S\phi(q)$ with $S=\sum_iv_i\phi(k_i)^\top\in\mathbb R^{d_v\times D}$, and the normaliser $z^\top\phi(q)$ with $z=\sum_i\phi(k_i)$ adds $D$ more numbers. The state size is of order $D$; the identity feature map $D=d_k$ is plain linear attention, and the softmax has $D=\infty$. cited Katharopoulos et al. write attention as a linear dot product of kernel feature maps and use $\phi(x)=\mathrm{elu}(x)+1$; they note that the exponential kernel's feature map is infinite-dimensional, so exact softmax attention cannot be linearised (Derivation D15). Schlag et al.'s DPFP map instead projects keys up to dimension $2d\nu$ to raise capacity, since a plain map keeps capacity at the key dimension. **One read, many writers.** The MLP and attention have the same read, $\sum_iv_i\,\sigma(k_i^\top q)$; they differ in *who writes the pairs and for how long they live* (Chapter 9) and in a normalisation that makes the attention weights compete. The linear layer, the MLP, the cache and the fast state are all the same recall operation applied to differently produced pairs. That is the unification. ### Where the unification is exact and where it is an analogy | Statement | Status | | --- | --- | | Causal linear attention equals a recurrent Hebbian state read by $S_tq_t$ | exact (Chapter 9; normaliser aside) | | The delta row is one SGD step on $\tfrac12\lVert Sk-v\rVert^2$ | exact (Chapter 10) | | One linear-attention layer over residual values is one gradient step | exact for linear attention and a linear regression task (Chapter 11); a construction for trained models only in small linear settings | | Softmax attention is a Nadaraya–Watson smoother | exact for the unnormalised exponential kernel with the normalisation included (Chapter 6) | | An MLP is $V^\top\sigma(Kx)$ | exact in form; its keys and values are *learned weights*, not written pair by pair, so the clean outer-product write holds only for the last layer of a linear network (Chapter 3) | | MLP and attention are "the same read" | the read is the same formula, but attention normalises its weights and the MLP does not, so their capacities in the table below measure different operations | | Softmax attention has a "state" | an analogy: its memory is the set of all pairs, which grows, and appending is not an update of a fixed object | | Mamba and GLA are the decay row | the decay forms are cited (Chapter 10); the identification with a key-value outer-product write needs the feature-map reading and is not derived here | | TTT-MLP is another cell of the grid | in the grid it has a fixed state and a nonlinear read, like an MLP, but its weights are updated at test time by gradient steps | The first three rows carry the argument of the note; the rest are useful frames, and Chapter 13 uses them as such. ### Putting a number on each choice | Choice | Effect | Number in this note | | --- | --- | --- | | Hebbian write, linear read | capacity of order $d_k$ | interference $\sqrt{(M-1)/d_k}$ (Chapter 3) | | Least-squares write | capacity $2d_k$ | $\sqrt{M/d_k-1}$ (Chapter 4) | | ReLU / ReLU² / ReLU³ read | $2d$, $\tfrac23d^2$, $\tfrac2{15}d^3$ | $128$, $2731$, $34\,953$ at $d=64$ (Chapter 5) | | Softmax read | $e^{2\beta(1-\beta/d)}$ | $1.2\times10^6$ at $d=64$, $\beta=8$ (Chapter 6) | | Fixed state | $O(d_kd_v)$ memory, $O(d_kd_v)$ per token | $6.7\times10^7$ at $n=4096$ (Chapter 9) | | Growing cache | $O(nd)$ memory, $O(nd)$ per token | crosses the weights at $n=6d$ (Chapter 9) | ## 13. The right mental models Seven statements, each pointing back at the chapter that derives it. 1. **A matrix is a memory.** $W=\sum_iv_ik_i^\top$: rows are addresses, columns are contents, and $Wq$ is "the contents of everything that looks like $q$" (Chapters 1 to 3). 2. **Training writes the slow memory by outer products.** SGD on a linear layer adds $-\eta\bar yx^\top$, a memory with key $x$ and value $-\eta\bar y$ (Chapter 3). 3. **A good write corrects what is stored.** Add, then add the error, then add the error in the geometry of the keys: Hebbian, delta, least squares. Least squares doubles the Hebbian capacity; the delta rule is its online, unpreconditioned approximation and stores the latest pair exactly (Chapter 4). 4. **A nonlinearity is a selectivity dial.** The sharper the score function, the more stored pairs a query separates; softmax's exponential buys capacity exponential in $\beta$ (Chapters 5 and 6). 5. **Attention is a soft dictionary rebuilt per sequence.** Keys and values are projections of the tokens; the output is a kernel-weighted average, and gradient descent moves keys toward the queries they help (Chapters 6 and 7). 6. **Two memories, two clocks.** Weights are slow and fixed-size; context is fast and grows; the projections $W_Q,W_K,W_V$ are the *program* that writes the fast memory (Chapter 9). 7. **A fixed-size fast state must forget.** A state of size $D$ has capacity of order $D$, and its update rule says how it forgets: never (add, and then it interferes), exponentially (decay), by overwriting (delta) or by learning (test-time training); only an unbounded cache avoids forgetting by construction (Chapters 10 and 12). ### Questions to ask of any new layer
One set of questions, seven layers. Click a layer or use the arrow keys: its state, how it is written, how it is read and what it forgets, in the same four rows every time.
- What is the **state**, and does its size grow with the sequence? - Who **writes** it (training, or the forward pass), and what rule: add, decay, correct? - What is the **read**: linear or nonlinear in the query, and what does the nonlinearity separate? - **Where does it interfere?** Which keys overlap, and what is the capacity at this width? - What does it **forget**, and is that forgetting chosen (a gate) or accidental (overwriting)? ### Failure modes the picture predicts | Symptom | Reading in the picture | | --- | --- | | A fixed-state model loses a fact from early in a long document | interference or decay in the state; capacity of order $d_k$ is exceeded | | Attention becomes uniform (diffuse) on a long context | the target's weight $e^\beta$ is small against the sum over $M$ other weights; and even when it is not, value-space noise $\sqrt M\,e^{\beta^2/d-\beta}$ is not small (Chapter 6) | | Attention saturates to one token and trains badly | too large $\beta$: argmax, vanishing gradient (Chapter 6) | | A model "knows" a fact but fails when it is rephrased (conjecture) | the query may not land near the stored key if the key's response region is narrow (Chapter 5 on biases); not derived here | | Adding a document to the prompt dilutes another | softmax weights compete per query and head: attention is a budget of one, though nothing is evicted from the cache | ## 14. An application: editing one fact in the slow memory If an MLP is a key-value memory, changing one stored fact is a *rank-one edit*. Let $W$ be the second matrix of an MLP, so a key $k$ (an internal feature vector) produces the output $Wk$, and let $C=E[kk^\top]$ be the (uncentred) second-moment matrix of the keys the layer sees on typical data. Here the *key* of Chapter 3 is the hidden activation that feeds the second MLP matrix (in Chapter 5 the keys were the rows of the first matrix; both views are consistent: the hidden activation is the vector of similarities to those keys). We want a new matrix $W'$ with $W'k^\star=v^\star$ for one key $k^\star$ and as little change as possible on typical keys: $$\begin{gathered} \min_{\Delta}\ \tfrac12\operatorname{tr}(\Delta C\Delta^\top) \\ \text{s.t.}\quad\Delta k^\star=v^\star-Wk^\star . \end{gathered}$$ derived The Lagrangian condition (Derivation D17) gives $\Delta C=\lambda k^{\star\top}$ for a vector $\lambda$, so $\Delta=\lambda k^{\star\top}C^{-1}$ and the constraint fixes $\lambda$: $$\boxed{\;W'=W+\frac{(v^\star-Wk^\star)\,(C^{-1}k^\star)^\top}{k^{\star\top}C^{-1}k^\star}\;}$$ a rank-one update whose *key* is the whitened $C^{-1}k^\star$ (the delta rule of Chapter 4, with the preconditioner of RLS). cited This is the form of the edit in ROME (Meng et al., 2022), who write $\hat W=W+\Lambda(C^{-1}k^\star)^\top$ with $C$ the uncentred second-moment matrix of keys estimated from a sample of Wikipedia text, treat the MLP's second matrix as a linear associative memory, take the key $k^\star$ to be the post-nonlinearity hidden activation at the last subject token, and find the value $v^\star$ by gradient optimisation rather than reading it off. MEMIT (Meng et al., 2023) is the batched, multi-layer version, with $C_0=\lambda E[kk^\top]$ for a scale $\lambda$ ($C$ above is proportional to it). ## 15. A Rosetta stone: one idea under many names The same computation has been discovered repeatedly, in different fields and under different names. Reading across them is the quickest way to make the unified view of Chapter 12 stick, because each field supplies a question that the others forgot to ask. | Name | Field | What it is in this note | | --- | --- | --- | | Correlation matrix memory, Hebbian learning | neuroscience, early associative memory | the outer-product write $W=\sum v_ik_i^\top$ (Chapter 3) | | Delta rule, Widrow–Hoff, LMS | adaptive filtering | the error-correcting write (Chapter 4) | | Recursive least squares, Kalman filter | estimation and control | the preconditioned delta rule (Chapter 4) | | Hopfield network | statistical physics | a memory read by iterating a fixed-point map (Chapter 6) | | Nadaraya–Watson kernel smoother | nonparametric statistics | softmax attention with an exponential kernel (Chapter 6) | | Hash table, dictionary | computer science | the hard limit $\beta\to\infty$ of attention (Chapter 6) | | Key-value memory network | question answering | an MLP or attention with learned keys and values (Chapter 5) | | Fast weights, fast weight programmer | neural-network history | linear attention: a slow net writes a fast matrix (Chapter 9) | | Linear RNN, state-space model | sequence modelling, control | the recurrence $S_t=A_tS_{t-1}+v_tk_t^\top$ (Chapters 9 and 10) | | Online learning, meta-learning | machine learning | test-time training and in-context gradient descent (Chapters 10 and 11) | | Policy gradient, REINFORCE | reinforcement learning | the gradient on attention logits (Chapter 7) | | Random-feature / kernel approximation | kernel methods | the feature-map compression of softmax (Chapter 12) | recalled The historical attributions behind this table (Hebb, Kohonen and Anderson for correlation memories; Widrow and Hoff for the delta rule; Hopfield; Nadaraya and Watson; Schmidhuber's fast weights) are from memory, and the dates are deliberately left out. The mapping to this note's chapters is the part I stand behind. ### What each field contributes - **Neuroscience and early associative memory** asked how many items a memory can hold and how it degrades: capacity and interference (Chapter 3). - **Estimation and control** asked how to update a memory optimally from one observation at a time: the delta rule and recursive least squares (Chapter 4). - **Statistics** asked what a lookup is estimating: attention is an estimator of a value at a query, with a kernel and a bandwidth (Chapter 6). - **Physics** asked what the memory's stable states are: patterns are minima of an energy, and retrieval is descent (Appendix D8). - **Sequence modelling** asked how to make the memory cheap: compress the growing cache into a fixed state, and ask what it must forget (Chapters 9 and 10). - **Meta-learning** asked what a model does when given examples: it learns from them in its forward pass (Chapter 11). ### The compressed form of the whole note Take a stream of key-value pairs $(k_t,v_t)$ and queries $q_t$. 1. **State.** Something that holds what has been written: a matrix $W$ or $S$, a set of pairs, the weights of a small model. 2. **Write.** A rule that changes the state when a pair arrives: add, decay, correct, or take a gradient step. 3. **Read.** A rule that returns a value for a query: linear, thresholded (ReLU powers), exponential-and-normalised (softmax), or a small network. 4. **Forget.** What the combination discards, and whether that choice is made or accidental. An MLP, an attention head, a linear-attention layer, a state-space layer and a test-time-training layer differ in the answers they give to these four questions and in *who writes* (training or the forward pass), and in nothing else that this note needs. ## 16. Open questions - **Capacity of trained layers** (Chapter 5): relate the random-key capacity to a measurable property of a trained MLP. - **Roles of $Q$, $K$ and $V$** (Chapter 7): when are keys and values effectively tied, and what does each carry that the others cannot? - **Intermediate memory sizes** (Chapter 9): a principled state of size between $d^2$ and $nd$, with a capacity law. - **Optimal fixed-size write** (Chapter 10): how close is a learned delta rule to recursive least squares? - **A capacity theory for learned keys** (Chapter 12): predict the retrievable associations from the write rule and read kernel on a data distribution. - **In-context gradient descent in softmax models** (Chapter 11): which tasks does it describe, and which does it not? ## 17. Exercises **1.** A memory has $d_k=16$ and stores $M=32$ random unit keys with independent unit values. What interference-to-signal ratio do you expect from the Hebbian store and from the least-squares store?
Answer Hebbian: $\sqrt{(M-1)/d_k}=\sqrt{31/16}=1.39$ (measured $1.38$). Least squares: $\sqrt{M/d_k-1}=\sqrt{32/16-1}=1.00$ (measured $1.01$).
**2.** Show that a sum of $r$ outer products has rank at most $r$ and give the matrix-product form.
Answer $\sum_{k=1}^r\alpha_k\rho_k^\top=AB$ with $A=[\alpha_1\cdots\alpha_r]\in\mathbb R^{m\times r}$ and $B$ the matrix with rows $\rho_k^\top$. Every column of $AB$ is a combination of the $r$ columns of $A$, so the column space has dimension at most $r$.
**3.** For $d=64$, estimate the capacity of a squared-ReLU memory and compare it with the exact formula that uses the fourth moment of a unit-vector dot product.
Answer Asymptotic: $\tfrac23d^2=2731$. Exact: $2/E[s^4]=2d(d+2)/3=2816$, where $E[s^4]=3/(d(d+2))$. The difference is the factor $(d+2)/d$.
**4.** At what context length does the key-value cache of a model with $d=8192$ and $L=80$ layers equal its parameter bytes, if both are stored in the same precision and $12d^2L$ is the parameter count?
Answer $n^\star=12d^2L/(2dL)=6d=49\,152$ tokens, independent of $L$. (This ignores grouped-query attention, which reduces the cache and moves the crossing to larger $n$.)
**5.** Starting from $W_0=0$ and $\eta=0.1$, three in-context pairs $(x_i,y_i)=(1,1),(2,2),(1,3)$ in one dimension. What does one linear-attention layer predict for $x_q=1$ and what is the least-squares answer?
Answer One layer: $\hat y=\eta\sum_iy_ix_ix_q=0.1\,(1\cdot1+2\cdot2+3\cdot1)\cdot1=0.8$. Least squares for $y=wx$: $w=\sum x_iy_i/\sum x_i^2=8/6=1.33$, so the answer is $1.33$. One step at this $\eta$ is far from converged; with $\eta=1/\sum x_i^2=1/6$ one step is exactly the least-squares solution in this one-dimensional case.
## Appendix A. Linear algebra toolkit Everything the main text uses about vectors and matrices, defined once. A reader comfortable with linear algebra can skip it and return when a term is unfamiliar. **Vectors, dot product, norm.** A *vector* $x\in\mathbb R^n$ is a list of $n$ numbers, written as a column. The *dot product* (inner product) of $x,y\in\mathbb R^n$ is $x^\top y=\sum_ix_iy_i$. The *norm* is $\lVert x\rVert=\sqrt{x^\top x}$, the length of $x$. A vector is a *unit vector* if $\lVert x\rVert=1$. For unit vectors $x^\top y=\cos\theta$, where $\theta$ is the angle between them, so the dot product is a *similarity*: $1$ for the same direction, $0$ for perpendicular ("orthogonal") ones, $-1$ for opposite ones. A set of vectors is *orthonormal* if they are unit vectors and pairwise orthogonal; in $\mathbb R^n$ there are at most $n$ of them. **Matrices.** A matrix $A\in\mathbb R^{m\times n}$ is a rectangular table with $m$ rows and $n$ columns; its *transpose* $A^\top$ swaps rows and columns, so $(AB)^\top=B^\top A^\top$. The *matrix-vector product* and the *matrix-matrix product* are the subject of Chapters 1 and 2. A matrix is *square* if $m=n$, *symmetric* if $A=A^\top$, and the *identity* $I$ has ones on the diagonal and zeros elsewhere. The *trace* $\operatorname{tr}A=\sum_iA_{ii}$ sums the diagonal; it satisfies $\operatorname{tr}(AB)=\operatorname{tr}(BA)$. **Norms of matrices.** The *Frobenius norm* is $\lVert A\rVert_F=\sqrt{\sum_{ij}A_{ij}^2}=\sqrt{\operatorname{tr}(A^\top A)}$: the length of $A$ read as one long vector. It is the "size of the error" whenever the note compares two matrices. **Rank, row space, column space.** The *column space* of $A$ is the set of all combinations $\sum_jc_jA_{\cdot j}$ of its columns, and the *rank* is its dimension: the number of linearly independent columns, which equals the number of independent rows. A matrix of rank $r$ can be written as a product $A=BC$ with $B\in\mathbb R^{m\times r}$ and $C\in\mathbb R^{r\times n}$, equivalently as a sum of $r$ outer products (Chapter 2). **Inverse.** A square matrix $A$ has an *inverse* $A^{-1}$ with $AA^{-1}=A^{-1}A=I$ exactly when its rank is $n$ ("full rank", "invertible"). Solving $Ax=b$ is $x=A^{-1}b$. **Least squares and the normal equations.** When $Ax=b$ has no exact solution, the *least-squares* solution minimises $\lVert Ax-b\rVert^2$. Setting the gradient to zero gives the *normal equations* $A^\top Ax=A^\top b$, so $x=(A^\top A)^{-1}A^\top b$ when $A$ has full column rank. When there are many exact solutions, the one of smallest norm is $x=A^\top(AA^\top)^{-1}b$; both are instances of the *pseudo-inverse*. **Orthogonal projection.** The matrix $P=A(A^\top A)^{-1}A^\top$ maps any vector to its closest point in the column space of $A$. It is symmetric and *idempotent* ($P^2=P$: projecting twice changes nothing), and $\operatorname{tr}P$ equals the rank of $A$. Chapter 4 uses all three properties. **Singular value decomposition (SVD).** Every matrix factors as $A=\sum_k\mu_ku_kv_k^\top$ with *singular values* $\mu_1\ge\mu_2\ge\dots\ge0$, orthonormal *left singular vectors* $u_k$ and orthonormal *right singular vectors* $v_k$. The number of nonzero $\mu_k$ is the rank, and $\lVert A\rVert_F^2=\sum_k\mu_k^2$ (the "energy" of $A$). **Sherman–Morrison formula.** For an invertible $R$ and a vector $k$, $(R^{-1}+kk^\top)^{-1}=R-\dfrac{Rkk^\top R}{1+k^\top Rk}$: the inverse after a rank-one change is a rank-one change of the inverse, without re-inverting. Chapter 4 uses it for recursive least squares. **Ridge regression.** Least squares with a penalty $\lambda\lVert x\rVert^2$ added; the solution $(A^\top A+\lambda I)^{-1}A^\top b$ always exists for $\lambda>0$. **Whitening.** Multiplying vectors by $C^{-1/2}$ (or, as in this note, one factor $C^{-1}$) where $C$ is their second-moment matrix, so that the transformed vectors are uncorrelated and of unit size in every direction. ## Appendix B. Calculus and probability toolkit **Gradient and chain rule.** For a scalar function $L(\theta)$ of a vector or matrix of parameters, the *gradient* $\partial L/\partial\theta$ has the same shape as $\theta$ and entry $\partial L/\partial\theta_j$; it points in the direction of steepest increase, so $\theta\leftarrow\theta-\eta\,\partial L/\partial\theta$ is a step of *gradient descent* with *step size* (learning rate) $\eta$. The *chain rule* says that if $L$ depends on $y$ and $y$ depends on $\theta$, then $\partial L/\partial\theta_j=\sum_i(\partial L/\partial y_i)(\partial y_i/\partial\theta_j)$. The matrix of all $\partial y_i/\partial\theta_j$ is the *Jacobian*. **Finite differences.** A numerical check of a gradient: $\partial L/\partial\theta_j\approx\big(L(\theta+h e_j)-L(\theta-h e_j)\big)/(2h)$ for small $h$. Every gradient formula in this note was verified this way. **Lagrange multipliers.** To minimise $f(x)$ subject to a constraint $g(x)=0$, find $x$ and $\lambda$ with $\nabla f=\lambda\nabla g$ and $g(x)=0$. Chapter 14 uses it for the rank-one edit. **Expectation, variance, independence.** $E[X]$ is the average of a random quantity $X$ over its distribution; $\operatorname{Var}X=E[X^2]-E[X]^2$. Two random quantities are *independent* if knowing one says nothing about the other; for independent zero-mean $X,Y$, $E[XY]=0$ and $\operatorname{Var}(X+Y)=\operatorname{Var}X+\operatorname{Var}Y$, which is why interference from many unrelated memories grows like the *square root* of their number. **Gaussian.** The normal distribution $\mathcal N(0,\sigma^2)$ with bell-shaped density; sums of many small independent terms are approximately Gaussian. A *standard Gaussian vector* has independent $\mathcal N(0,1)$ entries. **Monte Carlo.** Estimating an expectation by averaging over many random samples. The checks in this note use it with a fixed random seed so that the printed numbers are reproducible. **Jensen's inequality.** For a convex function such as the square, $f(E[X])\le E[f(X)]$; in particular $E[\sqrt{X}]\le\sqrt{E[X]}$, which is why a measured interference ratio sits slightly below $\sqrt{(M-1)/d}$. **Random unit vectors in high dimension.** Two independent random unit vectors in $\mathbb R^d$ are nearly orthogonal: their dot product $s$ has mean $0$ and $E[s^2]=1/d$ (Derivation D1). This near-orthogonality is the reason a $d$-dimensional space can hold many more than $d$ nearly distinct keys, and also the reason their small overlaps add up into interference. ## Appendix C. Machine-learning toolkit **Model, parameters, loss, SGD.** A *model* is a function with adjustable numbers (*parameters* or *weights*). A *loss* $L$ measures how wrong its output is on an example; *training* adjusts the parameters to reduce the loss by *stochastic gradient descent* (SGD): repeatedly take the gradient on one example (or a small batch) and step against it. *Backpropagation* is the chain rule applied layer by layer to get those gradients. **Linear layer and error signal.** A *linear layer* computes $y=Wx$. If $\bar y=\partial L/\partial y$ is the *error signal* arriving from above, then $\partial L/\partial W=\bar y\,x^\top$ (Derivation D14): the gradient of a linear layer is an outer product. **Token, embedding.** Text is cut into *tokens* (word pieces). An *embedding* maps each token to a vector $x\in\mathbb R^d$, the model *width* $d$. A sequence of $n$ tokens becomes $n$ vectors, stacked as the rows of an $n\times d$ matrix $X$. **Logit, softmax.** A *logit* is a raw score, any real number. The *softmax* turns logits $\ell_1,\dots,\ell_n$ into positive weights that sum to one: $\operatorname{softmax}(\ell)_i=e^{\ell_i}/\sum_je^{\ell_j}$. It keeps the order of the logits, exaggerates differences (a gap of $\Delta$ multiplies a weight by $e^\Delta$) and is unchanged if the same number is added to every logit. The *temperature* form $\operatorname{softmax}(\beta\ell)$ has *inverse temperature* $\beta$. **Activation functions.** A nonlinearity applied entrywise between linear layers. $\mathrm{ReLU}(s)=\max(0,s)$. *Squared ReLU* is $\mathrm{ReLU}(s)^2$. *Swish* (also called SiLU) is $s/(1+e^{-s})$, a smooth version of ReLU. A *gated linear unit* (GLU) multiplies two linear readings of the input, one passed through an activation (the *gate*): $f(x)=\mathrm{act}(Gx)\odot(Kx)$, where $\odot$ is entrywise multiplication; SwiGLU uses Swish as the activation. **MLP.** A *multilayer perceptron* is two linear layers with an activation between: $f(x)=W_2\,\sigma(W_1x)$. In a transformer it is applied to each position separately and is also called the *feed-forward layer*; its *hidden width* is the size of $\sigma(W_1x)$, usually $4d$. **Residual connection and the residual stream.** A *residual connection* adds a layer's input to its output, $x\leftarrow x+\mathrm{layer}(x)$. Stacking layers this way gives each position a running vector, the *residual stream*, that every layer reads from and adds to. **Layer normalisation (LayerNorm).** Rescales a vector to have zero mean and unit variance across its entries (with learned gain and shift) so that layer inputs stay at a stable scale. It is written $\mathrm{LN}$ in Chapter 8 and does not change the memory picture. **Autoregressive model, causal mask.** A language model that predicts each token from the earlier ones. In attention it is enforced by a *causal mask*: position $t$ may read only positions $i\le t$. **Positional information.** Attention over a set ignores order (Chapter 6), so position must be injected, by adding position vectors to the embeddings or by rotating the queries and keys (*rotary embeddings*, RoPE). The note's memory picture is unchanged by this; where it matters (the gauge freedom of Chapter 7) it says so. **KV cache.** During generation, the keys and values of all earlier tokens are stored so they need not be recomputed; this store is the fast memory of Chapter 9. **In-context learning.** A trained model adapting to a new task from examples placed in its prompt, with no weight update. Chapter 11 gives one mechanism. **Grouped-query attention (GQA).** Several query heads share one key head and one value head, shrinking the KV cache. **Mixture of experts (MoE)** replaces one MLP by many and routes each token to a few of them; it multiplies the slow memory without multiplying the per-token compute. **State-space models (SSMs).** Sequence layers built on a linear recurrence $S_t=A_tS_{t-1}+B_tx_t$ with a readout; Mamba is a *selective* version in which $A_t,B_t$ depend on the input. Chapter 10 shows how they sit in the unified table. **Policy gradient and baseline.** To increase the expected reward of a softmax distribution over choices one steps along $a_i(r_i-\bar r)$, the weight times the reward's excess over a *baseline* $\bar r$ (REINFORCE). Chapter 7 shows that the gradient on attention logits has exactly this form. ## Appendix D. Derivations Each derivation here is referred to from the main text. They are written to be followed with a pencil. ### D1. Near-orthogonality: $E[(k^\top k')^2]=1/d$ Let $k,k'$ be independent and uniformly distributed on the unit sphere of $\mathbb R^d$. The distribution is rotation invariant, so conditioning on $k'$ and rotating it to the first basis vector gives $k^\top k'\overset{d}{=}k_1$, the first coordinate of $k$. Since $\sum_ik_i^2=1$ and all coordinates are exchangeable, $E[k_1^2]=1/d$. $\square$ ### D2. Higher moments of a dot product of unit vectors Write $k=g/\lVert g\rVert$ with $g$ a standard Gaussian vector. Then $s^2=k_1^2=g_1^2/(g_1^2+\dots+g_d^2)$, and $g_1^2$ over the sum of $d$ squares is *Beta$(\tfrac12,\tfrac{d-1}2)$* distributed. The moments of a Beta$(a,b)$ variable $B$ are $E[B^p]=\prod_{j=0}^{p-1}\frac{a+j}{a+b+j}$, so with $a=\tfrac12$ and $a+b=\tfrac d2$ $$\begin{gathered} E[s^{2p}]=\prod_{j=0}^{p-1}\frac{\tfrac12+j}{\tfrac d2+j} \\ =\frac{1\cdot3\cdots(2p-1)}{d(d+2)\cdots(d+2p-2)}, \end{gathered}$$ which gives $\tfrac1d$, $\tfrac{3}{d(d+2)}$ and $\tfrac{15}{d(d+2)(d+4)}$ for $p=1,2,3$. $\square$ (The Beta-moment formula is standard; the numbers were also checked by Monte Carlo in Chapter 5.) ### D3. Least-squares memory and its projection Written out in the fold in Chapter 4 (the block titled "Derivation D3"). ### D4. Eckart–Young Written out in the fold in Chapter 2 (the block titled "Derivation D4"). ### D5. Recursive least squares from Sherman–Morrison Written out in the fold in Chapter 4 (the block titled "Derivation D5"). ### D6. Softmax Jacobian and the gradients of attention For $a_i=e^{\ell_i}/Z$ with $Z=\sum_je^{\ell_j}$: $\partial a_i/\partial\ell_j=\big(\delta_{ij}e^{\ell_i}Z-e^{\ell_i}e^{\ell_j}\big)/Z^2=a_i(\delta_{ij}-a_j)$. With $o=\sum_ja_jv_j$ and $g=\partial L/\partial o$, $$\begin{gathered} \frac{\partial L}{\partial\ell_i}=\sum_jg^\top v_j\frac{\partial a_j}{\partial\ell_i} \\ =\sum_jg^\top v_j\,a_j(\delta_{ij}-a_i) \\ =a_i\,g^\top v_i-a_i\,g^\top o=a_i(v_i-o)^\top g . \end{gathered}$$ Since $\ell_i=\beta q^\top k_i$: $\partial\ell_i/\partial k_i=\beta q$ and $\partial\ell_i/\partial q=\beta k_i$, and since $o$ depends on $v_i$ only through $a_iv_i$, $\partial L/\partial v_i=a_ig$. $\square$ ### D7. Capacity of the softmax memory Written out in the fold in Chapter 6 (the block titled "Derivation D7"). ### D8. The Hopfield energy and attention as a fixed point Written out in the fold in Chapter 6 (the block titled "Derivation D8"). ### D9. Only $W_QW_K^\top$ matters For invertible $G$, $(W_QG)(W_KG^{-\top})^\top=W_QGG^{-1}W_K^\top=W_QW_K^\top$, so every score $x_i^\top W_QW_K^\top x_j$ is unchanged. $\square$ ### D10. Why a linear read of a growing memory is a fixed-size state $\sum_{i\le t}v_i(k_i^\top q)=\big(\sum_{i\le t}v_ik_i^\top\big)q$ by associativity of matrix products, and $\sum_{i\le t}v_ik_i^\top$ is a $d_v\times d_k$ matrix whatever $t$ is. For a degree-$p$ polynomial read, $(k^\top q)^p=\langle k^{\otimes p},q^{\otimes p}\rangle$, so $\sum_iv_i(k_i^\top q)^p=\big(\sum_iv_i\,\mathrm{vec}(k_i^{\otimes p})^\top\big)\mathrm{vec}(q^{\otimes p})$ is a read of a fixed $d_v\times d_k^p$ state. $\square$ ### D11. A gradient step is a linear-attention readout See Chapter 11; the two lines there are the whole derivation. The generalisation to $k$ steps substitutes the residuals $y_i-W_{k-1}x_i$ for $y_i$ in the same expression. ### D12. Several layers are several steps After layer $k-1$ the weight matrix is $W_{k-1}$ and each example token holds the residual $r_i^{(k)}=y_i-W_{k-1}x_i$. A linear-attention layer with keys $x_i$, values $r_i^{(k)}$ and query $x_q$ outputs $W_{k-1}x_q+\eta\sum_ir_i^{(k)}(x_i^\top x_q)=W_{k-1}x_q-\eta\sum_i(W_{k-1}x_i-y_i)x_i^\top x_q=W_kx_q$, which is the prediction after one more gradient step. $\square$ ### D13. Variance of a dot product For independent zero-mean unit-variance entries, $\operatorname{Var}(q^\top k)=\sum_i\operatorname{Var}(q_ik_i)=\sum_iE[q_i^2]E[k_i^2]=d_k$. $\square$ ### D14. Gradient of a linear layer $y_i=\sum_jW_{ij}x_j$, so $\partial y_i/\partial W_{ij}=x_j$ and $\partial y_{i'}/\partial W_{ij}=0$ for $i'\ne i$; by the chain rule $\partial L/\partial W_{ij}=\bar y_ix_j$, i.e. $\partial L/\partial W=\bar y\,x^\top$. $\square$ ### D15. The exponential kernel has infinite feature dimension $e^{\beta q^\top k}=\sum_{p=0}^\infty\frac{\beta^p}{p!}(q^\top k)^p$, and by D10 each term $(q^\top k)^p=\langle q^{\otimes p},k^{\otimes p}\rangle$ is an inner product of $p$-fold tensor powers, so $e^{\beta q^\top k}=\langle\Phi(q),\Phi(k)\rangle$ with $\Phi(x)=\big(\sqrt{\beta^p/p!}\;x^{\otimes p}\big)_{p\ge0}$, a vector with a block for every degree $p$. Truncating at degree $P$ gives a finite feature map of dimension $\sum_{p\le P}d_k^p$, an approximation of the exact kernel and not the kernel itself. $\square$ ### D16. A policy gradient in four lines A distribution $a_i=\operatorname{softmax}(\ell)_i$ over choices $i$, each with reward $r_i$; the objective is $J=\sum_ia_ir_i$. Since $\partial a_i/\partial\ell_j=a_i(\delta_{ij}-a_j)$ (D6), $\partial J/\partial\ell_j=\sum_ir_ia_i(\delta_{ij}-a_j)=a_j(r_j-\bar r)$ with $\bar r=\sum_ia_ir_i$. The same expression comes from the *log-derivative trick* $\nabla J=E_{i\sim a}[r_i\nabla\log a_i]$ when the expectation is estimated from samples (REINFORCE). Subtracting any constant $b$ from every $r_i$ (a *baseline*) leaves $\partial J/\partial\ell_j$ unchanged in the exact sum, because $a_j(r_j-b-(\bar r-b))=a_j(r_j-\bar r)$, and it lowers the variance of the sampled estimate; the choice $b=\bar r$ is the mean baseline, and $r_j-\bar r$ is the *advantage*. For attention with reward $r_i=-v_i^\top g$ this gives $\partial L/\partial\ell_j=-\partial J/\partial\ell_j=a_j(v_j-o)^\top g$, the formula of Chapter 7. $\square$ ### D17. The rank-one edit Minimise $\tfrac12\operatorname{tr}(\Delta C\Delta^\top)$ subject to $\Delta k^\star=r$, where $r=v^\star-Wk^\star$ and $C$ is symmetric positive definite. The Lagrangian is $\tfrac12\operatorname{tr}(\Delta C\Delta^\top)-\lambda^\top(\Delta k^\star-r)$ for a multiplier vector $\lambda$. Its gradient with respect to $\Delta$ is $\Delta C-\lambda k^{\star\top}$, which vanishes at $\Delta=\lambda k^{\star\top}C^{-1}$. Substituting in the constraint, $\lambda\,(k^{\star\top}C^{-1}k^\star)=r$, so $\lambda=r/(k^{\star\top}C^{-1}k^\star)$ and $\Delta=r\,(C^{-1}k^\star)^\top/(k^{\star\top}C^{-1}k^\star)$ (using $C=C^\top$). The objective equals $E\lVert\Delta k\rVert^2$ for keys with second-moment matrix $C$, the mean squared change on typical keys; $C$ must be invertible, which needs more samples than dimensions. $\square$ ### D18. Heads add Write the concatenated head outputs as $H=[H_1\ \cdots\ H_h]$ with $H_m\in\mathbb R^{n\times d_k}$ and $W_O$ as the stack of row blocks $W_O^{(1)},\dots,W_O^{(h)}\in\mathbb R^{d_k\times d}$. Block-matrix multiplication gives $HW_O=\sum_mH_mW_O^{(m)}$. The parameter count of $W_Q,W_K,W_V$ is $3\cdot h\cdot d\cdot d_k=3d^2$ for $d_k=d/h$, plus $d^2$ for $W_O$. $\square$ ### D19. Softmax attention as a Gaussian-kernel smoother Written out in the fold in Chapter 6 (the block titled "Derivation D19"). ### D20. Eight facts about rank and projections used in the text (i) $\operatorname{rank}(AB)\le\min(\operatorname{rank}A,\operatorname{rank}B)$, since the columns of $AB$ are combinations of the columns of $A$. (ii) A sum of $r$ outer products has rank at most $r$. (iii) $QK^\top$ with $Q,K\in\mathbb R^{n\times d_k}$ has rank at most $d_k$, as does $W_QW_K^\top$. (iv) $\sum_jP_{ij}P_{jk}=P_{ik}$ for a projection, so $\sum_jP_{ij}^2=P_{ii}$ when $P$ is also symmetric. (v) $\operatorname{tr}P=\operatorname{rank}P$ for a projection, since its eigenvalues are 0 and 1. (vi) $\lVert A\rVert_F^2=\sum_k\mu_k^2$ because $\operatorname{tr}(A^\top A)$ sums the eigenvalues of $A^\top A$, which are $\mu_k^2$. (vii) A Frobenius norm is unchanged by multiplying by orthonormal matrices. (viii) For unit-norm independent values $v_i$ with zero mean, $E\lVert\sum_ic_iv_i\rVert^2=\sum_ic_i^2$, the identity behind every interference ratio in the note. ## Appendix E. Glossary Alphabetical. The number in brackets is the chapter or appendix where the term is used most. - **Activation function.** An entrywise nonlinearity between linear layers (ReLU, Swish, softmax is not one). [5, C] - **Advantage.** A reward minus a baseline; how much better a choice is than average. [7, C] - **Associative memory.** A store addressed by content: give it a key, get the value stored with it. [3] - **Attention (self-attention).** A sublayer in which each position builds a query, compares it with the keys of all positions and returns a weighted combination of their values. [6] - **Attention matrix $A$.** The $n\times n$ matrix of attention weights; row $i$ is the distribution that query $i$ puts on the keys. [6] - **Autoregressive.** Generating or predicting one token at a time from the earlier ones. [0, C] - **Backpropagation.** Computing gradients layer by layer with the chain rule. [C] - **Baseline.** A reference value subtracted from a reward to reduce the variance of a policy gradient. [7, C] - **Bias.** A constant vector added inside a layer; in an MLP it shifts a key's threshold. [5] - **Capacity.** The number of stored pairs at which interference equals signal (this note's operational definition). [3] - **Causal mask.** Forbids position $t$ from reading positions after it. [0] - **Context, context window.** The tokens a model can read; the length is $n$. [0] - **Cross-entropy loss.** Average of $-\log p(\text{true next token})$. [0] - **Delta rule.** Write the error $v-Wk$ rather than $v$; gradient descent on $\tfrac12\lVert Wk-v\rVert^2$. [4, 10] - **Dictionary (hash table).** A store that returns the value for the exactly matching key. [6] - **Embedding.** The learned vector assigned to a token. [0] - **Energy (Hopfield).** A function whose minima are the stored patterns; retrieval is descent on it. [6, D8] - **Error signal.** $\partial L/\partial y$, the gradient of the loss at a layer's output. [3, C] - **Fast weights.** A matrix produced per sequence by the forward pass, as opposed to the slow weights learned in training. [9] - **Feature map $\phi$.** A function applied to keys and queries so that $\phi(q)^\top\phi(k)$ approximates a similarity kernel. [9, 12] - **Feed-forward layer.** Another name for the MLP sublayer. [5] - **Frobenius norm.** Square root of the sum of squared entries of a matrix. [A] - **Gate.** A factor between 0 and 1 (or an activation) that decides how much of something passes. [5, 10] - **Gated linear unit (GLU).** A layer that multiplies two linear readings entrywise, one passed through an activation. [5, C] - **Gradient, gradient descent.** The vector of partial derivatives; stepping against it to reduce a loss. [B] - **Grouped-query attention.** Several query heads sharing one key head and one value head. [8] - **Head.** One copy of $(W_Q,W_K,W_V)$ with its own softmax; a layer runs several in parallel. [8] - **Hebbian rule.** Add $vk^\top$ for each pair: wiring together what occurs together. [3] - **Hidden state.** The fixed-size summary a recurrent layer carries from one token to the next. [9] - **Hopfield network.** A memory whose stored patterns are minima of an energy, retrieved by iterating an update. [6] - **In-context learning.** Adapting to a task from examples in the prompt without changing weights. [11] - **Induction head.** A head that finds the earlier occurrence of the current token and copies the token that followed it. [11] - **Interference.** The part of a recalled value contributed by the other stored pairs. [3] - **Jacobian.** The matrix of all partial derivatives of a vector function. [B, D6] - **Kernel.** A function $\kappa(q,k)$ measuring the similarity of two points; **kernel smoother**: a $\kappa$-weighted average of stored values. [6] - **Key, value, query.** The address a memory is matched by, the content it returns, and the question being asked. [3, 6] - **KV cache.** The stored keys and values of earlier tokens. [9] - **LayerNorm.** Rescales a vector to zero mean and unit variance across its entries, with learned gain and shift. [8, C] - **Least squares.** Choosing the parameters that minimise the sum of squared errors. [4, A] - **Logit.** A raw real-valued score before a softmax. [0, 6] - **Loss.** A scalar measuring error; training minimises it. [0, B] - **Mamba.** A selective state-space layer whose recurrence parameters depend on the input. [10] - **MLP.** Two linear layers with an activation between them; in a transformer, the feed-forward sublayer. [5] - **Mixture of experts.** Many MLPs of which only a few are used per token. [9] - **Nadaraya–Watson estimator.** A kernel smoother: $\sum_i\kappa(q,k_i)v_i/\sum_j\kappa(q,k_j)$. [6] - **Orthonormal.** Unit length and mutually perpendicular. [A] - **Outer product.** $uv^\top$, the matrix with entries $u_iv_j$. [2] - **Parameters (weights).** The learned numbers in a model. [0] - **Positional information.** What tells attention the order of tokens. [0] - **Preconditioner.** A matrix that reshapes a gradient step to the geometry of the problem. [4] - **Projection (orthogonal).** The map to the closest point of a subspace; $P^2=P$ and $P=P^\top$. [4, A] - **Rank.** The number of independent columns; the smallest number of outer products that sum to the matrix. [2] - **ReLU, squared ReLU.** $\max(0,s)$ and $\max(0,s)^2$. [5] - **Residual stream.** The running per-position vector to which every sublayer adds. [0, 8] - **Ridge regression.** Least squares with a penalty $\lambda\lVert W\rVert^2$. [4, A] - **SGD.** Stochastic gradient descent: step against the gradient of one example or batch. [3, C] - **Sherman–Morrison formula.** The inverse after a rank-one change, without re-inverting. [A, D5] - **Singular value decomposition.** $A=\sum\mu_ku_kv_k^\top$ with orthonormal $u_k$, $v_k$. [2, A] - **Softmax.** Positive weights summing to one, $e^{\ell_i}/\sum_je^{\ell_j}$. [0, 6] - **State-space model (SSM).** A layer built on the linear recurrence $S_t=A_tS_{t-1}+B_tx_t$. [10, C] - **Superposition.** Storing more features than dimensions by letting directions overlap; the interference of Chapter 3. [5] - **Temperature.** The scale $\beta$ in $\operatorname{softmax}(\beta\ell)$: small is flat, large is a hard maximum. [6] - **Test time.** Using the trained model; the weights do not change. **Test-time training** changes a fast state by gradient steps during use. [0, 10] - **Token.** One piece of text mapped to a vector. [0] - **Trace.** The sum of the diagonal; for a projection it equals the rank. [A] - **Unembedding.** The matrix turning a final vector into one logit per vocabulary item. [0] - **Value (see key).** - **Whitening.** Multiplying by the inverse of a second-moment matrix so that directions are decorrelated. [4, 14] ## Further reading "Read in full" means I opened the body of the source (the PDF or the page) in a separate check pass; "abstract" means only the abstract; "not opened" means the claim is from memory or from another source's description. **The post that frames the note** - [Alex Wang — *You could have invented transformers from memory*](https://alexwang.ai/posts/associative-memory/): outer-product, optimal-linear, MLP and attention stores with capacities and interference-to-signal ratios. Read in full. **Memory and attention** - [Vaswani et al. — *Attention is all you need*](https://arxiv.org/abs/1706.03762) (2017). Read in full (scaling footnote, post-norm, $d_{\mathrm{ff}}=2048$). - [Sukhbaatar, Szlam, Weston and Fergus — *End-to-end memory networks*](https://arxiv.org/abs/1503.08895) (2015). Read in full. - [Geva, Schuster, Berant and Levy — *Transformer feed-forward layers are key-value memories*](https://arxiv.org/abs/2012.14913) (2021). Read in full. - [Ramsauer et al. — *Hopfield networks is all you need*](https://arxiv.org/abs/2008.02217) (2020). Read in full. - [Elhage et al. — *A mathematical framework for transformer circuits*](https://transformer-circuits.pub/2021/framework/index.html) (2021). Read in full. - [Olsson et al. — *In-context learning and induction heads*](https://arxiv.org/abs/2209.11895) (2022). Read in full. - [Ainslie et al. — *GQA: training generalized multi-query transformer models*](https://arxiv.org/abs/2305.13245) (2023). Read in full. - [Xiong et al. — *On layer normalization in the transformer architecture*](https://arxiv.org/abs/2002.04745) (2020). Pre-norm versus post-norm. Read in full. - [Shazeer — *GLU variants improve transformer*](https://arxiv.org/abs/2002.05202) (2020). Read in full. **Fast weights, linear attention, state-space layers, test-time learning** - [Katharopoulos, Vyas, Pappas and Fleuret — *Transformers are RNNs*](https://arxiv.org/abs/2006.16236) (2020). Read in full. - [Schlag, Irie and Schmidhuber — *Linear transformers are secretly fast weight programmers*](https://arxiv.org/abs/2102.11174) (2021). Read in full. - [Yang, Wang, Zhang, Shen and Kim — *Parallelizing linear transformers with the delta rule over sequence length*](https://arxiv.org/abs/2406.06484) (2024). Read in full. - [Yang, Kautz and Hatamizadeh — *Gated delta networks: improving Mamba2 with delta rule*](https://arxiv.org/abs/2412.06464) (2024). Read in full. - [Yang et al. — *Gated linear attention transformers with hardware-efficient training*](https://arxiv.org/abs/2312.06635) (2023). Read in full. - [Gu and Dao — *Mamba: linear-time sequence modeling with selective state spaces*](https://arxiv.org/abs/2312.00752) (2023) and [Dao and Gu — *Transformers are SSMs*](https://arxiv.org/abs/2405.21060) (2024). Read in full. - [Sun et al. — *Learning to (learn at test time): RNNs with expressive hidden states*](https://arxiv.org/abs/2407.04620) (2024). Read in full. - [Behrouz, Zhong and Mirrokni — *Titans: learning to memorize at test time*](https://arxiv.org/abs/2501.00663) (2025). Read in full. - [Wang, Shi and Fox — *Test-time regression: a unifying framework for designing sequence models with associative memory*](https://arxiv.org/abs/2501.12352) (2025). Read in full. - Schmidhuber, [*Learning to control fast-weight memories*](https://doi.org/10.1162/neco.1992.4.1.131), Neural Computation (1992). Bibliographic data confirmed; body not opened. **In-context learning** - [von Oswald et al. — *Transformers learn in-context by gradient descent*](https://arxiv.org/abs/2212.07677) (2022). Read in full. **Editing facts** - [Meng, Bau, Andonian and Belinkov — *Locating and editing factual associations in GPT* (ROME)](https://arxiv.org/abs/2202.05262) (2022) and [*Mass-editing memory in a transformer* (MEMIT)](https://arxiv.org/abs/2210.07229) (2023). Read in full. **Capacity and length scaling (as the post states them)** - [Allen-Zhu and Li — *Physics of language models 3.3: knowledge capacity scaling laws*](https://arxiv.org/abs/2404.05405) (about 2 bits per parameter for synthetic facts at 1000 exposures) and [Morris et al. — *How much do language models memorize?*](https://arxiv.org/abs/2505.24832) (about 3.6 bits per parameter for random data). Both read in full, and they measure different things. - [Nakanishi — *Scalable-softmax*](https://arxiv.org/abs/2501.19399) (2025) and [*Critical attention scaling in long-context transformers*](https://arxiv.org/abs/2510.05554), which lists the $\log n$ and $(\log n)^2$ scalings; [Peng et al. — *YaRN*](https://arxiv.org/abs/2309.00071) for the $(0.1\ln s+1)^2$ factor. Su's note on entropy invariance (kexue.fm/archives/8823) I could not open, so its exact form is the post's description. - Amit, Gutfreund and Sompolinsky, [*Storing infinite numbers of patterns in a spin-glass model of neural networks*](https://doi.org/10.1103/PhysRevLett.55.1530), Phys. Rev. Lett. (1985): the classical Hopfield capacity near $0.14N$. Read in full. **Not opened** - Widrow and Hoff (1960) for the delta rule, as attributed in Schlag et al. and Gated DeltaNet; Hebb, Kohonen and Anderson for correlation memories; Nadaraya and Watson for the kernel smoother; Nichani, Lee and Bietti, Garcia et al., Li et al. and Zhong et al. as described by the post; the Eckart–Young and Weyl theorems and the Sherman–Morrison formula as standard results. **On this site** - [Reinforcement learning: a synthesis](rl.html): the policy-gradient identity of Chapter 7 appears there as the group-baseline gradient of GRPO (Chapter 18). - [Deriving the Prior-Data Fitted Network](../posts/pfn-derived.html): a transformer used as an in-context learner, the practical side of Chapter 11.