# Discrete Codebooks: From Vector Quantization and VQ-VAE to Residual Codes, Scalar Codes and Semantic IDs A neural network lives in continuous space: an image is a grid of real numbers, a song a vector of floats, a product an embedding. Yet some of the most visible systems of the last few years first turn these into a short string of integers drawn from a small table, and then model the string. Image generators that predict picture patches as words. Speech models that talk in eight integers every 80 milliseconds. Recommenders that name a video not by an arbitrary ID but by a three-number address and *generate* the address of what to show next. The table is a *codebook*, the integers are *codes* or *semantic tokens*, and the operation that produces them is *quantization*. This note is a self-contained guide to the whole idea, written to be read from the top and also looked things up in. It starts below the neural network, with the arithmetic of storing a number in a few bits, and builds up: why a codebook is the optimal answer to a precise question (Part A); how a large codebook can be assembled from small ones (Part B); the VQ-VAE that made discrete codes trainable inside a neural network, with its gradient trick, its losses, its failure mode and the fixes (Part C); the extensions, which are mostly answers to that failure mode: residual quantization, k-means residuals, finite scalar and lookup-free quantization, spherical codes, multi-scale and one-dimensional tokenizers (Part D); what the codes are used for in images and in audio, with the bit-accounting of each system (Part E); semantic IDs for recommendation and retrieval (Part F); how a sequence model is built on top of the codes (Part G); and what to measure, what is solved and what is open (Part H). Three questions run through it. *What problem does a discrete code solve?* It is not obvious: a continuous vector carries more information than an integer, and the step that makes the integer cannot be differentiated. *Is there a first-principles reason it works?* There is one, from the theory of lossy compression, and it also explains why the designs look the way they do. *What is the trend, and how much of it is evidence?* Which uses are supported, which are claims by their authors, and where the evidence points the other way. ## The story: Lantern's help centre Lantern, the start-up of the inference, retrieval and decision-model notes, now has a help centre of two million articles and a log of what each reader opened next. The team wants to predict the next article. The obvious design gives every article an ID and a 64-number vector in a table; the model reads the vectors of the last few articles and returns a score for every article. It has problems, met in order, and each one is an entry to a part of the note. | When | What goes wrong | The idea that explains it | Where | | --- | --- | --- | --- | | Week 1 | The table is 256 MB and a new article has no row | Why an ID table is the wrong object | A, F | | Week 2 | "Why not compress each vector to a few bits?" | Quantization: bits against error, and the codebook that is optimal for it | A | | Week 3 | "A codebook of 65,536 vectors won't train, and won't fit the data we have" | Building big codebooks from small ones: residual, product, grid | B | | Week 4 | The first trained codebook uses a fifth of its entries | The argmin has no gradient; the straight-through estimator; dead codes | C | | Week 5 | Three papers fix this three different ways | Residual codes, k-means residuals, finite scalar and lookup-free quantization | D | | Week 6 | Pictures and audio at the other end of the company already do this | What the same operation buys in vision and audio, and the bits it spends | E | | Week 8 | The next article generated as a three-number address | Semantic IDs, their collisions, and the tree behind them | F | | Week 9 | How is the sequence model on top supposed to read these codes? | Flattening, depth transformers, coarse-to-fine, constrained decoding | G | | Week 10 | Should we replace the retriever with this? | What to measure, what is solved, what is open | H | ## What is public and what is not Three kinds of source are used, and each claim says which. *Papers whose text was opened for this note* support the claims made from them; their titles and identifiers are in the reading list at the end. They include VQ-VAE and VQ-VAE-2, VQGAN and ViT-VQGAN, the discrete VAE of DALL-E, the residual quantizer of RQ-VAE, finite scalar quantization, the lookup-free quantizer of MAGVIT-v2, binary spherical quantization, SimVQ, the rotation trick, the one-dimensional tokenizer TiTok, visual autoregressive modelling, Chameleon, the audio codecs SoundStream, EnCodec, DAC and Mimi (in the Moshi paper), AudioLM, HuBERT, the retrieval papers DSI and GenRet, and the recommendation papers TIGER, the YouTube study of semantic IDs, LC-Rec, LETTER, HSTU and OneRec with its technical report. *Results the authors report about their own systems* are reported as theirs; the production claims of OneRec and of the YouTube study are the companies' own, not independent measurements. *Classical results* (Shannon's rate–distortion function, Lloyd and Max, Panter and Dite, Gish and Pierce, the straight-through estimator, hierarchical softmax, product quantization) are stated from memory where the note says so. Blue marks what is stored or observed, copper is a cost or the thing that goes wrong, violet is a bound or a quantity being compared. ## Notation | Symbol | Meaning | | --- | --- | | $x$, $z_e$, $z_q$ | an input; the encoder's continuous output; the codebook vector that replaces it | | $K$ | number of entries in one codebook; $e_1,\dots,e_K$ are the entries, each of $d$ numbers | | $L$ | number of codes per item: residual stages, or positions in a semantic ID | | $R$ | rate: bits spent per stored number (or, in Part A, per vector) | | $D$ | distortion: mean squared error between the input and its reconstruction | | $N$, $M$ | number of items, and number of distinct IDs available ($M=K^L$) | | $\mathrm{sg}[\cdot]$ | stop-gradient: the identity going forward, zero derivative going back | | $\beta$, $\gamma$ | commitment weight; decay of a moving average | | $r_l$ | the residual entering stage $l$ of a residual quantizer, with $r_1=z_e$ | | $H$, $\mathrm{PP}$ | entropy of code usage and its exponential, the perplexity $\exp(H)$ | The collisions, admitted: $D$ is distortion in Part A, the embedding size of a codebook entry in some of the VQ-VAE formulas, the stack depth of a residual quantizer in Part D and a discriminator in C9; $d$ is the number of numbers per vector. $L$ is the number of residual stages or positions in an ID, the number of levels of a scalar-quantizer channel, and the loss $\mathcal L$. $K$ is the codebook size, except in "top-$K$" and "Pass@$K$", where it is a count of generated items. $N$ is a number of items, of latents, of counts in a moving average or of centroids, as each section says. $R$ is a rate, not the rotation of the rotation trick. $q$ is a posterior, a quantizer or a transport plan, again as each section says. ## A. First principles: what a codebook is for ### A1. Three things an integer is good for A continuous vector carries more information than an integer, so the integer must buy something. Three things are claimed in the literature, and they are different in kind. 1. **An interface.** A sequence model with a softmax output wants a finite set of symbols to choose from, a cross-entropy loss to train on and sampling to generate with. Text has that natively. Pixels, sound and products do not, and a codebook gives them a vocabulary. The VQ-VAE paper argues that discrete representations are a natural fit for many modalities (language is discrete, speech is typically written as symbols, images can be described by language), pairs the codes with an autoregressive prior, and reports generating images, video and speech with it. 2. **An address.** Naming an item by a short string of codes, instead of by a row in a table that grows with the catalogue, lets a model generate the name, share statistical strength between items with a common prefix, and name an item it has never seen. This is the use in recommendation (Part F). 3. **A budget.** An integer of $R$ bits is an exact measure of storage and of the bitrate of a stream. A codec is built from it: audio at a fixed number of kilobits per second (Part E). None of the three needs the code to be discrete in itself. The honest summary is that *discreteness is a convenient interface to machinery built for symbols*, and Part E ends with work that questions whether the interface is necessary. The rest of this part asks what is lost when a vector is replaced by one of $K$ entries, because every later design is a way of paying less for the same loss. ### A2. One number, $R$ bits Start with one real number $x$ and spend $R$ bits on it. That allows $2^R$ distinct values to remember, so a *codebook* of $K=2^R$ levels $e_1<\dotsDerivation: the distortion of a uniform quantizer, and why a bit is worth 6 dB The claim is that a uniform quantizer with $2^R$ levels over a range $W$ has $D\approx\frac{W^2}{12}2^{-2R}$, so one more bit divides $D$ by four. **Step 1: the step.** $2^R$ levels over a range $W$ means a step $\Delta=W/2^R$. **Step 2: the error.** If the density of $x$ is nearly flat inside each cell, the error $x-\hat x$ is uniform on $[-\Delta/2,\Delta/2]$. **Step 3: its mean square.** $$D=\frac1\Delta\int_{-\Delta/2}^{\Delta/2}u^2\,du=\frac{\Delta^2}{12}.$$ **Step 4: put the rate in.** $$D=\frac{W^2}{12}\,2^{-2R}.$$ **Step 5: one more bit.** $R\to R+1$ multiplies $D$ by $2^{-2}=\tfrac14$. In decibels, $10\log_{10}4=6.02$ dB. **What this does not say.** The flat-density step is the high-rate approximation. For few levels the cells are wide, the density is not flat inside them and the formula is wrong; Figure 1 below uses the exact optimum instead. ### A3. The best any scheme can do: the rate–distortion bound How far can any scheme, not only a uniform grid, push the error for a given number of bits? For a Gaussian number the answer has a short proof.
Derivation: no code of $R$ bits per number gives a mean square error below $\sigma^2 2^{-2R}$ The claim is that if $X\sim\mathcal N(0,\sigma^2)$ is stored in $R$ bits per number and read back as $\hat X$, then $D=\mathbb E(X-\hat X)^2\ge\sigma^2\,2^{-2R}$. The argument is the standard one for the rate–distortion function of a Gaussian source (Shannon; recalled, with the steps written out). **Step 1: rate bounds information.** A code of $R$ bits per number has at most $2^{R}$ values per number, so the reconstruction carries at most $R$ bits about the input: $R\ge I(X;\hat X)$, the mutual information, measured in bits. (For a block of many numbers coded together the same holds per number.) **Step 2: expand the information.** $I(X;\hat X)=h(X)-h(X\mid\hat X)$, with $h$ the differential entropy. **Step 3: condition on the reconstruction.** Given $\hat X$, the uncertainty in $X$ is the uncertainty of the error: $h(X\mid\hat X)=h(X-\hat X\mid\hat X)\le h(X-\hat X)$, because conditioning cannot raise entropy. **Step 4: the Gaussian maximises entropy.** Among all variables with mean square $D$, the Gaussian has the largest differential entropy, $\tfrac12\log_2(2\pi e D)$. So $h(X-\hat X)\le\tfrac12\log_2(2\pi eD)$. **Step 5: combine.** With $h(X)=\tfrac12\log_2(2\pi e\sigma^2)$, $$R\ \ge\ \tfrac12\log_2(2\pi e\sigma^2)-\tfrac12\log_2(2\pi eD)=\tfrac12\log_2\frac{\sigma^2}{D},$$ which rearranges to $D\ge\sigma^22^{-2R}$. **What this does not say.** It is a lower bound, and it says nothing about how to reach it. That a code exists that comes arbitrarily close needs a second argument (random coding over long blocks), not given here. It also says nothing about the *cost* of the code: the closer to the bound, the longer the blocks and the larger the codebook, which is the subject of Part B.
The bound has the same shape as the uniform quantizer's: $D$ falls by a factor of four for every bit, 6.02 dB. It is the dashed violet line in Figure 1, and no code of $R$ bits per number, scalar or vector, goes below it.
Each bit is worth up to 6 dB; the best scalar codebook gets close to that slope but not onto the bound, and splitting the bits into two stages costs more. Error of a standard Gaussian number against the bits spent on it. The violet dashed line is the bound $2^{-2R}$. The blue line is the best codebook of $2^R$ levels, found by Lloyd's algorithm. The copper line splits the bits into two stages of $2^{R/2}$ levels, the second trained on what the first left. Drag the ring along the bits axis: at 4 bits the best codebook of 16 numbers is 20.2 dB down against a bound of 24.1; at 8 bits it is 43.9 dB down against 48.2, a gap of 4.3 dB, and the two-stage version with 32 numbers instead of 256 is 37.3 dB down.
Two features of the curves matter for everything after. The best codebook runs parallel to the bound at about 6 dB per bit, so the *shape* of the trade is fixed: error falls by a constant factor per bit. And it sits above the bound by a gap that grows to 4.35 dB, because $D\cdot2^{2R}$ for the best scalar codebook tends to a constant $\sqrt3\pi/2\approx2.72$ (Panter and Dite; recalled, and the model shows the trend: 2.43 at 4 bits, 2.64 at 6, 2.70 at 8). ### A4. Finding the best codebook: Lloyd's algorithm Which levels should the codebook hold? Two facts settle it.
Derivation: Lloyd's two conditions, and why alternating them cannot make things worse The claim is that the best codebook for a distribution satisfies two conditions, and that a loop which enforces them in turn lowers the distortion at every step. **Step 1: fix the codebook, choose the assignment.** The error of sending $x$ to level $e_k$ is $\|x-e_k\|^2$. For each $x$ the smallest is the nearest level. So the best cells are the nearest-neighbour cells (the Voronoi cells of the codebook), with boundaries halfway between neighbouring levels. **Step 2: fix the cells, choose the levels.** For cell $C_k$ the contribution to $D$ is $\mathbb E[\|x-e_k\|^2\mathbf 1_{x\in C_k}]$. Differentiating with respect to $e_k$ and setting the derivative to zero gives $$e_k=\mathbb E[x\mid x\in C_k],$$ the mean of the cell, its *centroid*. The second derivative is positive, so this is the minimum. **Step 3: alternate.** Each of the two steps minimises $D$ over one of its two arguments with the other held, so neither raises it. $D$ is bounded below by zero and never rises, so it converges. **What this does not say.** The limit is a local optimum, not necessarily the global one. A level that starts with an empty cell has no points to average and never moves (Step 2 has nothing to work with). That is the seed of the dead-code problem in Part C.
On data this is the *k-means* algorithm (Lloyd 1957 and 1982 for the signal-processing form, Max 1960 for the Gaussian tables; recalled). For a standard Gaussian it gives the distortions 0.3634, 0.1175, 0.03455 and 0.0095 at 1, 2, 3 and 4 bits, which the model reproduces and which agree with the classical tables. Everything later in the note that learns a codebook is a version of this loop: assign each point to its nearest entry, move each entry to the mean of its points. The VQ-VAE does it by gradient steps (C5), the moving-average variant does it online (C5), and RQ-Kmeans (D2) just runs it. ### A5. Vectors: the cell shape, and the exponent that matters Move to vectors of $d$ numbers and a codebook of $K$ vectors. The rate is $\log_2K$ bits for the vector, so $\log_2K/d$ bits per number. How does the error scale?
Derivation: in $d$ dimensions the error falls as $K^{-2/d}$, and the cell shape sets the constant The claim is that at high rate, for data with a smooth density, a good vector codebook of $K$ entries has mean square error per number $D\approx G\,V^{2/d}K^{-2/d}$ on a region of volume $V$ with a nearly flat density, where $G$ depends only on the shape of the cells. **Step 1: the cells.** $K$ cells share a region of volume $V$, so a typical cell has volume $V/K$. **Step 2: the error in a cell.** The squared distance from a point to the centre of its cell scales as the cell's linear size squared, which is $(V/K)^{2/d}$ in $d$ dimensions. Write the mean over the cell, per coordinate, as $G\,(V/K)^{2/d}$, where $G$ is the *normalised second moment* of the cell shape: the same for every cell of the same shape, whatever its size. **Step 3: the exponent.** $D\approx G\,V^{2/d}K^{-2/d}$. With $K=2^{R}$ for the vector, $D\propto2^{-2R/d}$: *per number*, each bit of *rate per number* ($R/d$) buys the same factor of four, 6.02 dB. The rate that matters is the rate per number, not per vector. **Step 4: the shape.** For a cube $G=1/12$, which is the scalar case of A2. For a ball in high dimension $G\to1/(2\pi e)\approx0.0585$. The ratio is $2\pi e/12=1.42$, or **1.53 dB**: the most that rounder cells can buy over a cubic grid, however large the dimension. **What this does not say.** The approximation needs a high rate, a smooth density and a region where it is flat; the codebooks of this note are small and the data clustered, so the formula is a guide to the exponent, not a prediction of any figure here.
Two consequences shape the rest. First, the *unit* is bits per number: a 16-bit code for a vector of 4 numbers is 4 bits per number, for a vector of 256 numbers it is 0.06 bits per number, and the error they leave is very different. Second, of the 4.35 dB gap of A3 between the best scalar codebook and the bound, 1.53 dB is the shape of the cell, which vectors recover as the dimension grows. The other 2.8 dB belongs to a different choice, spending *exactly* $R$ bits on every number: a code that gives likely values shorter names (entropy coding) needs only the Gish–Pierce constant $\pi e/6$ in place of $\sqrt3\pi/2$ at high rate (recalled). That brings us to usage. ### A6. Rate is entropy, not $\log_2K$: why unused codes cost bits A codebook of $K$ entries *can* name $\log_2K$ bits, but a code that is used unevenly *carries* less. If entry $k$ is used with probability $p_k$, the information the code carries per symbol is the entropy $H=-\sum_kp_k\log_2p_k\le\log_2K$, with equality only when every entry is used equally. The *perplexity* $2^H$ (or $e^{H}$ with natural logarithms) is the number of equally used entries that would carry the same information. Two things follow. - **Dead codes lower the real rate.** The DAC paper says it plainly: a codebook with a large unused part has an implicit reduction in target bitrate, which translates to poor reconstruction. A codebook of 1,024 entries of which 24 are used is, for the purposes of Figure 1, a codebook of about 24. - **A prior is an entropy coder.** After the codes are produced, a sequence model fit to them assigns each code a probability. The bits needed to write the data down with that model are its cross-entropy, which is at most $\log_2K$ per code and falls as the model learns structure (Part C, the ELBO view). Codes that are used evenly and independently leave the prior nothing to learn. Usage is therefore a first-class quantity, measured as perplexity, utilisation (the share of entries used) or entropy, and every extension in Part D can be read as a way of raising it. ### A7. What "error" means once the loss is perceptual All of the above measures error as mean square. The tokenizers for images and audio of Parts C and E are trained with perceptual and adversarial losses instead, which changes what the distortion $D$ is: it becomes a distance in a feature space, or the output of a discriminator. The qualitative shape of the trade (more bits, less error; a codebook that is used unevenly wastes bits) survives. The 6 dB-per-bit slope does not carry over, because it is a statement about mean square error, so a figure such as Figure 1 should be read as a model of the quantizer, not of a trained image tokenizer. ## B. Four ways to build a large codebook ### B1. Why not one table of $2^{16}$ vectors Take a codebook of $K=2^R$ vectors of $d$ numbers. Four costs grow with $K$: - the table itself, $Kd$ numbers; - the search at encoding time, $Kd$ multiply–adds per vector to find the nearest entry; - the data needed to train it: an entry that no point falls near is never trained, and an entry fit to a handful of points memorises them; - the softmax of any model that predicts the code, which is $K$ wide. For $K=65{,}536$ and $d=256$ the table is 16.8 million numbers and each vector costs 16.8 million multiply–adds to encode. The residual codebooks in the papers below hold 256 to 8,192 entries each; single codebooks reach 16,384 in VQGAN and ViT-VQGAN's 8,192, and the lookup-free quantizer of MAGVIT-v2 goes to $2^{18}$ by removing the table altogether. The way out is to build a large *implicit* codebook from small explicit ones. There are three ways, plus the flat table as the reference. | Construction | What is stored | What it represents | Encoding cost | Example | | --- | --- | --- | --- | --- | | **Flat**: one codebook | $K$ vectors | $K$ points | $Kd$ | VQ-VAE, VQGAN | | **Residual**: quantize, then quantize what is left | $L$ codebooks of $K$ | up to $K^L$ sums | $LKd$ | RQ-VAE, SoundStream, TIGER | | **Product**: split the vector into parts, quantize each | $P$ codebooks of $K$, each for $d/P$ numbers | $K^P$ concatenations | $Kd$ | product quantization for nearest-neighbour search (Jégou et al. 2011; recalled) | | **Scalar grid**: round every number to a few levels | nothing, or $d$ small tables | $\prod_iL_i$ grid points | $d$ | finite scalar quantization, lookup-free quantization |
Four ways to reach a large vocabulary. Residual and product tables multiply a few small tables; the scalar grid needs no table at all. Each buys capacity by assuming something about the data.
### B2. Residual quantization Write $r_1=z$. Quantize it with a codebook $C^{(1)}$ to get the index $c_1=\arg\min_k\|r_1-e^{(1)}_k\|$, take what is left, $r_2=r_1-e^{(1)}_{c_1}$, quantize that with a second codebook, and so on. After $L$ stages the vector is approximated by the *sum* $\hat z=\sum_{l=1}^Le^{(l)}_{c_l}$, and the item is the tuple $(c_1,\dots,c_L)$.
Each stage codes the remainder. Press Next to follow one vector.
Derivation: what residual quantization stores, what it represents, and why each stage cannot raise the error The claim is that $L$ stages of $K$ entries store $LKd$ numbers and address up to $K^L$ different reconstructions, and that the error after stage $l$ is the mean square of the residual $r_{l+1}$. **Step 1: the reconstruction.** After $l$ stages, $\hat z^{(l)}=\sum_{j\le l}e^{(j)}_{c_j}$, and by the recursion the residual is $r_{l+1}=z-\hat z^{(l)}$. So the error of the $l$-stage code is exactly $\|r_{l+1}\|^2$: the residual *is* the error, which is why the recursion is natural. **Step 2: the count.** A reconstruction is determined by the tuple of indices, one of $K$ choices per stage, so there are at most $K^L$ distinct reconstructions. Only $LK$ vectors are stored. Some tuples may never occur, and different tuples may give nearly the same sum, so $K^L$ is the *capacity*, not the number of useful cells. **Step 3: each stage helps.** At stage $l$ the codebook is fit to the residuals $r_l$ by k-means. A codebook with $K\ge1$ entries can place one entry at the mean of the residuals, which gives mean square error equal to the variance of $r_l$, at most $\mathbb E\|r_l\|^2$. The best $K$-entry codebook does at least as well, so $\mathbb E\|r_{l+1}\|^2\le\mathbb E\|r_l\|^2$ in expectation for stage-wise trained codebooks. **Step 4: the cost of a stage.** Encoding stage $l$ compares $r_l$ with $K$ entries, $Kd$ operations, so $LKd$ in all, against $K^Ld$ for a flat codebook of the same capacity. **What this does not say.** Step 3 holds for each stage trained on the residuals it receives; it does not say the total error equals that of a flat codebook with $K^L$ entries. It does not, as the experiment below shows. And the greedy encoding (take the nearest entry at stage 1, then at stage 2) is not the best tuple for a given $z$: a pair whose stage-1 entry is not the nearest can leave a smaller final error. Beam search over the first stage or two improves encoding at extra cost (recalled; the papers below use the greedy rule).
Three properties make residual codes worth their loss, and each reappears in the note. 1. **A hierarchy.** The first code names a coarse cell and later codes refine it, so two items with the same first code are close and the codes form a tree. TIGER draws this on the Amazon Beauty catalogue: the first code tracks the coarse category and the later ones the finer categories (F3). 2. **An adjustable rate.** Dropping the last stages gives a coarser version of the same message. SoundStream trains its quantizer with dropout over stages so that one model runs from 3 to 18 kbps (E3). 3. **A short list of small softmaxes.** A model predicting one code at a time faces a $K$-wide softmax $L$ times instead of a $K^L$-wide one once (F4). **Shared or separate codebooks.** Whether stage $l$ uses its own codebook or all stages share one is a design choice made differently by the main users. RQ-VAE shares a single codebook across all depths, giving two reasons: the total size is the only quantity to tune, and every entry is available at every depth, so an entry can be used where it helps most. TIGER keeps a separate codebook per level, because the norm of the residuals falls with the level, so different levels work at different granularities. ### B3. Product quantization Split the vector into $P$ parts of $d/P$ numbers and quantize each part with its own codebook of $K$ entries. The code is the tuple of $P$ indices and the reconstruction their concatenation. It stores $PK(d/P)=Kd$ numbers, the same as a flat codebook, and addresses $K^P$ points. The price is an assumption: that the parts are *independent enough* that the best cell for the whole is the product of the best cells for the parts. Structure that couples the parts, such as clusters that are not aligned with the split, is lost. Its main home is nearest-neighbour search, where distances to the codes are computed from small precomputed tables (recalled; not opened here), and it is the ancestor of the multi-codebook designs above. ### B4. The scalar grid Round every number of the vector separately to a few levels. With $L_i$ levels for coordinate $i$ the capacity is $\prod_iL_i$ cells, boxes on a grid. If the levels are fixed by a rule and nothing is learned this is a *lattice* and the table disappears: finite scalar quantization (D3) and lookup-free quantization (D4) are the two forms in the literature. If each coordinate gets its own learned levels, it is the best *product of scalar quantizers*, which is the baseline used in Figure 2 below. ### B5. An experiment in four dimensions Which of the four wins? Figure 2 compares them on vectors of four numbers, at 4, 6 and 8 bits per vector (1, 1.5 and 2 bits per number), on two kinds of data: independent Gaussian numbers and points in 16 clusters. Each method is trained on 3,000 points and scored on 3,000 other points, by the error in decibels below the signal. The sample is small on purpose: it makes the *cost of training data* visible, which is the third item of B1.
On held-out points the flat codebook's lead shrinks or disappears; on clustered data it is large, and structure that a method cannot see costs it several decibels. Bars are the error on 3,000 held-out points, in dB below the signal (longer is better); the black tick is the error on the 3,000 training points; the violet dashed line is the rate–distortion bound for Gaussian numbers. Drag the ring along the bits axis. At 8 bits on Gaussian data the flat codebook of 256 vectors reaches 11.4 dB on its training points but only 9.9 held out, where a two-part product quantizer gets 9.8, a scalar grid 9.5 and two residual stages of 16 get 9.1; the bound is 12.0. On clustered data at 8 bits the flat codebook gets 17.2 held out, two residual stages of 16 get 16.2, the product quantizer 14.1 and the scalar grid 11.4.
Reading it, in order. 1. **A flat codebook overfits first.** At 8 bits the flat codebook has 256 vectors to fit with 3,000 points, about twelve points each. Its error on the training points is 11.4 dB and on new points 9.9 dB. The product quantizer, with two codebooks of 16 vectors each fit to two-number halves, shows no such gap (9.83 on the training points, 9.78 held out). Fewer parameters per cell is the point of the structured constructions, and this is what B1's third cost looks like. 2. **Residual quantization loses to the flat codebook at equal bits.** On Gaussian data at 8 bits, two stages of 16 are 0.8 dB behind the flat codebook on held-out points (9.1 against 9.9) and four stages of 4 are 1.5 dB behind (8.3), and the training-set gap is 1.9 to 2.9 dB. This is Figure 1's finding in four dimensions: the loss is real, smaller on held-out data than on the training data. 3. **Structure the method cannot see costs bits.** The scalar grid has the rounded cells of a cubic lattice and cannot follow clusters: on the clustered data at 4 bits it reaches 4.0 dB where the flat codebook reaches 11.0. The product quantizer follows the clusters only as far as they are aligned with its split, and falls 3 dB behind the flat codebook at 8 bits (14.1 against 17.2). The residual quantizer finds them: its first stage can capture the clusters and later stages refine inside them, so it is about one decibel behind the flat codebook with two stages of 16 (16.2 against 17.2). 4. **No method reaches the bound.** The Gaussian bound is 12.0 dB at 8 bits. The best held-out result is 9.9 dB, 2.2 dB short, and the shortfall includes both the shape of the cells (up to 1.53 dB at best, A5) and the finite training set. These are four-dimensional, independent-coordinate, small-sample results with a Gaussian built from twelve uniforms. They say how the costs of A and B trade, not how any of the trained networks of later parts behave; but the lesson, that a structured codebook is a bet on structure in the data, and a flat one a bet on having enough points, is what the designs below turn on. ### B6. Choosing | Prefer | When | | --- | --- | | Flat | the data is low-dimensional or clustered, a few thousand entries are enough, and there is plenty of data per entry | | Residual | a hierarchy is wanted (an address, a coarse-to-fine bitrate), a very large capacity is needed from small tables, or the data is clustered at several scales | | Product | the coordinates are close to independent in the chosen split, or the main use is fast distance computation | | Scalar grid | training a codebook is the problem: usage must be uniform by construction, or the vocabulary has to be very large ($2^{16}$ and above) | ## C. The VQ-VAE and what it needed to be trainable ### C1. The picture A VQ-VAE (van den Oord, Vinyals and Kavukcuoglu 2017) is an autoencoder with a quantizer in the middle. An encoder maps the input $x$ to a continuous output $z_e(x)$, the quantizer replaces it by the nearest of $K$ codebook vectors $e_1,\dots,e_K\in\mathbb R^D$, and a decoder reconstructs the input from that vector: $$k=\arg\min_j\|z_e(x)-e_j\|_2,\qquad z_q(x)=e_k,\qquad \hat x=\mathrm{Dec}(z_q).$$
An autoencoder with a quantiser in the middle. Press Next. The index $k$ of the chosen vector is the code.
For images the encoder is convolutional and produces a *grid* of such vectors, each replaced by its own index; the paper uses a 32×32 grid for ImageNet and 8×8×10 for CIFAR10. It is trained in two stages: first the autoencoder, with a uniform prior over codes, then an autoregressive model (PixelCNN for images, WaveNet for audio) on the indices.
Tokenize first, model the codes second. What stage 1 cannot reconstruct, stage 2 cannot generate (C11).
The paper compresses 128×128×3 ImageNet images into a 32×32 grid with $K=512$, which is $128\cdot128\cdot3\cdot8$ bits of pixels against $32\cdot32\cdot9$ bits of code, a reduction of about 42.7 in bits (the paper rounds it to 42.6). Three things set it apart from earlier discrete-latent VAEs: discrete codes, a prior learned afterwards, and no *posterior collapse* (a VAE with a powerful decoder learning to ignore its latents). It reports 4.67 bits per dimension on CIFAR10 against 4.51 for a continuous VAE and 5.14 for the discrete VIMCO baseline, all as lower bounds. ### C2. The probabilistic view: the KL term is the rate The VQ-VAE is a variational autoencoder whose posterior is deterministic. The posterior $q(z=k\mid x)$ is one-hot at the nearest entry, and the prior $p(z)$ is uniform over the $K$ entries. The evidence bound of any VAE is $\log p(x)\ge\mathbb E_q[\log p(x\mid z)]-\mathrm{KL}(q(z\mid x)\,\|\,p(z))$.
Derivation: with a one-hot posterior and a uniform prior the KL term is the constant $\log K$, which is the number of nats in the code The claim is that for a latent $z\in\{1,\dots,K\}$ with posterior concentrated on one value and a uniform prior, $\mathrm{KL}(q\|p)=\log K$ for every input, so the bound is the reconstruction term minus the constant $\log K$ per latent. **Step 1: the posterior.** $q(z=k\mid x)=1$ for $k=\arg\min_j\|z_e(x)-e_j\|$ and $0$ otherwise (the paper's equation 1). **Step 2: the prior.** $p(z=k)=1/K$ for every $k$. **Step 3: the divergence.** $$\mathrm{KL}(q\|p)=\sum_kq(k)\log\frac{q(k)}{p(k)}=1\cdot\log\frac{1}{1/K}=\log K.$$ Only the term with $q=1$ contributes. **Step 4: with $N$ latents.** For a grid of $N$ independent latents the divergence is $N\log K$ nats, or $N\log_2K$ bits. **Step 5: read it as a rate.** $N\log_2K$ is exactly the number of bits needed to write down the code with a fixed-length code (A6). So the bound is *minus a distortion term minus a rate term*, and the rate term is fixed by the architecture: it does not depend on $x$ or on the parameters, so it drops out of the gradient. The training is then pure reconstruction, at a fixed rate, which is the problem of Part A. **What this does not say.** The two extra terms of the training loss (the codebook and commitment terms, C4) are *not* part of this bound. They are added to make the training work, and they are not derived from the evidence. And once an autoregressive prior is fit afterwards, the code costs less than $N\log_2K$ bits (the prior's cross-entropy), which is why the paper's likelihood is evaluated with that prior as $\log p(x)\ge\log p(x\mid z_q)\,p(z_q)$.
Two consequences. The VQ-VAE is a rate–distortion machine in the exact sense of A, with the rate *chosen* by the grid size and $K$. And it makes the *prior* the place where structure is exploited: since the autoencoder spends the full $\log_2K$ bits whatever the content, the model of the codes in the second stage is what recovers the savings (Part G). ### C3. The gradient: the straight-through estimator The $\arg\min$ has no useful derivative. A small move of $z_e$ either changes nothing (same nearest entry) or jumps (a different entry), so the derivative is zero almost everywhere and undefined on the cell boundaries. Left alone, the encoder would learn nothing. The VQ-VAE copies the decoder's gradient from $z_q$ straight to $z_e$ in the backward pass: the *straight-through estimator* (Bengio, Léonard and Courville 2013; recalled). In code, $$z_q^{\mathrm{st}}=z_e+\mathrm{sg}[e_k-z_e],$$ which equals $e_k$ going forward and has the derivative of $z_e$ going back. The paper observes that $\nabla_zL$ tells the encoder how to change its output to lower the loss, possibly into a different cell on the next pass, and that the simple estimator worked well.
The gradient skips the argmin. Press Next.
Why should this work? One argument is exact for a uniform quantizer.
Derivation: a quantizer smoothed by one cell of noise has slope one, which is what straight-through assumes The claim is that for a uniform quantizer of step $s$, the expected output when a uniform dither of width $s$ is added to the input equals the input, so the derivative of the *smoothed* quantizer is one. That is the assumption the straight-through estimator makes: it treats the quantizer as having slope one, as the smoothed quantizer does. **Step 1: the quantizer.** $q(z)=s\,\mathrm{round}(z/s)$. **Step 2: the smoothed quantizer.** Let $u$ be uniform on $[-s/2,s/2]$ and define $\bar q(z)=\mathbb E_u\,q(z+u)$. **Step 3: its value.** For a given $z$, $z+u$ ranges over an interval of width $s$ centred on $z$. The rounded value takes the neighbouring grid levels with probabilities that are linear in $z$ (the share of the interval that falls in each cell), so $\bar q(z)=z$. The staircase averaged against one step of noise is the identity. **Step 4: the derivative.** $\bar q'(z)=1$ for all $z$. Replacing $q'$ by one in the chain rule is the straight-through rule. This is the slope of the smoothed quantizer, not the exact derivative of the smoothed loss $\mathbb E_u\,f(q(z+u))$, which equals $f'(z)$ only when $f$ is linear. **Step 5: what it needs.** Equality in step 3 uses the uniform grid. A codebook with unequal cells (a learned VQ codebook) does not have $\bar q(z)=z$ exactly, so for a VQ-VAE the argument is an approximation that improves as the cells get small relative to the decoder's curvature. **What this does not say.** The estimator still ignores *where the cell boundaries are*: the gradient has no term for what happens when $z_e$ crosses one, which is why a code can flip back and forth, and why the commitment term of C4 exists.
A one-parameter example makes the claim numerical. Let the input be $x\sim\mathcal N(0,1)$, the encoder a single scale $w$ ($z=wx$), the quantizer a uniform grid of step $\Delta=1$, and the decoder the inverse scale ($\hat x=q/w$). Then the loss depends only on the effective step $s=\Delta/w$ and is the distortion of a uniform quantizer with that step, which can be computed exactly. The true gradient with respect to $w$ goes through the staircase; the straight-through gradient treats $q$ as the identity and equals $-2L/w$. Figure 3 compares the two.
The straight-through gradient is the true gradient to four decimals while the step is no larger than the spread of the data, and over-states it once the quantizer is coarse. The ratio of the straight-through gradient to the exact gradient, for one encoder scale, against the quantizer step in units of the standard deviation of the data. Both gradients point downhill at every step shown. Drag the ring: at a step of 1 the loss is 0.0833, the high-rate value step²/12, and the ratio is 1.0000; at 2 the loss is 0.330 against 0.333 for the formula and the ratio 1.046; at 3 the loss is 0.648 against 0.75 and the ratio 1.52.
The dither argument motivates the rule but does not explain this figure. What the figure shows is that, for a fine quantizer, where the loss follows the high-rate formula $L=s^2/12$, the exact derivative with respect to the scale is $-2L/w$, and the straight-through value is the same $-2L/w$. The agreement holds up to a step as large as one standard deviation of the data because the formula is accurate there; the correction to $s^2/12$ for a Gaussian is of order $e^{-2\pi^2/s^2}$ (recalled, from the Poisson sum), which is negligible at $s=1$. Only for a coarse quantizer does the estimator drift (a 5 percent over-statement at step 2, 52 percent at step 3). So the straight-through gradient is an *accurate* first-order estimate when the quantization error is small relative to the scale of the signal, and an unreliable one in the regime of very few codes, which is where the extensions of Part D live. The example has a fixed grid and a single parameter: it says nothing about the interaction of a *learned* codebook with the encoder, which is where the next two sections go. ### C4. The three-term loss The VQ-VAE objective, in the paper's equation 3 (with the reconstruction term written as a loss), $$\mathcal L=\log p(x\mid z_q)\;\to\;\|x-\hat x\|^2\;+\;\|\mathrm{sg}[z_e(x)]-e\|_2^2\;+\;\beta\,\|z_e(x)-\mathrm{sg}[e]\|_2^2,$$ has three terms, and each trains a different set of parameters.
Three terms, three destinations. The decoder is trained by reconstruction; the encoder also through the copy, and by the commitment term; the codebook only by its own term.
The codebook entries get no gradient from the data term, because the straight-through path skips them, so the codebook term is one step of Lloyd's algorithm (A4) done by gradient. The codebook is dimensionless, so entries can drift while the encoder's output grows; the commitment term makes the encoder commit. The paper reports the training robust to $\beta$ between 0.1 and 2.0, uses $\beta=0.25$ throughout, and says the right value depends on the scale of the reconstruction loss. With $N$ latents the codebook and commitment terms are averaged over the $N$. The later papers (VQGAN, ViT-VQGAN, RQ-VAE, TIGER) keep the same terms with $\beta=0.25$ in several cases, and replace the squared-error reconstruction by perceptual and adversarial losses. ### C5. Learning the codebook: gradient or moving average The codebook term moves each entry toward the encoder outputs assigned to it. The optimal value has a closed form.
Derivation: the moving-average update is online k-means, with a memory of about a hundred batches The claim is that the exponential-moving-average update of the VQ-VAE appendix tracks the centroid of each entry's points, and that with decay $\gamma=0.99$ the average has a half-life of about 69 steps. **Step 1: the goal.** For entry $e_i$ with assigned encoder outputs $z_{i,1},\dots,z_{i,n_i}$ the loss $\sum_j\|z_{i,j}-e_i\|^2$ is minimised by their mean, $e_i=\frac1{n_i}\sum_jz_{i,j}$ (the centroid condition of A4). **Step 2: the problem with minibatches.** A minibatch sees only a few of each entry's points, so the mean over the whole data cannot be computed directly. Keep running estimates of the numerator and denominator instead. **Step 3: the update.** With $n_i^{(t)}$ the number of vectors in batch $t$ assigned to entry $i$, $$N_i^{(t)}=\gamma N_i^{(t-1)}+(1-\gamma)\,n_i^{(t)},\qquad m_i^{(t)}=\gamma m_i^{(t-1)}+(1-\gamma)\sum_jz_{i,j}^{(t)},\qquad e_i^{(t)}=\frac{m_i^{(t)}}{N_i^{(t)}}.$$ $N_i$ is a running count and $m_i$ a running sum; their ratio is a running mean. This is the paper's equations 6 to 8. **Step 4: the memory.** A batch's weight is multiplied by $\gamma$ each step, so it is halved after $\ln(0.5)/\ln\gamma$ steps: 69.0 steps for $\gamma=0.99$, and the total weight of all past batches is $1/(1-\gamma)=100$ batches. **Step 5: a consequence for dead codes.** If no point is assigned to entry $i$ for a long stretch, $N_i$ decays toward zero and $m_i$ with it, but their *ratio* $e_i$ stays where it was (both shrink at the same rate until floating-point or the smoothing terms of an implementation intervene). The entry is neither updated nor removed: it stays dead. **What this does not say.** The paper uses the loss version in its experiments and mentions the moving average as an option, and the VQ-VAE-2 paper uses the moving average with the default $\gamma=0.99$. The two are the same algorithm at different speeds; neither addresses the problem of an entry that nobody chooses.
### C6. Dead codes: why a codebook shrinks silently An entry moves only when some point is assigned to it, by gradient or by moving average. If the entries start spread over a wider range than the data, those out in the tails have no points and get no update of their own: they are *dead*, and the codebook shrinks silently to the few entries that happen to be in the data. Figure 4 shows the mechanism in its cleanest form.
Unused entries are wasted bits. A healthy codebook is used across most of its entries; a collapsed one keeps only the few that the encoder outputs happen to reach.
Codes started outside the data stay outside it. 64 codes begin evenly spread on an interval of the given half-width; the data are a standard Gaussian, drawn as the grey curve. A code is blue if a batch of 1,024 points lands in its cell in expectation, and hollow copper otherwise. The second row is after 50 steps of Lloyd's algorithm. Drag the ring along the spread axis: at spread 3, which just covers the data, 60 codes are alive; at 8, 24 are alive at the start and 30 after training, with a perplexity of 20.4, the number of equally used codes that would give the same entropy. A code whose cell holds no points does not move by itself; a few come alive only because the codes next to them move into the gap, and the outermost codes, which own the empty tails, drift outward.
The summary number is the *perplexity* $\exp(H)$ of the usage distribution $H=-\sum_kp_k\ln p_k$: 64 codes used equally have perplexity 64, one code used always has perplexity 1. The figure is the clean case, a one-dimensional Gaussian with exact cell masses; real encoders are not Gaussian, but the mechanism does not depend on that. **A sharper diagnosis.** The SimVQ paper (Zhu et al. 2024) traces the collapse to what it calls *disjoint optimisation*: only the nearest codes receive gradient, so the same subset moves closer to the encoder distribution, "somewhat akin to" a cocoon effect, while the rest of the codebook is never updated and stays under-used. It motivates the work by the scale at which this bites: Chameleon's image codebook is limited to 8k, far below the 128k vocabulary of a language model like LLaMA3, because large codebooks do not train. **The fixes in the papers.** They fall into five groups, plus a sixth that removes the table.
Six ways out. Five repair the table; the last deletes it (D3, D4).
1. **Start the codes where the data are.** TIGER sets the first codebook by running k-means on the first training batch and trains for many epochs, reporting that this gives high codebook usage (at least 80 percent); audio codecs use k-means initialisation and randomised restarts of unused entries (as the DAC paper describes the existing practice). 2. **Look up in a small space.** ViT-VQGAN projects the encoder output to a low-dimensional space *for the lookup only*, then maps the matched code back to the high-dimensional embedding; and it $\ell_2$-normalises both encoder outputs and codebook entries so that Euclidean distance becomes cosine similarity. Its ablation (Table 4 of that paper) is the cleanest evidence in the literature: | Lookup dimension | 256 | 128 | 64 | 32 | 16 | 8 | 4 | 32, no $\ell_2$ normalisation | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Codebook usage | 4% | 14% | 37% | 96% | 95% | 96% | 96% | 2% | | FID of the generated images | 3.68 | 2.77 | 2.50 | 1.55 | 1.50 | 1.52 | 3.68 | 5.44 | Usage is the percentage of codes used for a batch of 256 test images, averaged over the test set, with a codebook of 8,192 entries; the numbers are the authors'. A 32-dimensional lookup with normalisation uses 96 percent of the codes; the same width without normalisation uses 2 percent. The DAC audio codec adopts both tricks and reports that they remove the need for k-means initialisation and restarts, with a lookup in 8 or 32 dimensions against embeddings of 1,024. 3. **Re-parameterise the codebook.** SimVQ writes each entry as a linear combination of a small learned basis, $\hat e_i=c_iW$ with $W\in\mathbb R^{d\times d}$ shared by all entries, and trains $W$ with the coefficients $c_i$ frozen at their random initial values. Now every update to $W$ moves *every* entry, rotating and stretching the whole space, so no entry is left behind. The paper describes this as one linear layer and reports 100 percent codebook utilisation in its tables at sizes from 1,024 up to 262,144 entries (its own results), and shows three regimes: updating the coefficients only (vanilla VQ, when $W=I$), updating $W$ only (all entries adapt), or both. 4. **Change the gradient.** The rotation trick (Fifty et al. 2024) replaces the straight-through copy by a rotation and rescaling that carries the encoder output onto its entry, treated as a constant in the backward pass, so the angle and the size between the two reach the gradient. The authors' complaint is that the copy loses everything about how far and in which direction the output is from its entry. They report better reconstruction, higher codebook use and lower quantization error across 11 training setups. 5. **Add a term that rewards use.** An entropy penalty on the usage (D4) rewards even use directly. The LFQ paper cites the entropy loss of the MaskGIT tokenizer as its inspiration; the FSQ paper lists entropy penalties among the extras that its method avoids. **One further fix: remove the table.** The codes of Part D3 and D4 (finite scalar and lookup-free quantization) have no codebook to collapse. ### C7. Relaxations: the discrete VAE of DALL-E The other way to train a discrete latent is to relax it. The DALL-E paper trains a *discrete VAE* that compresses each 256×256 RGB image into a 32×32 grid of tokens with 8,192 possible values, so 1,024 tokens of 13 bits, 13,312 bits per image. It does not use the straight-through route of the VQ-VAE; it uses the Gumbel-softmax relaxation (Jang et al. 2016; Maddison et al. 2016): the encoder outputs 8,192 logits at each position and the decoder is fed a *soft* mixture of codebook vectors with a temperature $\tau$ that is annealed from 1 down to 1/16 (the appendix gives the schedule), where the relaxation is nearly tight. The paper reports that the evidence bound became hard to optimise because the posterior is discrete, that annealing schedules for the temperature and step size were important for closing the gap between the relaxed and the true bound, and that 1×1 convolutions at the ends of the decoder helped it generalise from the relaxed to the true bound. At sampling time the tokens are the argmax of the encoder logits, without noise. It also speculates, in a footnote, that for small values of the KL weight the noise from the relaxation makes the optimiser reduce codebook usage early in training. The VQ-VAE paper made the opposite finding for the *soft-to-hard* relaxations of earlier compression work: the authors could not train with them from scratch, because the decoder was always able to invert the continuous relaxation so that no actual quantisation took place. The two statements are consistent: a relaxation works when the temperature is annealed to the discrete limit and the training is checked against the true bound; it fails when the decoder can use the slack. ### C8. VQ-VAE-2: a hierarchy of codes VQ-VAE-2 (Razavi, van den Oord and Vinyals 2019) stacks two levels of quantization for 256×256 images. The encoder first transforms and downsamples the image by a factor of 4 to a 64×64 representation, then by a factor of two more to a top-level 32×32 map; both maps are quantized. The *top* code models global structure and the *bottom* code, conditioned on it, local detail. The decoder reads both. The model is trained as a VQ-VAE with the same three-term loss and the moving-average codebook update with $\gamma=0.99$. The paper reports that the representation is over 30 times smaller than the image and that training and sampling the prior in latent space is also 30 times faster than in pixel space. A PixelCNN-family prior with self-attention (PixelSnail) is fit to the top level, and a smaller conditional prior to the bottom level; the paper argues that the hierarchical factorisation also makes it possible to train larger priors, because the bottom level needs only local receptive fields. Two lessons recur later. A *hierarchy of codes* separates what is global from what is local, which is what residual codes (D1), the audio hierarchies of E4 and the multi-scale tokenizers of D7 do in different ways. And the cost of the tokenizer is the *size* of the sequence the prior must model: 32×32 and 64×64 maps are 1,024 and 4,096 positions. ### C9. VQGAN: a perceptual codebook VQGAN (Esser, Rombach and Ommer 2020) changes the *loss*, not the quantizer. The squared error of the VQ-VAE is replaced by a perceptual reconstruction loss, and an adversarial term from a patch-based discriminator is added to keep the reconstruction sharp at high compression. The objective is $\min_{E,G,Z}\max_D\ \mathbb E\big[\mathcal L_{VQ}+\lambda\,\mathcal L_{GAN}\big]$ with an adaptive weight $$\lambda=\frac{\nabla_{G_L}[\mathcal L_{rec}]}{\nabla_{G_L}[\mathcal L_{GAN}]+\delta},$$ the ratio of the gradient norms of the two losses with respect to the last layer of the decoder, with $\delta=10^{-6}$ for numerical stability. So the discriminator is neither ignored nor allowed to overwhelm the reconstruction. A transformer then models the sequence of indices in raster order as $p(s)=\prod_ip(s_i\mid s_{Derivation: how much attention the spatial and depth split saves The claim is that for $T$ positions and $D$ codes each, the pairwise attention of one flat sequence is $(TD)^2$ and that of the two-level design is $T^2+TD^2$. **Step 1: the flat sequence.** All $TD$ tokens attend to each other (causally, so half of the pairs; the factor is the same on both sides and is dropped). The count is $(TD)^2=T^2D^2$. **Step 2: the spatial level.** The spatial transformer sees only $T$ positions: $T^2$ pairs. **Step 3: the depth level.** At each of the $T$ positions the depth transformer sees $D$ codes: $D^2$ pairs per position, $TD^2$ in all. **Step 4: a number.** For the paper's $T=64$ and $D=4$ the flat count is $256^2=65{,}536$ and the split count is $64^2+64\cdot16=5{,}120$, a factor of 12.8 fewer. A 16×16 grid of single codes would cost $256^2=65{,}536$, the same as the flat 8×8×4 sequence: the same bits, and the same cost, unless the sequence is split. **What this does not say.** It counts attention pairs only. The depth transformer is small, so its feed-forward cost is small; the spatial transformer's width and the number of layers set the rest, and nothing here compares quality. The paper also addresses *exposure bias*: at sampling time the codes at deeper levels are harder to predict, since they encode finer detail, and errors accumulate with depth. It defines a categorical distribution over codes conditioned on the residual, $Q_\tau(k\mid z)\propto\exp(-\|z-e(k)\|^2/\tau)$, and uses it for *soft labelling* of the target codes, supervising the transformer with the geometry of the codebook, and for stochastic sampling of the codes during training. Results (class-conditional ImageNet, FID without rejection sampling, as the authors report): the 480M-parameter RQ-Transformer reaches 15.72, matching the 1.4B VQ-GAN's 15.78, and the 1.4B version reaches 11.56. The comparisons are the authors' own. ### D2. RQ-Kmeans: residual quantization with no gradient The industrial recommenders of Part F use a simpler version. OneRec's technical report tokenizes items with *RQ-Kmeans* (Luo et al. 2024, as cited there): for layer $l=1,\dots,L_t$, with the first residual the item's embedding, run k-means on the residuals of all items to get $N_t$ centroids, assign each item to its nearest centroid, subtract it, and move to the next layer. There is no encoder, no decoder and no gradient: the "codebook learning" is Lloyd's algorithm of A4, applied $L_t=3$ times. The report compares it with RQ-VAE on a three-layer codebook of 8,192 entries per layer: | | RQ-VAE | RQ-Kmeans | | --- | --- | --- | | Reconstruction loss | 0.0548 | 0.0410 | | Codebook utilisation, layers 1 / 2 / 3 | 1.0000 / 0.9963 / 0.9958 | 1.0000 / 1.0000 / 1.0000 | | Token distribution entropy, layers 1 / 2 / 3 | 8.3892 / 8.4805 / 8.6037 | 8.9191 / 8.7770 / 8.7276 | The reconstruction loss is 25.18 percent lower. The entropies are in a unit the report does not name; the natural logarithm of 8,192 is 9.011, which fits the numbers (it is the largest value an entropy of 8,192 symbols can take in nats), so reading them as nats gives a perplexity of $e^{8.9191}\approx7{,}500$ of 8,192 codes for the first layer of RQ-Kmeans against $e^{8.3892}\approx4{,}400$ for RQ-VAE: 91 percent of the codes in use, against 54. That reading is this note's inference. These are the report's own numbers for its own data, and its embeddings (multimodal features aligned with collaborative signals) differ from those of the RQ-VAE it compares against, so it is not a controlled comparison of the two quantizers. The reason this can work is worth stating, because it follows from A4 and C6. Lloyd's algorithm *initialised from the data* and run to convergence has no dead-code problem in the sense of C6 (entries start at data points) and no gradient-estimator bias. What it gives up is *joint* training with an encoder and a decoder: the embedding to be quantized is fixed beforehand, so the codes cannot adapt the embedding to be easier to quantize. ### D3. Finite scalar quantization FSQ (Mentzer et al. 2023) removes the codebook. Project the encoder output to a few dimensions $d$ (the paper says typically fewer than 10), bound each number, and round it to one of $L_i$ values: $$\hat z_i=\mathrm{round}\big(f_i(z_i)\big),\qquad f(z)=\lfloor L/2\rfloor\tanh(z)\ \text{(odd }L\text{)}.$$ The paper's listing handles even $L$ with an offset of $\tfrac12$ and a small margin ($\epsilon=10^{-3}$), so that $f$ maps onto exactly $L$ rounded values. The implied codebook is the product of the per-dimension sets, of size $\prod_iL_i$, and a vector maps to an integer by reading its digits in a mixed-radix number system with basis $(1,L_1,L_1L_2,\dots)$. Gradients use the straight-through estimator on `round`, as in C3. There is nothing to look up, nothing to collapse in the sense of C6, and no commitment loss, because every dimension is used by construction.
A table against a grid. FSQ replaces the search over K vectors by independent rounding.
Table 1 of the paper recommends level sets to match common codebook sizes, with the heuristic $L_i\ge5$: | Target size | Levels | Product | | --- | --- | --- | | $2^8$ | 8, 6, 5 | 240 | | $2^{10}$ | 8, 5, 5, 5 | 1,000 | | $2^{12}$ | 7, 5, 5, 5, 5 | 4,375 | | $2^{14}$ | 8, 8, 8, 6, 5 | 15,360 | | $2^{16}$ | 8, 8, 8, 5, 5, 5 | 64,000 | The authors report that FSQ is competitive with VQ on image generation and on the UViM tasks (depth, colourisation, panoptic segmentation), marginally worse on the UViM ones, and does not suffer from codebook collapse. On codebook use their comparison is stark: VQ falls below 50 percent usage for codebooks larger than $2^{11}$ and never uses more than about $2^{10}$ codes for larger ones, while FSQ uses more than $2^{15}$ of $2^{16}$. They also observe diminishing gains from ever larger codebooks, because the compression cost of the representation keeps growing. "No collapse by construction" has a precise meaning, and Figure 5 shows its limit. Every *level* of every channel exists, but whether the data *land* on all of them depends on the spread of the encoder's output relative to the bound. If $z$ is small, $\tanh$ is nearly linear and everything rounds to the central levels; if $z$ is large, $\tanh$ saturates and mass piles on the extreme levels.
The grid cannot collapse, but how evenly it is used depends on the scale of what feeds it. Each panel is one channel of an FSQ with levels 8, 5, 5, 5 (1,000 codes); the bars are the share of Gaussian encoder outputs of the given spread that fall on each level after the paper's bounding and rounding. Drag the ring along the scale axis: at a spread of 1 the perplexity is 942 of 1,000 and 500 codes receive at least one point in a batch of 1,024; at 0.2 the perplexity is 20 and 59 codes are alive; at 6 the perplexity is 111 and 136 are alive.
The encoder is trained, so it can learn to put its outputs where they are useful; the figure says the *quantizer* is not the thing that guarantees even use, the scale of what feeds it is, and that a norm layer or a loss on the scale of $z$ is part of a working FSQ. The paper does not claim otherwise. What FSQ gives up is a learned geometry: the cells are a fixed grid in the latent space, and any structure has to be learned by the encoder that feeds the grid rather than by the codebook. ### D4. Lookup-free quantization MAGVIT-v2's lookup-free quantizer (Yu et al. 2023) is the binary case of the same idea, with the embedding dimension of the codebook set to zero so the codebook is an integer set of size $K$. The latent is decomposed into $\log_2K$ independent dimensions, each quantized to $\{-1,+1\}$ by the sign function, $q(z_i)=\mathrm{sign}(z_i)$, and the token index is read from the signs as a binary number, $$\mathrm{Index}(z)=\sum_{i=1}^{\log_2K}2^{i-1}\,\mathbf 1[z_i>0].$$ A vocabulary of $2^{18}=262{,}144$ is 18 signs. The authors report that, for their lookup-free quantizer, reconstruction and generation both improve consistently as the vocabulary grows, a property they did not see for ordinary VQ, in which a very large vocabulary can hurt the language model that has to predict the codes. They also factorise the prediction layer so that a small transformer can predict a large vocabulary. To keep all codes in use the paper adds an entropy penalty, $$\mathcal L_{entropy}=\mathbb E\big[H(q(z))\big]-H\big[\mathbb E\,q(z)\big],$$ computed with a soft version of the quantizer. Under the independence of the dimensions the first entropy is a sum over the $\log_2K$ dimensions, which makes this term cheap at any size. The second entropy, of the average code distribution, is estimated directly up to about $K=2^{18}$; beyond that the paper splits the dimensions into groups to save memory.
Derivation: the entropy penalty is minus the mutual information between the input and its code The claim is that with a soft code distribution $q(\cdot\mid z)$, the penalty $\mathbb E[H(q(\cdot\mid z))]-H(\mathbb E[q(\cdot\mid z)])$ equals $-I(Z;Q)$, so minimising it maximises the information the code carries about the input. **Step 1: the two entropies.** For a batch, $q(\cdot\mid z)$ is the distribution over codes that the quantizer assigns to the input $z$. The first term is the average over inputs of the entropy of that distribution, $H(Q\mid Z)$. The second term is the entropy of the *average* distribution over the batch, which is the marginal distribution of the code, $H(Q)$. **Step 2: the identity.** $I(Z;Q)=H(Q)-H(Q\mid Z)$. **Step 3: the sign.** $\mathcal L_{entropy}=H(Q\mid Z)-H(Q)=-I(Z;Q)$. **Step 4: what each part does.** Minimising $H(Q\mid Z)$ makes each input's assignment *confident* (close to one-hot), so the quantizer is decisive. Maximising $H(Q)$ makes the *batch* use the codes evenly, which is the usage of A6. Both are the same thing, information. **What this does not say.** The information is measured on soft assignments, and the trained quantizer uses hard signs; the gap between the two is a relaxation the paper manages with a soft quantizer. And a high $I(Z;Q)$ says the code is informative about $z$, not that it is informative about what the decoder needs.
### D5. Binary spherical quantization BSQ (Zhao et al. 2024) fixes two things in LFQ: the entropy term is expensive at large vocabularies, and the quantization error is not bounded. It projects the embedding to a lower-dimensional *unit hypersphere*, $u=v/\|v\|$, and quantizes each axis to a sign, scaled so that the result is again on the sphere: $$\hat u=\tfrac1{\sqrt L}\,\mathrm{sign}(u),$$ with $\mathrm{sign}(0)=1$ and a straight-through estimator. The codebook is the $2^L$ corners of a cube scaled to the sphere.
Derivation: the quantization error of a spherical sign code is bounded The claim is that for unit $u\in\mathbb R^L$ and $\hat u=\mathrm{sign}(u)/\sqrt L$, $\|u-\hat u\|^2=2-\frac2{\sqrt L}\sum_i|u_i|\le2-\frac2{\sqrt L}$. **Step 1: both vectors have unit length.** $\|u\|=1$ by construction and $\|\hat u\|^2=L\cdot\frac1L=1$. **Step 2: expand.** $\|u-\hat u\|^2=\|u\|^2+\|\hat u\|^2-2u\cdot\hat u=2-2u\cdot\hat u$. **Step 3: the inner product.** $u\cdot\hat u=\frac1{\sqrt L}\sum_iu_i\,\mathrm{sign}(u_i)=\frac1{\sqrt L}\sum_i|u_i|$. **Step 4: bound the sum from below.** Because $\|u\|_1\ge\|u\|_2$, $\sum_i|u_i|\ge1$, so the error is at most $2-2/\sqrt L$. **What this does not say.** The bound is a worst case over all unit $u$ and shrinks toward 2 as $L$ grows (the worst case is a point as far from every corner as possible). What matters in practice is the *typical* error, which for an encoder trained to put its outputs near corners is much smaller. The paper's claim is that the approximation error is bounded, not that it is small.
The paper states three advantages: it is parameter-efficient (no explicit codebook), it scales to arbitrary token dimensions, and it is compact, compressing visual data by up to $100\times$ with minimal distortion. Its factorised entropy computation, again from the independence of the binary axes, is what makes the entropy term cheap at large $L$. It reports state-of-the-art reconstruction on image and video benchmarks at 2.4 times the throughput of the best earlier methods, and, by fitting an autoregressive prior over the codes for arithmetic coding, video compression comparable to standard codecs. These are the authors' results. ### D6. SimVQ and the rotation trick, as extensions Both were introduced in C6 and belong in the family tree: SimVQ re-parameterises the codebook so that all entries train (D-axis: *usage*), and the rotation trick changes the gradient (D-axis: *the estimator*). Neither changes the capacity of the codebook, and both are reported against the same vanilla VQ baseline in their own papers. ### D7. Multi-scale codes and next-scale prediction Visual autoregressive modelling (VAR, 2024) changes the *order* in which codes are predicted. Instead of the raster-scan next-token prediction of C9, it predicts a coarse-to-fine sequence of *scales*, "next-scale prediction": a whole low-resolution map of codes first, then a larger one conditioned on it, and so on. The abstract reports, on ImageNet 256×256, an improvement of the FID over an autoregressive baseline from 18.65 to 1.73 and of the inception score from 80.4 to 350.2, with 20 times faster inference, outperforming the Diffusion Transformer on several measures, and power-law scaling with linear correlation coefficients near −0.998. The tokenizer behind it is a multi-scale quantizer (recalled, and not opened beyond the abstract: each scale quantizes what the coarser scales left), which makes it the spatial analogue of the residual code of B2: the "depth" of the stack is resolution. These are the authors' benchmark results. ### D8. One-dimensional tokenizers All the grid tokenizers above produce tokens tied to image patches (2D), so the number of tokens is set by the downsampling factor: 256, 1,024 or 4,096 for a 256×256 image (for factors 16, 8, 4; the TiTok paper). TiTok (Yu et al. 2024) drops the grid, takes a one-dimensional sequence of latent tokens, and reports that an image can be represented with as few as 32 tokens, against 256 for a VQGAN and 65,536 pixels, for reconstruction and for generation. It exploits what the authors call region redundancy. The practical meaning for this note is that the *number of tokens* and the *number of bits per token* are separate dials, and that the sequence the prior has to model can be short. ### D9. The family at a glance | Quantizer | What is learned | How gradients pass | Collapse risk | Hierarchy | Typical sizes in the sources | Used in | | --- | --- | --- | --- | --- | --- | --- | | VQ (flat) | $K$ entries | straight-through; codebook loss or EMA | high: C6 | none | 512 to 16,384 | VQ-VAE, VQ-VAE-2, VQGAN | | VQ, factorised and $\ell_2$-normalised | entries in a small lookup space | same | low (96% usage in the ablation) | none | 8,192 | ViT-VQGAN, DAC | | VQ, re-parameterised (SimVQ) | basis $W$ and coefficients | same | none reported (100%) | none | 1,024 to 262,144 | SimVQ | | Residual (RQ-VAE) | $K$ entries shared by $D$ depths | straight-through, EMA | moderate | coarse to fine | 16,384 × depth 2 to 16 | RQ-VAE, SoundStream, EnCodec, DAC, Mimi, TIGER | | RQ-Kmeans | $L$ codebooks, no network | none (k-means) | none in the report | coarse to fine | 8,192 × 3 | OneRec | | FSQ | nothing; a grid of levels | straight-through on `round` | none by construction; depends on scale (Figure 5) | none | 240 to 64,000 | MaskGIT and UViM experiments of the paper | | LFQ | nothing; signs | straight-through; entropy loss | low with the entropy loss | none | up to $2^{18}$ | MAGVIT-v2 | | BSQ | nothing; signs on a sphere | straight-through; bounded error | low | none | arbitrary dimension | BSQ-ViT | | Gumbel-softmax (dVAE) | $K$ entries by relaxation | reparameterised, annealed | needs care (C7) | none | 8,192 | DALL-E | ## E. Where the codes go: images and audio, and the bits they spend ### E1. The bit accounting of a tokenizer Every tokenizer in the sources is a choice of how many tokens, from how large a vocabulary. Multiplying gives the bits per item, and dividing into the bits of the raw data gives the compression. The table uses only numbers stated in the papers. | System | Input | Tokens | Vocabulary | Bits per input | Raw bits | Compression | | --- | --- | --- | --- | --- | --- | --- | | VQ-VAE (ImageNet) | 128×128×3 image | 32×32 | 512 | 9,216 | 393,216 | 42.7× | | DALL-E dVAE | 256×256×3 image | 32×32 | 8,192 | 13,312 | 1,572,864 | 118× | | VQGAN | 256×256×3 image | 16×16 | 1,024 | 2,560 | 1,572,864 | 614× | | VQGAN, larger codebook | 256×256×3 image | 16×16 | 16,384 | 3,584 | 1,572,864 | 439× | | RQ-VAE | 256×256×3 image | 8×8×4 | 16,384 | 3,584 | 1,572,864 | 439× | | Chameleon | 512×512×3 image | 1,024 | 8,192 | 13,312 | 6,291,456 | 473× | Raw bits are at 8 bits per colour channel. Two readings. First, the same bits can be spent in different shapes: RQ-VAE's 8×8×4 stack and VQGAN's 16×16 grid with the same vocabulary spend 3,584 bits each, and RQ-VAE gets there with a quarter of the positions (D1). Second, the compression is large (hundreds of times) and the reconstructions are good enough that the tokenizers are used to generate pictures: the bits in the table are the entire message the generator has to write. ### E2. Images: generation on top of the codes
The tokenizer is both ends of the pipe. Hover a box to follow the path.
**Autoregressive on a grid.** The VQ-VAE with a PixelCNN prior (C1), VQGAN with a transformer prior (C9) and the RQ-Transformer (D1) are all the same two-stage plan: tokenize, then model the tokens. Reported quality moves with the tokenizer at least as much as with the prior: VQGAN's reconstruction FID falls from 7.94 to 4.98 as the codebook grows from 1,024 to 16,384 entries on the same grid, and to 1.45 with a 64×64 grid (C9), and RQ-VAE's from 17.95 to 1.83 as depth grows (D1). **One vocabulary for text and pictures.** Chameleon (the Chameleon Team 2024) trains a tokenizer that encodes a 512×512 image into 1,024 discrete tokens from a codebook of 8,192 and puts them in one vocabulary with the text tokens: a vocabulary of 65,536 that includes the 8,192 image tokens, learned with a subword tokenizer for the text. Image and text are then a single token stream for an early-fusion model, the strongest form of the "interface" argument of A1. It is also where the codebook size bites: the SimVQ paper cites Chameleon's 8k image codebook as being far below the 128k vocabularies of language models. **The tokenizer is the ceiling.** MAGVIT-v2 (Yu et al. 2023) argues that language models trail diffusion models in visual generation largely because of the tokenizer. With its lookup-free quantizer (D4), a vocabulary of $2^{18}$ and a design that gives images and video a shared vocabulary, the authors report that language models using it outperform diffusion models on ImageNet and Kinetics benchmarks. The title of the paper is the thesis, "Language Model Beats Diffusion: Tokenizer is Key to Visual Generation", and it is the authors' result on their benchmarks. **Coarse to fine instead of raster order.** VAR (D7) keeps the discrete tokens and changes the order, predicting whole scales; its authors report both better FID and 20 times faster inference than an autoregressive baseline. **A counter-trend: continuous tokens.** Li, Tian, Li, Deng and He (2024) argue that vector-quantized tokens are not a necessity for autoregressive image generation: each token's distribution can be modelled with a small diffusion process instead of a softmax, which removes the discrete tokenizer. They report that this works for standard autoregressive and for masked autoregressive models. Two readings are open. Discrete codes were a convenience for the cross-entropy interface (A1), and a different loss removes the need. Or the discrete interface still wins on speed, scale or tooling, which this paper does not test. Whichever is right for images, it does not touch the second reason of A1, the *address*, which has no continuous counterpart: an item still needs a name. ### E3. Audio: a bitrate you can read off A neural audio codec is a VQ-VAE whose quantizer is residual, trained end to end with adversarial and spectral losses, and its output is judged by a bitrate. The arithmetic is the same as E1, with time in place of space: frames per second times codebooks times bits per code. | Codec | Sample rate | Frame rate | Codebooks | Entries each | Bitrate | Compression of 16-bit audio | | --- | --- | --- | --- | --- | --- | --- | | SoundStream, the AudioLM example | not given | 50 Hz | 4 | 1,024 | 2.0 kbps | not given | | EnCodec | 24 kHz | 75 Hz | 2, 4, 8, 16, 32 | 1,024 | 1.5, 3, 6, 12, 24 kbps | 256× at 1.5 kbps | | DAC | 44.1 kHz | 86.1 Hz | 9 | 1,024 | 7.75 kbps (reported as 8) | 91× | | Mimi (Moshi) | 24 kHz | 12.5 Hz | 8 | 2,048 | 1.1 kbps | 349× | EnCodec's frame rate and codebook size come from its description (75 latent steps per second at 24 kHz, at most 32 codebooks of 1,024 entries, bandwidths of 1.5, 3, 6, 12 and 24 kbps at 24 kHz); each codebook costs $75\times10=750$ bits per second, so 2, 4, 8, 16 and 32 codebooks give those five bandwidths. DAC reports compressing 44.1 kHz audio 90 times at about 8 kbps, with a stride of 512 giving 86 frames a second and 9 codebooks. The computation $44{,}100/512\times9\times10=7{,}752$ bits per second reproduces the order of that figure, and 16-bit audio at 44.1 kHz is 705.6 kbps, a factor of 91. SoundStream's own abstract gives a range of 3 to 18 kbps, and the AudioLM paper's example of $50\times4\times\log_21024=2{,}000$ bits per second shows the formula. **One model, many bitrates.** SoundStream trains with *quantizer dropout*: for each training example it samples how many of the residual stages to keep, so one model works at every bitrate from 3 to 18 kbps with negligible loss compared with fixed-bitrate models, the authors report; EnCodec selects a number of stages per batch in the same spirit. The DAC paper found that dropping stages on every example hurts quality at full bandwidth, and applies the dropout to each example with probability $p=0.5$, which it reports closes the gap to the full-bandwidth model while keeping the lower bitrates. This is B2's second property, an adjustable rate, in use. **Improved usage.** DAC's central change is the lookup trick of C6, borrowed from ViT-VQGAN: lookup in a low-dimensional space (8 or 32 dimensions against embeddings of 1,024) with $\ell_2$-normalised vectors. It reports that EnCodec, trained at 24 kbps, still under-uses its codebooks (shown by the per-codebook entropy of its Figure 1) and that the new design trained with the original VQ-VAE losses, without k-means initialisation or restarts, uses them better. ### E4. Semantic tokens and acoustic tokens There are two kinds of audio token, and they are made differently. *Acoustic tokens* come from a codec such as SoundStream or Mimi: they are learned for reconstruction and carry fine detail. *Semantic tokens* come from a self-supervised speech model. AudioLM (Borsos et al. 2022) computes them by taking activations from an intermediate layer of a w2v-BERT model, 1,024-dimensional vectors at 25 Hz (one every 40 ms), and training **k-means** with $K$ clusters on them; the token is the index of the nearest centroid. A semantic token is therefore a plain flat codebook of A4 on a frozen embedding, and a rate of $25\times\log_21024=250$ bits per second. HuBERT's training targets are of the same kind: its offline clustering step starts from k-means with 100 clusters on MFCC features and refines with further iterations on learned features. The HuBERT abstract describes the model as predicting the labels of masked frames from these cluster assignments. AudioLM combines them in a hierarchy, because the two kinds serve conflicting needs: long-range coherence from the semantic tokens, fidelity from the acoustic ones.
Semantic first, then acoustic. Press Next.
Mimi folds the hierarchy into one codec: it **distils** semantic information from a non-causal self-supervised model into the first of its eight codebooks, so that the same causal network produces semantic and acoustic tokens and can stream, and the other seven carry acoustic detail. The Moshi paper observes that, with eight codebooks at 12.5 Hz, the sequence is 100 tokens per second, so five minutes of audio would be 30,000 time steps, which is why the model generates the eight codes of a frame in a small hierarchy (a depth transformer, in RQ-Transformer's terms, D1) and not flat. The structure is the same as D1's residual stack: the first code is the *what* and the rest the *how*, and the reason to separate them is to let the sequence model work on the part that has long-range structure. ### E5. Text already is a codebook A tokenizer for text such as byte-pair encoding is a learned codebook of subword strings, fit to the data by a rule that merges frequent pairs. The analogy is useful in one respect: in all three modalities a fixed vocabulary of a few thousand to a few hundred thousand symbols is chosen by a trade between sequence length and vocabulary size. The MAGVIT-v2 authors make the comparison themselves: the visual vocabularies in use were smaller than the 200,000 or more of a language model's. The YouTube study (F5) draws the connection the other way, using SentencePiece, a text tokenizer, to merge frequent runs of semantic-ID codes. ### E6. Where the evidence points | Use | What the code is | Why it is discrete | Evidence for it | Against or open | | --- | --- | --- | --- | --- | | Image generation | patch tokens | softmax and language-model tooling; one stream with text | VQGAN, RQ-VAE, MAGVIT-v2, VAR (authors' benchmarks) | continuous tokens with a diffusion loss (Li et al.) | | Audio codec | residual codes per frame | a fixed, adjustable bitrate; streaming | SoundStream, EnCodec, DAC, Mimi (authors' tests) | none opened here | | Speech and music generation | semantic plus acoustic tokens | long-range structure from the semantic tokens | AudioLM, Moshi | none opened here | | Recommendation, retrieval | an address per item | generation, cold start, memory | TIGER, YouTube study, OneRec (Part F) | see F9 | ## F. Semantic IDs: naming items by their content ### F1. What is wrong with an ID A recommender must refer to items. The usual way is an *item ID*: an integer, and a row of an embedding table indexed by it. The row is learned from the interactions the item has had, so an item with few interactions has a poor row, and a new item has none. Singh et al. (2023) describe the two obvious repairs for a ranking model at YouTube and what each costs. Randomly hashing IDs into a smaller table saves memory but generalises poorly to new and long-tail items; the production ranking model they study uses on the order of ten million hash buckets for on the order of a hundred million videos. Feeding a frozen content embedding instead fixes the generalisation but reduces *memorisation*, the ability to learn the particulars of one popular item, and hurts quality. A Semantic ID is the compromise: compact discrete codes learned from the frozen content embeddings and used where the random ID was. Hashing collides items at random; a semantic ID collides (and shares prefixes between) items that are *alike*, which is what a ranking model wants from its collisions. ### F2. Before semantic IDs: documents as identifiers The idea of *generating an identifier* came to retrieval first. The Differentiable Search Index (Tay et al. 2022) trains a single Transformer to map a query directly to a document identifier, "all information about the corpus is encoded in the parameters of the model", and studies how identifiers should be represented: naively as text strings of integers, as unstructured *atomic* identifiers (one token per document), and as simple structured semantic identifiers built from the content. GenRet (Sun et al. 2023) argues that rule-based identifiers, such as a document's title or the result of clustering BERT embeddings, often fail to capture the document's semantics, and *learns* the tokenization instead with a discrete autoencoder of three parts: a tokenization model that produces identifiers, a reconstruction model that rebuilds the document from its identifier, and a sequence-to-sequence retrieval model that generates identifiers for queries. That is the VQ-VAE recipe of Part C applied to documents. The honest status for search is the study of Pradeep et al. (2023): on the full MS MARCO collection of 8.8 million passages, with models up to 11 billion parameters, generative retrieval is competitive with dual encoders on small corpora, but scaling to millions of passages "remains an important and unsolved challenge". Synthetic queries as document representations mattered most, and adding parameters helped little. This is search, not recommendation, but it shows what the addressing idea has to beat: a nearest-neighbour search over embeddings, which is cheap, mature and indifferent to catalogue size. ### F3. TIGER: generate the address of the next item TIGER (Rajput et al. 2023) turns the idea into a retriever for recommendation. Every item gets a Semantic ID. A user's history becomes a sequence of such IDs, and a transformer encoder–decoder is trained to *generate the ID of the next item*, a few tokens, instead of scoring every item.
Generate the address, then look it up. Press Next.
The pieces, as the paper gives them: - A frozen content embedding of each item goes through an MLP encoder with hidden layers of 512, 256 and 128 units to a latent of 32 numbers. - A residual quantizer with $L=3$ levels, each with $K=256$ entries of 32 numbers, a *separate* codebook per level (B2), with $\beta=0.25$ and the loss $\mathcal L=\|x-\hat x\|^2+\sum_l\big(\|\mathrm{sg}[r_l]-e_{c_l}\|^2+\beta\|r_l-\mathrm{sg}[e_{c_l}]\|^2\big)$ with $r_l$ the residual entering level $l$, trained until at least 80 percent of the codebook is used, with k-means initialisation on the first batch. - Items that share all three codes receive a fourth code, 0, 1, …, to make the ID unique; items with no collision get 0. A lookup table maps IDs back to items. - The datasets are three Amazon review categories of about 12,000 to 18,000 items each. At generation time beam search returns the top candidates. - The authors compare Semantic IDs from RQ-VAE with locality-sensitive hashing and find RQ-VAE better; hierarchical k-means, they say, loses the meaning between clusters; plain VQ-VAE codes did similarly for retrieval but lose the hierarchy that gives the later abilities. Using IDs of six codes from codebooks of 64 instead of four from 256, they find the metrics robust, at a longer input sequence. They report that TIGER outperforms the baselines on all three datasets, with up to 29 percent better NDCG@5 than SASRec and 17.3 percent better Recall@5 than S³-Rec on Beauty; the baseline numbers are taken from an earlier paper's published results (Zhou et al.), except P5, which they re-ran. For cold start they remove 5 percent of the test items from training, give them Semantic IDs from the trained quantizer, and report that with a cap of 10 percent of unseen items in the candidate list TIGER does better than a nearest-neighbour search in the same content-embedding space (their Figure 5). All of these are the authors' numbers on three small public datasets. ### F4. The first-principles reading: a learned tree A flat recommender computes a score for each of $N$ items and normalises: a softmax over $N$. A Semantic ID factorises the same distribution by the chain rule over its codes: $$P(\text{item})=\prod_{l=1}^{L}P(c_l\mid c_1,\dots,c_{l-1},\ \text{history}).$$
The code tuple is a path in a learned tree. Each level is one softmax over $K$ codes.
Derivation: the chain rule over codes is a tree, and costs $KL$ logits instead of $N$ The claim is that if every item has a unique code tuple $(c_1,\dots,c_L)$ with $c_l\in\{1,\dots,K\}$, then predicting the tuple one code at a time defines a proper distribution over items using $KL$ logits per path, not $N$. **Step 1: items as leaves.** Draw a tree whose root has $K$ children, each with $K$ children, down $L$ levels. A tuple is a path from the root; the unique tuple of an item is its leaf. The tree has $K^L\ge N$ leaves, and some are empty. **Step 2: probabilities by the chain rule.** Any distribution over tuples factorises as $P(c_1)P(c_2\mid c_1)\cdots P(c_L\mid c_{ This is not new: it is a *hierarchical softmax* (Morin and Bengio 2005, recalled), in which the tree is fixed by a hand-made hierarchy or by Huffman coding on frequency. What is new is where the tree comes from: it is *learned from content*, so that items with the same prefix are similar, and it is shared across the catalogue by one set of codebooks. ### F5. The YouTube study: semantic IDs inside a ranking model Singh et al. do not generate IDs; they use them as *features* in a production ranking model for the next video to watch. The semantic IDs come from a 2048-dimensional content embedding produced by a video encoder (a VideoBERT-backbone transformer over audio and visual features) quantized with an RQ-VAE of $L$ levels. Using the codes directly as a single feature would still need a table of $K^L$ rows, so they hash *sub-pieces* of the code sequence, in two ways. - **N-grams.** Group the codes into overlapping or consecutive groups of $N$ and give each group its own learned embedding; an item's representation is the sum of its groups' embeddings. The table for an $N$-gram has $K^N$ rows, which grows exponentially with $N$, so the study keeps $N\le2$. - **SentencePiece.** Learn a vocabulary of merged runs of codes from the data with the SentencePiece tokenizer, the same way a text tokenizer merges frequent character pairs, so frequent code runs get their own embedding and rare ones are split. The abstract reports that SentencePiece outperformed the hand-made n-grams, and the body argues it balances *generalisation* (short pieces, shared across items) against *memorisation* (long pieces, specific to few items). The prefix structure is what makes this work: a short piece is shared by every item in a coarse cluster, which is what a new video can borrow from, and a long piece pins down nearly one item. They report that in a real YouTube ranking model Semantic IDs improved generalisation on new and long-tail items without costing overall quality, and state that they "can replace the direct use of video IDs". The numbers are the authors' own. ### F6. LC-Rec and LETTER: fixing collisions and adding collaborative signal Two papers change how the tree is built. **LC-Rec** (Zheng et al. 2023) builds item indices with a residual quantizer, feeds them to a large language model as extra tokens, and fine-tunes the model on recommendation and alignment tasks. Its change to the tokenizer is about collisions. The usual fix (TIGER's) adds one more level to the tree and gives each colliding item a distinct supplementary code; the paper objects that this puts semantically irrelevant distributions in the last level. It instead constrains the assignment at the *last* level to be *uniform*: for a batch $B$ of last-level residuals and $K$ codewords it solves $$\min_q\ \sum_{r\in B}\sum_{k=1}^Kq(k\mid r)\,\|r-v_k\|^2\quad\text{s.t.}\quad\sum_kq(k\mid r)=1,\quad\sum_{r\in B}q(k\mid r)=\frac{|B|}{K},$$ an optimal-transport problem whose cost is the squared distance, solved with the Sinkhorn–Knopp algorithm. The constraint says every codeword receives the same number of items.
Derivation: balanced assignment as optimal transport, and how Sinkhorn solves it The claim is that forcing equal load at the last level is a transport problem between the items and the codewords, and that alternately rescaling rows and columns of a kernel matrix solves its entropy-regularised form. **Step 1: the plan.** Let $q_{rk}\ge0$ say how much of item $r$ goes to codeword $k$. Each item sends one unit: $\sum_kq_{rk}=1$. Each codeword receives $|B|/K$: $\sum_rq_{rk}=|B|/K$. A matrix with these marginals is a *transport plan*. **Step 2: the cost.** Sending item $r$ to codeword $k$ costs $c_{rk}=\|r-v_k\|^2$. The best plan minimises $\sum_{rk}q_{rk}c_{rk}$ under the two constraints. Without the second constraint, each item would take its nearest codeword and the loads could be uneven. **Step 3: regularise.** Adding an entropy term $\varepsilon\sum q\log q$ (the standard way to make the problem fast; recalled, the paper states only that it uses Sinkhorn–Knopp) makes the optimal plan have the form $q_{rk}=a_r\,e^{-c_{rk}/\varepsilon}\,b_k$ for two positive vectors $a,b$. **Step 4: Sinkhorn–Knopp.** Let $G_{rk}=e^{-c_{rk}/\varepsilon}$. Alternate: set $a_r=1\big/\sum_kG_{rk}b_k$ so each row sums to one; then set $b_k=\tfrac{|B|}{K}\big/\sum_rG_{rk}a_r$ so each column sums to $|B|/K$. Each step fixes one set of marginals exactly, and the iteration converges to a plan with both. **Step 5: read off the codes.** Assign each item the codeword with the largest $q_{rk}$ (or sample from its row). **What this does not say.** The balance costs distortion: an item may be sent to a codeword that is not its nearest, so the reconstruction error rises. How much depends on how uneven the natural clusters are, and the paper does not give the figure in the text opened here. And a balanced *last level* spreads items over the leaves of each prefix; it does not remove collisions if there are more items than leaves in some prefix.
**LETTER** (Wang et al. 2024) is a learnable tokenizer for generative recommendation that adds three regularisers to the RQ-VAE: a semantic one (the RQ-VAE reconstruction), a *collaborative* one (a contrastive alignment loss that pulls the quantized embedding toward the item's embedding from a collaborative-filtering model), and a *diversity* loss against code-assignment bias. It also proposes a ranking-guided generation loss for the sequence model. The motivations in the abstract are that item tokenizers built from content alone lack the signal of user behaviour and suffer from assignment bias. ### F7. OneRec: the tokenizer and the model in production OneRec (Deng et al. 2025 and the OneRec Technical Report) takes the generative idea to a whole pipeline at Kuaishou. It encodes a user's behaviour history and decodes the videos the user may want, session by session, with a mixture-of-experts transformer, and aims to *replace the cascade of retrieval and ranking* with one model. Its tokenizer is the part that matters here. **Tokenizer.** The report says the item space is billion-scale and ever-growing, which prevents generating atomic identifiers, and tokenizes items into coarse-to-fine semantic IDs with a reduced and fixed vocabulary, for knowledge transfer among similar items and better generalisation to new ones. It argues that earlier solutions (TIGER, LC-Rec) build the IDs from content features alone, neglecting collaborative signals and giving poorer reconstruction. Its tokenizer therefore first aligns multimodal representations of each video with collaborative signals, then quantizes with RQ-Kmeans (D2): three layers of 8,192 centroids, so $8{,}192^3\approx5.5\times10^{11}$ possible IDs, and the comparison with RQ-VAE of D2 (reconstruction loss 25 percent lower, usage 1.0 in all layers). **Scaling the codebook and the search.** Expanding the codebook from 8,192 to 32,768 improves playtime metrics significantly and interaction metrics slightly, evaluated with reward-based metrics because the next-token loss cannot be compared across vocabularies of different size (the cross-entropy is over a larger candidate set). At inference the model generates $K$ items per request (Pass@K): the report finds consistent improvements from $K=8$ to 512 and marginal ones from 512 to 1,024, and deploys $K=512$. **Legality.** The report defines the *legality ratio* as the proportion of generated semantic-ID sequences that map to a real item, and notes that the space of ID sequences ($N_t^{L_t}$) is far larger than the number of videos, so illegal generations are possible. It finds that reinforcement learning with its preference-alignment method *increases* illegal outputs, attributing it to a "squeezing effect" of negative advantages that compresses probability mass onto the current best output so that some legal tokens end up with probabilities comparable to illegal ones, and addresses it with a format reward. The lesson for F4 is that the valid-ID constraint is not a corner case: it interacts with the training. **Deployment claims.** The first OneRec paper reports a 1.6 percent increase in watch time in the main scene of Kuaishou. The technical report describes a one-week experiment on 5 percent of traffic in two short-video scenarios: the pure generative model with preference alignment matched the performance of the whole complex system, and adding reward-model selection gave statistically significant gains of +0.54 and +1.24 percent in app stay time and +0.05 and +0.08 percent in seven-day lifetime, after which deployment was expanded to about 25 percent of queries per second; a local-services scenario reports a 21.01 percent growth in GMV. These are the company's own measurements and were not independently checked. **A different route to generative recommendation.** HSTU (Zhai et al. 2024) reformulates recommendation as *sequential transduction* in a generative modelling framework ("Generative Recommenders") and proposes a new architecture for high-cardinality, non-stationary streams of user actions, reporting that it scales with compute where deep learning recommendation models do not. Its abstract does not mention discrete codes. It is a reminder that "generative recommender" and "semantic ID" are separable ideas. ### F8. Collisions, bits and memory The tree must have room for every item. Figure 6 is the arithmetic for Lantern's two million articles with 64-number vectors and $K=256$.
Three codes of 256 hold two million articles in 24 bits against a minimum of 20.9, but even spreading would leave one in nine sharing an ID. Top bar: bits in the ID, with the violet mark at $\log_2N=20.9$, the fewest bits that can tell two million items apart. Middle bar: the share of items that have at least one other item with the same ID, assuming the items fall evenly over the $256^L$ possible IDs, which is the best case. Bottom bars: memory for a 64-number vector per item at 16 bits (blue) against $L$ codebooks plus one byte per code per item (violet). Drag the ring along the codes axis: at three codes, 11.2 percent of items collide and memory falls from 256 MB to 6.1 MB; at four, 0.047 percent collide and memory is 8.1 MB.
Three things follow. 1. **The floor is $\log_2N$ bits.** An ID that tells $N$ items apart needs at least $\log_2N$ bits. With two million items that is 20.9, so two codes of 8 bits (16) cannot do it and three (24) can. OneRec's three layers of 8,192 carry $3\times13=39$ bits. 2. **Slack is the price of uneven use.** Even at 24 bits, throwing two million items evenly into $2^{24}$ cells leaves 11 percent sharing a cell, the birthday effect. Real items are not even: similar items fall near each other by design, so collisions are more frequent than this best case. The size of that effect was not measured here. TIGER's answer is the extra code; LC-Rec's is the balanced last level (F6); the cost of the first is a longer sequence, of the second a higher reconstruction error. 3. **Memory is the headline.** Replacing a 256 MB table by about 6 MB of codebooks and codes is the saving a hashed or quantized ID gives. It matters when the catalogue is hundreds of millions of items; at two million, 256 MB is small, and the case for Semantic IDs rests on generalisation more than on memory. ### F9. Decoding: beam search, validity and the width you pay for A tree of $K^L$ leaves is searched by *beam search*: keep the $b$ best prefixes after each code, expand each by the $K$ codes, keep the best $b$ again. A request costs about $bL$ forward steps of the decoder instead of one scoring of every item. Two quantities govern quality: the beam width $b$ (the Pass@K of the OneRec report), and the validity of what the beam returns. TIGER measures invalid IDs at 0.1 to 1.6 percent of the top 10 and 0.3 to 6 percent of the top 20, a rate that rises with depth into the ranking, since lower-ranked candidates are lower-probability paths that are more likely to leave the set of real IDs. A prefix tree of valid IDs (a trie) constrains the beam so that only codes with a valid continuation can be chosen, which removes invalid outputs without retraining; it costs a lookup per step and a trie in memory of the size of the catalogue, and it changes the probabilities the model assigns because mass on invalid branches is renormalised away (this paragraph's reasoning is the note's, not a source's). ### F10. Drift and evaluation Semantic IDs depend on the encoder that produced the embeddings and on the codebooks trained on them. If either is retrained, the codes of the same item change, and every model that consumed the old codes has to be retrained or the mapping frozen. Whether an IDs-as-features model (F5) or a generative one (F3, F7) tolerates this better is not answered by the sources opened here. Evaluation has two layers, the *tokenizer* and the *recommender*; the tokenizer is judged by the quantities of Part H (utilisation, entropy, collision rate, reconstruction, prefix purity), the recommender by the usual ranking metrics against a tuned retrieval baseline. The OneRec report evaluates its tokenizer with three metrics (reconstruction loss, codebook utilisation, token-distribution entropy); TIGER inspects the first code against product categories. ### F11. What is solved, claimed and open | Problem | Status | What the sources show | | --- | --- | --- | | Memory of an embedding table | Solved by construction | $L$ codebooks plus $L$ bytes per item (Figure 6) | | Generalisation to new and long-tail items | Reported by the authors | Singh et al. in YouTube ranking; TIGER cold start on Amazon data | | Generating an item instead of scoring all | Demonstrated at small scale; reported at one company's scale | TIGER on about 10 to 20 thousand items; OneRec's own deployment figures | | Uniqueness of IDs | Handled | TIGER's extra token; LC-Rec's balanced last level | | Invalid IDs | Rare, not zero; interacts with RL | 0.1 to 1.6 percent of top-10 in TIGER; OneRec's format reward | | Collaborative signal in the codes | Addressed | LETTER; OneRec's aligned embeddings | | Catalogue changes and retraining | Open | not in the sources opened: when items arrive or the encoder is retrained, codes may move | | Scale to millions of items in public benchmarks | Open | no TIGER-style result with controlled comparison found at that scale; the passage study is a caution | | Quality against a tuned embedding retriever | Open | the sources compare with other sequential recommenders, not against a tuned nearest-neighbour baseline at large scale | ## G. Modelling the codes The tokenizer produces a grid, a stack, or a sequence of integers; a second model has to learn their distribution. The design of that model is where most of the cost of a discrete-code system sits, and it is shaped by the choices of Parts B to D. ### G1. The prior is an entropy coder After training the tokenizer, a sequence model assigns each code a probability given the codes before it. Writing the data down with that model costs $\sum_t-\log_2p(c_t\mid c_{Answers 1. $\log_2(10^8)=26.6$ bits, so 4 codes of 8 bits (32 bits). With 3 codes (24 bits) even a perfectly even spread leaves 99.7 percent of items sharing an ID. 2. $4\times11\times12.5=550$ bits per second. 3. $8\times5\times5\times5=1{,}000$ codes, $\log_2 1000=9.97$ bits. 4. With an even spread, $2\times10^6-E[\text{occupied cells}]=114{,}610$, which is 5.7 percent of the articles. The 11.2 percent of Figure 6 counts every article that shares a cell, including the first of each pair. 5. Table: 12.8 GB. Codes: 0.4 GB (400 million bytes) plus 131 KB of codebooks. 6. Half-life $\ln0.5/\ln0.999=693$ steps, window $1/(1-\gamma)=1{,}000$ batches. 7. $1{,}024\times\log_2300=8{,}427$ bits, about 9 percent fewer than 9,216 (the first stage spent the full rate; the prior recovers a little). 8. Each codebook is $75\times10=750$ bits per second; 12 give 9,000 bits per second, 9 kbps. 9. Growing from $2^8$ to $2^{12}$ entries adds 4 bits to the vector, which is 1 bit per number for $d=4$: $6.02$ dB. For $d=16$ it is 0.25 bit per number: 1.5 dB. The same bits buy less in more dimensions (A5). 10. Flat: $(256\times8)^2=4{,}194{,}304$. Split: $256^2+256\times64=81{,}920$. A factor of 51. 11. $H=0.5(1)+0.25(2)+0.125(3)\cdot2=1.75$ bits; perplexity $2^{1.75}=3.36$ of 4 entries. 12. $2-2/\sqrt{18}=1.53$ (against a distance of at most 2 between two unit vectors).
### H8. Further reading Entries with a link support the claims made from them at the level of the abstract or text opened. Entries without a link were not opened. **Quantizers and tokenizers.** - van den Oord, Vinyals and Kavukcuoglu 2017, *Neural Discrete Representation Learning*, [arXiv:1711.00937](https://arxiv.org/abs/1711.00937). - Razavi, van den Oord and Vinyals 2019, *Generating Diverse High-Fidelity Images with VQ-VAE-2*, [arXiv:1906.00446](https://arxiv.org/abs/1906.00446). - Ramesh et al. 2021, *Zero-Shot Text-to-Image Generation* (DALL-E), [arXiv:2102.12092](https://arxiv.org/abs/2102.12092). - Esser, Rombach and Ommer 2020, *Taming Transformers for High-Resolution Image Synthesis*, [arXiv:2012.09841](https://arxiv.org/abs/2012.09841). - Yu et al. 2021, *Vector-quantized Image Modeling with Improved VQGAN*, [arXiv:2110.04627](https://arxiv.org/abs/2110.04627). - Lee, Kim, Kim, Cho and Han 2022, *Autoregressive Image Generation using Residual Quantization*, [arXiv:2203.01941](https://arxiv.org/abs/2203.01941). - Mentzer, Minnen, Agustsson and Tschannen 2023, *Finite Scalar Quantization: VQ-VAE Made Simple*, [arXiv:2309.15505](https://arxiv.org/abs/2309.15505). - Yu et al. 2023, *Language Model Beats Diffusion: Tokenizer is Key to Visual Generation*, [arXiv:2310.05737](https://arxiv.org/abs/2310.05737). - Zhao, Xiong and Krähenbühl 2024, *Image and Video Tokenization with Binary Spherical Quantization*, [arXiv:2406.07548](https://arxiv.org/abs/2406.07548). - Yu, Weber et al. 2024, *An Image is Worth 32 Tokens for Reconstruction and Generation* (TiTok), [arXiv:2406.07550](https://arxiv.org/abs/2406.07550). - Fifty et al. 2024, *Restructuring Vector Quantization with the Rotation Trick*, [arXiv:2410.06424](https://arxiv.org/abs/2410.06424). - Zhu et al. 2024, *Addressing Representation Collapse in Vector Quantized Models with One Linear Layer* (SimVQ), [arXiv:2411.02038](https://arxiv.org/abs/2411.02038). - *Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction*, [arXiv:2404.02905](https://arxiv.org/abs/2404.02905). - Li, Tian, Li, Deng and He 2024, *Autoregressive Image Generation without Vector Quantization*, [arXiv:2406.11838](https://arxiv.org/abs/2406.11838). - Chameleon Team 2024, *Chameleon: Mixed-Modal Early-Fusion Foundation Models*, [arXiv:2405.09818](https://arxiv.org/abs/2405.09818). **Audio.** - Zeghidour et al. 2021, *SoundStream: An End-to-End Neural Audio Codec*, [arXiv:2107.03312](https://arxiv.org/abs/2107.03312). - Défossez et al. 2022, *High Fidelity Neural Audio Compression* (EnCodec), [arXiv:2210.13438](https://arxiv.org/abs/2210.13438). - Kumar et al. 2023, *High-Fidelity Audio Compression with Improved RVQGAN* (DAC), [arXiv:2306.06546](https://arxiv.org/abs/2306.06546). - Borsos et al. 2022, *AudioLM: a Language Modeling Approach to Audio Generation*, [arXiv:2209.03143](https://arxiv.org/abs/2209.03143). - Hsu et al. 2021, *HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units*, [arXiv:2106.07447](https://arxiv.org/abs/2106.07447). - Défossez et al. 2024, *Moshi: a speech-text foundation model for real-time dialogue* (introduces Mimi), [arXiv:2410.00037](https://arxiv.org/abs/2410.00037). **Recommendation and retrieval.** - Tay et al. 2022, *Transformer Memory as a Differentiable Search Index*, [arXiv:2202.06991](https://arxiv.org/abs/2202.06991). - Sun et al. 2023, *Learning to Tokenize for Generative Retrieval* (GenRet), [arXiv:2304.04171](https://arxiv.org/abs/2304.04171). - Rajput et al. 2023, *Recommender Systems with Generative Retrieval* (TIGER), [arXiv:2305.05065](https://arxiv.org/abs/2305.05065). - Singh et al. 2023, *Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations*, [arXiv:2306.08121](https://arxiv.org/abs/2306.08121). - Zheng et al. 2023, *Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation* (LC-Rec), [arXiv:2311.09049](https://arxiv.org/abs/2311.09049). - Wang et al. 2024, *Learnable Item Tokenization for Generative Recommendation* (LETTER), [arXiv:2405.07314](https://arxiv.org/abs/2405.07314). - Zhai et al. 2024, *Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations* (HSTU), [arXiv:2402.17152](https://arxiv.org/abs/2402.17152). - Deng et al. 2025, *OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment*, [arXiv:2502.18965](https://arxiv.org/abs/2502.18965); and the *OneRec Technical Report*, [arXiv:2506.13695](https://arxiv.org/abs/2506.13695). - Pradeep et al. 2023, *How Does Generative Retrieval Scale to Millions of Passages?*, [arXiv:2305.11841](https://arxiv.org/abs/2305.11841). **Not opened here.** Lloyd (1982) and Max (1960) on optimal quantizers; Panter and Dite (1951) and Gish and Pierce (1968) on the high-rate formulas; Shannon (1959) on rate–distortion; Bengio, Léonard and Courville (2013) on the straight-through estimator; Jang et al. and Maddison et al. (2016) on the Gumbel-softmax; Morin and Bengio (2005) on hierarchical softmax; Jégou, Douze and Schmid (2011) on product quantization; Chang et al. (2022) on MaskGIT; Copet et al. (2023) on delay patterns in music generation; Cuturi (2013) on Sinkhorn distances; Luo et al. (2024) on RQ-Kmeans, as cited by the OneRec report. **Companion notes.** [The Art of LLM Inference](inference) for the cost of the softmax and the sequence lengths above, and [The Art of Retrieval-Augmented Generation](rag) for the nearest-neighbour baseline to beat. ### Changelog - Version 1: first draft; residual codes, semantic IDs and the first three figures. - Version 2: full rewrite as a self-contained deep dive: first-principles derivations, the four ways to build a codebook (with a four-dimensional experiment), the VQ-VAE in full (probabilistic view, straight-through estimator, losses, moving averages, dead codes and their fixes, relaxations, VQ-VAE-2, VQGAN, ViT-VQGAN), the extensions (RQ-VAE, RQ-Kmeans, FSQ, LFQ, BSQ, SimVQ, rotation trick, VAR, TiTok), bit accounting for images and audio, semantic IDs for retrieval and recommendation (DSI, GenRet, TIGER, YouTube, LC-Rec, LETTER, OneRec, HSTU), modelling the codes, diagnostics and exercises.