# VAEs, Diffusion and Flow Matching: One Idea in Three Languages Three families of models dominate the generation of images, audio and video. A *variational autoencoder* turns data into a code and back, and is trained on a bound on the likelihood. A *diffusion model* destroys data with noise in a thousand small steps and learns to undo each. A *flow* moves noise onto the data with a smooth deterministic map, and *flow matching* trains that map by regression. They look like three machines (one pass to decode, hundreds of network evaluations per sample, an integrated differential equation) from three literatures with three vocabularies: evidence lower bound, score, velocity field. How many network evaluations does it take to draw one sample, and what is the best likelihood any of the three could reach? The first answer differs by a factor of a thousand between them; the second, for the small example used throughout, is the same for all three, 1.058 nats, and is reached only if the model is exact. That is the first hint that they are one thing. This note argues that they are one thing seen from three sides, and derives the identities that make it so. Each connection is an equation. Read it from the top or look things up. Part A says what a generative model has to do; B builds the VAE, C the flow, D the diffusion model as a VAE with a fixed encoder, E the same model as score estimation and as an ODE, F flow matching. G puts the pieces on one map, with latent diffusion, guidance, discrete data and what to monitor. H has the limits, exercises and sources. ## The answer in six sentences 1. A diffusion model is a hierarchical VAE with a *fixed* encoder: a chain of Gaussian noising steps, each latent the same size as the data (D1–D3). 2. Its training loss is the VAE's evidence lower bound, term by term, and each term is a squared error of a *denoiser*: the best guess of the clean data from a noisy copy (D3–D5). 3. A denoiser, a noise predictor and a score (the gradient of the log density of the noisy data) are the same object up to a known rescaling, by Tweedie's formula (E3). 4. The reverse of the noising process can be run as a stochastic differential equation or as a deterministic ordinary one, and both give the same density at every noise level; so every diffusion model is also a continuous normalizing flow, with exact likelihoods (E5–E7). 5. Flow matching trains that ordinary equation's velocity field directly by regression on straight-line or Gaussian paths. For Gaussian paths its network is the diffusion network in other coordinates, and what differs is the path, the time weighting, the parameterisation and the sampler (F2–F3). 6. What is genuinely different is not the family but a handful of choices: how much of the work the encoder does, how the noise levels are weighted, and how many network evaluations a sample costs (G1). ## The example that runs through the note One dimension is enough to see every identity, and the exact answer can be computed. The data are a mixture of two Gaussians: 40 percent of the mass in a peak of standard deviation 0.3 around −1.2, 60 percent in a peak of standard deviation 0.4 around 1.0. Its mean is 0.12, its variance 1.294, its differential entropy 1.058 nats. Noised as $x_t=\alpha_t x_0+\sigma_t\varepsilon$ it is again a two-Gaussian mixture at every noise level, so the density, its gradient, the best denoiser and the flow velocity are closed forms. What a one-dimensional example cannot show is what makes images hard: the data live near a thin surface in a space of millions of dimensions, a network must generalise from a finite set, and errors compound over hundreds of steps. Part H returns to this. ## How to read the notes Colour is a role, the same in the figures and the equations. Blue marks the data and the signal ($x_0$, $\alpha_t$, the data density). Copper marks noise and everything a network must learn or estimate ($\varepsilon$, $\sigma_t$, $\hat x_0$, $v_\theta$, the model $p_\theta$). Violet marks a derived law that follows from the other two: the score, the signal-to-noise ratio, the exact reverse step. Uncoloured symbols are bookkeeping (indices, times, expectations). In a display equation, follow the colours to see which parts are given, which are learned and which are consequences. The schema figures use the same code: blue is data, amber is noise, codes and the learned model, purple a derived law, red an obstacle or a cost, green a result. Derivations are folded, closed by default. Open questions are in boxes marked *Open question*. ## Notation | Symbol | Meaning | | --- | --- | | $x_0$, $x_t$ | a clean data point; the same point after noise at time $t\in[0,1]$ ($t=0$ data, $t=1$ pure noise) | | $\varepsilon$ | a standard Gaussian noise vector, independent of $x_0$ | | $\alpha_t$, $\sigma_t$ | signal and noise scale of the path $x_t=\alpha_tx_0+\sigma_t\varepsilon$; dots are time derivatives | | $\mathrm{snr}_t$, $\lambda_t$ | signal-to-noise ratio $\alpha_t^2/\sigma_t^2$ and its logarithm $\lambda_t=\log\mathrm{snr}_t$ | | $p_t$ | the density of $x_t$; $p_0$ is the data density | | $\nabla\log p_t(x)$ | the score of the noised data | | $\hat x_0(x,t)$ | a denoiser, the guess of $x_0$ from $x_t=x$: the exact one, $\mathbb E[x_0\mid x_t=x]$, or a network's approximation of it, as the text says; $\hat\varepsilon$ is the matching noise estimate | | $v_t(x)$, $u_t$ | the marginal velocity field of the flow, and the conditional velocity of one path, $u_t(x_t\mid x_0,\varepsilon)$ | | $z$ | a VAE latent; $q_\phi(z\mid x)$ is the encoder and $p_\theta(x\mid z)$ the decoder; $\tau^2$ is the decoder's noise variance | | $\mathcal L$ | a loss or bound; the subscript says which | | $\mathrm{mmse}(\mathrm{snr})$ | the mean square error of the best denoiser at that signal-to-noise ratio | | $h$, $I$ | differential entropy and mutual information, in nats | The collisions, admitted. $\sigma$ is the noise scale of the path, never a decoder's standard deviation ($\tau$). Lower-case $s$ is an earlier time in D2, an encoder's standard deviation in B3 and a coupling layer's scale in C2. $v$ is a velocity, a variance in fact (a) of A3, and the output of "v-prediction" (D4, a different quantity). $\lambda$ is the log signal-to-noise ratio and the loss weight $\lambda(t)$ of D6 and E7. $t$ is an integer step in D1 and D3 and a continuous time elsewhere. $a$ is an encoder coefficient in B3 and a drift in E5; $w$ a decoder weight in B3, an importance weight in B4, a guidance weight in G3, and $dw$ a Wiener increment in E4; $f$ a flow map in C, a drift in E, a test function in B2; $z$ a VAE latent and, in F1, the pair $(x_0,\varepsilon)$; $r$ a signal-to-noise ratio in D5 and $1-t$ in E5. $\beta_t$ is the DDPM schedule, $\beta$ the rate weight of the $\beta$-VAE. $p_t$ is the exact noised data and $p_\theta$ a model. ## A. What a generative model has to do ### A1. The target A generative model is a distribution $p_\theta$ that we can sample from and that is close to the distribution $p_{\rm data}$ the examples came from. The standard notion of close is the Kullback–Leibler divergence, $$\mathrm{KL}(\htmlClass{eq-obs}{p_{\rm data}}\,\|\,\htmlClass{eq-hid}{p_\theta})=\mathbb E_{x\sim \htmlClass{eq-obs}{p_{\rm data}}}\big[\log \htmlClass{eq-obs}{p_{\rm data}}(x)-\log \htmlClass{eq-hid}{p_\theta}(x)\big]=\underbrace{-h(\htmlClass{eq-obs}{p_{\rm data}})}_{\text{fixed}}+\underbrace{\mathbb E\big[-\log \htmlClass{eq-hid}{p_\theta}(x)\big]}_{\text{what we train}}.$$ The first term does not depend on the model, so minimising the divergence is minimising the average negative log-likelihood, whose floor is the entropy $h(p_{\rm data})$, reached when the model is exact. For the two-peak example the entropy is 1.058 nats; a model at 1.2 nats is 0.142 above the floor, and that gap is the divergence. Image papers quote bits per dimension: divide by $\ln 2$ and by the number of dimensions, after accounting for the width of the discretisation bins. Likelihood is the criterion that motivates every model in this note, but only classical flows train on it directly; the VAE trains on a bound and diffusion models on weighted denoising losses, and the reason is the subject of the next section. ### A2. Three obstacles and three ways round The obvious model is a network that outputs a density, trained by maximum likelihood. It meets three obstacles, and each family is a different answer to one of them. **The integral.** A model with a hidden code $z$ defines $p_\theta(x)=\int p_\theta(x\mid z)\,p(z)\,dz$, which has no closed form for a neural decoder. The *variational autoencoder* bounds it with one sampled code and learns which codes to sample (Part B). **The normalising constant.** A density $e^{-E_\theta(x)}/Z_\theta$ needs $Z_\theta=\int e^{-E_\theta}dx$, another intractable integral. The gradient $\nabla_x\log p_\theta(x)=-\nabla_xE_\theta(x)$ does not contain it: this is the *score*. Learn it and a sample comes from walking uphill on the log density with noise added; but the score of the data is only defined, and only well estimated, near the data. *Diffusion models* learn the score after noise has been added, at every noise level (Part E). **The invertibility.** A smooth invertible map $x=f_\theta(z)$ from a simple distribution gives the exact density by change of variables, at the price of an architecture that is invertible with a cheap Jacobian determinant. A *normalizing flow* pays it (Part C). A *continuous* flow loosens it by solving a differential equation, and *flow matching* removes the need to simulate that equation in training (Part F).
Each family answers one obstacle and pays for it. Red is the obstacle, amber the way round, dashed what is given up. Hover a box to follow one column.
The diffusion and flow columns are not separate: diffusion models turn out to be both a very deep VAE and a flow. ### A3. Gaussian facts used everywhere Four small facts carry most of the algebra of the note. They are collected here, with proofs of (b) and (c) in the fold ((a) and (d) are standard), so that later derivations can cite them by letter. **(a) Noising a Gaussian gives a Gaussian.** If $x_0\sim\mathcal N(\mu,v)$ and $x_t=\alpha x_0+\sigma\varepsilon$ then $x_t\sim\mathcal N(\alpha\mu,\alpha^2v+\sigma^2)$. Sums of independent Gaussians are Gaussian and variances add. **(b) The Kullback–Leibler divergence between Gaussians.** $$\mathrm{KL}\big(\mathcal N(m_1,s_1^2)\,\|\,\mathcal N(m_2,s_2^2)\big)=\ln\frac{s_2}{s_1}+\frac{s_1^2+(m_1-m_2)^2}{2s_2^2}-\frac12 .$$ **(c) The posterior of a linear Gaussian model.** If $z\sim\mathcal N(0,1)$ and $x\mid z\sim\mathcal N(wz,\tau^2)$ then $z\mid x\sim\mathcal N\!\big(\tfrac{w}{w^2+\tau^2}x,\ \tfrac{\tau^2}{w^2+\tau^2}\big)$ and $x\sim\mathcal N(0,w^2+\tau^2)$. **(d) Reparameterisation.** A sample from $\mathcal N(m,s^2)$ is $m+s\varepsilon$ with $\varepsilon\sim\mathcal N(0,1)$, so a quantity computed from the sample is a differentiable function of $m$ and $s$ with the randomness moved out.
Derivation of (b) and (c) **Claim (b).** The divergence between two univariate Gaussians has the closed form above. **Step 1: write the definition.** $\mathrm{KL}=\mathbb E_1[\ln p_1(x)-\ln p_2(x)]$ with $p_i=\mathcal N(m_i,s_i^2)$. **Step 2: expand the logs.** $$\ln p_1(x)-\ln p_2(x)=\ln\frac{s_2}{s_1}-\frac{(x-m_1)^2}{2s_1^2}+\frac{(x-m_2)^2}{2s_2^2}.$$ **Step 3: take the expectation under $p_1$.** The middle term has expectation $-\tfrac12$ since $\mathbb E_1(x-m_1)^2=s_1^2$. For the last term, $\mathbb E_1(x-m_2)^2=s_1^2+(m_1-m_2)^2$ by splitting $x-m_2=(x-m_1)+(m_1-m_2)$ and noting the cross term has mean zero. **Step 4: add.** $\ln\frac{s_2}{s_1}-\frac12+\frac{s_1^2+(m_1-m_2)^2}{2s_2^2}$, which is the claim. **Claim (c).** The posterior of $z$ given $x$. **Step 1: the joint is Gaussian.** $z$ and $x=wz+\tau\eta$ are linear in independent standard Gaussians, so $(z,x)$ is jointly Gaussian with $\mathrm{Var}(z)=1$, $\mathrm{Cov}(z,x)=w$, $\mathrm{Var}(x)=w^2+\tau^2$. **Step 2: condition.** For a jointly Gaussian pair with zero means, $z\mid x$ is Gaussian with mean $\frac{\mathrm{Cov}(z,x)}{\mathrm{Var}(x)}x=\frac{w}{w^2+\tau^2}x$ and variance $\mathrm{Var}(z)-\frac{\mathrm{Cov}(z,x)^2}{\mathrm{Var}(x)}=1-\frac{w^2}{w^2+\tau^2}=\frac{\tau^2}{w^2+\tau^2}$. **What this does not say.** Both are one-dimensional. In $d$ dimensions with covariance matrices the algebra is the same with matrix inverses, and the diagonal case is $d$ copies of the scalar case.
The same four facts hold coordinate by coordinate for a Gaussian with diagonal covariance, which is how images are handled throughout this literature: noise is added independently to every pixel. ### A4. Why noise makes the score learnable One more observation, because it is the idea behind the whole diffusion family. Data in high dimension occupy a thin region and $\log p_{\rm data}$ is minus infinity (or very steep) off it, so a network trained only on data never sees the places where a sampler will start. Gaussian noise of scale $\sigma$ smooths the density into the data density convolved with a Gaussian: positive everywhere, with a finite score that for large $\sigma$ is nearly that of one Gaussian. Small $\sigma$ keeps the fine structure; large $\sigma$ carries only the coarse layout. A score model at *every* noise level covers every place a sampler might be, and the levels are linked because one step to the next is itself Gaussian. The noised-data figure of Part E shows it. **Exercise A1.** For the two-peak example, what is the best possible average negative log-likelihood, how many bits per number is that, and how much worse is the best single Gaussian?
Answer The floor is the entropy of the data, 1.058 nats, which is 1.527 bits per number (divide by $\ln2$). The best single Gaussian has the data's mean and variance, 0.12 and 1.294, and its average negative log-likelihood is its entropy $\tfrac12\ln(2\pi e\cdot1.294)=1.548$ nats. The single Gaussian is therefore 0.490 nats above the floor: that is what ignoring the two peaks costs. Among all densities with a given variance the Gaussian has the largest entropy, so the two-peak density being below 1.548 is expected.
## B. The variational autoencoder ### B1. The bound Give the model a hidden code $z$ with a simple prior $p(z)$, usually a standard Gaussian, and a decoder $p_\theta(x\mid z)$. The likelihood integrates over all codes, which needs the posterior $p_\theta(z\mid x)$, as intractable as the integral. So introduce an *encoder* $q_\phi(z\mid x)$ that we choose and can sample from, and compare it with the true posterior: $$\log \htmlClass{eq-hid}{p_\theta}(x)=\underbrace{\mathbb E_{\htmlClass{eq-hid}{q_\phi}(z\mid x)}\Big[\log\frac{\htmlClass{eq-hid}{p_\theta}(x,z)}{\htmlClass{eq-hid}{q_\phi}(z\mid x)}\Big]}_{\text{evidence lower bound, }\mathcal L(x)}+\mathrm{KL}\big(\htmlClass{eq-hid}{q_\phi}(z\mid x)\,\|\,\htmlClass{eq-hid}{p_\theta}(z\mid x)\big).$$ The divergence is never negative, so $\mathcal L(x)\le\log p_\theta(x)$, and the gap is exactly how far the encoder is from the true posterior. One objective trains both networks: over the decoder it improves the likelihood, over the encoder it shrinks the gap. Split the joint differently and the bound is a trade between two costs: $$\mathcal L(x)=\underbrace{\mathbb E_{\htmlClass{eq-hid}{q_\phi}(z\mid x)}\big[\log \htmlClass{eq-hid}{p_\theta}(x\mid z)\big]}_{\text{reconstruction}}-\underbrace{\mathrm{KL}\big(\htmlClass{eq-hid}{q_\phi}(z\mid x)\,\|\,p(z)\big)}_{\text{rate}} .$$
The VAE is an encoder, a sampled code and a decoder; the bound charges two costs. Reconstruction rewards a code the decoder can rebuild from; the rate charges for how far the code moved from the prior. Step through it, or hover a box.
Averaged over $x$, the rate upper-bounds the information the code carries about $x$ (it equals it when the prior matches the average encoder). It is the bits per vector of the note on discrete codebooks, and the same trade.
Derivation of the identity and of the second form **Claim.** For any $q_\phi(z\mid x)$ that is positive wherever $p_\theta(z\mid x)$ is, $\log p_\theta(x)=\mathcal L(x)+\mathrm{KL}(q_\phi\|p_\theta(z\mid x))$. **Step 1: Bayes' rule.** For any code $z$, $p_\theta(x,z)=p_\theta(z\mid x)\,p_\theta(x)$, so $\log p_\theta(x)=\log p_\theta(x,z)-\log p_\theta(z\mid x)$. **Step 2: average over $q_\phi$.** The left side does not depend on $z$, so averaging both sides over $z\sim q_\phi(\cdot\mid x)$ leaves it unchanged: $$\log \htmlClass{eq-hid}{p_\theta}(x)=\mathbb E_q\big[\log \htmlClass{eq-hid}{p_\theta}(x,z)-\log \htmlClass{eq-hid}{p_\theta}(z\mid x)\big].$$ **Step 3: add and subtract $\log q_\phi$.** $$\log \htmlClass{eq-hid}{p_\theta}(x)=\mathbb E_q\Big[\log\frac{\htmlClass{eq-hid}{p_\theta}(x,z)}{\htmlClass{eq-hid}{q_\phi}(z\mid x)}\Big]+\mathbb E_q\Big[\log\frac{\htmlClass{eq-hid}{q_\phi}(z\mid x)}{\htmlClass{eq-hid}{p_\theta}(z\mid x)}\Big].$$ The first term is $\mathcal L(x)$. The second is the divergence from $q_\phi$ to the posterior, which is non-negative. **Step 4: the second form.** Write $p_\theta(x,z)=p_\theta(x\mid z)p(z)$ inside the first term: $$\mathcal L(x)=\mathbb E_q[\log \htmlClass{eq-hid}{p_\theta}(x\mid z)]+\mathbb E_q\Big[\log\frac{p(z)}{\htmlClass{eq-hid}{q_\phi}(z\mid x)}\Big]=\mathbb E_q[\log \htmlClass{eq-hid}{p_\theta}(x\mid z)]-\mathrm{KL}(\htmlClass{eq-hid}{q_\phi}(z\mid x)\|p(z)).$$ **What this does not say.** Nothing here says the gap is small. It is zero only if the encoder family contains the true posterior and the optimisation finds it. Everything the VAE gets wrong that is not the decoder's fault is in that gap.
### B2. Why the reparameterisation matters To train the encoder we need the gradient of an expectation under $q_\phi$ with respect to $\phi$. The score-function estimator, $\nabla_\phi\mathbb E_q[f(z)]=\mathbb E_q[f(z)\nabla_\phi\log q_\phi(z)]$, works for any $f$ but has a large variance. Kingma and Welling (2013) instead write the sample as a differentiable function of the parameters and independent noise, $z=m_\phi(x)+s_\phi(x)\varepsilon$, so the gradient passes through $f$. The difference is large. For $q=\mathcal N(\mu,1)$ and $f(z)=z^2$ the true derivative of $\mu^2+1$ is $2\mu$; the pathwise estimator $2(\mu+\varepsilon)$ has variance 4, the score-function estimator $(\mu+\varepsilon)^2\varepsilon$ has variance $\mu^4+14\mu^2+15$, which at $\mu=1$ is 30: the same mean, 7.5 times the variance. This is the reason the VAE trains with one sample per example. It is also the reason that models with discrete latent variables are harder to train: there is no differentiable path from a code index to a parameter, and the discrete case needs either a relaxation or the straight-through trick (B8). ### B3. The one VAE that can be solved exactly Take one-dimensional data $x\sim\mathcal N(0,V)$, a prior $z\sim\mathcal N(0,1)$, a linear decoder $x\mid z\sim\mathcal N(wz,\tau^2)$ and a linear encoder $q(z\mid x)=\mathcal N(ax,s^2)$. Every term of the bound is then an expectation of a quadratic, so everything is closed form. The model is the one-dimensional case of probabilistic principal component analysis. What it shows is what the VAE objective does when nothing goes wrong, and it turns out to contain the rate–distortion function, the same curve as in part A of the note on [discrete codebooks](codebook). **Result.** Maximise the bound over the encoder, the decoder weight $w$, for fixed decoder noise $\tau^2$. - If $\tau^2Derivation of the optimum **Claim.** With data variance $V$ and decoder noise $\tau^2$, the maximum of the bound is as stated above, and for $\tau^2
Figure 1. The best linear VAE sits exactly on Shannon's curve. Data of variance 2. The curve on the left is the least rate at which a Gaussian can be reproduced to a given mean square error, $\tfrac12\ln(2/D)$. Each position of the ring is the optimal VAE for one decoder noise variance; the copper dot shows its rate and its error, which lie on the curve with $D=\tau^2$. On the right, the distribution of the code for one observation, against the prior (dashed). At noise variance 2 or more the two coincide and the code is empty. Drag the ring: from 0.1 to 1.5 the negative log-likelihood does not change at all, 1.766 nats, while the rate falls from 1.50 to 0.14 nats.
Three consequences are worth stating plainly. **The likelihood does not choose the representation.** For every $\tau^2$ between 0.1 and 1.5 the optimal model is the same density and the bound has the same value, 1.766 nats. What changes is the split between code and decoder noise: at $\tau^2=0.25$ the code carries 1.04 nats and the error is 0.25; at $\tau^2=1$ it carries 0.35 nats and the error is 1. A likelihood score cannot tell them apart. Alemi et al. (2018) showed in general that maximum likelihood does not by itself yield a good representation, and read the two terms of the bound as a rate and a distortion. **The decoder noise is a price.** Multiply the bound by $\tau^2$ and it becomes $-\tfrac12\text{squared error}-\tau^2\cdot\text{rate}$. So $\tau^2$ is the exchange rate between error and bits, the Lagrange multiplier of the rate–distortion problem, and exactly the weight $\beta$ of the $\beta$-VAE (Higgins et al. 2017, cited from memory). Choosing the decoder variance and choosing $\beta$ are the same decision. **Posterior collapse in its simplest form.** When the decoder can explain the data alone (noise variance at least the data variance), the best code is empty. Bowman et al. (2016) saw the real version in a language model: with a powerful decoder the code is ignored, and they countered it by annealing the weight of the rate term from zero and by dropping words from the decoder's input. The linear model shows this is not an optimisation pathology but the exact optimum when the price of information exceeds its value. ### B4. Gaps, and a tighter bound The bound can be loose for two reasons the linear model hides. The *approximation gap*: the encoder family cannot contain the true posterior (a diagonal Gaussian cannot represent two modes). The *amortisation gap*: one network predicts the posterior for every $x$ instead of optimising it for each. Both sit in the single quantity $\mathrm{KL}(q\|p(z\mid x))$. Burda et al. (2016) tightened the bound by drawing $K$ codes and averaging the importance weights $w_k=p(x,z_k)/q(z_k\mid x)$ inside the logarithm: $$\mathcal L_K(x)=\mathbb E_{z_{1:K}\sim q}\Big[\log\frac1K\sum_{k=1}^Kw_k\Big],\qquad \mathcal L_1\le\mathcal L_2\le\cdots\le\log \htmlClass{eq-hid}{p_\theta}(x).$$ Jensen's inequality gives $\mathcal L_K\le\log\mathbb E[\tfrac1K\sum w_k]=\log p_\theta(x)$ because the weights have mean $p_\theta(x)$, and the same argument applied to subsets gives the ordering. The cost is $K$ decoder evaluations per example. The paper reports that the resulting models use more of their latent capacity. The idea matters later because a diffusion model's bound has the same character: many cheap terms whose sum approaches the likelihood as their number grows (D5). ### B5. What the VAE does badly, and why **Blur.** With a Gaussian decoder the reconstruction term is a squared error, and if one code fits two different images the best output is their average, which is neither. A VAE with a simple decoder gives soft, averaged samples. The cures are a richer decoder or latent structure (B6), or a decoder that is itself generative, which leads to latent diffusion (G2). **The hole in the prior.** Sampling uses the prior, but the decoder is trained on codes from the *aggregate* posterior $\mathbb E_x q(z\mid x)$. If they differ, samples start where the decoder has not been. In the linear model they do not: at the optimum $a^2V+s^2=1$. For nonlinear models the mismatch is a standing concern, and the rate term keeps it small. **One step.** The VAE decodes in one pass: fast, and the root of the blur, since one call must turn a code into a whole image. ### B6. Deeper VAEs A single latent layer asks one decoder to do everything. The remedy is a stack, $p(z_L)\,p(z_{L-1}\mid z_L)\cdots p(x\mid z_1)$, with an encoder that runs the other way; the bound gains one rate term per layer. Vahdat and Kautz (2020) built NVAE and report 2.91 bits per dimension on CIFAR-10 (2.93 without flows in the prior), the best among non-autoregressive likelihood models then. Child (2021) scaled depth further: the very deep VAE beats the PixelCNN in likelihood, samples thousands of times faster, and in theory a deep enough VAE can represent an autoregressive model. Keep this picture in mind: the diffusion model of Part D is a hierarchical VAE with as many latents as one likes, each the size of the data, a fixed encoder, and one network shared by all layers. ### B7. The discrete case, briefly If the encoder outputs a one-hot code over $K$ entries and the prior is uniform, the rate is $\log K$ nats for every input, so the code's information is capped by its size. That is the VQ-VAE of the note on discrete codebooks, with the straight-through gradient in place of the reparameterisation. With a learned prior over its codes it is a two-stage model; a VAE whose continuous latent is then modelled by diffusion is the other, and the one that dominates image generation (G2). **Exercise B1.** For $q=\mathcal N(\mu,1)$ and $f(z)=z^2$, at what $\mu$ is the score-function estimator's variance 4 times the pathwise one? **Exercise B2.** Data have variance 3. A linear VAE is trained with decoder noise variance 0.5. How many nats and bits does the code carry, and at what noise variance does it collapse?
Answers **B1.** The ratio is $(\mu^4+14\mu^2+15)/4$. At $\mu=0$ it is $15/4=3.75$, already below 4, and it grows with $\mu$, so the ratio of 4 is reached at $\mu^4+14\mu^2-1=0$, which gives $\mu^2=-7+\sqrt{50}=0.0711$ and $\mu=0.267$. At $\mu=1$ the ratio is 7.5. **B2.** The rate is $\tfrac12\ln(3/0.5)=0.896$ nats, which is $1.292$ bits, and the reconstruction error is $0.5$. The code collapses at any noise variance of 3 or more.
## C. Flows ### C1. Change of variables Suppose $x=f(z)$ where $z$ has a known density $p_z$ and $f$ is smooth, invertible, with a smooth inverse. Probability is conserved: a small interval of $z$ values maps to a small interval of $x$ values carrying the same probability mass, so if the map stretches the interval the density thins out by the same factor. In one dimension, $$p_x(x)=p_z\big(f^{-1}(x)\big)\,\Big|\frac{d f^{-1}}{dx}\Big|,\qquad \log p_x(x)=\log p_z(z)-\log|f'(z)|\quad\text{with } z=f^{-1}(x).$$ In $d$ dimensions the derivative becomes the Jacobian matrix and the stretch factor its determinant, $\log p_x(x)=\log p_z(z)-\log|\det\partial f/\partial z|$. A model of this kind is a *normalizing flow* (Rezende and Mohamed 2015). It gives the exact likelihood of any data point, with no integral and no bound, and sampling is one forward pass: draw $z$, apply $f$. A one-line check of the formula: if $z\sim\mathcal N(0,1)$ and $x=2z+1$, then at $x=3$ we have $z=1$, and $\log p_x(3)=\log\phi(1)-\ln2=-1.419-0.693=-2.112$, which is the log density of $\mathcal N(1,4)$ at 3. In one dimension there is exactly one increasing map pushing $p_z$ onto $p_x$, the *quantile map* $f=F_x^{-1}\circ F_z$, so every increasing flow that gets the density right is this map. For the two-peak example it sends the median $z=0$ to the data median, 0.613, and stretches the region of $z$ that lands in the gap between the peaks until almost no probability remains there. This will matter: correct flows from different schedules and objectives agree on the map and differ only in the path along which points travel (F5). ### C2. Making the determinant cheap For a generic $d\times d$ Jacobian the determinant costs $O(d^3)$, hopeless for images. Flows are designed so that it does not. The standard device is a *triangular* Jacobian, whose determinant is the product of the diagonal. *Coupling layers* (Dinh et al. 2015 in NICE, 2017 in RealNVP) split the vector into two halves, leave the first alone and transform the second with a scale and shift computed from the first by an arbitrary network: $$y_1=x_1,\qquad y_2=x_2\odot\exp\big(s(x_1)\big)+t(x_1),\qquad \log|\det J|=\sum_i s(x_1)_i .$$ The network that computes $s$ and $t$ can be as large as one likes, because it is never inverted: $x_2=(y_2-t(y_1))\odot\exp(-s(y_1))$. Stacking layers with the halves swapped lets every coordinate depend on every other. NICE used the additive version, RealNVP added the scale, Glow (Kingma and Dhariwal 2018) an invertible $1\times1$ convolution as a learned permutation. *Autoregressive* flows use a triangular Jacobian the other way, each coordinate a function of the earlier ones. The restriction is real: a coupling layer moves half the coordinates at a time, its network may not see what it modifies, and every layer keeps the dimension of the data, with no bottleneck to discard detail. Flows were competitive in likelihood for a time, and the NVAE paper lists them among the models that beat VAEs before deep VAEs closed the gap. For high-resolution images diffusion models overtook them (Part D) and, as Part E shows, contain a flow whose map is not restricted at all. ### C3. A flow in continuous time If the discrete layers of a flow get thin and numerous, the composition becomes the solution of an ordinary differential equation $dx/dt=v_t(x)$ with a network velocity (Chen et al. 2018, *Neural ODEs*). With $t=0$ data and $t=1$ noise, integrating forward carries data to noise and sampling integrates backward; the map $f$ of C1 is that backward map. Invertibility is free: a smooth velocity field can be integrated either way. The determinant is replaced by something much cheaper, the *instantaneous change of variables*: $$\frac{d}{dt}\log p_t\big(x(t)\big)=-\nabla\!\cdot \htmlClass{eq-der}{v_t}\big(x(t)\big)=-\operatorname{tr}\frac{\partial \htmlClass{eq-der}{v_t}}{\partial x}.$$ The log density of a point changes along its trajectory at a rate equal to minus the divergence of the velocity at that point. The accumulated change over a trajectory is an integral of the trace of the Jacobian, which can be estimated without forming the matrix (Grathwohl et al. 2019, FFJORD, used Hutchinson's trace estimator for this).
Derivation: the continuity equation and the change of log density **Claim.** If a density $p_t$ is transported by the velocity $v_t$, then $\partial_tp_t=-\partial_x(p_tv_t)$, and along a trajectory $dx/dt=v_t(x)$ the log density changes at rate $-\partial_xv_t$. **Step 1: conservation of mass.** The probability inside an interval $[a,b]$ is $\int_a^bp_t\,dx$. It changes only by what flows through the two ends, at rate $p_tv_t$ per unit time. So $$\frac{d}{dt}\int_a^bp_t\,dx=p_t(a)\htmlClass{eq-der}{v_t(a)}-p_t(b)\htmlClass{eq-der}{v_t(b)}=-\int_a^b\partial_x(p_tv_t)\,dx.$$ **Step 2: the equation.** The interval is arbitrary, so the integrands agree: $\partial_tp_t=-\partial_x(p_tv_t)$. In $d$ dimensions this reads $\partial_tp_t=-\nabla\!\cdot(p_tv_t)$ and is called the *continuity equation*. It is the single equation behind the likelihood of a continuous flow, the probability-flow ODE of Part E and the training target of Part F. **Step 3: follow one point.** Let $x(t)$ solve $dx/dt=v_t(x)$. By the chain rule $$\frac{d}{dt}p_t(x(t))=\partial_tp_t+\htmlClass{eq-der}{v_t}\,\partial_xp_t=-\partial_x(p_tv_t)+\htmlClass{eq-der}{v_t}\partial_xp_t=-p_t\,\partial_xv_t .$$ **Step 4: divide by $p_t$.** $\frac{d}{dt}\log p_t(x(t))=-\partial_xv_t(x(t))$, which is the claim (in $d$ dimensions, the divergence). **What this does not say.** The formula gives the log density along a trajectory that starts at a point whose density is known. It does not give a density at a point without integrating the whole trajectory back to such a starting point.
Figure 2. The density follows the points. Forty-one evenly spaced points of the data space are carried by the exact velocity field of the two-peak example along the straight-line noising path. Top: their positions over time. Bottom: the density at the chosen time, with the points as ticks; where ticks crowd together the density is high. The readout follows the copper point: its log density at the start, plus the integral of minus the divergence of the velocity along its trajectory, equals the log density of the noised data at its position. At $t=0.5$: $-0.52-0.21=-0.73$. At $t=1$ the density is exactly Gaussian.
### C4. What the continuous flow buys and what it costs
A continuous flow trades the architecture for a solver. Both carry noise to data; only the left needs an invertible design, only the right needs many network calls per sample.
The continuous formulation drops the restriction: $v_t(x)$ may be any function from $\mathbb R^d\times[0,1]$ to $\mathbb R^d$, and the likelihood computation is clean. Two costs remain. Sampling and likelihood need an ODE solver, so a sample is tens or hundreds of network evaluations. And the original training was by maximum likelihood, which needs the solver *during training* with a backward pass through it; that limited how far continuous flows scaled, and the Flow Matching paper presents its method as training them at unprecedented scale. The break came from asking whether a velocity field could be trained without simulating it (Part F). Before that, a result that is not obvious and not in the flow literature of 2018: the ODE need not be learned by likelihood at all, because a diffusion model has one inside it. **Exercise C1.** In one dimension a standard Gaussian is mapped by $f(z)=z+z^3/3$. What is the density of $x$ at $x=0$? **Exercise C2.** Where does the exact one-dimensional flow of the two-peak example send $z=0$, and where does it send $z=-1$?
Answers **C1.** At $x=0$ we have $z=0$ and $f'(z)=1+z^2=1$, so the density is $\phi(0)/1=0.399$. **C2.** The map is $F_{\rm data}^{-1}\circ\Phi$. $z=0$ has $\Phi=0.5$, and the data median is 0.613. $z=-1$ has $\Phi=0.1587$, which is below the first peak's mass of 0.4, so it lands inside the left peak at $x=-1.28$.
## D. Diffusion models as hierarchical VAEs ### D1. The forward process Start from a data point $x_0$ and add a little Gaussian noise, then a little more, many times. The standard form (Sohl-Dickstein et al. 2015, Ho et al. 2020) shrinks the previous value slightly so that the scale stays bounded: $$x_t=\sqrt{1-\beta_t}\,x_{t-1}+\sqrt{\beta_t}\,\htmlClass{eq-hid}{\varepsilon_t},\qquad t=1,\dots,T .$$ Because sums of Gaussians are Gaussian (fact (a) of A3), the value after $t$ steps given the start is also a Gaussian whose mean is a scaled copy of $x_0$: $$x_t=\htmlClass{eq-obs}{\alpha_t}x_0+\htmlClass{eq-hid}{\sigma_t}\htmlClass{eq-hid}{\varepsilon},\qquad \htmlClass{eq-obs}{\alpha_t}=\prod_{u\le t}\sqrt{1-\beta_u},\quad \htmlClass{eq-hid}{\sigma_t}^2=1-\htmlClass{eq-obs}{\alpha_t}^2 .$$
One chain, read in two directions. Noising has no parameters; the generator undoes it one Gaussian step at a time. Hover a box to follow its arrows.
This one equation is the heart of the practical method: to train, one never runs the chain; one picks a $t$, draws $\varepsilon$ and forms $x_t$ directly. Ho et al. use $T=1000$ steps with $\beta_t$ rising linearly from $10^{-4}$ to $0.02$; the signal that survives to the last step is $\alpha_T=0.0064$ of the original and the log signal-to-noise ratio there is $-10.1$, so the last latent is almost, but not exactly, standard Gaussian noise (D6 returns to the mismatch). It is cleaner to take any positive functions $\alpha_t,\sigma_t$ with $\mathrm{snr}_t=\alpha_t^2/\sigma_t^2$ decreasing in $t$. Every schedule is such a pair: variance-preserving ($\alpha^2+\sigma^2=1$), variance-exploding ($\alpha=1$, $\sigma$ growing), cosine, and the straight-line "rectified" schedule $\alpha_t=1-t$, $\sigma_t=t$ of Part F. The Markov structure is kept: going from time $s$ to a later time $t$ is the Gaussian step $$x_t=\htmlClass{eq-obs}{\alpha_{t\mid s}}\,x_s+\htmlClass{eq-hid}{\sigma_{t\mid s}}\,\htmlClass{eq-hid}{\varepsilon},\qquad \htmlClass{eq-obs}{\alpha_{t\mid s}}=\frac{\htmlClass{eq-obs}{\alpha_t}}{\htmlClass{eq-obs}{\alpha_s}},\quad \htmlClass{eq-hid}{\sigma_{t\mid s}}^2=\htmlClass{eq-hid}{\sigma_t}^2-\htmlClass{eq-obs}{\alpha_{t\mid s}}^2\htmlClass{eq-hid}{\sigma_s}^2 .$$ ### D2. The reverse step is Gaussian when the start is known Running the chain backward is the generative model. The difficulty is that the reverse step $q(x_s\mid x_t)$ depends on the whole data distribution. But if the clean point $x_0$ were also known, the reverse step would be simple, and that is what the training signal uses. **Result.** For $sDerivation of the reverse step **Claim.** The posterior of an earlier state given a later state and the start is the Gaussian above. **Step 1: Bayes' rule and the Markov property.** Given $x_0$, the chain is Markov, so $$q(x_s\mid x_t,\htmlClass{eq-obs}{x_0})\propto q(x_t\mid x_s)\,q(x_s\mid \htmlClass{eq-obs}{x_0}).$$ Both factors are Gaussian in $x_s$: $q(x_t\mid x_s)=\mathcal N(\alpha_{t\mid s}x_s,\sigma_{t\mid s}^2)$ and $q(x_s\mid x_0)=\mathcal N(\alpha_sx_0,\sigma_s^2)$. **Step 2: multiply Gaussians in $x_s$.** The product of two Gaussian functions of $x_s$ is Gaussian with the *precisions* adding. The first factor, as a function of $x_s$, has precision $\alpha_{t\mid s}^2/\sigma_{t\mid s}^2$ and is centred at $x_t/\alpha_{t\mid s}$; the second has precision $1/\sigma_s^2$ and centre $\alpha_sx_0$. The total precision is $$\frac1{\htmlClass{eq-der}{\tilde\sigma^2}}=\frac{\htmlClass{eq-obs}{\alpha_{t\mid s}}^2}{\htmlClass{eq-hid}{\sigma_{t\mid s}}^2}+\frac1{\htmlClass{eq-hid}{\sigma_s}^2}=\frac{\htmlClass{eq-obs}{\alpha_{t\mid s}}^2\htmlClass{eq-hid}{\sigma_s}^2+\htmlClass{eq-hid}{\sigma_{t\mid s}}^2}{\htmlClass{eq-hid}{\sigma_{t\mid s}}^2\htmlClass{eq-hid}{\sigma_s}^2}=\frac{\htmlClass{eq-hid}{\sigma_t}^2}{\htmlClass{eq-hid}{\sigma_{t\mid s}}^2\htmlClass{eq-hid}{\sigma_s}^2},$$ using $\sigma_t^2=\sigma_{t\mid s}^2+\alpha_{t\mid s}^2\sigma_s^2$ from D1. That is the variance in the claim. **Step 3: the mean.** The combined mean is the precision-weighted average of the centres: $$\htmlClass{eq-der}{\tilde\mu}=\htmlClass{eq-der}{\tilde\sigma^2}\Big(\frac{\htmlClass{eq-obs}{\alpha_{t\mid s}}^2}{\htmlClass{eq-hid}{\sigma_{t\mid s}}^2}\cdot\frac{x_t}{\htmlClass{eq-obs}{\alpha_{t\mid s}}}+\frac{\htmlClass{eq-obs}{\alpha_s}x_0}{\htmlClass{eq-hid}{\sigma_s}^2}\Big)=\frac{\htmlClass{eq-hid}{\sigma_{t\mid s}}^2\htmlClass{eq-hid}{\sigma_s}^2}{\htmlClass{eq-hid}{\sigma_t}^2}\Big(\frac{\htmlClass{eq-obs}{\alpha_{t\mid s}}x_t}{\htmlClass{eq-hid}{\sigma_{t\mid s}}^2}+\frac{\htmlClass{eq-obs}{\alpha_s}x_0}{\htmlClass{eq-hid}{\sigma_s}^2}\Big),$$ which expands to the two coefficients in the claim. **What this does not say.** At generation time $x_0$ is not known, so this step cannot be taken. It is the *target* the learned reverse step is trained to match. ### D3. A diffusion model is a hierarchical VAE with a fixed encoder Now line up the pieces with Part B. The noised variables $x_1,\dots,x_T$ are the latent layers. The encoder is the forward chain $q(x_{1:T}\mid x_0)=\prod_tq(x_t\mid x_{t-1})$, with no parameters. The generator is the chain run backward from a standard Gaussian, $p_\theta(x_{0:T})=p(x_T)\prod_{t=1}^Tp_\theta(x_{t-1}\mid x_t)$, with Gaussian steps whose mean a network predicts. It is a hierarchical VAE of depth $T$ (B6), but every latent has the dimension of the data, the encoder is fixed, and one network with the layer index as input serves all layers. Kingma et al. (2021) open with the same observation.
Same three parts, two settings. The diffusion model fixes the encoder and trades one small code for many data-sized ones.
Because it is a VAE, it has an evidence lower bound, and the bound splits into three kinds of term: $$-\log \htmlClass{eq-hid}{p_\theta}(\htmlClass{eq-obs}{x_0})\le\underbrace{\mathrm{KL}\big(q(x_T\mid \htmlClass{eq-obs}{x_0})\,\|\,p(x_T)\big)}_{\text{prior term}}+\underbrace{\mathbb E\big[-\log \htmlClass{eq-hid}{p_\theta}(\htmlClass{eq-obs}{x_0}\mid x_1)\big]}_{\text{reconstruction}}+\sum_{t=2}^T\underbrace{\mathbb E\,\mathrm{KL}\big(q(x_{t-1}\mid x_t,\htmlClass{eq-obs}{x_0})\,\|\,\htmlClass{eq-hid}{p_\theta}(x_{t-1}\mid x_t)\big)}_{\text{diffusion terms}} .$$ The prior term has no parameters and is near zero when the last latent is nearly standard Gaussian; the reconstruction term concerns the least noisy step. Each diffusion term compares the learned reverse step with the exact reverse step *given the start* of D2. Both are Gaussians, so each term is a divergence between two Gaussians, computed in A3.
Derivation of the bound **Claim.** The negative log-likelihood is bounded above by the sum of the prior, reconstruction and diffusion terms. **Step 1: the bound for a chain of latents.** The VAE bound of B1 with latent $x_{1:T}$ and encoder $q(x_{1:T}\mid x_0)$ is $$-\log \htmlClass{eq-hid}{p_\theta}(\htmlClass{eq-obs}{x_0})\le\mathbb E_q\Big[-\log\frac{\htmlClass{eq-hid}{p_\theta}(x_{0:T})}{q(x_{1:T}\mid \htmlClass{eq-obs}{x_0})}\Big]=\mathbb E_q\Big[-\log p(x_T)-\sum_{t=1}^T\log\frac{\htmlClass{eq-hid}{p_\theta}(x_{t-1}\mid x_t)}{q(x_t\mid x_{t-1})}\Big].$$ **Step 2: condition the forward steps on $x_0$.** For $t\ge2$ the chain is Markov given $x_0$, so $q(x_t\mid x_{t-1})=q(x_t\mid x_{t-1},x_0)$, and Bayes' rule inverts it: $$q(x_t\mid x_{t-1},\htmlClass{eq-obs}{x_0})=\frac{q(x_{t-1}\mid x_t,\htmlClass{eq-obs}{x_0})\,q(x_t\mid \htmlClass{eq-obs}{x_0})}{q(x_{t-1}\mid \htmlClass{eq-obs}{x_0})}.$$ **Step 3: substitute.** In the sum over $t=2,\dots,T$ the ratios $q(x_t\mid x_0)/q(x_{t-1}\mid x_0)$ telescope to $q(x_T\mid x_0)/q(x_1\mid x_0)$. The $t=1$ term has $q(x_1\mid x_0)$ in its denominator and cancels the remainder. What is left is $$\mathbb E_q\Big[-\log\frac{p(x_T)}{q(x_T\mid \htmlClass{eq-obs}{x_0})}-\log \htmlClass{eq-hid}{p_\theta}(\htmlClass{eq-obs}{x_0}\mid x_1)-\sum_{t=2}^T\log\frac{\htmlClass{eq-hid}{p_\theta}(x_{t-1}\mid x_t)}{q(x_{t-1}\mid x_t,\htmlClass{eq-obs}{x_0})}\Big].$$ **Step 4: identify.** The first term is the divergence $\mathrm{KL}(q(x_T\mid x_0)\|p(x_T))$, the second the reconstruction, the third a sum of divergences $\mathrm{KL}(q(x_{t-1}\mid x_t,x_0)\|p_\theta(x_{t-1}\mid x_t))$ averaged over $x_t$. **What this does not say.** It does not say the bound is tight. The learned reverse step is a Gaussian; the true reverse step given only $x_t$ is a mixture over the possible starts, and Figure 3 below measures the cost of that approximation.
### D4. Each diffusion term is a denoising error Give the generator the same Gaussian shape as the exact reverse step, with the unknown clean point replaced by a network's guess $\hat x_0(x_t,t)$ and the same variance $\tilde\sigma^2$: $p_\theta(x_s\mid x_t)=\mathcal N(\tilde\mu(x_t,\hat x_0),\tilde\sigma^2)$. The two Gaussians then share a variance and differ only in the coefficient of $x_0$ in the mean, so by fact (b) of A3 the divergence is a squared error: $$\mathrm{KL}=\frac{(\text{coefficient of }\htmlClass{eq-obs}{x_0})^2\,\|\htmlClass{eq-obs}{x_0}-\htmlClass{eq-hid}{\hat x_0}\|^2}{2\htmlClass{eq-der}{\tilde\sigma^2}}=\frac12\big(\htmlClass{eq-der}{\mathrm{snr}_s}-\htmlClass{eq-der}{\mathrm{snr}_t}\big)\,\|\htmlClass{eq-obs}{x_0}-\htmlClass{eq-hid}{\hat x_0}(x_t,t)\|^2 .$$
Derivation of the weight **Claim.** The diffusion term for a step from $t$ back to $s$ is $\tfrac12(\mathrm{snr}_s-\mathrm{snr}_t)\,\mathbb E\|x_0-\hat x_0\|^2$. **Step 1: the difference of means.** From D2 the mean depends on $x_0$ only through $\frac{\alpha_s\sigma_{t\mid s}^2}{\sigma_t^2}x_0$, so $\tilde\mu(x_t,x_0)-\tilde\mu(x_t,\hat x_0)=\frac{\alpha_s\sigma_{t\mid s}^2}{\sigma_t^2}(x_0-\hat x_0)$. **Step 2: the divergence of two equal-variance Gaussians.** From fact (b), $\mathrm{KL}=\frac{(m_1-m_2)^2}{2\tilde\sigma^2}$, so $$\mathrm{KL}=\frac{\htmlClass{eq-obs}{\alpha_s}^2\htmlClass{eq-hid}{\sigma_{t\mid s}}^4}{\htmlClass{eq-hid}{\sigma_t}^4}\cdot\frac{\htmlClass{eq-hid}{\sigma_t}^2}{2\htmlClass{eq-hid}{\sigma_{t\mid s}}^2\htmlClass{eq-hid}{\sigma_s}^2}\|\htmlClass{eq-obs}{x_0}-\htmlClass{eq-hid}{\hat x_0}\|^2=\frac{\htmlClass{eq-obs}{\alpha_s}^2}{2\htmlClass{eq-hid}{\sigma_s}^2}\cdot\frac{\htmlClass{eq-hid}{\sigma_{t\mid s}}^2}{\htmlClass{eq-hid}{\sigma_t}^2}\|\htmlClass{eq-obs}{x_0}-\htmlClass{eq-hid}{\hat x_0}\|^2 .$$ **Step 3: simplify the second ratio.** $\sigma_{t\mid s}^2=\sigma_t^2-\alpha_{t\mid s}^2\sigma_s^2$, so $\frac{\sigma_{t\mid s}^2}{\sigma_t^2}=1-\frac{\alpha_t^2\sigma_s^2}{\alpha_s^2\sigma_t^2}=1-\frac{\mathrm{snr}_t}{\mathrm{snr}_s}$. **Step 4: multiply.** $\frac{\alpha_s^2}{2\sigma_s^2}=\tfrac12\mathrm{snr}_s$, so the coefficient is $\tfrac12\mathrm{snr}_s(1-\mathrm{snr}_t/\mathrm{snr}_s)=\tfrac12(\mathrm{snr}_s-\mathrm{snr}_t)$. **What this does not say.** It assumes the generator's variance is the exact reverse variance $\tilde\sigma^2$, which is what DDPM does with a fixed variance. A learned variance (Nichol and Dhariwal 2021) changes the term and can tighten the bound.
Three equivalent ways to ask the network for the same thing follow from $x_t=\alpha_tx_0+\sigma_t\varepsilon$: - **Clean-data prediction**: output $\hat x_0$. - **Noise prediction**: output $\hat\varepsilon$, with $\hat x_0=(x_t-\sigma_t\hat\varepsilon)/\alpha_t$. The errors are linked by $\|x_0-\hat x_0\|^2=e^{-\lambda_t}\|\varepsilon-\hat\varepsilon\|^2$, so $\|\varepsilon-\hat\varepsilon\|^2=\mathrm{snr}_t\|x_0-\hat x_0\|^2$. - **Velocity prediction**: output the combination $\alpha_t\varepsilon-\sigma_tx_0$ (Salimans and Ho 2022; a different object from the flow velocity of Part F, though for one schedule they are related). The ELBO's diffusion term in noise form has a particularly simple weight. Multiply the weight on the clean-data error by $e^{-\lambda_t}$: $$\tfrac12(\htmlClass{eq-der}{\mathrm{snr}_s}-\htmlClass{eq-der}{\mathrm{snr}_t})\,e^{-\htmlClass{eq-der}{\lambda_t}}\|\htmlClass{eq-hid}{\varepsilon}-\htmlClass{eq-hid}{\hat\varepsilon}\|^2=\tfrac12\big(e^{\htmlClass{eq-der}{\lambda_s}-\htmlClass{eq-der}{\lambda_t}}-1\big)\|\htmlClass{eq-hid}{\varepsilon}-\htmlClass{eq-hid}{\hat\varepsilon}\|^2\approx\tfrac12(\htmlClass{eq-der}{\lambda_s}-\htmlClass{eq-der}{\lambda_t})\|\htmlClass{eq-hid}{\varepsilon}-\htmlClass{eq-hid}{\hat\varepsilon}\|^2 .$$ For small steps, the bound is a *constant* weight, one half, on the noise-prediction error per unit of log signal-to-noise ratio. The true bound weights every noise level equally on the log-SNR axis. This is the single most useful fact for reading the diffusion literature, because every common training loss is a different weighting of the same integrand. ### D5. The bound as information, and the cost of few steps The ELBO's diffusion term, with the exact denoiser in place of a network, has a meaning. The error of the best possible denoiser at signal-to-noise ratio $r$ is the *minimum mean square error*, $\mathrm{mmse}(r)=\mathbb E\|x_0-\mathbb E[x_0\mid x_t]\|^2$. In the continuous-time limit the sum of the diffusion terms is $$\frac12\int \htmlClass{eq-der}{\mathrm{mmse}}(r)\,dr=I\big(\htmlClass{eq-obs}{x_0};x_{\rm least\ noisy}\big)-I\big(\htmlClass{eq-obs}{x_0};x_{\rm most\ noisy}\big),$$ the information about the data that the noising process destroys. This is the *I-MMSE relation* of Guo, Shamai and Verdú (2005, cited from memory): the derivative of the mutual information between a signal and a noisy copy with respect to the signal-to-noise ratio is half the minimum mean square error. Kong et al. (2023) generalise it to relate the data distribution exactly to an optimal denoising regression. Kingma et al. (2021) arrive from the other side: the continuous-time bound depends on the schedule only through the signal-to-noise ratio at the two ends, since the shape in between is a change of variable in the integral. So the model's loss is the information destroyed by noise, to be re-learned by denoising. A perfect denoiser makes the bound equal to the entropy; an imperfect one pays the difference. For continuous data the information in the least noisy copy grows without limit as the noise goes to zero, so the diffusion terms alone diverge; the reconstruction term of D3 cancels that growth, and this is why the area under the integrand in Figure 3 keeps growing to the right.
Check on a Gaussian, where everything is explicit **Claim.** For $x_0\sim\mathcal N(0,v)$, $I(x_0;\sqrt r\,x_0+\varepsilon)=\tfrac12\ln(1+rv)$ and its derivative with respect to $r$ is $\tfrac12\mathrm{mmse}(r)$. **Step 1: the information.** The noisy copy $y=\sqrt r\,x_0+\varepsilon$ has variance $rv+1$ and the noise has variance one. For Gaussians, $I=h(y)-h(\varepsilon)=\tfrac12\ln\big(2\pi e(rv+1)\big)-\tfrac12\ln(2\pi e)=\tfrac12\ln(1+rv)$. **Step 2: the best denoiser.** The posterior of $x_0$ given $y$ has variance $\frac{v}{1+rv}$ by fact (c) of A3 after scaling, so $\mathrm{mmse}(r)=\frac{v}{1+rv}$. **Step 3: differentiate.** $\frac{d}{dr}\tfrac12\ln(1+rv)=\frac{v}{2(1+rv)}=\tfrac12\mathrm{mmse}(r)$. **What this does not say.** For non-Gaussian data there is no closed form, and the relation is a theorem, not something to re-derive here.
Figure 3. With the exact denoiser the bound approaches the entropy, but only as the steps get small. Top: the integrand, one half times the signal-to-noise ratio times the best denoiser's error, per unit of log signal-to-noise ratio; its area is the information gained. The violet bars are what $T$ steps add to the bound; a step of width $\Delta\lambda$ counts the integrand $(e^{\Delta\lambda}-1)/\Delta\lambda$ times too much, so wide steps overshoot. Bottom: the bound minus the data's entropy of 1.058 nats, on a log scale, for steps equally spaced in log signal-to-noise ratio (blue) and in time on the straight-line path (copper). At 16 steps the excess is 18.3 nats against 215,000; at 256 steps it is 0.52 nats against 806.
Figure 3 puts numbers on the cost of few steps. The bound has the right limit, but at $T=16$ it is 18 nats above the entropy even with steps even in log signal-to-noise ratio, falling roughly as $1/T$ (2.4 at 64, 0.52 at 256). With steps even in time on the straight-line path, the first step must cross the whole high-signal end and the excess is orders of magnitude larger. So a likelihood bound needs many small steps, and where they sit matters as much as how many: a schedule is, among other things, a choice of where to put steps. This is why DDPM uses $T=1000$, and why *sampling* with 1000 steps was the original cost of the method (E6). (The excess also contains a second effect: the generator's variance is fixed at $\tilde\sigma^2$, which understates the variance of the true reverse step because the true reverse step is a mixture over possible clean points. In the two-peak example this matters most at high noise, where the reverse step is bimodal.) ### D6. Weightings, schedules and why "simple" works The DDPM paper trains on the simpler loss $\mathbb E\|\varepsilon-\hat\varepsilon\|^2$ with $t$ uniform, dropping the weights. The authors' reason: this down-weights the small-$t$ terms that teach the network to denoise tiny amounts of noise, so it can concentrate on harder tasks at larger $t$. The effect is large. On CIFAR-10 the true bound with fixed variance reaches at most 3.70 bits per dimension but an FID of 13.51 and an Inception score of 7.67; the simple loss gives 3.75 bits, an FID of 3.17 and an Inception score of 9.46: 0.05 bits of likelihood for a factor of four in FID. Express every loss as a weight on the same integrand and the picture is clear. Per unit time, the weight each objective puts on the clean-data error is: | Objective | Weight on $\lVert x_0-\hat x_0\rVert^2$ per unit time | Weight on $\lVert\varepsilon-\hat\varepsilon\rVert^2$ per unit log-SNR | | --- | --- | --- | | the ELBO | $\tfrac12\lvert d\,\mathrm{snr}_t/dt\rvert$ | $\tfrac12$ (flat) | | noise prediction, $t$ uniform | $\mathrm{snr}_t$ | $1/\lvert d\lambda_t/dt\rvert$ | | clean-data prediction, $t$ uniform | $1$ | $e^{-\lambda_t}/\lvert d\lambda_t/dt\rvert$ | | velocity prediction, $t$ uniform, variance-preserving | $\mathrm{snr}_t+1$ | $(1+e^{-\lambda_t})/\lvert d\lambda_t/dt\rvert$ | Take the continuous variance-preserving schedule with $\beta$ from 0.1 to 20. Relative to the bound, the simple noise-prediction loss has weight $2/|d\lambda/dt|$: 0.013 around $t=0.01$, 0.099 around $t=0.1$, 0.183 around $t=0.5$, 0.111 around $t=0.9$. Levels near the data get about 14 times less relative weight than the middle, the DDPM down-weighting in numbers. Kingma and Gao (2023) show that common diffusion objectives are weighted integrals of ELBOs over noise levels, and that a monotone weight equals the ELBO plus Gaussian noise augmentation; they report new monotone weightings with state-of-the-art FID on high-resolution ImageNet. Karras et al. (2022) split the design space into a weighting $\lambda(\sigma)$, a distribution of noise levels, a parameterisation and a sampler, separate from the network, and report an FID of 1.79 on class-conditional CIFAR-10. Schedules have a failure mode of their own. One that leaves residual signal at $t=1$ trains the network on inputs with a trace of the data, then asks it to start from pure noise. Lin et al. (2023) argue that common schedules do this, limiting what the model can generate (very dark images are their example), and propose zero terminal signal-to-noise ratio with velocity prediction. DDPM's linear schedule leaves 0.0064 at the last step, small but not zero. The cosine schedule of Nichol and Dhariwal (2021) adds noise more slowly; they also learn the reverse variances and train with a hybrid loss in which the true bound has weight 0.001.
Figure 4. The schedule decides where the sampler spends its steps. Top: log signal-to-noise ratio against time for four schedules; the chosen one is violet and the names are under the slider. Middle: its signal and noise scales (for the variance-exploding schedule noise is divided by ten so that it fits). The readout gives the distance of the sampler's output from the data density with 8 and with 32 Euler steps, and with 8 Heun steps. Drag the ring: the 8-step Euler errors are 0.996 for variance-exploding, 0.041 for variance-preserving, 0.019 for the cosine schedule and 0.101 for the straight-line schedule, and with 8 Heun steps the straight line reaches 0.006. The ranking is for this example and these solvers, not a statement about images.
### D7. What is learned, and what the limit is For small enough steps the reverse of a diffusion has the same form as the forward process (Feller 1949, used by Sohl-Dickstein et al. to justify Gaussian reverse steps), so the Gaussian reverse step becomes exact; that is why the excess in Figure 3 falls with $T$. The network then has one job, the best guess $\hat x_0(x_t,t)=\mathbb E[x_0\mid x_t]$ at every noise level, and the model is determined by this one function. Part E shows what it is: the gradient of a log density. **Exercise D1.** With DDPM's linear $\beta$ schedule from $10^{-4}$ to 0.02 over 1000 steps, what is $\prod(1-\beta_t)$ and the log signal-to-noise ratio at the last step? **Exercise D2.** The data are $\mathcal N(0,1)$. How much does the diffusion loss (with the exact denoiser) add between signal-to-noise ratio 1 and 99?
Answers **D1.** $\prod(1-\beta_t)=4.04\times10^{-5}$, so $\alpha_T=0.0064$ and $\lambda_T=\ln\frac{\bar\alpha}{1-\bar\alpha}=-10.12$. **D2.** The loss is $\tfrac12\int\mathrm{mmse}(r)\,dr$ over the range, and for unit variance $\mathrm{mmse}(r)=\frac1{1+r}$, which gives $\tfrac12\ln\frac{1+99}{1+1}=\tfrac12\ln50=1.956$ nats.
## E. Diffusion models as score estimation, as an SDE and as an ODE ### E1. The object being learned Part D ended with one function, the best guess $\hat x_0(x,t)=\mathbb E[x_0\mid x_t=x]$ of the clean data at every noise level. This part shows that it is the gradient of a log density, and that the same function drives both a stochastic and a deterministic generator. The *score* of a density $p$ is $\nabla_x\log p(x)$. It points uphill in log density and needs no normalising constant, since the gradient removes it. A score model of the data would suffice to sample (walking uphill with noise added, *Langevin dynamics*) but is hard to learn for the reason of A4. Song and Ermon (2019) perturb the data with Gaussian noise at several levels and learn each level's score in one network conditioned on the level, which is exactly the family $p_t$ of the forward process. On a two-mode toy, Langevin dynamics with the *exact* score recovers the relative weight of the modes wrongly while an annealed version, from large to small noise, recovers it: between two modes the score is nearly zero and says nothing about how much mass sits on each side, while at higher noise the modes connect and the score carries the proportions. ### E2. Denoising score matching How does one train a network to output $\nabla\log p_t$ when $p_t$ is unknown? The route of Vincent (2011, which I cite from memory; the idea is also in the DDPM paper's Section 3.2) uses the fact that the score of the *conditional* density is known in closed form. Given $x_0$, the noised point is $x_t\sim\mathcal N(\alpha_tx_0,\sigma_t^2)$, so $$\nabla_{x_t}\log q(x_t\mid \htmlClass{eq-obs}{x_0})=-\frac{x_t-\htmlClass{eq-obs}{\alpha_t}x_0}{\htmlClass{eq-hid}{\sigma_t}^2}=-\frac{\htmlClass{eq-hid}{\varepsilon}}{\htmlClass{eq-hid}{\sigma_t}}.$$ The conditional score is just the noise, rescaled. The claim is that regressing a network on this known conditional target gives, in the limit, the score of the marginal $p_t$ that we do not know.
Derivation: regressing on the conditional score recovers the marginal score **Claim.** The function minimising $\mathbb E\|f(x_t,t)+\varepsilon/\sigma_t\|^2$ is $\nabla\log p_t$. **Step 1: the marginal score is an average of conditional scores.** The marginal density is $p_t(x)=\int q(x\mid x_0)p_0(x_0)\,dx_0$. Differentiate: $$\nabla p_t(x)=\int\nabla q(x\mid \htmlClass{eq-obs}{x_0})\,\htmlClass{eq-obs}{p_0}(\htmlClass{eq-obs}{x_0})\,dx_0=\int q(x\mid \htmlClass{eq-obs}{x_0})\,\nabla\log q(x\mid \htmlClass{eq-obs}{x_0})\,\htmlClass{eq-obs}{p_0}(\htmlClass{eq-obs}{x_0})\,dx_0.$$ Divide by $p_t(x)$ and recognise the posterior $p(x_0\mid x)=q(x\mid x_0)p_0(x_0)/p_t(x)$: $$\htmlClass{eq-der}{\nabla\log p_t(x)}=\mathbb E\big[\nabla\log q(x\mid \htmlClass{eq-obs}{x_0})\ \big|\ x_t=x\big].$$ **Step 2: least squares learns a conditional mean.** For any random target $Y$ and any function $f$ of the input $x$, $$\mathbb E\|f(x)-Y\|^2=\mathbb E\|f(x)-\mathbb E[Y\mid x]\|^2+\mathbb E\|Y-\mathbb E[Y\mid x]\|^2,$$ because the cross term has zero mean. The last term does not depend on $f$. So the minimiser is $f(x)=\mathbb E[Y\mid x]$. **Step 3: combine.** Take $Y=\nabla\log q(x_t\mid x_0)=-\varepsilon/\sigma_t$. By Step 1, $\mathbb E[Y\mid x_t]=\nabla\log p_t(x_t)$, and by Step 2 the regression finds it. **What this does not say.** The loss does not go to zero at the optimum: the second term of Step 2 is the variance of the target that no function of $x_t$ can explain. This residual reappears in flow matching (Part F), where Figure 7 shows its size.
This is the same trick as the VAE's bound in another form: a quantity we cannot compute (the marginal score) is the average of quantities we can (the conditional scores), and a squared-error loss against the sampled quantity learns the average. It is used again, with the velocity in place of the score, in Part F. The noise-prediction loss of DDPM *is* this loss: predicting $\varepsilon$ is predicting $-\sigma_t$ times the score. ### E3. Tweedie's formula: the denoiser and the score are one function Step 1 of the derivation above, read for the Gaussian conditional, gives immediately $$\htmlClass{eq-der}{\nabla\log p_t(x)}=\mathbb E\Big[-\frac{x-\htmlClass{eq-obs}{\alpha_t}x_0}{\htmlClass{eq-hid}{\sigma_t}^2}\ \Big|\ x\Big]=-\frac{x-\htmlClass{eq-obs}{\alpha_t}\htmlClass{eq-hid}{\hat x_0}(x,t)}{\htmlClass{eq-hid}{\sigma_t}^2}\quad\Longleftrightarrow\quad\htmlClass{eq-hid}{\hat x_0}(x,t)=\frac{x+\htmlClass{eq-hid}{\sigma_t}^2\,\htmlClass{eq-der}{\nabla\log p_t(x)}}{\htmlClass{eq-obs}{\alpha_t}}.$$ This is *Tweedie's formula* (attributed to Robbins in 1956 and Efron in 2011; I cite both from memory). The best denoising of a noisy point is the point itself, moved along the score by the noise variance and rescaled. So the clean-data predictor, the noise predictor and the score are one function in three costumes: | Network output | in terms of the score $\nabla\log p_t$ | in terms of the denoiser $\hat x_0$ | | --- | --- | --- | | score | $\nabla\log p_t(x)$ | $(\alpha_t\hat x_0-x)/\sigma_t^2$ | | noise $\hat\varepsilon$ | $-\sigma_t\nabla\log p_t(x)$ | $(x-\alpha_t\hat x_0)/\sigma_t$ | | velocity (Part F) | $\dfrac{\dot\alpha_t}{\alpha_t}x+\Big(\dfrac{\dot\alpha_t\sigma_t^2}{\alpha_t}-\sigma_t\dot\sigma_t\Big)\nabla\log p_t$ | $\dot\alpha_t\hat x_0+\dot\sigma_t\hat\varepsilon$ |
One function, four costumes. By Tweedie's formula each is a rescaling of the denoiser; purple marks the derived ones. The table gives the formulas.
A numerical instance on the two-peak example: at $t=0.5$ on the straight-line path ($\alpha=\sigma=0.5$) the score at $x=0$ is 0.336. Tweedie gives $\hat x_0=(0+0.25\cdot0.336)/0.5=0.168$, and the noise estimate is $-0.5\cdot0.336=-0.168$, consistent with $(x-\alpha\hat x_0)/\sigma=(0-0.084)/0.5$.
Figure 5. Noise merges the peaks, and the best denoiser tells them apart. The straight-line path with the two-peak data. Top: the density of the noised data (blue) against the clean density (dashed). Middle: the score, the slope of the log of the top curve (violet). Bottom: the best guess of the clean value given the noisy one (violet), against the identity (dashed). At small noise the guess snaps to the nearest peak and the score is steep and sign-changing; at $t=0.5$ the density is a single hump, the guess moves smoothly from the left peak to the right one, and at $t=1$ the density is Gaussian and the guess is the data mean. Drag the ring along $t$: the three curves are the same function seen three ways.
### E4. The forward process as a stochastic differential equation Now take the steps to zero. Write the noising process as a differential equation for $x_t$ with a drift pulling the signal down and a noise source of strength $g_t$ (Song et al. 2021): $$dx=f_t\,x\,dt+g_t\,dw,\qquad f_t=\frac{\htmlClass{eq-obs}{\dot\alpha_t}}{\htmlClass{eq-obs}{\alpha_t}},\quad g_t^2=\frac{d\htmlClass{eq-hid}{\sigma_t}^2}{dt}-2f_t\htmlClass{eq-hid}{\sigma_t}^2 .$$ Every schedule $(\alpha,\sigma)$ of D1 corresponds to one such equation and the other way round. The variance-preserving process of DDPM has $f=-\beta_t/2$ and $g^2=\beta_t$. The variance-exploding process has $f=0$, so only noise is added.
Derivation: the drift and diffusion of a given schedule **Claim.** The process $dx=f_tx\,dt+g_t\,dw$ has the marginal $x_t=\alpha_tx_0+\sigma_t\varepsilon$ when $f=\dot\alpha/\alpha$ and $g^2=d\sigma^2/dt-2f\sigma^2$. **Step 1: the mean.** Taking expectations of the equation gives $d\,\mathbb E x/dt=f\,\mathbb Ex$, so $\mathbb Ex_t=x_0\exp\int f=\alpha_tx_0$ when $f=\dot\alpha/\alpha$ and $\alpha_0=1$. **Step 2: the variance.** For a linear equation $dx=fx\,dt+g\,dw$ the variance $V_t$ obeys $dV/dt=2fV+g^2$: the drift scales the existing variance, and the noise adds $g^2dt$. **Step 3: match to $\sigma_t^2$.** Setting $V_t=\sigma_t^2$ gives $d\sigma^2/dt=2f\sigma^2+g^2$, which is the claim. **Check for the variance-preserving process.** There $\sigma^2=1-\alpha^2$ and $\alpha=e^{-B/2}$ with $B'=\beta$. Then $d\sigma^2/dt=-2\alpha\dot\alpha=\beta\alpha^2$. The right side is $2(-\beta/2)\sigma^2+g^2=-\beta(1-\alpha^2)+g^2$, which equals $\beta\alpha^2$ if $g^2=\beta$. **What this does not say.** The discrete DDPM is the Euler discretisation of this equation with $\beta_t$ as the step-size-scaled $\beta$; the continuous version needs a limit in which $T\to\infty$ and $\beta_t\to0$ together.
### E5. Two ways back: a stochastic equation and an ordinary one Anderson (1982) showed that the reverse in time of a diffusion is again a diffusion, and Song et al. (2021) apply it here: the reverse-time equation is $$dx=\big[f_tx-g_t^2\,\htmlClass{eq-der}{\nabla\log p_t(x)}\big]\,dt+g_t\,d\bar w,$$ run from $t=1$ down to $t=0$, with $d\bar w$ noise running backward. The only thing that depends on the data is the score, so a model of the score at every noise level is a model of the reverse process. Song et al. also derive a deterministic equation with the same marginals, the *probability-flow ODE*: $$\frac{dx}{dt}=f_tx-\tfrac12g_t^2\,\htmlClass{eq-der}{\nabla\log p_t(x)}.$$ Running this ODE from a Gaussian sample at $t=1$ down to $t=0$ gives a sample from the data density, and running it forward gives a deterministic encoding of a data point into noise.
Derivation: the ODE and the SDE have the same marginals **Claim.** The probability-flow ODE, and a whole family of reverse equations with noise, all keep the density at every time equal to $p_t$. **Step 1: the forward density equation.** The density of $dx=fx\,dt+g\,dw$ obeys the Fokker–Planck equation (in one dimension; $d$ dimensions replace derivatives by divergences and gradients) $$\partial_tp=-\partial_x(fx\,p)+\tfrac12g^2\partial_x^2p .$$ **Step 2: rewrite the second-order term as a drift.** Since $\partial_xp=p\,\partial_x\log p$, $$\tfrac12g^2\partial_x^2p=\partial_x\big(\tfrac12g^2p\,\partial_x\log p\big),\qquad\text{so}\qquad\partial_tp=-\partial_x\Big[\big(fx-\tfrac12g^2\partial_x\log p\big)\,p\Big].$$ This is the continuity equation of C3 with velocity $v=fx-\tfrac12g^2\partial_x\log p$. An ODE with this velocity transports $p_t$ in the same way as the SDE did: this is the probability-flow ODE. **Step 3: reverse the clock.** Let $r=1-t$ and $\tilde p_r=p_{1-r}$. Then $\partial_r\tilde p=-\partial_tp=-\partial_x\big[w\,\tilde p\big]$ with $w=-v$, again a continuity equation: the ODE run backward. **Step 4: add noise in the reverse direction.** An equation $dx=a\,dr+\gamma\,dW$ in the variable $r$ has $\partial_r\tilde p=-\partial_x(a\tilde p)+\tfrac12\gamma^2\partial_x^2\tilde p=-\partial_x\big[(a-\tfrac12\gamma^2\partial_x\log\tilde p)\tilde p\big]$ by the same rewriting. It reproduces the marginals if $a-\tfrac12\gamma^2\partial_x\log\tilde p=w$, that is $$a=-fx+\tfrac12(g^2+\gamma^2)\,\partial_x\log p .$$ **Step 5: read off the special cases.** $\gamma=0$ gives the ODE. $\gamma=g$ gives $a=-fx+g^2\partial_x\log p$, which in the original time is the reverse-time SDE above. Any $\gamma\ge0$ in between works, with more or less noise injected and more or less score used to correct it. **What this does not say.** The statement is about the exact score. A learned score has errors, and the ODE and the SDE then differ in how errors propagate: the ODE carries them along a path, the SDE adds fresh noise at each step that partly washes out earlier errors (E6).
The consequence: *a diffusion model contains a continuous normalizing flow*. Its velocity is read off the score by the table in E3 with no extra training, and the machinery of Part C applies: the likelihood of a data point is the log density of a Gaussian at its encoding plus the integral of the divergence along the trajectory. Song et al. use this for exact likelihoods via the probability-flow ODE and a trace estimator, reporting 2.99 bits per dimension on CIFAR-10 for a new SDE variant, and FID 2.20 for their best samples. The ODE also gives the diffusion model what the VAE has, a deterministic invertible map between data and code. Part F shows the other side: the same field can be learned by regression. ### E6. Samplers: discretisations of these two equations
One score, two ways back. Both give the same density at every noise level. Every sampler is one of them with finitely many steps.
Every sampler is a way of integrating one of these equations in finitely many steps. - **DDPM's ancestral sampling** discretises the reverse SDE (Song et al., their Appendix E). - **DDIM** (Song, Meng and Ermon 2021) generalises DDPM to non-Markovian forward processes with the same training objective; its deterministic member is an Euler step of the probability-flow ODE in the variable rescaled by $\alpha_t$, and the authors report 10 to 50 times faster sampling. The plain Euler sampler of Figure 6 is a close relative, not DDIM itself. - **Higher-order solvers.** Karras et al. (2022) integrate the ODE with Heun's second-order method (two evaluations per step) and add controlled noise back at each step ("churn"). Song et al. use *predictor–corrector* samplers, where a Langevin step corrects the SDE step.
Figure 6. The deterministic sampler is accurate with few steps; the stochastic one needs more and has a floor. The variance-preserving path, exact velocity. Top: eleven starting points and their paths, exact in grey and with the chosen number of Euler steps in copper. Bottom: the distance of the samples from the data density against the number of steps, for Euler (copper), Heun (blue, two evaluations per step) and the Euler–Maruyama stochastic sampler (violet); the dashed line is the distance an exact sample of the same size would have. Euler's error falls by a factor of 4.3, 2.6 and 2.0 over the successive doublings of the steps: 0.175 at 4, 0.041 at 8, 0.016 at 16, 0.008 at 32. At 16 steps the stochastic sampler is at 0.050.
Figure 6 is less simple than it is usually told. With *exact* velocities Euler's error roughly halves with each doubling of steps, as a first-order method should (successive factors 4.3, 2.6 and 2.0). Heun's error is lower per step (0.029 at 8 steps against Euler's 0.041), but each step costs two evaluations, and at *equal evaluations* it has no clear advantage on this variance-preserving path (16 evaluations: 0.016 for Euler, 0.029 for Heun at 8 steps; 32: 0.008 and 0.009); on the straight-line path it does better (F4). The stochastic sampler is three times worse than the ODE at 16 steps, falls to 0.017 at 32 and stops improving (0.029 at 64): the finite sample, whose own distance is 0.011, and the noise left at the last step set a floor. It cannot win here and often wins on images because our velocity is exact, so there are no errors for injected noise to wash out; a learned model has errors, and noise that partly erases them can be worth the extra steps. The toy cannot show that, and the note claims nothing beyond the sources named. ### E7. Why this is the same model as in Part D Two sentences close the loop with the VAE view. Training on the denoising loss with weight $\lambda(t)$ is a weighted integral of the quantity the ELBO is made of (D5, Kingma and Gao). Huang et al. (2021) give the continuous-time version, the ELBO of a diffusion model being a denoising score matching loss up to a model-independent term, and Song et al. (2021, *Maximum likelihood training of score-based diffusion models*) show that the weight $g_t^2$ makes the loss an upper bound on the negative log-likelihood. So an ELBO, a denoising loss and a score-matching loss are one object with different weights. **Exercise E1.** On the straight-line path at $t=0.5$ the score at $x=0$ is 0.336. What are $\hat x_0$ and $\hat\varepsilon$ there? **Exercise E2.** Show that for the variance-preserving schedule the probability-flow ODE is $\dot x=-\tfrac\beta2\big(x+\nabla\log p_t(x)\big)$.
Answers **E1.** $\hat x_0=(x+\sigma^2\,\text{score})/\alpha=(0+0.25\cdot0.336)/0.5=0.168$ and $\hat\varepsilon=-\sigma\,\text{score}=-0.168$. **E2.** For this schedule $f=-\beta/2$ and $g^2=\beta$, so the probability-flow ODE of E5 is $\dot x=fx-\tfrac12g^2\nabla\log p=-\tfrac\beta2x-\tfrac\beta2\nabla\log p=-\tfrac\beta2(x+\nabla\log p)$.
## F. Flow matching ### F1. Training a flow by regression Part C ended with a continuous flow whose velocity $v_t(x)$ is a network, and a cost: maximum likelihood integrates the ODE inside every training step. Flow matching *chooses* a family of densities $p_t$ from data ($t=0$) to noise ($t=1$) and asks for the velocity whose ODE carries each into the next; by the continuity equation of C3 it exists and satisfies $\partial_tp_t=-\nabla\!\cdot(p_tv_t)$. We could regress a network onto it if we could evaluate it, but it depends on the unknown data density. The idea of Lipman et al. (2023), reached independently by Liu et al. (2022) and Albergo and Vanden-Eijnden (2023), is that as in denoising score matching the marginal velocity is an average of conditional velocities that *are* known. (A convention: the flow matching papers put noise at $t=0$ and data at $t=1$. This note keeps $t=0$ for data and $t=1$ for noise so that the formulas match Part D. To convert, replace $t$ by $1-t$ and flip the sign of the velocity.) **The construction.** Pick a data point $x_0$ and a noise vector $\varepsilon$ and connect them by a path, for instance the Gaussian family of D1, $$x_t=\htmlClass{eq-obs}{\alpha_t}x_0+\htmlClass{eq-hid}{\sigma_t}\htmlClass{eq-hid}{\varepsilon},\qquad \htmlClass{eq-der}{u_t}(x_t\mid \htmlClass{eq-obs}{x_0},\htmlClass{eq-hid}{\varepsilon})=\frac{dx_t}{dt}=\htmlClass{eq-obs}{\dot\alpha_t}x_0+\htmlClass{eq-hid}{\dot\sigma_t}\htmlClass{eq-hid}{\varepsilon} .$$ The speed of this one path is its *conditional velocity* $u_t$, known exactly. (Lipman et al. condition on the data point alone and take the path to be Gaussian around it; conditioning on the noise vector as well, as here, gives the same marginal velocity.) Averaging over all the paths through a given place gives the marginal velocity, $$\htmlClass{eq-der}{v_t(x)}=\mathbb E\big[\htmlClass{eq-der}{u_t}\ \big|\ x_t=x\big],$$ and the conditional flow matching loss $$\mathcal L_{\rm CFM}(\theta)=\mathbb E_{t,\htmlClass{eq-obs}{x_0},\htmlClass{eq-hid}{\varepsilon}}\big\|\htmlClass{eq-hid}{v_\theta}(x_t,t)-\htmlClass{eq-der}{u_t}(x_t\mid \htmlClass{eq-obs}{x_0},\htmlClass{eq-hid}{\varepsilon})\big\|^2$$ has the same gradient in $\theta$ as the intractable loss that regresses on $v_t$ itself.
Training never runs the ODE; sampling does. The network regresses on the velocity of a path through a random pair, then the same network drives the solver.
Derivation: the average of the conditional velocities generates the marginal path **Claim.** (1) With $v_t(x)=\mathbb E[u_t\mid x_t=x]$, the ODE $\dot x=v_t(x)$ transports $p_t$, the density of the noised data. (2) The loss $\mathcal L_{\rm CFM}$ differs from $\mathbb E\|v_\theta-v_t\|^2$ by a constant independent of $\theta$. **Step 1: continuity for each conditional path.** Fix $x_0$ and $\varepsilon$. The point $x_t$ moves deterministically with velocity $u_t$. For the pair $(x_0,\varepsilon)$ drawn from their joint law, write $q_t(x\mid z)$ for the density of $x_t$ given the pair $z=(x_0,\varepsilon)$; in the Gaussian family with $\varepsilon$ included in the conditioning this is a point mass moving along the path, and its density obeys $\partial_tq_t(x\mid z)=-\nabla\!\cdot\big(q_t(x\mid z)\,u_t(x\mid z)\big)$ by C3. **Step 2: average over the pair.** Multiply by the density $p(z)$ of the pair and integrate. The marginal density is $p_t(x)=\int q_t(x\mid z)p(z)\,dz$, so $$\partial_tp_t(x)=-\nabla\!\cdot\int q_t(x\mid z)\,\htmlClass{eq-der}{u_t}(x\mid z)\,p(z)\,dz .$$ **Step 3: recognise a conditional mean.** The integral equals $p_t(x)\,\mathbb E[u_t\mid x_t=x]=p_t(x)v_t(x)$, since $q_t(x\mid z)p(z)/p_t(x)$ is the posterior of $z$ given $x$. So $\partial_tp_t=-\nabla\!\cdot(p_tv_t)$: the continuity equation for $v_t$, which proves (1). **Step 4: the loss.** As in Step 2 of the derivation in E2, for a random target $Y=u_t$ and a function $f$ of $x_t$, $\mathbb E\|f-Y\|^2=\mathbb E\|f-\mathbb E[Y\mid x_t]\|^2+\mathbb E\|Y-\mathbb E[Y\mid x_t]\|^2$. With $f=v_\theta$ the first term is $\mathbb E\|v_\theta-v_t\|^2$, and the second does not involve $\theta$, which proves (2). **What this does not say.** The ODE with the marginal velocity transports the *marginals* $p_t$; it does not reproduce the individual paths. Two conditional paths that cross at a point have different velocities there, and the marginal velocity is their average.
Figure 7. A network trained on targets that disagree learns their average, and the loss cannot go below the disagreement. The straight-line path, $x_t=(1-t)x_0+t\varepsilon$, whose target is $\varepsilon-x_0$. Top: each copper dot is the target for one random pair at the chosen time, plotted at the noisy position it came from; the violet curve is the exact average, the velocity field. Bottom: the variance left in the target after the best possible prediction, against time, below the mean square of the target itself (violet, 2.31). At $t=0.5$ it is 2.05, 89 percent of the total; it is 1.10 at $t=0.2$, 2.23 at $t=0.6$ and 1.57 at $t=0.9$. Drag the ring: the dots scatter widely but their average is a smooth curve.
Figure 7 shows both parts. The dots are sampled targets; no function of position fits them, because the pairs arriving at a place disagree about where they go. The curve is their average, the velocity of the flow. The lower panel is the second term of Step 4: the loss at the best network is *not* zero but the height of the copper curve, between 1.1 and 2.2 against a total mean square of 2.31 for the example, highest where the modes merge. So a flow matching or diffusion training curve is flat from early on and its final value says little about quality; the information is in the difference from a floor that is invisible. The noise-prediction loss of DDPM is the same, since E2 is the same argument. ### F2. For Gaussian paths flow matching is diffusion Take the Gaussian conditional path above. The marginal velocity is the conditional mean of $\dot\alpha_tx_0+\dot\sigma_t\varepsilon$ given $x_t$, which is $$\htmlClass{eq-der}{v_t(x)}=\htmlClass{eq-obs}{\dot\alpha_t}\,\htmlClass{eq-hid}{\hat x_0}(x,t)+\htmlClass{eq-hid}{\dot\sigma_t}\,\htmlClass{eq-hid}{\hat\varepsilon}(x,t),$$ linear in the denoiser and the noise estimate of Part D. By Tweedie's formula (E3) it can be written in terms of the score, $$\htmlClass{eq-der}{v_t(x)}=\frac{\htmlClass{eq-obs}{\dot\alpha_t}}{\htmlClass{eq-obs}{\alpha_t}}x+\Big(\frac{\htmlClass{eq-obs}{\dot\alpha_t}\htmlClass{eq-hid}{\sigma_t}^2}{\htmlClass{eq-obs}{\alpha_t}}-\htmlClass{eq-hid}{\sigma_t}\htmlClass{eq-hid}{\dot\sigma_t}\Big)\htmlClass{eq-der}{\nabla\log p_t(x)}.$$ Compare with the probability-flow ODE of E5. *The velocity field that flow matching learns on a Gaussian path is exactly the probability-flow ODE of the diffusion model with that schedule.* There is nothing to choose between them as models; they are one vector field written in different variables.
Derivation: the two vector fields are the same **Claim.** For the path $x_t=\alpha_tx_0+\sigma_t\varepsilon$, the marginal velocity $v_t$ equals the drift of the probability-flow ODE of E5. **Step 1: the diffusion side.** From E4 the schedule has $f_t=\dot\alpha_t/\alpha_t$ and $g_t^2=2\sigma_t\dot\sigma_t-2f_t\sigma_t^2$. The probability-flow ODE is $\dot x=f_tx-\tfrac12g_t^2\nabla\log p_t(x)$. **Step 2: expand the coefficient of the score.** $$-\tfrac12g_t^2=-\htmlClass{eq-hid}{\sigma_t}\htmlClass{eq-hid}{\dot\sigma_t}+f_t\htmlClass{eq-hid}{\sigma_t}^2=\frac{\htmlClass{eq-obs}{\dot\alpha_t}\htmlClass{eq-hid}{\sigma_t}^2}{\htmlClass{eq-obs}{\alpha_t}}-\htmlClass{eq-hid}{\sigma_t}\htmlClass{eq-hid}{\dot\sigma_t} .$$ **Step 3: the flow matching side.** The marginal velocity is $v_t=\dot\alpha_t\hat x_0+\dot\sigma_t\hat\varepsilon$. Tweedie's formula (E3) gives $\hat x_0=(x+\sigma_t^2\nabla\log p_t)/\alpha_t$ and $\hat\varepsilon=-\sigma_t\nabla\log p_t$. Substituting, $$\htmlClass{eq-der}{v_t}=\frac{\htmlClass{eq-obs}{\dot\alpha_t}}{\htmlClass{eq-obs}{\alpha_t}}x+\Big(\frac{\htmlClass{eq-obs}{\dot\alpha_t}\htmlClass{eq-hid}{\sigma_t}^2}{\htmlClass{eq-obs}{\alpha_t}}-\htmlClass{eq-hid}{\sigma_t}\htmlClass{eq-hid}{\dot\sigma_t}\Big)\htmlClass{eq-der}{\nabla\log p_t(x)}.$$ **Step 4: compare.** The coefficient of $x$ is $f_t$ and the coefficient of the score is the expression of Step 2. The two vector fields are identical. **What this does not say.** It does not say the two training losses are the same: they weight the noise levels differently (F3). It says only that the function being learned is the same.
The Flow Matching paper says this in its abstract: its Gaussian probability paths subsume the diffusion paths as particular instances, and training diffusion models with flow matching, in the authors' report, is a more robust and stable alternative. ### F3. What is different in practice If the model is the same, what separates a diffusion implementation from a flow matching one? Five choices, each changeable alone.
Diffusion against flow matching is five separate choices, not two families. The usual setting of each is shown; any box can be swapped alone.
**The weighting.** On the straight-line path the flow matching loss puts weight $1/t^2$ on the clean-data error and the ELBO $(1-t)/t^3$; the ratio is $t/(1-t)=\sigma/\alpha$: 0.111 at $t=0.1$, 1 at $t=0.5$, 9 at $t=0.9$. Relative to the bound, uniform-$t$ flow matching puts nine times less weight on the least noisy levels and nine times more on the noisiest. **The distribution of $t$.** Esser et al. (2024) study rectified flow for text-to-image generation and bias $t$ toward perceptually relevant intermediate scales, for instance with a *logit-normal* distribution; in their large-scale study the best setup outperformed the diffusion formulations they compared with (their words: superior performance). The weighting of D6 and the distribution of $t$ move the same dial. **The output.** Velocity, noise, clean data or score: the same function reparameterised (E3), changing the conditioning of the regression and stability at the ends of the time range. **The path and the sampler.** Diffusion uses a variance-preserving or variance-exploding path (D1); rectified flow uses the straight line $\alpha_t=1-t$, $\sigma_t=t$. A flow matching model is sampled by an ODE solver, a diffusion model by an ODE solver or an SDE, and the sampler literature (E6) transfers to either.
Derivation: the weight of the velocity loss against the ELBO's **Claim.** On the path $\alpha_t=1-t$, $\sigma_t=t$, the velocity loss weights $\|x_0-\hat x_0\|^2$ by $1/t^2$ per unit time and the ELBO by $(1-t)/t^3$. **Step 1: the velocity error in terms of the denoiser error.** With $v=\dot\alpha\hat x_0+\dot\sigma\hat\varepsilon$ and $\hat\varepsilon=(x-\alpha\hat x_0)/\sigma$, $$v=\Big(\htmlClass{eq-obs}{\dot\alpha}-\frac{\htmlClass{eq-hid}{\dot\sigma}\htmlClass{eq-obs}{\alpha}}{\htmlClass{eq-hid}{\sigma}}\Big)\htmlClass{eq-hid}{\hat x_0}+\frac{\htmlClass{eq-hid}{\dot\sigma}}{\htmlClass{eq-hid}{\sigma}}x,$$ so the difference between two velocity guesses with the same $x$ is $\big(\dot\alpha-\dot\sigma\alpha/\sigma\big)$ times the difference of the denoisers, and $$\|v-\htmlClass{eq-hid}{\hat v}\|^2=\Big(\htmlClass{eq-obs}{\dot\alpha}-\frac{\htmlClass{eq-hid}{\dot\sigma}\htmlClass{eq-obs}{\alpha}}{\htmlClass{eq-hid}{\sigma}}\Big)^2\|\htmlClass{eq-obs}{x_0}-\htmlClass{eq-hid}{\hat x_0}\|^2 .$$ **Step 2: evaluate on the straight line.** $\dot\alpha=-1$, $\dot\sigma=1$, $\alpha/\sigma=(1-t)/t$, so $\dot\alpha-\dot\sigma\alpha/\sigma=-1-\frac{1-t}{t}=-\frac1t$. The weight is $1/t^2$. **Step 3: the ELBO's weight.** By D4 it is $\tfrac12|d\,\mathrm{snr}/dt|$ with $\mathrm{snr}=(1-t)^2/t^2$. Then $$\frac{d\,\mathrm{snr}}{dt}=-\frac{2(1-t)}{t^2}-\frac{2(1-t)^2}{t^3}=-\frac{2(1-t)}{t^3}\big(t+(1-t)\big)=-\frac{2(1-t)}{t^3},$$ so the ELBO's weight is $(1-t)/t^3$. **Step 4: the ratio.** $\dfrac{1/t^2}{(1-t)/t^3}=\dfrac{t}{1-t}=\dfrac\sigma\alpha$. **What this does not say.** It compares the weights per unit time at uniform sampling of $t$. A different distribution of $t$ (the logit-normal below) changes the effective weight again.
So the contrasts are a path, a weighting, a time distribution, an output and a solver, chosen separately. ### F4. Straight conditional paths, curved marginal flow The appeal of the straight path is that straight trajectories integrate exactly in one step. Two facts qualify it. First, the straight lines are *conditional* paths, one per pair of data point and noise vector, and they cross. The learned ODE is the *average* over the paths through a point, and its trajectories cannot cross (a smooth velocity field has a unique solution through every point). In one dimension a trajectory cannot overtake another, so the end-to-end map of the exact flow is the quantile map of C1 whatever the path; different paths give different trajectories to that same map. Second, the averaged trajectories are not straight. On the two-peak example the exact trajectories of the straight-line path have a total length 1.77 times the distance between their endpoints, and an 8-step Euler sampler on this path has error 0.101, against 0.019 for the cosine path (D6). With a first-order solver the straight conditional path gives no few-step advantage here; with Heun's second-order solver it does: 8 Heun steps reach 0.006 on the straight-line path, against 0.029 for the variance-preserving and 0.009 for the cosine path, because its trajectories, though not straight, are smooth. What buys few-step sampling is *rectification* (Liu et al. 2022): after training a flow, generate pairs (noise, flow output) and train a new flow on straight paths between those pairs. The pairs are a deterministic coupling, their lines cross far less, the authors prove the transport cost does not increase, and they report that repeating the procedure gives increasingly straight paths. This "reflow" step, not the straight conditional path alone, is the gain. ### F5. Beyond Gaussian paths Nothing in F1 uses the Gaussian form of the path: it needs a conditional path with known velocity and a way to sample the conditioning variable. Albergo and Vanden-Eijnden (2023) build flows from "stochastic interpolants" between *any* two densities with a quadratic loss, noting that a Gaussian base yields a diffusion model and its score, so diffusion can be bypassed. Lipman et al. (2023) use the optimal-transport displacement between a noise and a data sample, and the *Flow Matching Guide* (2024) collects the extensions: Riemannian manifolds, discrete state spaces through continuous-time Markov chains, "generator matching". Pairing noise and data by minibatch optimal transport, so that conditional paths cross less, is a further line this note does not cover. ### F6. One step and a few steps The last family gives up the ODE solver. *Progressive distillation* (Salimans and Ho 2022) distils a deterministic sampler into one with half the steps, repeatedly, and introduces the velocity parameterisation for stability at few steps. *Consistency models* (Song et al. 2023) learn a function mapping any point on a probability-flow trajectory to its origin, which gives one-step generation by design and multistep sampling to trade compute for quality; they are trained by distillation or alone. Both build on the ODE view of Part E and are not reviewed beyond their abstracts. **Exercise F1.** On the straight-line path, what is the conditional velocity $u_t$ for $x_0=1$ and $\varepsilon=-0.5$? Is it the same at every $t$? **Exercise F2.** At $t=0.5$ on the straight-line path, what weight does the flow matching loss give to the clean-data error, and how does it compare with the ELBO's?
Answers **F1.** $u_t=\dot\alpha x_0+\dot\sigma\varepsilon=-1\cdot1+1\cdot(-0.5)=-1.5$, the same at every $t$: the path is a straight line traversed at constant speed. **F2.** The flow matching weight is $1/t^2=4$ and the ELBO's is $(1-t)/t^3=0.5/0.125=4$. They are equal at the midpoint of the path, and differ by the factor $t/(1-t)$ elsewhere.
## G. The map, and what the map leaves out ### G1. One table The parts above fit on one page. Each family is a column; the figure shows what each learns and what a sample costs, the table what it is trained on and what it can compute.
The families differ in who does the work and how often the network is called. Amber is learned, grey fixed, red the many-pass cost.
| | VAE | Normalizing flow | Diffusion (D, E) | Flow matching (F) | | --- | --- | --- | --- | --- | | Loss | the bound, reconstruction and rate | exact likelihood | a weighted denoising error; the bound if the weight is right | a weighted regression of velocity | | Likelihood | a bound, with a gap | exact | a bound; exact through the ODE (E5) | exact through the ODE (C3) | | Deterministic latent code | encoder mean (approximate) | exact | exact through the ODE | exact through the ODE | Reading down the third and fourth columns, the diffusion and flow-matching models have the same vector field (F2), the same exact likelihood computation (C3) and the same deterministic code, and differ in loss weighting, path and habits of use. Reading across to the first, the diffusion model is a VAE whose encoder is frozen to a noising chain and whose depth is large (D3), so each term is closed form and no encoder is learned. Three dials remain, and the families are different settings of them. 1. **How much of the work the encoder does.** In a VAE the encoder is learned and does as much as the model needs; the cost is the gap of B4 and the difficulty of making a one-step decoder good. In a diffusion model the encoder is fixed and does nothing but add noise, and the entire modelling burden falls on the denoiser, which is run many times. 2. **How the noise levels are weighted.** The likelihood weights every level equally on the log signal-to-noise axis (D4). Perceptual quality is served better by other weights (D6, F3), at some cost in likelihood. 3. **The number of network evaluations per sample.** One for a VAE or a distilled model, and between ten and a thousand for a diffusion or flow model, depending on the path, the solver and the amount of distillation. ### G2. Latent diffusion: the two-stage model that won A diffusion model in pixel space spends most of its capacity on detail the eye cannot see, and every sample costs hundreds of passes over a full-resolution image. Rombach et al. (2022) split the job: first train an autoencoder that removes imperceptible detail, then train the diffusion model in its latent space. In the language of this note it is the rate–distortion trade of B3 applied once, to choose the latent's size, and then a generative model fitted to the result.
Two stages: compress once, diffuse in the small space. The caption or class enters only the denoiser.
The autoencoder is a VAE or VQ-VAE with a very small weight on the rate term. Rombach et al. describe a *KL-reg* variant that "imposes a slight KL-penalty" towards a standard normal, and a *VQ-reg* variant with a quantisation layer in the decoder, as in the VQGAN of the note on discrete codebooks. They add a perceptual and an adversarial loss so the decoder does not blur (the cure for B5), choose a downsampling factor $f=2^m$, and rescale the latent to controlled variance. A figure in the paper compares $f=4$ with DALL-E's $f=8$ and VQGAN's $f=16$. The diffusion model then models the distribution of latents, a much easier problem than pixels, with a many-step denoiser instead of a one-step decoder. Peebles and Xie (2023) replaced the convolutional U-Net by a transformer on latent patches (*DiT*), reporting a strong negative correlation between forward-pass cost and FID, and an FID of 2.27 on class-conditional ImageNet at $256\times256$, obtained with classifier-free guidance at scale 1.5 in their convention, which is $w=0.5$ in the $(1+w)$ form below. Esser et al. (2024) use the rectified-flow loss of Part F in an autoencoder's latent space, with a transformer in which text and image tokens have separate weights. Seen from the codebook note, the second stage can be made tractable two ways: quantise the latent into a vocabulary and model it autoregressively (dead codes, a ceiling set by the tokenizer), or keep it continuous and use a diffusion or flow model (a solver, paid for in evaluations). ### G3. Conditioning and guidance To generate from a *conditional* distribution $p(x\mid c)$ (a class, a caption), give the network $c$ as an extra input and train as before. A separate question is how to make the output follow the condition *more* closely than the data distribution does. Bayes' rule gives a handle: $$\htmlClass{eq-der}{\nabla\log p(x\mid c)}=\htmlClass{eq-der}{\nabla\log p(x)}+\htmlClass{eq-der}{\nabla\log p(c\mid x)}.$$ The conditional score is the unconditional score plus the gradient of the log-probability a classifier gives to $c$. Strengthen that term with a weight $w\ge0$ and samples are pushed towards regions where $c$ is likely. Ho and Salimans (2022) do it without a classifier: one network sometimes sees the condition and sometimes a null token, and the two outputs are combined at sampling time as $\tilde\varepsilon=(1+w)\,\varepsilon_\theta(x,c)-w\,\varepsilon_\theta(x)$.
Guidance is two passes and an extrapolation. $w=0$ is the plain conditional model.
Derivation: what the guided score points towards **Claim.** The guided score $(1+w)\nabla\log p(x\mid c)-w\nabla\log p(x)$ is the score of the density $\propto p(x\mid c)^{1+w}p(x)^{-w}=p(x\mid c)\,p(c\mid x)^{w}/p(c)^{w}$, at the data end ($t=0$). **Step 1: write the density.** $\log\tilde p=(1+w)\log p(x\mid c)-w\log p(x)+\text{const}$. Its gradient in $x$ is the guided score. **Step 2: rewrite with Bayes.** $p(x\mid c)/p(x)=p(c\mid x)/p(c)$, so $p(x\mid c)^{1+w}p(x)^{-w}=p(x\mid c)\big(p(c\mid x)/p(c)\big)^{w}$. The factor $p(c\mid x)^w$ is the implicit classifier's probability raised to the power $w$: the density is the class density, re-weighted towards points the classifier is sure about. **Step 3: the caveat about time.** The identity holds for the densities of the *clean* data. At noise level $t$ the network supplies the scores of the *noised* densities $p_t(x\mid c)$ and $p_t(x)$. Noising does not commute with taking powers and ratios of densities: $p_t(x\mid c)^{1+w}p_t(x)^{-w}$ is not the noised version of the tilted density. So the guided field is the score of the right object only at $t=0$, and the sampler, which integrates it over all $t$, is not guaranteed to land on the tilted density. **What this does not say.** It does not say guidance is wrong. It says the tilted density is a description of the rule's intent, not of its output.
Figure 8. Guidance sharpens samples towards the class, and overshoots the density it was meant to reach. Two Gaussian classes with centres at $\pm0.7$ and spread 0.5; samples are drawn for the right-hand class with exact scores and a 256-step sampler. Grey dashed: all data. Blue: the class density. Violet dashed: the density $p(x\mid c)^{1+w}p(x)^{-w}$, normalised. Copper: the samples. At weight 0 the samples match the class density (distance 0.002). At weight 1 their mean is 1.02 and their spread 0.36; the target's are 0.79 and 0.44, and the distance between samples and target is 0.225. At weight 4 the samples have spread 0.27 and a classifier calls 99.9 percent of them right-hand, against 96.7 percent for the target.
Figure 8 shows two things at once. Guidance does what it is for: as $w$ grows the samples concentrate away from the other class and a classifier is more and more confident about them (0.88 at $w=0$, 0.99 at $w=1$, 0.999 at $w=3$). It also costs diversity: the spread of the samples falls from 0.50 to 0.36 at $w=1$ and 0.27 at $w=4$. And the sampler overshoots its stated target: at $w=1$ the samples sit further to the right and are tighter than the density $p(x\mid c)^{1+w}p(x)^{-w}$. The caveat of Step 3 is the reason, and the example is as simple as one can make it. This is a property of the rule as written, observed here with exact scores; whether and how it matters for images, where the networks are inexact and the weights are tuned by eye, is a different question that this note does not address. ### G4. Discrete data Everything above assumes real-valued data. For tokens (text, code, the codes of the note on discrete codebooks) the Gaussian machinery does not apply. The VAE has the VQ-VAE (B7); diffusion has analogues that corrupt tokens, for instance by masking, and a reverse process that restores them; flow matching has one on continuous-time Markov chains, developed in the sixth and seventh chapters of the Flow Matching Guide. This note derives none of them, and the identities of Parts D to F are for continuous variables: do not assume each carries over. ### G5. What to watch when training one - **The loss is a poor guide.** Its floor is well above zero (F1) and the curve flattens early. Watch sample quality, and the loss *at each noise level* (the integrand of Figure 3), not the average. - **Weight and time distribution are one decision.** Write the effective weight on the clean-data error against signal-to-noise ratio (the table in D6) and see where it puts the effort. - **Check the end of the schedule.** The signal-to-noise ratio at $t=1$ should be near zero, and the sampler should start from exactly the distribution training ended at (D6). - **Report quality against evaluations.** One number at one solver and step count does not say whether the model or the sampler is better. Plot quality against network evaluations, for the ODE and the SDE, as in Figure 6. - **Sample from a moving average of the weights**, not the last iterate; common practice (cited from memory). - **Likelihood is a different test.** Evaluate it by the ODE with the bound at the weight of D4, and expect it to disagree with sample quality (D6, DDPM's two losses). ### G6. Which to use | You need | A reasonable choice | Why | | --- | --- | --- | | Fast sampling, one pass | a VAE with a good decoder, or a distilled diffusion or flow model | Part B; F6 | | The best sample quality at moderate compute | latent diffusion or flow matching, the straight-line path, a tuned time distribution, a second-order solver | G2, F3, E6 | | Likelihoods, compression | a diffusion model trained on the bound with a learned schedule and many steps | D5, Kingma et al. | | A latent for editing or retrieval | a VAE (one pass), or the ODE encoding of a diffusion model | B, E5 | | Control by a condition, with fidelity over diversity | a conditional model with guidance | G3 | | Discrete data | a VQ-VAE with an autoregressive prior, or a discrete diffusion model | B7, G4 | ## H. Limits, exercises and sources ### H1. What the note cannot tell you - **One dimension is not images.** Every figure uses a density on the line whose denoiser is exact. Real data live near a thin surface in a space of millions of dimensions, the networks are inexact in ways that depend on the noise level, and a sampler's errors compound over hundreds of steps. The identities of Parts C to F do not depend on the dimension; the sizes of the gaps in Figures 3, 4 and 6 do. - **An exact denoiser is not a good one.** For a finite training set the exact denoiser of the *empirical* distribution moves a noisy point towards a weighted average of the training points, and at small noise it snaps to the nearest one: following it reproduces the training set. That follows from the posterior formula in A3 with point masses. That trained networks do not behave this way, and why, is a question of what the network cannot represent, not of the loss. - **The rankings are for this example.** Which schedule needs fewest steps (Figure 4), whether Heun beats Euler at equal evaluations (Figure 6), and how far the guided sampler overshoots (Figure 8) are measured on a two-peak density with exact scores. They show that the effects exist and what produces them. They are not benchmarks. - **Likelihood and quality come apart.** The DDPM table of D6 is the evidence: the loss with the better bound has the worse FID. A note on likelihood cannot say which model draws better pictures. - **Not covered.** Discrete diffusion beyond a mention, video, the training of consistency models, energy-based models, generative adversarial networks, autoregressive models, Schrödinger bridges, and the engineering of large runs (resolution-dependent noise schedules, mixed precision, parallelism). ### H2. More exercises **Exercise H1.** Using Figure 3: how many steps, placed evenly in log signal-to-noise ratio, bring the bound to within one nat of the entropy? **Exercise H2.** For the two overlapping classes of Figure 8 and guidance weight 1, compare the samples with the density $p(x\mid c)^{1+w}p(x)^{-w}$. **Exercise H3.** Why is the loss of a trained flow matching network not a measure of how good it is? **Exercise H4.** Two models are trained on the two-peak data, one with the straight-line path and one with the variance-preserving path, both with exact velocities, and sampled with a thousand Euler steps. What should the two sets of samples look like, and what changes if each is sampled with four steps?
Answers **H1.** The bound minus the entropy is 2.37 nats at 64 steps, 1.08 at 128 and 0.52 at 256. 128 steps leaves 0.08 nats too much and 256 is the first of the grid that is within a nat. **H2.** The density has mean 0.79, spread 0.44 and a classifier calls 93.3 percent of its mass right-hand. The samples have mean 1.02, spread 0.36 and 98.6 percent. The distance between samples and density is 0.225, against 0.002 at weight 0: the sampler pushes further than the rule it implements. **H3.** F1: the loss at the best network equals the variance of the target that no network can remove, which for the two-peak example is 1.1 to 2.2 of a total of 2.31, depending on the time. The loss curve therefore flattens at a large value, and the interesting part, the difference from the floor, is not visible. A network far from optimal and one close to it both have a loss near the floor. **H4.** With a thousand steps both approximate the same map from noise to data (C1: in one dimension any increasing map carrying the Gaussian to the data is the quantile map), so the two sets of samples should be indistinguishable from the data and from each other. With four steps the discretisation errors differ by path (D6, Figure 4): at 4 steps the variance-preserving path is at 0.175 and the straight-line path at 0.197 on the example (Euler), both visibly wrong.
### H3. Reading list Each of these was opened. For the following only the abstract and the passages cited in the text were read: NVAE, Very Deep VAEs, Importance Weighted Autoencoders, Generating Sentences from a Continuous Space, Fixing a Broken ELBO, NICE, Real NVP, Glow, rectified flow, stochastic interpolants, the Flow Matching Guide (its contents and the cited chapters), progressive distillation, consistency models, DiT, the Stable Diffusion 3 paper beyond its noise-sampling section, Information-Theoretic Diffusion and the flawed-schedules paper. Years are those of publication; the arXiv identifiers date several of them one year earlier. **Variational autoencoders.** - Kingma and Welling 2013, *Auto-Encoding Variational Bayes*, [arXiv:1312.6114](https://arxiv.org/abs/1312.6114). - Burda, Grosse and Salakhutdinov 2016, *Importance Weighted Autoencoders*, [arXiv:1509.00519](https://arxiv.org/abs/1509.00519). - Bowman et al. 2016, *Generating Sentences from a Continuous Space*, [arXiv:1511.06349](https://arxiv.org/abs/1511.06349). - Alemi et al. 2018, *Fixing a Broken ELBO*, [arXiv:1711.00464](https://arxiv.org/abs/1711.00464). - Vahdat and Kautz 2020, *NVAE: A Deep Hierarchical Variational Autoencoder*, [arXiv:2007.03898](https://arxiv.org/abs/2007.03898). - Child 2021, *Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images*, [arXiv:2011.10650](https://arxiv.org/abs/2011.10650). **Flows.** - Rezende and Mohamed 2015, *Variational Inference with Normalizing Flows*, [arXiv:1505.05770](https://arxiv.org/abs/1505.05770). - Dinh, Krueger and Bengio 2015, *NICE*, [arXiv:1410.8516](https://arxiv.org/abs/1410.8516). - Dinh, Sohl-Dickstein and Bengio 2017, *Density Estimation using Real NVP*, [arXiv:1605.08803](https://arxiv.org/abs/1605.08803). - Kingma and Dhariwal 2018, *Glow*, [arXiv:1807.03039](https://arxiv.org/abs/1807.03039). - Chen et al. 2018, *Neural Ordinary Differential Equations*, [arXiv:1806.07366](https://arxiv.org/abs/1806.07366). - Grathwohl et al. 2019, *FFJORD*, [arXiv:1810.01367](https://arxiv.org/abs/1810.01367). **Diffusion and score models.** - Sohl-Dickstein et al. 2015, *Deep Unsupervised Learning using Nonequilibrium Thermodynamics*, [arXiv:1503.03585](https://arxiv.org/abs/1503.03585). - Song and Ermon 2019, *Generative Modeling by Estimating Gradients of the Data Distribution*, [arXiv:1907.05600](https://arxiv.org/abs/1907.05600). - Ho, Jain and Abbeel 2020, *Denoising Diffusion Probabilistic Models*, [arXiv:2006.11239](https://arxiv.org/abs/2006.11239). - Song et al. 2021, *Score-Based Generative Modeling through Stochastic Differential Equations*, [arXiv:2011.13456](https://arxiv.org/abs/2011.13456). - Song, Meng and Ermon 2021, *Denoising Diffusion Implicit Models*, [arXiv:2010.02502](https://arxiv.org/abs/2010.02502). - Nichol and Dhariwal 2021, *Improved Denoising Diffusion Probabilistic Models*, [arXiv:2102.09672](https://arxiv.org/abs/2102.09672). - Kingma et al. 2021, *Variational Diffusion Models*, [arXiv:2107.00630](https://arxiv.org/abs/2107.00630). - Huang et al. 2021, *A Variational Perspective on Diffusion-Based Generative Models and Score Matching*, [arXiv:2106.02808](https://arxiv.org/abs/2106.02808). - Song et al. 2021, *Maximum Likelihood Training of Score-Based Diffusion Models*, [arXiv:2101.09258](https://arxiv.org/abs/2101.09258). - Karras et al. 2022, *Elucidating the Design Space of Diffusion-Based Generative Models*, [arXiv:2206.00364](https://arxiv.org/abs/2206.00364). - Kingma and Gao 2023, *Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation*, [arXiv:2303.00848](https://arxiv.org/abs/2303.00848). - Kong et al. 2023, *Information-Theoretic Diffusion*, [arXiv:2302.03792](https://arxiv.org/abs/2302.03792). - Lin et al. 2023, *Common Diffusion Noise Schedules and Sample Steps are Flawed*, [arXiv:2305.08891](https://arxiv.org/abs/2305.08891). **Flow matching.** - Lipman et al. 2023, *Flow Matching for Generative Modeling*, [arXiv:2210.02747](https://arxiv.org/abs/2210.02747). - Liu, Gong and Liu 2022, *Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow*, [arXiv:2209.03003](https://arxiv.org/abs/2209.03003). - Albergo and Vanden-Eijnden 2023, *Building Normalizing Flows with Stochastic Interpolants*, [arXiv:2209.15571](https://arxiv.org/abs/2209.15571). - Lipman et al. 2024, *Flow Matching Guide and Code*, [arXiv:2412.06264](https://arxiv.org/abs/2412.06264). - Salimans and Ho 2022, *Progressive Distillation for Fast Sampling of Diffusion Models*, [arXiv:2202.00512](https://arxiv.org/abs/2202.00512). - Song et al. 2023, *Consistency Models*, [arXiv:2303.01469](https://arxiv.org/abs/2303.01469). **Latent diffusion and guidance.** - Ho and Salimans 2022, *Classifier-Free Diffusion Guidance*, [arXiv:2207.12598](https://arxiv.org/abs/2207.12598). - Rombach et al. 2022, *High-Resolution Image Synthesis with Latent Diffusion Models*, [arXiv:2112.10752](https://arxiv.org/abs/2112.10752). - Peebles and Xie 2023, *Scalable Diffusion Models with Transformers*, [arXiv:2212.09748](https://arxiv.org/abs/2212.09748). - Esser et al. 2024, *Scaling Rectified Flow Transformers for High-Resolution Image Synthesis*, [arXiv:2403.03206](https://arxiv.org/abs/2403.03206). **Not opened here.** Luo (2022), the tutorial *Understanding Diffusion Models: A Unified Perspective*; Hyvärinen (2005) on score matching; Vincent (2011) on denoising score matching and its link to autoencoders; Anderson (1982) on the reverse of a diffusion; Robbins (1956) and Efron (2011) on the formula that this note calls Tweedie's; Feller (1949), as cited by Sohl-Dickstein et al.; Guo, Shamai and Verdú (2005) on the I-MMSE relation, as cited by Kong et al.; Higgins et al. (2017) on the $\beta$-VAE; Dhariwal and Nichol (2021) on classifier guidance; Cremer, Li and Duvenaud (2018) on the amortisation gap; the discrete diffusion papers and the work on minibatch optimal-transport couplings. The statements about them in the text come from memory or from the papers that cite them. **Companion notes.** [Discrete Codebooks](codebook), whose VQ-VAE is the discrete case of Part B and whose tokenizers are the first stage of the models of G2; [The Art of LLM Inference](inference) for what a many-step sampler costs in network evaluations. ### Changelog - Version 1: first draft. The VAE, flows, diffusion as a hierarchical VAE, score and SDE views, flow matching, and the map between them; eight figures on an exactly solvable one-dimensional example.