# Deriving the Prior-Data Fitted Network

You have twelve measurements and one missing answer. A line could explain them. So could a curve. With little data, choosing one explanation too early is a consequential decision. Can a neural network learn to carry several explanations, use the measurements to reweight them, and predict the missing answer—without updating its weights on your dataset?

<figure class="mfig pf-figure pf-wide" id="pf-context" aria-label="The coefficient posterior and predictive regression update together as context arrives">
<div class="pf-plot"></div>
<figcaption><b>Conditioning gathers plausible lines.</b> Drag the handle to reveal measurements: the ellipse of coefficients shrinks and the family of lines tightens, while the band keeps the noise of the next observation.</figcaption>
</figure>

That is the problem behind a **prior-data fitted network**, or PFN. The network is trained on many small prediction problems before it sees yours. What it learns across those problems is a way to learn *within* a new problem. The dataset becomes an input to the network, alongside the point whose answer you want.

The derivation has three steps. Bayes' rule says what the ideal answer is. A training objective teaches a network to produce that answer from simulated tasks. A transformer that reads tables supplies the machinery. Two extensions then follow almost for free: change the question from observation to intervention, as in Do-PFN, or represent a time series as a prediction task, as in ForecastPFN and TabPFN-TS. A last section combines both for the media mix model, where the prior is unusually well known and where the earlier essay ran into the limits of what data alone can say.

The figures use small exact models, so every moving quantity has a known meaning; they are not outputs of a TabPFN or Do-PFN checkpoint. Where a figure has a ring handle, drag it, or focus the figure and use the arrow keys.

## A prior is a distribution over whole worlds

One colour code runs through the essay, the same as in the figures, and the media-mix essay uses its own: <span class="k-world">violet</span> is the hidden task, <span class="k-data">blue</span> is the observed context, and <span class="k-query">copper</span> is the query and the answer we want.

Let a dataset be $D=\{(x_i,y_i)\}_{i=1}^{n}$. The usual picture picks a model family and fits its parameters to this particular $D$. Begin one level earlier instead: imagine a process that generates entire datasets. Draw a hidden task parameter $\phi$ from a distribution $\pi$, then draw many rows under that *same* $\phi$:

$$
\htmlClass{eq-world}{\phi}\sim\pi(\htmlClass{eq-world}{\phi}),\qquad
y_i\sim p(y_i\mid x_i,\htmlClass{eq-world}{\phi}).
$$

For a family of lines, $\phi=(b,w)$ and $y_i=b+w x_i+\epsilon_i$. The intercept and slope are drawn once per dataset; the noise $\epsilon_i$ is drawn again for each row. Drawing a new slope for every row would describe a different problem, because the rows would no longer reveal one shared line.

<figure class="mfig pf-figure" id="pf-worlds" aria-label="Three examples of a prior over complete prediction tasks">
<div class="pf-plot"></div>
<figcaption><b>A prior is a distribution over whole tasks, not over rows.</b> Each curve is one possible task; the highlighted one generates a dataset, its rows being the blue points.</figcaption>
</figure>

A **prior** is the distribution over these worlds before the current measurements arrive. It can be a density, or only a sampler: a simulator that draws a causal graph, equations on its edges, noise levels and a target column is already a prior over tabular problems. Synthetic data does not remove assumptions. It makes them executable. A learner that has only seen smooth functions has reason to continue a smooth curve; one trained on thresholds has reason to preserve abrupt changes.

## Evidence reweights the worlds

Suppose there are only three possible slopes $w_j$. The probability of the observations under slope $j$ is its **likelihood**, and Bayes' rule turns it into a new weight:

$$
\underbrace{p(\htmlClass{eq-world}{w_j}\mid \htmlClass{eq-data}{D})}_{\text{posterior weight}}
=
\frac{\overbrace{p(\htmlClass{eq-data}{D}\mid \htmlClass{eq-world}{w_j})}^{\text{likelihood}}
      \overbrace{\pi(w_j)}^{\text{prior weight}}}
{\sum_k p(D\mid w_k)\pi(w_k)}.
$$

With independent Gaussian errors of variance $\sigma^2$, the log-likelihood is, up to a constant,

$$
\log p(D\mid w_j)
=-\frac{1}{2\sigma^2}\sum_i(y_i-w_jx_i)^2+\text{constant}.
$$

So the familiar squared residual has a probabilistic meaning: a line earns less weight when its errors would be surprising at the assumed noise level. An imperfect line is not discarded; it keeps whatever weight its residuals leave it.

<figure class="mfig pf-figure" id="pf-posterior" aria-label="Moving one observation recomputes posterior weights over three slopes">
<div class="pf-plot"></div>
<figcaption><b>A measurement reweights the worlds.</b> Drag it vertically: each line keeps weight in proportion to how well it explains the point, and the three weights sum to one.</figcaption>
</figure>

## A prediction averages what remains possible

We want an outcome $y_*$ at a query input $x_*$. Even after seeing $D$ we may not know which world generated it, so we average each world's outcome distribution with its posterior weight:

$$
p(\htmlClass{eq-query}{y_*}\mid \htmlClass{eq-query}{x_*},\htmlClass{eq-data}{D})
=\int p(\htmlClass{eq-query}{y_*}\mid \htmlClass{eq-query}{x_*},\htmlClass{eq-world}{\phi})\,p(\htmlClass{eq-world}{\phi}\mid \htmlClass{eq-data}{D})\,d\htmlClass{eq-world}{\phi}.
$$

This is the **posterior predictive distribution**. The posterior $p(\phi\mid D)$ is a distribution over hidden explanations; the predictive is a distribution over the next answer. A predictor can approximate the second without ever exposing the first. (We treat $x_*$ as a fixed experimental design, so it carries no evidence about $\phi$. When inputs depend on the hidden world, the weight becomes $p(\phi\mid D,x_*)$; the causal section needs that distinction.)

<figure class="mfig pf-figure" id="pf-mixture" aria-label="The predictive distribution mixes the outcomes of surviving hypotheses">
<div class="pf-plot"></div>
<figcaption><b>Surviving worlds disagree most far from the data.</b> Drag the query along <i>x</i>: the predictive density is the weighted mixture of what each line predicts there.</figcaption>
</figure>

The mean is a useful summary but not the distribution. Two competing explanations can put most of their probability on either side of it, and reporting only the average would erase exactly the disagreement that matters.

### An exact ruler

For linear regression we can compute everything. Here the hidden task is the coefficient vector $\beta=\phi$. Write $h(x)=(1,x)^\top$, stack the features into a matrix $H$, and assume

$$
\beta\sim\mathcal N(0,S_0),\qquad
y=H\beta+\epsilon,\qquad
\epsilon\sim\mathcal N(0,\sigma^2 I).
$$

Multiplying prior and likelihood adds their quadratic terms in $\beta$; completing the square gives

$$
S_n=(S_0^{-1}+\sigma^{-2}H^\top H)^{-1},\qquad
m_n=S_n\sigma^{-2}H^\top y,
$$

and integrating the coefficients out gives the query distribution

$$
y_*\mid x_*,D\sim
\mathcal N\!\left(
h_*^\top m_n,
\underbrace{\sigma^2}_{\text{measurement noise}}
+\underbrace{h_*^\top S_nh_*}_{\text{uncertainty about the line}}
\right).
$$

These are the calculations in the opening figure. More data shrinks $S_n$ but never removes the next measurement's noise. The model is a ruler for checking learned approximations; richer priors do not come with a small matrix inverse.

## Train a network to do the averaging

Suppose we can generate tasks but cannot evaluate their posterior predictive integral. We can still generate the *correct answer to a prediction problem*: sample a world, sample a dataset from it, hide one label, and keep that label as the training target. With a network $q_\theta(y_*\mid x_*,D)$, one pretraining episode is

$$
\htmlClass{eq-world}{\phi}\sim\pi,\quad
\htmlClass{eq-data}{D}\sim p(\cdot\mid \htmlClass{eq-world}{\phi}),\quad
(\htmlClass{eq-query}{x_*},\htmlClass{eq-query}{y_*})\sim p(\cdot\mid \htmlClass{eq-world}{\phi}),\quad
\ell=-\log q_\theta(\htmlClass{eq-query}{y_*}\mid \htmlClass{eq-query}{x_*},\htmlClass{eq-data}{D}).
$$

The context and the query come from the same world. The query label reaches the loss but never the network's input. Repeat over many worlds and update $\theta$ to lower the average loss. This is the original PFN construction. [Müller et al., ICLR 2022](https://arxiv.org/html/2112.10510v7)

<figure class="mfig pf-figure" id="pf-episode" aria-label="A sampled world supplies context points and a held-out answer; the answer is only read against the predictor's density">
<div class="pf-plot"></div>
<figcaption><b>Training scores a prediction against an answer it never saw.</b> Drag the slope of the violet world. The density at x* is computed from the blue points alone; the copper bar from y* to the curve is q(y*), and the score is its negative log.</figcaption>
</figure>

No posterior is ever computed as a target. The simulator supplies ordinary outcomes, and across enough episodes those outcomes reveal which distributions are good predictions for each kind of context.

### Why log loss has the right answer

Minimizing the expected log loss makes the network's answer equal the simulator's own conditional distribution, which is the Bayesian predictive when the simulator is the prior.

<figure class="mfig pf-figure" id="pf-score" aria-label="Expected log loss is minimized at the generating class probability">
<div class="pf-plot"></div>
<figcaption><b>Expected log loss is lowest at the true probability.</b> Drag the network's probability: the minimum sits at 0.7, not at the most confident answer.</figcaption>
</figure>

<details class="more"><summary>Derivation: why log loss has the right answer</summary>

Write $\mathcal I=(D,x_*)$ for everything the predictor sees and $p_\pi(y\mid \mathcal I)$ for the simulator's true conditional distribution. Averaging first over the possible answers to the same visible problem,

$$
\begin{aligned}
\mathcal L(\theta)
&=\mathbb E_{\mathcal I}\!\left[
-\int p_\pi(y\mid \mathcal I)\log q_\theta(y\mid \mathcal I)\,dy
\right]\\
&=\mathbb E_{\mathcal I}\!\left[
H\big(p_\pi(\cdot\mid \mathcal I)\big)
+\operatorname{KL}\!\left(
p_\pi(\cdot\mid \mathcal I)\,\Vert\,q_\theta(\cdot\mid \mathcal I)
\right)\right],
\end{aligned}
$$

where the second form adds and subtracts $\log p_\pi(y\mid \mathcal I)$ inside the integral. The entropy does not depend on $\theta$. The KL divergence is nonnegative and zero only when the two distributions agree. So the population optimum, with enough capacity and a suitable output family, is the simulator's posterior predictive.

</details>

That is an idealized statement about the expected objective, not a guarantee about a finite network. Finite simulation, limited capacity, output discretization, optimization, and a mismatch between synthetic and real tasks can all leave a gap. The loss also matters: squared error has optimum $\mathbb E[Y\mid \mathcal I]$, the conditional mean, so it trains a point estimate rather than a distribution. That difference separates the forecasting extensions later.

### What has been amortized?

A conventional Bayesian procedure runs an inference computation for every new dataset. A PFN pays for many such tasks once, during pretraining, and stores an approximate prediction algorithm in $\theta$. At use time the new dataset changes the activations and the prediction; $\theta$ stays fixed. This is **in-context learning**, and "no task-specific gradient updates" is the precise claim. Pretraining still required optimization, a longer context still costs computation and memory, and practical TabPFN inference adds preprocessing and ensembles around the single pass described here.

## How a transformer reads a table

### Rows become tokens, tables become grids

A table is not a sentence. Rows are examples, columns have different meanings, and the target is partly missing. Reordering complete examples should not change the problem, so row position must not be a signal.

The first TabPFN classifier embedded each context pair $(x_i,y_i)$ as one token and the query as $(x_*,\text{missing})$. The missing flag matters: an unobserved target is not a target equal to zero. [Hollmann et al., ICLR 2023](https://arxiv.org/html/2207.01848v6) The 2025 architecture instead keeps a grid of vectors, one per example and feature group, with the target as an extra column. Attention along a row lets the model describe an example; attention down a column lets it compare examples. [Hollmann et al., Nature 2025](https://www.nature.com/articles/s41586-024-08328-6)

<figure class="mfig pf-figure pf-wide" id="pf-table" aria-label="A numeric table locates the sources and receivers of two indexed attention matrices">
<div class="pf-plot"></div>
<figcaption><b>One grid, read along two axes.</b> In blue, the answer cell of a context row weighs the cells of its own row. In copper, the query's answer cell weighs the four context rows. Tap a row to change it.</figcaption>
</figure>

The worked network below has four labelled examples, one query, three input columns and one target column. Each feature group is one scalar and each vector has four channels, so the arithmetic stays visible. The projection matrices are fixed illustrative weights, not a pretrained model, but the operations are genuine attention, residual additions and normalization. In the released v2 architecture, raw columns are normally grouped in pairs and each position carries a much larger vector. [Official v2 implementation](https://github.com/PriorLabs/TabPFN/blob/9d9d8725168810e5f9de76bc68e0d4d6688ae26f/src/tabpfn/architectures/tabpfn_v2.py)

### Attention is a learned weighted retrieval

Stack the vectors as rows of $Z$. With column-vector projections $q=W_Qh$, learned matrices produce queries, keys and values:

$$
Q=ZW_Q^\top,\qquad K=ZW_K^\top,\qquad V=ZW_V^\top.
$$

For one query vector $q$, compare it with each permitted key $k_i$, scale, exponentiate and normalize:

$$
\alpha_i=
\frac{\exp(q^\top k_i/\sqrt{d_k})}
{\sum_{j\ \mathrm{permitted}}\exp(q^\top k_j/\sqrt{d_k})},
\qquad o=\sum_i\alpha_i v_i.
$$

Keys decide what is retrieved; values decide what is carried.

<figure class="mfig pf-figure" id="pf-attention" aria-label="The direction of a query vector determines attention weights on four visible key-value pairs">
<div class="pf-plot"></div>
<figcaption><b>Attention is a soft dictionary lookup.</b> Turn the query: alignment with a key raises its weight and draws the output toward that key's value.</figcaption>
</figure>

Here $q$ and the four keys are hand-chosen two-dimensional vectors, so alignment is visible as an angle. A trained PFN learns its projections from representations of the table. Note what $\alpha_i$ is: a weight on an *input representation*, not automatically $p(\phi\mid D)$, a weight on a whole possible world. The training objective says which prediction the stack is encouraged to approximate; one attention map does not prove which inference algorithm it learned.

### Two axes, one operation

Let $h_{ij}\in\mathbb R^d$ be the vector for example $i$ and feature group $j$. **Feature attention** holds $i$ fixed: the tokens are the feature groups of one row, including the target slot. For destination $j$,

$$
a^{\mathrm{feat}}_{i,jk}
=\operatorname{softmax}_{k}\!\left(
\frac{(W_Q^{\mathrm{feat}}h_{ij})^\top
      (W_K^{\mathrm{feat}}h_{ik})}{\sqrt{d_k}}
\right),\qquad
u_{ij}=\sum_k a^{\mathrm{feat}}_{i,jk}W_V^{\mathrm{feat}}h_{ik}.
$$

This step never reads another example. For the query's target token, it gathers the query's observed inputs into the place the answer will be read from. In the toy, each vector starts as $h=(z,m,c,1)^\top$: a standardized value, a missingness flag, a column identifier and a constant. The query's target has $z=0$ and $m=1$, an encoding of absence rather than a supplied answer.

<figure class="mfig pf-figure pf-wide" id="pf-feature-attention" aria-label="Feature attention multiplies four within-row value vectors by their normalized weights">
<div class="pf-plot"></div>
<figcaption><b>Within a row, the query target gathers its own features.</b> Each row of V is one feature, weighted and summed per channel. Tap a channel, or drag the query value.</figcaption>
</figure>

The result is added to the old representation and normalized, $\bar h=\operatorname{LN}(h+u)$, so attention refines a vector rather than overwriting it.

**Row attention** holds the feature group $j$ fixed and varies the example. Each destination forms a query from $\bar h_{ij}$, and its keys and values come from the labelled context examples $\ell=1,\ldots,n$:

$$
a^{\mathrm{row}}_{j,i\ell}
=\operatorname{softmax}_{\ell\le n}\!\left(
\frac{(W_Q^{\mathrm{row}}\bar h_{ij})^\top
      (W_K^{\mathrm{row}}\bar h_{\ell j})}{\sqrt{d_k}}
\right),\qquad
r_{ij}=\sum_{\ell=1}^{n}a^{\mathrm{row}}_{j,i\ell}
W_V^{\mathrm{row}}\bar h_{\ell j}.
$$

This is more expressive than a comparison of one raw column: $\bar h_{\ell j}$ has already gathered the other columns of example $\ell$, so in the target column a context token describes both its outcome and the features under which it occurred.

<figure class="mfig pf-figure pf-wide" id="pf-row-attention" aria-label="Row attention multiplies enriched context target vectors by their normalized weights">
<div class="pf-plot"></div>
<figcaption><b>Across rows, the same arithmetic gathers examples.</b> The sources are now the four context examples in the target column. Tap a channel, or drag the shared query value.</figcaption>
</figure>

The arithmetic is the one just seen; only the sources changed. Feature and row stages have separate projections, so changing the axis changes the question asked even though the mechanism is the same. The query's own target token is not among the keys; its state survives through the residual connection. A per-cell MLP then transforms each vector separately and moves no information between cells. Repeating these blocks and applying a head to the query's final target vector produces the prediction.

### Two steps are enough to spread information

Any context feature can reach the query's answer in two moves, one along each axis.

<figure class="mfig pf-figure pf-wide" id="pf-receptive-field" aria-label="A selected context feature reaches the query by a horizontal feature-attention edge and a vertical row-attention edge">
<div class="pf-plot"></div>
<figcaption><b>Two axis steps connect any context feature to the answer.</b> Tap a feature: its route runs along its own row, then down the target column. Q₂ is blocked as a source for Q₁.</figcaption>
</figure>

<details class="more"><summary>Derivation: the two-step route</summary>

Take an observed value in row $\ell$, column $k$, far from the query's target. Feature attention can carry it into the context target token, and row attention can then carry that token into the query's target:

$$
h_{\ell,k}\quad\longrightarrow\quad
\bar h_{\ell,y}\quad\longrightarrow\quad
\tilde h_{*,y}.
$$

</details>

Attending along each axis separately, instead of over every cell at once, keeps tables with many rows and columns affordable. The saving comes from giving communication a structured route, not from assuming the features are independent.

### What the network must not see

The matrix form of attention is

$$
\operatorname{Attention}(Z)
=\operatorname{softmax}\!\left(
\frac{QK^\top}{\sqrt{d_k}}+M
\right)V,
$$

where $M_{ij}=0$ permits a read and $M_{ij}=-\infty$ forbids it; a forbidden entry gets zero probability. In the across-row stage, context and query rows read context keys only. Query features still enter their own representations through encoding, within-row attention and residuals, but a hidden query target never enters the input. Excluding queries from the keys stops one query from relaying information to another, which is different from the left-to-right mask of next-token generation. [TabPFN, Appendix E.2](https://arxiv.org/html/2207.01848v6#A5.SS2)

<figure class="mfig pf-figure" id="pf-mask" aria-label="An attention mask permits context reads and isolates test queries">
<div class="pf-plot"></div>
<figcaption><b>The mask decides who may read whom.</b> Filled cells are permitted reads; query columns, including each query's own, are masked.</figcaption>
</figure>

This concerns the attention computation. A full pipeline may also preprocess several query rows together, so strict independence of a whole software call has to be checked there too. The toy fixes its preprocessing statistics from the context to keep the property checkable.

Order is the second thing the network must ignore. Permuting the context rows by a matrix $P$ permutes keys and values together, and softmax is equivariant to that permutation:

$$
\operatorname{softmax}(q(PK)^\top)\,PV
=\operatorname{softmax}(qK^\top)\,V.
$$

<figure class="mfig pf-figure" id="pf-permutation" aria-label="Two orders of the same weighted values have different intermediate sums and the same final sum">
<div class="pf-plot"></div>
<figcaption><b>Reordering the rows reorders the additions, not the sum.</b> The same four weight–value pairs reach the same endpoint through different partial sums.</figcaption>
</figure>

Feature order needs more care, because swapping columns must keep each column's meaning and apply the same rearrangement to context and queries. The practical 2025 TabPFN models are not fully permutation invariant; their inference ensembles shuffle feature order to approximate the property. [Nature, inference details](https://www.nature.com/articles/s41586-024-08328-6#Sec12)

## What comes out of the network

For classification, a head predicts one logit per class and a softmax gives probabilities. For regression the outcome is continuous, so the head needs a density representation. A useful PFN construction divides the target axis into bins $B_k=[b_k,b_{k+1})$, predicts their masses $m_k$ with a softmax, and spreads each mass uniformly across its bin:

$$
q_\theta(y\mid \mathcal I)
=\frac{m_k(\mathcal I)}{b_{k+1}-b_k},
\qquad y\in B_k.
$$

Height is density; **area** is probability. A bin twice as wide needs half the height to carry the same mass. The loss for a target in bin $k$ is $-\log m_k+\log(b_{k+1}-b_k)$, and for fixed boundaries the width term does not change the gradient with respect to the logits. The original PFN work calls this a Riemann distribution. [PFN, Appendix E.2](https://arxiv.org/html/2112.10510v7#A5.SS2)

<figure class="mfig pf-figure" id="pf-density" aria-label="A shaded probability density and its linked cumulative distribution share an outcome threshold">
<div class="pf-plot"></div>
<figcaption><b>Probability is area under the density, and height on the cumulative curve.</b> Drag c: the shaded area above equals F(c) below, and unequal bin widths show up as unequal slopes.</figcaption>
</figure>

Means, quantiles and intervals follow from the predicted distribution; the [appendix](#appendix-a-pfn-for-regression-start-to-finish) writes them out. Real regression heads also need a policy for tails, normalization and transformed targets; the bounded example draws no infinite tail it does not have. An inaccurate density can mean the network has not learned the inference, or that the synthetic prior confidently describes the wrong kinds of problems. More optimization addresses only the first, and calibration on a simulator is not proof of calibration on an arbitrary real table.

## Change the target: from prediction to intervention

Suppose the table has a treatment $A$, an outcome $Y$ and background features $X$. An ordinary predictor answers a question about the records in which treatment happened to equal $a$,

$$
p(Y\mid A=a,X=x,D),
$$

while a causal question asks what happens when we *set* treatment to $a$:

$$
p(Y\mid do(A=a),X=x,D).
$$

They differ because treatment selection can carry information about unobserved causes of the outcome. In a structural causal model (SCM) each variable is generated by an assignment $Z_j=f_j(Z_{\mathrm{pa}(j)},U_j)$. An intervention replaces the assignment for $A$ with $A:=a$ and leaves the others running.

<figure class="mfig pf-figure" id="pf-surgery" aria-label="An intervention replaces the treatment assignment and removes its incoming causal edge">
<div class="pf-plot"></div>
<figcaption><b>An intervention replaces one assignment and leaves the rest.</b> Observing A keeps its cause U; setting A cuts that arrow.</figcaption>
</figure>

Training an observational PFN on SCM data does not by itself teach it intervention queries; the loss must reward the answer to the question we actually want. **Do-PFN** therefore samples an SCM $\psi$, generates an *observational* context from it, and generates query targets under interventions on that same SCM. The network learns to predict interventional outcomes from observational context plus the intervention request. [Robertson et al., NeurIPS 2025](https://arxiv.org/html/2506.06039v3)

Restrict $X$ to pre-treatment features and choose the intervention externally. The Bayesian target is

$$
p_\pi(\htmlClass{eq-query}{y}\mid do(A=a),\htmlClass{eq-query}{x},\htmlClass{eq-data}{D^{\mathrm{obs}}})
=\int p_{\htmlClass{eq-world}{\psi}}(\htmlClass{eq-query}{y}\mid do(A=a),\htmlClass{eq-query}{x})
\,p_\pi(\htmlClass{eq-world}{\psi}\mid \htmlClass{eq-data}{D^{\mathrm{obs}}},\htmlClass{eq-query}{x})\,d\htmlClass{eq-world}{\psi},
$$

and, as before, a network can approximate this mixture from simulated targets and log loss. The prior now ranges over mechanisms and graphs, and the query covariates stay in the weights: seeing a person's features is evidence about which world produced them.

This amortizes causal prediction. It does not make every effect identifiable. Take two worlds with $U\sim\mathcal N(0,1)$ and independent zero-mean noise $\epsilon\sim\mathcal N(0,\sigma^2)$:

$$
\begin{array}{lll}
\text{World A:}& A=U,&Y=2A+\epsilon,\\
\text{World B:}& A=U,&Y=2U+\epsilon.
\end{array}
$$

Their observed tables have exactly the same distribution: in both, $Y=2A+\epsilon$. Under $do(A=a)$, though, World A has mean outcome $2a$ and World B has mean zero, because setting $A$ does not move $U$.

<figure class="mfig pf-figure" id="pf-identification" aria-label="Observationally identical causal models give different interventional means">
<div class="pf-plot"></div>
<figcaption><b>Identical data can hide different causal answers.</b> Drag the assigned treatment: both worlds fit the same cloud, yet their interventional means disagree.</figcaption>
</figure>

If the worlds start with equal prior weight they keep equal weight after any number of observational rows, so the prior-relative mean is the average of their interventional means. A longer context cannot manufacture the missing information. Familiar assumptions (consistency, conditional exchangeability, positivity) justify replacing a causal mean by an adjusted observational one; a prior over SCMs is a different, explicit source of structure, and its appropriateness is part of the scientific argument.

## Put time into the task

### Time as table features, or as simulated worlds

A forecasting problem gives a history $y_0,\ldots,y_T$ and asks about the future. We can build a table whose features say *when* each observation occurred:

$$
\htmlClass{eq-data}{D_T}=\{(g(t),y_t):t\le T\},\qquad
g(t)=\left(
t,\sin\frac{2\pi t}{P},\cos\frac{2\pi t}{P},z_t
\right).
$$

Here $P$ is a candidate seasonal period and $z_t$ holds covariates known for the query time. The sine–cosine pair encodes phase (sine alone would give different phases the same value), and the running index can represent change across cycles. Future times become rows with missing targets. Shuffling whole rows does not erase time, because $t$ travels with its target; shuffling the targets alone would.

<figure class="mfig pf-figure" id="pf-time-features" aria-label="Moving a future query shows its calendar and seasonal feature row">
<div class="pf-plot"></div>
<figcaption><b>A forecast question is a table row with the answer missing.</b> Drag the future time: its features are known, its outcome is the question mark.</figcaption>
</figure>

This is the route of **TabPFN-TS**: a pretrained tabular regressor with temporal feature engineering. The paper's v4 formulation uses calendar, seasonal and known-future covariate features, no autoregressive lags, so future rows can be queried in parallel, and it needs no time-series-specific pretraining. [Hoo et al., 2026 version](https://arxiv.org/html/2501.02945v4) The representation carries an inductive bias: a Fourier feature makes repeating phase easy to see, and a running index supplies a coordinate for trend without guaranteeing extrapolation.

**ForecastPFN** takes the other route. It draws synthetic series with trend, seasonality and noise and pretrains a forecaster on them. Its implemented objective is squared error, so it learns a conditional point estimate; it should not be called a density-output PFN merely because it shares the idea. [Dooley et al., NeurIPS 2023](https://proceedings.nips.cc/paper_files/paper/2023/file/0731f0e65559059eb9cd9d6f44ce2dd8-Paper-Conference.pdf)

<figure class="mfig pf-figure pf-wide" id="pf-time-priors" aria-label="Temporal synthetic worlds lead to a mean target while time-feature tables support predictive densities">
<div class="pf-plot"></div>
<figcaption><b>ForecastPFN changes the training worlds; TabPFN-TS changes the input.</b> The curves are toy targets, not outputs of either checkpoint.</figcaption>
</figure>

The comparison organizes every extension: one can change the prior, the input representation, the target, the loss or the inference procedure. Two systems can share a transformer and solve different statistical problems.

### A seasonal posterior we can calculate

To inspect the inference problem without a checkpoint in the way, use a small Bayesian seasonal regression,

$$
y_t=\beta_0+\beta_1(t/24)
+\beta_2\sin(2\pi t/12)
+\beta_3\cos(2\pi t/12)+\epsilon_t,
$$

with a Gaussian prior on $\beta$. This is the linear model derived earlier with four features; the formulas for $m_n$ and $S_n$ still apply, since the curve is nonlinear in time but linear in the unknown coefficients.

<figure class="mfig pf-figure" id="pf-forecast" aria-label="An exact seasonal Bayesian forecast updates as more history is revealed">
<div class="pf-plot"></div>
<figcaption><b>More history narrows the forecast, but never the noise of the next point.</b> Drag the boundary: the mean and band come from an exact four-coefficient posterior, not from TabPFN-TS.</figcaption>
</figure>

The toy makes its assumption visible: the period of twelve is built into the feature map. If the process changes regime, or the period is wrong, a narrow band can still be wrong. A PFN that has amortized a richer prior can express richer patterns, but its generalization still has to be checked on the processes of interest.

### A band is not a path

Knowing every future marginal $q_h(y_{T+h}\mid D_T)$ does not determine the joint law of a trajectory. In the seasonal toy, one posterior draw of $\beta$ used at every future time gives a coherent path; drawing independently from each time's marginal breaks that shared explanation.

<figure class="mfig pf-figure" id="pf-joint" aria-label="Coherent and independent draws have identical marginal uncertainty but different temporal dependence">
<div class="pf-plot"></div>
<figcaption><b>Identical marginal bands can hide different paths.</b> Both panels have the same uncertainty at each time; on the left every path shares one sampled coefficient vector, on the right each time is drawn independently.</figcaption>
</figure>

Write the four-component design vector as

$$
h(t)=(1,\ t/24,\ \sin(2\pi t/12),\ \cos(2\pi t/12))^\top.
$$

The latent seasonal mean then has covariance

$$
\operatorname{Cov}(f_s,f_t\mid D_T)
=h(s)^\top S_nh(t),
$$

and independent marginal sampling sets every off-diagonal term to zero. That changes the answer to questions about cumulative demand or simultaneous threshold crossings. Per-future-row predictive distributions, as in TabPFN-TS, should not be silently treated as a joint trajectory model; adding a joint construction is a further modeling decision.

### Keep the future out of the context

At forecast origin $T$ the context may contain targets measured at or before $T$. Calendar features and scheduled events can be known for later dates. Future realized demand, or a lag that depends on a future target, is not known merely because a table can hold it.

<figure class="mfig pf-figure" id="pf-leakage" aria-label="Forecasts from past targets alone and from the same history plus one forbidden future target are compared on matched axes">
<div class="pf-plot"></div>
<figcaption><b>One future target changes the forecast, and the band hides it.</b> Drag T: the lower fit also receives the diamond, so its mean shifts and its band narrows using information unavailable at T.</figcaption>
</figure>

Estimate periods, scalers and other transformations from the available history, evaluate on chronological or rolling origins, and, if a covariate's future is unknown, supply a forecast or scenario rather than its later realization, with its added uncertainty.

## A case worth building: the media mix model

The [media mix model essay](mmm-derived.html) showed that two years of weekly sales cannot settle how an effect fades, and that a budget which follows demand makes advertising look better than it is. Its remedy was outside information: a holdout, a lift test, an on-off schedule. Such questions are decided by the prior and by whatever identifying variation the context contains, which is the situation a PFN is built for, because it makes the prior an explicit, trained object.

Three things line up.

- **The prior is known.** The MMM essay derived the mechanisms: adstock, a saturating response, trend, seasonality. A prior over worlds is a draw of these with their parameters, a budget rule and noise. This is much narrower than the prior of a generic table model, and the narrower the prior, the more a network trained on it has to offer over a general one. That is a hypothesis to test, not a result.
- **The data are small.** The MMM essay counts about a hundred weekly observations for a dozen parameters. That is the regime where conditioning on a few rows matters and per-brand refitting is the cost.
- **The target can be causal.** "Budgets follow demand" is a structural causal model with a latent demand that drives both spend and sales. Sample such worlds with varying confounding strength, record sales under $do(\text{spend plan})$, and the Do-PFN construction applies: the network learns to answer the interventional question from observational weeks, averaged over how endogenous the budget might be. Lift tests and geo holdouts enter as extra context rows. The test to run is whether a network trained on worlds with and without such rows narrows its interval when an experiment is present and widens it when none is.

It does not make endogeneity go away. Without variation the demand did not choose, the effect is not identified, and the network returns the prior's answer. The claim to test is calibration: because the simulator records the true contribution of every channel, coverage can be measured, including when the real prior differs from the training prior. Carryover also runs across rows, which favors the second option of the previous section: simulate whole time series as tasks, rather than adding lag features to a table.

PyMC Labs has open-sourced the prior side: [`pymc-generator`](https://github.com/pymc-labs/pymc-generator), an alpha simulator of marketing worlds with confounding, mediation, carryover and saturation curves, the exact decomposition of the outcome, stacked padded corpora of worlds, and a plug-in oracle model to compare against (a reference, not the exact posterior). Its stated limits are a Gaussian outcome with identity link and a baseline with no seasonal term, though treatments and covariates can carry a seasonal sinusoid and a trend. I found no released trained network and no benchmark against a Bayesian MMM, so this section is a proposal rather than a result.

## What stays the same

The common object is a **prediction problem presented as input**. The prior chooses which problems the learner sees repeatedly. The context supplies evidence about this problem. The query states which answer is missing. The objective decides which summary or distribution is rewarded.

| Construction | What changes | The question learned |
| --- | --- | --- |
| PFN principle | Simulated task distribution | Predict a hidden answer from a context |
| TabPFN | Tabular priors, architecture, and output head | Predict an outcome in a new table |
| Do-PFN | Interventional targets from sampled SCMs | Predict an intervention's outcome from observational context |
| ForecastPFN | Temporal simulator and a squared-error objective | Predict a future conditional mean |
| TabPFN-TS, cited formulation | Time features around a pretrained tabular regressor | Predict each future row's outcome distribution |
| Proposed: PFN for MMM | Simulated marketing worlds with confounded budgets and experiments in context | Predict sales under a spend plan, and each channel's contribution |

There are two levels of learning. Pretraining changes the weights across tasks; in-context inference uses a new task's observations with those weights fixed. The Bayesian derivation explains the target of that learned procedure, and the transformer supplies a flexible mechanism for approximating it.

The starting question was how twelve measurements can change a network's answer without changing its weights. They are not an instruction to retrain the network. They are part of the function it was trained to evaluate.

## Appendix: a PFN for regression, start to finish

Regression is the case the figures use, so it is worth stating once as a recipe. The [exact ruler](#an-exact-ruler) is its special case with a linear function and Gaussian noise.

**1. Prior.** Draw a hidden function $f$ and a noise level, then draw $n+m$ rows $x_i$ and outcomes $y_i=f(x_i)+\epsilon_i$. Everything the network will ever know about regression is in the family of $f$ and $\epsilon$ that this step can produce.

**2. Episode.** Show the network $n$ rows as the context $D$ and hold out the other $m$ as queries. Each held-out $y_*$ is a training target; it never enters the input.

**3. Scale.** Standardize the outcomes of each episode, for example by the context mean and spread, so one set of bins serves every task. This is a design choice: it fixes what "the same shape at a different scale" means to the network.

**4. Head and loss.** Predict bin masses $m_k$ and minimize $-\log m_k+\log(b_{k+1}-b_k)$ for the bin that holds $y_*$, as in the previous section.

**5. Read the answer.** The predicted density is piecewise constant, so its cumulative function is piecewise linear. With $M_k=\sum_{j<k}m_j$,

$$
F(y)=M_k+m_k\,\frac{y-b_k}{b_{k+1}-b_k}\quad(y\in B_k),
\qquad
\mathbb E[y_*]\approx\sum_k m_k\,\frac{b_k+b_{k+1}}{2},
$$

and the $p$-quantile is $b_k+\dfrac{p-M_k}{m_k}(b_{k+1}-b_k)$ for the bin with $M_k\le p<M_{k+1}$. A 90% interval takes $p=0.05$ and $p=0.95$. The mean is approximate because mass beyond the outer bins needs a tail rule.

**6. Check against the ruler.** Under the linear-Gaussian prior the exact predictive is $\mathcal N\!\big(h_*^\top m_n,\ \sigma^2+h_*^\top S_nh_*\big)$. A trained network's median and interval should match it, and the gap is the amortization error. The log score on held-out episodes measures it directly, because the exact predictive has the best expected score.

Training on squared error instead returns only the conditional mean. That is the ForecastPFN choice, and it is why such a network cannot draw a band. The bin head costs a few hundred outputs and returns the whole distribution, which is what the question "how sure are you?" needs.

## Further reading

- [Müller et al. — *Transformers Can Do Bayesian Inference*](https://arxiv.org/html/2112.10510v7). The original PFN objective and predictive-density construction.
- [Hollmann et al. — *TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second*](https://arxiv.org/html/2207.01848v6). The first tabular classification specialization.
- [Hollmann et al. — *Accurate predictions on small data with a tabular foundation model*](https://www.nature.com/articles/s41586-024-08328-6). The 2025 cell-based architecture and regression extension.
- [Robertson et al. — *Do-PFN: In-Context Learning for Causal Effect Estimation*](https://arxiv.org/html/2506.06039v3). Observational contexts with interventional supervision.
- [Dooley et al. — *ForecastPFN: Synthetically-Trained Zero-Shot Forecasting*](https://proceedings.nips.cc/paper_files/paper/2023/file/0731f0e65559059eb9cd9d6f44ce2dd8-Paper-Conference.pdf). A temporal prior with a point-prediction objective.
- [Hoo et al. — *From Tables to Time: Extending TabPFN-v2 to Time Series Forecasting*](https://arxiv.org/html/2501.02945v4). The temporal-feature approach; this essay refers to the January 2026 paper version rather than assuming every subsequent package uses the same configuration.
- [PyMC Labs — `pymc-generator`](https://github.com/pymc-labs/pymc-generator). An open-source generator of synthetic marketing worlds with ground-truth decompositions.
- [The media mix model, derived](mmm-derived.html). The mechanisms and the identification limits that motivate the last section.