# A/B Tests and Quasi-Experiments from First Principles
An online shop redesigns its checkout page and wants to know whether the new page sells more. The easy case is the one it controls: show the new page to a random half of the visitors and compare. Then the harder cases arrive. A feature cannot be forced on anyone, only advertised in an email. Free shipping goes to every customer with a loyalty score of 600 or more, and nobody chose the cutoff for the sake of an experiment. A price change goes live in three cities and not in the others. Each of these looks like a different method with a different name, and each is the same idea applied to a different lever.
A coin flip points at the truth; self-selection points somewhere else, and more visitors only make it surer. The curves show where the estimated effect lands over many repeated tests, the bars below show a typical 95% interval. The new page adds 2 to spend per visitor. When visitors choose the page themselves, the keen ones choose it, and the estimate lands near 5. Drag the handle to change the visitors per arm.
The idea has two moves. **Subtract what the groups share**, so that whatever made them different in the first place cancels. **Divide by how far the design moved the treatment**, so that an effect of being assigned becomes an effect of being treated. This essay derives a coin flip, a covariate measured before the test, a nudge, a cutoff and a before-and-after comparison from the same potential outcomes, and shows the two moves in each. One colour code runs through the text and the figures: blue is what we observe, copper is a hidden quantity such as the outcome a visitor did not get, and violet is the effect we derive.
## Assumptions
1. **No spillovers, one version.** A visitor's outcome depends only on that visitor's own treatment, and "the new page" means one thing. If treated and untreated visitors compete for the same stock, or talk to each other, the comparisons below answer a different question.
2. **Consistency.** The outcome we observe is the potential outcome of the treatment the visitor actually received.
3. **Independent assignment (A/B test, nudge).** The coin does not know anything about the visitor. It is independent of both potential outcomes.
4. **Exclusion and no defiers (nudge).** The nudge changes the outcome only by changing who takes the treatment, and it never pushes anyone away from it.
5. **Continuity (cutoff).** Without the treatment, the average outcome would pass through the cutoff without a jump.
6. **Parallel trends (before and after).** Without the treatment, the treated group would have moved by the same amount as the untreated group.
Each design swaps the assumption it needs for one it can defend, and the data cannot prove any of them. Assumption 3 is the only one that a person can make true by acting, which is why the first section is the most valuable.
## A coin flip: the missing half becomes ignorable
**Teaser.** The shop's analyst compares visitors who used the new checkout with visitors who used the old one and finds a gap of 5 in spend per visitor. The designer says it is the page. The analyst suspects that the visitors who found the new page were the keen ones. Neither can check by looking at the gap.
Give each visitor $i$ two potential outcomes: $Y_i(1)$, the spend with the new page, and $Y_i(0)$, the spend with the old one. Only one is seen. The effect we want is the average over visitors,
$$
\htmlClass{eq-der}{\tau}=\mathbb E\big[\htmlClass{eq-hid}{Y(1)}-\htmlClass{eq-hid}{Y(0)}\big].
$$
Let a coin $Z_i\in\{0,1\}$ decide the page. The observed spend is $Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0)$, and the estimate is the difference of the two arms' average spend,
$$
\hat\tau=\bar Y_{Z=1}-\bar Y_{Z=0}.
$$
The figure shows what that estimate does over many repeated tests. With a coin, it centres on the true effect, and its spread shrinks as visitors accumulate. If visitors pick their own page, it centres on the true effect plus the difference between the people who pick and the people who do not, and the spread shrinking only makes the wrong centre more certain. At 800 visitors per arm the honest interval holds the truth 95% of the time, and the self-chosen one holds it 15% of the time; at 1,600 per arm it holds it 1% of the time.
Derivation: why the difference of means is the effect, how noisy it is, and what a biased comparison does
**Step 1: write each arm's average with potential outcomes.** In the arm that got the new page the observed spend is $Y(1)$, and in the other arm it is $Y(0)$. So the expected value of the estimate is
$$
\begin{aligned}
\mathbb E[\hat\tau]&=\mathbb E[Y\mid Z=1]-\mathbb E[Y\mid Z=0] \\
&=\mathbb E[Y(1)\mid Z=1]-\mathbb E[Y(0)\mid Z=0].
\end{aligned}
$$
**Step 2: use the coin.** The coin is independent of the pair $(Y(0),Y(1))$. Conditioning on an independent variable changes nothing, so
$$
\begin{aligned}
\mathbb E[Y(1)\mid Z=1]&=\mathbb E[Y(1)], \\
\mathbb E[Y(0)\mid Z=0]&=\mathbb E[Y(0)].
\end{aligned}
$$
**Step 3: read off the effect.** Substituting back,
$$
\mathbb E[\hat\tau]=\mathbb E[Y(1)]-\mathbb E[Y(0)]=\htmlClass{eq-der}{\tau}.
$$
In words, the group that got the old page is a fair picture of what the group that got the new page would have done without it. The unobservable $Y(0)$ of the treated has been replaced by the observable $Y(0)$ of the untreated, and the swap is free because of the coin.
**Step 4: see what a self-chosen comparison adds.** If the visitors choose, write $T$ for the choice and repeat Step 1 with $T$ in place of $Z$. Adding and subtracting what the choosers would have spent without the page gives
$$
\begin{aligned}
\mathbb E[Y\mid T=1]-\mathbb E[Y\mid T=0]&=\tau_{\text{chosen}} \\
&\quad+\underbrace{\mathbb E[Y(0)\mid T=1]-\mathbb E[Y(0)\mid T=0]}_{b}.
\end{aligned}
$$
The term $b$ compares two kinds of people on the same outcome, the spend without the new page. It is zero when the choice is unrelated to that spend, which the coin guarantees and self-selection does not. In the figure $b=3$, so the self-chosen estimate centres on $2+3=5$.
**Step 5: how noisy the estimate is.** Each arm has $n$ visitors whose spend has variance $\sigma^2$, so each arm's average has variance $\sigma^2/n$. The two arms contain different people, so their averages are independent and the variances add:
$$
\operatorname{Var}(\hat\tau)=\frac{\sigma^2}{n}+\frac{\sigma^2}{n}=\frac{2\sigma^2}{n},
\qquad
s=\sigma\sqrt{\frac{2}{n}}.
$$
With $\sigma=20$ and $n=800$, $s=20\sqrt{2/800}=1.0$.
**Step 6: the interval.** By the central limit theorem the estimate is close to normal with mean $\tau$ and standard deviation $s$, so the interval $\hat\tau\pm1.96\,s$ holds $\tau$ with probability 95%.
**Step 7: the interval around a biased centre.** If the estimate is centred on $\tau+b$ instead, the interval holds $\tau$ only when the estimate lands within $1.96\,s$ of $\tau$, which happens with probability
$$
\begin{aligned}
\Pr\big(\text{interval holds }\tau\big)&=\Phi\!\Big(1.96-\frac{b}{s}\Big) \\
&\quad-\Phi\!\Big(-1.96-\frac{b}{s}\Big),
\end{aligned}
$$
where $\Phi$ is the standard normal cumulative distribution. The ratio $b/s=b\sqrt{n}/(\sigma\sqrt2)$ grows with $\sqrt n$, so the probability falls toward zero as the test grows. With $b=3$ it is 15% at $n=800$ and 1% at $n=1{,}600$. More data cannot repair a biased comparison; only a different comparison can.
**Step 8: how many visitors to detect a given effect.** Call a result real when $\hat\tau$ exceeds $1.96\,s$, and ask for an 80% chance of that when the true effect is $\tau$. The estimate is centred on $\tau$ with spread $s$, and it exceeds $1.96\,s$ with probability 80% when $\tau=(1.96+0.84)\,s$. Solving for $n$,
$$
n=\frac{2\,(1.96+0.84)^2\,\sigma^2}{\tau^2}.
$$
The factor $1/\tau^2$ is why small lifts are expensive: halving the lift to detect quadruples the visitors.
Worked example: how long must the checkout test run?
The new page is expected to add 2 to spend per visitor, and spend has a standard deviation of 20. Then $n=2(2.8)^2\cdot 20^2/2^2\approx 1{,}570$ visitors per arm. With 500 visitors per arm per day that is about three days, and the test should run to a whole number of weeks so that each weekday appears equally often in both arms. At the 800 per arm of the figure the chance of catching a lift of 2 is only about 52%, which is why a "flat" result from a small test says little.
## Subtract a number you measured before: variance reduction
**Teaser.** The test needs 1,570 visitors per arm because spend varies a lot between visitors. But much of that variation was there before the test started: big spenders last month are big spenders this month. If last month's spend were subtracted out, what would remain to explain?
Let $X_i$ be visitor $i$'s spend in the weeks *before* the test. Because it was measured before the coin was flipped, the coin cannot have changed it, and the two arms have the same average $X$ in expectation. The adjusted estimate subtracts the arms' difference in $X$, scaled by a number $\theta$:
$$
\begin{aligned}
\hat\tau_\theta&=\big(\bar Y_{Z=1}-\bar Y_{Z=0}\big) \\
&\quad-\theta\big(\bar X_{Z=1}-\bar X_{Z=0}\big).
\end{aligned}
$$
Subtracting the arms' difference in $X$ cannot bias the estimate, because that difference is zero on average. It can remove noise, because a lucky draw of big spenders into one arm shows up in $X$ too. The best $\theta$ makes the adjusted variance $(1-\rho^2)$ times the raw variance, where $\rho$ is the correlation between $X$ and $Y$.
Subtracting the past narrows the curve without moving it. Both curves centre on the true effect. The adjusted one is narrower by the factor $\sqrt{1-\rho^2}$, so the same chance of catching the effect needs $1-\rho^2$ times as many visitors. Drag the handle to change how well the pre-test metric predicts the outcome.
At a correlation of 0.6 the adjusted variance is 64% of the raw one. The test needs about 1,005 visitors per arm instead of 1,570, or the same 1,570 give a 94% chance of catching the lift instead of 80%.
Derivation: why the adjustment is unbiased for every $\theta$, and which $\theta$ is best
**Step 1: the adjustment has mean zero.** Randomisation makes the coin independent of $X$, so each arm's average $X$ has the same expected value $\mu_X$:
$$
\mathbb E\big[\bar X_{Z=1}-\bar X_{Z=0}\big]=\mu_X-\mu_X=0.
$$
**Step 2: the estimate stays unbiased.** Taking expectations of $\hat\tau_\theta$ and using Step 1 and the result of the coin-flip section,
$$
\begin{aligned}
\mathbb E[\hat\tau_\theta]&=\mathbb E[\bar Y_{Z=1}-\bar Y_{Z=0}] \\
&\quad-\theta\,\mathbb E[\bar X_{Z=1}-\bar X_{Z=0}] \\
&=\tau-0.
\end{aligned}
$$
This holds for every $\theta$, so the choice of $\theta$ is free and can be spent on reducing noise.
**Step 3: the variance as a function of $\theta$.** Define the adjusted outcome of one visitor $Y_i-\theta X_i$. Its variance is
$$
\operatorname{Var}(Y-\theta X)=\sigma_Y^2-2\theta\,\operatorname{Cov}(X,Y)+\theta^2\sigma_X^2.
$$
The two arms are independent samples of $n$ visitors each, so, as before, the variance of the difference of their averages doubles and divides by $n$:
$$
\operatorname{Var}(\hat\tau_\theta)=\frac{2}{n}\Big(\sigma_Y^2-2\theta\,\operatorname{Cov}(X,Y)+\theta^2\sigma_X^2\Big).
$$
**Step 4: minimise over $\theta$.** The expression is a parabola in $\theta$. Setting its derivative to zero,
$$
-2\operatorname{Cov}(X,Y)+2\theta\sigma_X^2=0
\quad\Longrightarrow\quad
\theta^\ast=\frac{\operatorname{Cov}(X,Y)}{\sigma_X^2}.
$$
This is the slope of the line that predicts $Y$ from $X$.
**Step 5: the value at the minimum.** Substituting $\theta^\ast$ back and writing $\operatorname{Cov}(X,Y)=\rho\,\sigma_X\sigma_Y$,
$$
\begin{aligned}
\operatorname{Var}(\hat\tau_{\theta^\ast})&=\frac{2}{n}\Big(\sigma_Y^2-\frac{\operatorname{Cov}(X,Y)^2}{\sigma_X^2}\Big) \\
&=\frac{2\sigma_Y^2}{n}\,(1-\rho^2).
\end{aligned}
$$
**Step 6: the cost in visitors.** The number of visitors needed for a given power is proportional to the variance, so it is multiplied by $1-\rho^2$. At $\rho=0.6$ that is $0.64\times1{,}570\approx1{,}005$.
**Step 7: the same thing as a regression.** The line in Step 4 is what a regression of $Y$ on the arm and on $X$ estimates; its coefficient on the arm equals the adjusted difference, with $\theta$ estimated by that regression. So the "trick" is the ordinary regression adjustment, and its justification here comes entirely from randomisation, not from the regression being a correct model of spend.
**Step 8: the one condition.** $X$ must be fixed before the coin is flipped. If $X$ were measured after the page was shown, the page could change it, Step 1 would fail, and the term subtracted would carry away part of the real effect. That is the mediator in the causal-graph picture: a variable on the path from treatment to outcome must not be adjusted for.
## When you can only nudge: divide by how far you moved the treatment
**Teaser.** The shop builds a wishlist feature. It cannot force anyone to use it. It can send an email to a random half of the customers saying the feature exists. Some of them try it; some of the others find it on their own. Comparing users with non-users is the wrong comparison. What is the right one?
Three kinds of customer exist, and nobody can tell which one a given customer is. **Always-users** use the feature whether or not they get the email. **Never-users** do not use it either way. **Nudged users**, the compliers, use it only if the email arrives. The email is a coin flip, so its two arms contain the same mix of the three kinds. The users-versus-non-users comparison does not: the users include every always-user, who are keen customers, and the non-users include every never-user, who are not.
Compare the arms, then divide. Comparing users with non-users overstates the effect because keen customers use the feature anyway. Comparing the two email arms is honest but small, since it is diluted by everyone the email did not move. Dividing by the share the email did move gives the effect for those who were moved. Drag the handle to change that share.
At a nudged share of 30% the users-versus-non-users gap is 17.1 and the true effect is 6. The email arms differ by only 1.8, which is the effect of 6 diluted to 30% of the customers. Dividing by 0.3 returns 6. The price is precision: the range around the divided estimate is $1/0.3$ times as wide as the range around the arm gap, so seeing the effect as clearly as with full take-up needs 11 times as many visitors.
Derivation: the ratio of two differences is the effect on the nudged customers
**Step 1: potential treatment and potential outcome.** Let $Z$ be the email (a coin), $D(z)$ whether the customer would use the feature if the email status were $z$, and $Y(d)$ their spend if they use it ($d=1$) or not ($d=0$). We observe $D=D(Z)$ and $Y=Y(D)$. The three kinds of customer are
$$
\begin{aligned}
\text{always-user: }&D(0)=1,\ D(1)=1, \\
\text{nudged user: }&D(0)=0,\ D(1)=1, \\
\text{never-user: }&D(0)=0,\ D(1)=0.
\end{aligned}
$$
Assumption 4 (no one is pushed *away*, so $D(1)\ge D(0)$ for everybody) rules out a fourth kind who uses the feature only without the email.
**Step 2: what the email does to one customer's spend.** With the email the spend is $Y(D(1))$; without it, $Y(D(0))$. For an always-user or a never-user the two are the same number, because $D(1)=D(0)$, so the difference is zero (this is the exclusion assumption: the email alone does nothing). For a nudged user, the difference is $Y(1)-Y(0)$. Both cases are covered by
$$
Y(D(1))-Y(D(0))=\big(D(1)-D(0)\big)\big(Y(1)-Y(0)\big).
$$
**Step 3: the difference between the arms.** The email is independent of everything about the customer, so the arm averages differ by the average of Step 2:
$$
\begin{aligned}
\mathbb E[Y\mid Z=1]-\mathbb E[Y\mid Z=0] \\
&=\mathbb E\Big[\big(D(1)-D(0)\big)\big(Y(1)-Y(0)\big)\Big] \\
&=\Pr(\text{nudged})\cdot\mathbb E\big[Y(1)-Y(0)\mid\text{nudged}\big].
\end{aligned}
$$
The product is non-zero only for nudged users, who are a share $c=\Pr(\text{nudged})$ of the customers. This difference is called the **intent-to-treat** effect: it is the effect of sending the email, and it is diluted by $c$.
**Step 4: the same difference for treatment itself.** Take the same two arms and ask how much more of them used the feature:
$$
\begin{aligned}
\mathbb E[D\mid Z=1]-\mathbb E[D\mid Z=0]&=\mathbb E\big[D(1)-D(0)\big] \\
&=\Pr(\text{nudged})=c.
\end{aligned}
$$
This is the **first stage**: how far the design moved the treatment.
**Step 5: divide.** The ratio of the two differences is
$$
\begin{aligned}
\frac{\mathbb E[Y\mid Z=1]-\mathbb E[Y\mid Z=0]}{\mathbb E[D\mid Z=1]-\mathbb E[D\mid Z=0]}
&=\frac{c\cdot\mathbb E[Y(1)-Y(0)\mid\text{nudged}]}{c} \\
&=\htmlClass{eq-der}{\mathbb E\big[Y(1)-Y(0)\mid\text{nudged}\big]}.
\end{aligned}
$$
The quantity is the effect on the customers the nudge actually moved. It is called the local average treatment effect, and the formula is the Wald ratio.
**Step 6: why users against non-users fails.** Among customers who used the feature are all the always-users (base spend 60 in the figure) and the nudged half of the nudged users (base 50). Among non-users are all the never-users (base 40) and the half of the nudged users who got no email (base 50). With $c=0.3$ and always-users 20% of the customers, the average spend, base plus the effect for those who use it (62 for always-users, 56 for nudged users), is $(0.2\cdot62+0.15\cdot56)/0.35=59.4$ among users and $(0.5\cdot40+0.15\cdot50)/0.65=42.3$ among non-users. The gap is 17.1, and most of it is a difference between kinds of customer rather than an effect of the feature.
**Step 7: the price of dividing.** The numerator has standard error $s=\sigma\sqrt{2/n}$, as in the coin-flip section; with $\sigma=20$ and $n=10{,}000$ per arm, $s=0.28$. To first order the denominator is a known number $c$, so the ratio's standard error is $s/c$: $0.94$ at $c=0.3$ and $2.8$ at $c=0.1$. Matching the precision of a test with full take-up needs $s/c$ to equal $s$, which takes $1/c^2$ times as many visitors: 11 times at $c=0.3$, 100 times at $c=0.1$. A nudge that moves few people is a weak instrument, and a ratio of two small numbers is unstable.
**Step 8: who the answer is for.** The effect on always-users (2 in the figure) and on never-users never appears, because the email does not change whether they use the feature, so their effect leaves no trace in the difference between the arms. The answer, 6, describes the nudged customers only, and it differs from the effect on everyone.
Worked example: did the onboarding email raise subscriptions?
A random half of new users receive an email explaining the premium trial. Some start the trial anyway; most of those who start it do so after the email. The comparison of subscribers with non-subscribers cannot be trusted, because the people who start a premium trial unprompted are the committed ones. The comparison of the two email arms can be: it says how much the email raised spend per user. Divide it by the difference in trial-start rate between the arms, and the answer is the effect of the trial on the users the email persuaded to start one. If that difference in trial-start rate is small, say 2 percentage points, the divided estimate has a range 50 times wider than the arm comparison, and the right decision may be that the test cannot answer the question.
## A cutoff is a nudge in disguise
**Teaser.** Free shipping goes to every customer with a loyalty score of 600 or more. A customer at 601 and a customer at 599 are nearly the same person, and only one of them gets free shipping. The shop never flipped a coin, but the rule did.
Write $X$ for the score and $D=\mathbf 1[X\ge 0]$ for treatment, with the score centred at the cutoff. Near the cutoff, who falls on which side is close to luck. The assumption is that without the treatment the average outcome would pass through the cutoff smoothly, so any jump at the cutoff is the effect. The data cannot reach the cutoff exactly, so the estimate uses a window of width $h$ on each side, and the window creates the central trade-off: a narrow window has few customers and a noisy estimate; a wide window has customers who are no longer alike, and a biased one.
A window trades bias for noise. Top: the true average outcome (copper), the window (shaded), the difference of window averages (dashed) and the lines fitted inside the window (blue, with the violet segment as the estimate). Bottom: the typical error of each estimator by window, with the best window marked. Drag the handle to change the window.
At a window of 0.4 the difference of window averages reads 8.8, because the outcome rises with the score and the right side of the window sits higher than the left. Fitting a line on each side and reading both lines at the cutoff gives 4.7. The true jump is 5. The line wins because it removes the slope, and it ends up with a wider best window and a smaller error: about 1.35 at a window of 0.55, against 1.64 at a window of 0.11 for the plain averages.
Derivation: why the jump is the effect, and how the window trades bias against noise
**Step 1: the jump is the effect.** Write $m_1(x)=\mathbb E[Y(1)\mid X=x]$ and $m_0(x)=\mathbb E[Y(0)\mid X=x]$. Only $m_1$ is seen to the right of the cutoff and only $m_0$ to the left, so the data show
$$
\begin{aligned}
\lim_{x\downarrow0}\mathbb E[Y\mid X=x]&=m_1(0^+), \\
\lim_{x\uparrow0}\mathbb E[Y\mid X=x]&=m_0(0^-).
\end{aligned}
$$
If both functions are continuous at 0 (assumption 5), then $m_1(0^+)=m_1(0)$ and $m_0(0^-)=m_0(0)$, and the difference of the limits is
$$
\htmlClass{eq-der}{m_1(0)-m_0(0)}=\mathbb E\big[Y(1)-Y(0)\mid X=0\big].
$$
The swap is the same one as in the coin flip: the untreated customers just to the left stand in for the treated customers just to the right.
**Step 2: a finite window.** With finitely many customers there is no limit, only averages over a window. In the figure the true average outcome is $m(x)=50+8x+K x^2$ plus a jump of 5 at the cutoff, with $K=12$ on the right and $K=0$ on the left. Inside a window of width $h$ the scores are uniformly spread.
**Step 3: the difference of window averages.** On the right the average of $m$ over $[0,h]$ is the average of each term:
$$
\begin{aligned}
\text{right}&=50+5+8\cdot\frac h2+12\cdot\frac{h^2}{3}.
\end{aligned}
$$
On the left, over $[-h,0]$, the linear term averages to $-h/2$ times 8 and the curvature term vanishes:
$$
\text{left}=50-8\cdot\frac h2.
$$
The difference, the estimate, is therefore
$$
\hat\tau_{\text{avg}}=5+\underbrace{8h+4h^2}_{\text{bias}}.
$$
The slope of the outcome in the score moves the bias linearly in $h$. At $h=0.4$ the bias is $3.2+0.64=3.84$, which is the 8.8.
**Step 4: fit a line on each side.** A line fitted over the window removes the slope. The best straight line to $x^2$ over $[0,h]$ has slope $\operatorname{Cov}(x,x^2)/\operatorname{Var}(x)$. With $\mathbb E[x]=h/2$, $\mathbb E[x^2]=h^2/3$ and $\mathbb E[x^3]=h^3/4$,
$$
\begin{aligned}
\operatorname{Cov}(x,x^2)&=\frac{h^3}{4}-\frac h2\cdot\frac{h^2}{3}=\frac{h^3}{12}, \\
\operatorname{Var}(x)&=\frac{h^2}{3}-\frac{h^2}{4}=\frac{h^2}{12}.
\end{aligned}
$$
So the slope is $h$, and the intercept at the cutoff is $\mathbb E[x^2]-h\,\mathbb E[x]=h^2/3-h^2/2=-h^2/6$. The straight part of $m$ is fitted exactly and contributes no bias. The bias from curvature is $-K h^2/6$ on each side, and the difference between sides is
$$
\hat\tau_{\text{line}}=5\underbrace{-\frac{12-0}{6}h^2}_{\text{bias}}=5-2h^2.
$$
At $h=0.4$ that is $5-0.32=4.68$. The bias is now quadratic in $h$ and does not depend on the slope.
**Step 5: noise of the plain averages.** If the scores are uniform with $N$ customers over a range of 2, the density is $N/2$ per unit of score and each side of the window holds $n_s=Nh/2$ customers. An average of $n_s$ customers whose outcome has standard deviation $\sigma$ has variance $\sigma^2/n_s$. Two sides add:
$$
\operatorname{Var}(\hat\tau_{\text{avg}})=\frac{2\sigma^2}{Nh/2}=\frac{4\sigma^2}{Nh}.
$$
**Step 6: noise of the lines.** The intercept of a fitted line has variance $\sigma^2\big(1/n_s+\bar x^2/S_{xx}\big)$, where $\bar x=h/2$ and $S_{xx}=n_sh^2/12$. Then
$$
\operatorname{Var}(\text{intercept})=\frac{\sigma^2}{n_s}\Big(1+\frac{h^2/4}{h^2/12}\Big)=\frac{4\sigma^2}{n_s},
$$
four times that of a plain average, because the line must also estimate a slope. Two sides give
$$
\operatorname{Var}(\hat\tau_{\text{line}})=\frac{8\sigma^2}{Nh/2}=\frac{16\sigma^2}{Nh}.
$$
**Step 7: the best window.** The typical error is $\sqrt{\text{bias}^2+\text{variance}}$. With $N=2000$ and $\sigma=10$,
$$
\begin{aligned}
\text{averages: }&\ (8h+4h^2)^2+\frac{0.2}{h}, \\
\text{lines: }&\ (2h^2)^2+\frac{0.8}{h}.
\end{aligned}
$$
The first is smallest near $h=0.11$ with an error of about 1.64. The second is smallest near $h=0.55$ with an error of about 1.35. The lines tolerate a wider window because their bias grows with $h^2$ rather than $h$, and the wider window pays for the extra noise.
**Step 8: when the cutoff only nudges.** If crossing the cutoff raises the chance of treatment without making it certain (free shipping is offered but must be claimed), the jump in the outcome is the effect of the nudge. Dividing it by the jump in the share treated gives the effect for the customers moved by the cutoff: the Wald ratio of the previous section, taken at the cutoff.
Worked example: does free shipping at a loyalty score of 600 raise orders?
Customers with 600 points or more get free shipping, and the analyst plots the average number of orders in the following quarter against the score. The check before any estimate is that nothing else changes at 600 (no other benefit starts there), and that customers could not tune their score to land just above it, which a smooth count of customers across the cutoff supports. Then the estimate compares fitted lines on both sides inside a window, and the window is varied to show the answer does not depend on one choice. The result is the effect of free shipping for customers near a score of 600. It says nothing about customers at 300 or at 900, whose reaction may be quite different.
## Difference-in-differences is the same subtraction, twice
**Teaser.** A new delivery option starts in three cities and not in others. Orders in the three cities rise. Orders everywhere were rising anyway.
The treated cities were different before the change, and everything changed over time. Two subtractions remove the two nuisances. Writing each city's outcome as a level that belongs to the city plus a change that belongs to the date plus the effect,
$$
Y_{it}=\alpha_i+\lambda_t+\tau\,D_{it}+\varepsilon_{it}.
$$
Subtracting a city's own earlier value removes $\alpha_i$, what the city shares with itself. Subtracting the same change in the untreated cities removes $\lambda_t$, what all cities share on a date. What remains is $\tau$.
Derivation: two subtractions, and the assumption they need
**Step 1: the first difference removes the city.** For a treated city, the change from before to after is
$$
\begin{aligned}
\mathbb E[Y_{\text{after}}-Y_{\text{before}}\mid\text{treated}]&=\lambda_{\text{after}}-\lambda_{\text{before}}+\tau.
\end{aligned}
$$
The city's own level $\alpha_i$ appears in both periods and cancels. What is left is the change that belongs to the date, plus the effect.
**Step 2: the second difference removes the date.** For an untreated city the same change is
$$
\mathbb E[Y_{\text{after}}-Y_{\text{before}}\mid\text{untreated}]=\lambda_{\text{after}}-\lambda_{\text{before}}.
$$
Subtracting it from the treated change,
$$
\begin{aligned}
\text{DiD}&=(\lambda_{\text{after}}-\lambda_{\text{before}}+\tau)-(\lambda_{\text{after}}-\lambda_{\text{before}}) \\
&=\htmlClass{eq-der}{\tau}.
\end{aligned}
$$
**Step 3: what was assumed.** The model gave the treated and untreated cities the *same* $\lambda_t$. That is the parallel-trends assumption, written without a model: without the treatment the treated cities' average would have changed by the same amount as the untreated cities'. The treated cities' own change is a number we saw, and the untreated cities' change is the number we use to imagine what they would have done.
**Step 4: the same pattern.** The step that removed $\alpha_i$ and the step that removed $\lambda_t$ are the "subtract what is shared" move. In the coin flip it removed the visitor's baseline by making the arms alike; in the adjustment it removed last month's spend; here it removes the city and the date. The denominator is 1 because the treatment turns on fully for the treated group.
The assumption can be checked in part by plotting the gap between the groups over the weeks before the change, which should be flat. Synthetic control is a close relative that builds the imagined comparison from a weighted blend of untreated units chosen to track the treated unit's history; I describe it from memory and have not checked a reference.
## One trick, many designs
Every design here estimates a ratio of two differences, and what changes is the lever that produces the difference.
| design | what you control | subtract | divide by | what you assume |
| --- | --- | --- | --- | --- |
| A/B test | who gets the treatment | the control arm's average | 1 | the coin was fair and everyone followed it |
| adjustment with a pre-test metric | nothing extra | the arms' difference in the pre-test metric | 1 | the metric was measured before the coin |
| nudge | who gets the encouragement | the unnudged arm's average | the difference in take-up | the nudge acts only through take-up, and pushes nobody away |
| cutoff | nothing; a rule exists | the average just to the left | the jump in take-up, if it is not 1 | nothing else jumps at the cutoff |
| before and after, with a comparison group | the timing, or nothing | the group's own past, then the comparison group's change | 1 | parallel trends |
Derivation: the same identity under several names
**Step 1: the general statement.** Let $Z$ be any variable that moves the treatment and that is, at the point where we compare, as good as random with respect to the potential outcomes. Let $\Delta$ denote "the value where $Z$ is on minus the value where it is off". Under the assumptions of the nudge section,
$$
\htmlClass{eq-der}{\frac{\Delta\,\mathbb E[Y]}{\Delta\,\mathbb E[D]}}=\mathbb E\big[Y(1)-Y(0)\mid\text{moved by }Z\big].
$$
**Step 2: the coin flip.** Let $Z$ be the treatment itself. Then $D=Z$, $\Delta\,\mathbb E[D]=1$, and the ratio is the difference of arm averages.
**Step 3: the nudge.** $Z$ is the email and $D$ is take-up. This is Step 5 of the nudge section.
**Step 4: the cutoff.** $Z=\mathbf 1[X\ge0]$ and $\Delta$ is the difference of the limits from the right and the left of the cutoff. If the rule fixes treatment exactly at the cutoff, $\Delta\,\mathbb E[D]=1$; if it only nudges, the ratio is the fuzzy version.
**Step 5: before and after.** Let $Z$ be the pair (treated group, after the start date) and $\Delta$ the double difference. The denominator is 1 when the treatment switches on fully, and independence is replaced by parallel trends, because the groups were not formed by a coin.
**Step 6: the pre-test adjustment.** It adds to the numerator a quantity that is zero on average, $\theta\,\Delta\,\mathbb E[X]$, and so does not change the ratio's centre; it changes only the noise. This is the one design that is a refinement rather than a source of identification.
**Step 7: why one idea.** In each case the numerator is something we see in two groups that differ only in the lever, and the denominator tells us how much treatment that lever moved. The assumption in the last column is what lets us say that the two groups differ *only* in the lever. The design chooses the assumption, and the arithmetic stays the same.
## A playbook
**Every causal estimate is a number we saw minus a number we imagined, divided by how much treatment moved.** A coin flip imagines the missing outcome from the other arm. A nudge imagines it for the customers who were moved. A cutoff imagines it from the customers just across the line. Before and after imagines it from the group's own past plus everyone else's change.
1. **Name the lever you have.** Can you decide who gets the treatment? Can you only encourage it? Does a rule with a threshold decide it? Did the change simply start on a date? The lever picks the design.
2. **State the contrast and the population.** Effect on whom (everyone, the customers the nudge moved, the customers near the cutoff), against what, and over what period.
3. **Write the assumption as a sentence a sceptic could attack.** "The coin was fair." "The email changes spend only through the trial." "Nothing else starts at 600 points." "The treated cities would have moved with the others."
4. **Run the check the assumption implies.**
- Coin: the arms have the sizes the coin promised, and the same pre-test averages.
- Nudge: the nudge moved a meaningful share of customers, which is the first stage.
- Cutoff: customers do not pile up on one side, and other variables do not jump at the cutoff.
- Before and after: the gap between the groups was flat before the change.
5. **Reduce noise with what was measured before.** A pre-test metric costs nothing and shortens the test.
6. **Report the effect, the people it is for, and the range.** A range that is wide because the first stage is weak is information, not a failure of the report.
| lever | design | estimates the effect for | typical failure |
| --- | --- | --- | --- |
| you assign | A/B test | everyone in the test | people leave the test, or the arms influence each other |
| you encourage | nudge and ratio | customers the nudge moved | the nudge moves almost nobody |
| a rule with a threshold | cutoff | customers near the threshold | customers manipulate their score, or a second benefit starts there |
| a start date | before and after | the treated group | a shock hits only the treated group |
## What the story cannot tell you
The coin flip protects against everything that happened before the flip, and nothing that happens after it. Visitors who see a slow new page may leave before they are counted, and then the arms no longer have the same kinds of people. Checking that the two arms have the sizes the coin promised catches many of these failures before any effect is computed.
The 5% promise in the A/B section holds for one look at a size fixed in advance. Looking every day and stopping when the gap first crosses the line breaks it, and the appendix on A/B tests in [the distributions essay](dist-derived.html) shows how quickly. A test that tracks a long-lived behaviour can also measure a novelty effect: the first weeks of a new page are not the steady state.
Everything above assumed that one visitor's treatment does not change another's outcome. In a marketplace, a discount shown to half of the buyers can take stock from the other half, and then the control arm is no longer a picture of the world without the discount. Designs that randomise whole markets or regions, or that run the change at different times, exist for this case and are beyond this essay.
The effects here are local. The nudge gives the effect for the customers who were moved. The cutoff gives it for customers at the cutoff. The answer to "what would happen if we rolled it out to everyone" needs a further assumption that the local effect carries over, and it should be stated as one.
## Further reading
- Imbens and Angrist — *Identification and estimation of local average treatment effects*, Econometrica 62(2), 467–475, 1994. The paper that introduced the local average treatment effect, the effect for those whose treatment the instrument can move. I confirmed the citation and the main result from search listings of the paper; I did not reread its proofs. The NBER listing gives the authors in the opposite order.
- Deng, Xu, Kohavi and Walker — *Improving the sensitivity of online controlled experiments by utilizing pre-experiment data*, WSDM 2013, pp. 123–132. The pre-test adjustment, called CUPED. I confirmed the citation and that it reduces metric variance using pre-experiment data; the derivation above is my own write-up.
- Imbens and Lemieux — *Regression discontinuity designs: a guide to practice*, Journal of Econometrics 142(2), 615–635, 2008. A practical guide to the design, which traces it to Thistlethwaite and Campbell, 1960. I confirmed the citation and the abstract; I did not read the sections on choosing a window.
- Imbens and Rubin — *Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction*, Cambridge University Press, 2015. A textbook on the potential-outcomes framework, with parts on randomised experiments and on assignment with noncompliance. I confirmed the title, publisher and part structure from catalogue listings; I have not read the chapters.
- Kohavi, Tang and Xu — *Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing*, Cambridge University Press, 2020. A practitioner's book on running experiments at scale. I confirmed the citation and scope from the publisher's description; I have not read the chapters.