# Deriving Matching and Difference-in-Differences from Potential Outcomes An online shop launches a loyalty app. A month later the analyst reports that customers who installed it spend about 12 dollars more per month than customers who did not, and asks whether the app is worth the engineering time. A second question arrives a quarter later: free shipping was switched on in three cities and orders went up. Was it the shipping? Both questions are about a cause, and both are easy to answer wrongly with data the shop already has. The way to answer them is to say precisely what is missing from the data, and then to borrow it from somewhere you can defend.
What you see is not what the app did. App users spend more than others, but part of the gap is that the keen customers are the ones who install the app. The copper dashes show the app users as they would have been without it. Drag the handle to change how strongly keen customers adopt.
The route has five steps. Every customer has two outcomes, one with the treatment and one without, and only one is ever seen. A causal graph says which variables must be held fixed to recover the missing one. **Matching** fills it in with customers who look the same on those variables; **difference-in-differences** fills it in with the treated group's own past, corrected by what happened to everyone else. A short playbook says which to use and what to check. One colour code runs through the text and the figures: blue is what we observe, copper is a hidden quantity such as the outcome a customer did not get, and violet is the effect we derive. ## Assumptions 1. **No spillovers, one version.** A customer's outcome depends only on that customer's own treatment, and "the app" means one thing. If one customer's installing the app changes another's spending, or free shipping in one city pulls orders out of the next, the comparisons below answer a different question. 2. **Consistency.** The outcome we observe is the potential outcome of the treatment actually received. 3. **Conditional independence (matching).** Among customers with the same measured characteristics $X$, who got the treatment is unrelated to what their outcomes would have been. This is the assumption that nothing unmeasured drove both the choice and the outcome. 4. **Overlap (matching).** At every value of $X$ that occurs among the treated, some untreated customers exist. 5. **Parallel trends (difference-in-differences).** Without the treatment, the treated group's average outcome would have moved by the same amount as the untreated group's. 6. **No anticipation.** The treated group does not change its behaviour before the treatment starts. Assumptions 3 and 5 are the ones that carry the weight. Matching leans on 3 and difference-in-differences on 5, and each can be checked a little but never proven. ## Two worlds for every customer: potential outcomes **Teaser.** Maya installed the app and spent 61 dollars last month. What would she have spent without it? Nobody can look. Give each customer $i$ a treatment indicator $T_i$, equal to 1 if they installed the app and 0 if not. Give them two **potential outcomes**: $Y_i(1)$, the spend if they install it, and $Y_i(0)$, the spend if they do not. The effect of the app on customer $i$ is the difference, $$ \htmlClass{eq-der}{\tau_i}=\htmlClass{eq-hid}{Y_i(1)}-\htmlClass{eq-hid}{Y_i(0)}. $$ We observe only one of the two: $$ \htmlClass{eq-obs}{Y_i}=T_i\,Y_i(1)+(1-T_i)\,Y_i(0). $$ That is the whole difficulty, the **fundamental problem of causal inference**: for every customer, one of the two numbers in $\tau_i$ is missing, and no amount of data about other customers recovers it without an assumption. A table of four customers shows the shape. | customer | installed | $Y(1)$ | $Y(0)$ | effect $\tau_i$ | | --- | --- | --- | --- | --- | | Maya | yes | 61 | ? | ? | | Ben | no | ? | 44 | ? | | Chen | yes | 57 | ? | ? | | Dara | no | ? | 52 | ? | Because individual effects are out of reach, we ask about averages. Two averages matter here: $$ \begin{aligned} \text{ATE}&=\mathbb E[Y(1)-Y(0)], \\ \htmlClass{eq-der}{\text{ATT}}&=\mathbb E\big[Y(1)-Y(0)\mid T=1\big]. \end{aligned} $$ The **ATE** is the effect averaged over every customer. The **ATT**, the average effect on the treated, is the effect averaged over those who actually installed the app. It is the right target when the decision is about the people who chose the treatment: did the app help the customers who adopted it? It is also the quantity that matching and difference-in-differences are built to recover. If the effect is the same for everyone the two coincide; if the keen customers gain more, the ATT is larger than the ATE. The obvious estimate is the difference of observed means, $\mathbb E[Y\mid T=1]-\mathbb E[Y\mid T=0]$. The figure at the top shows why it is wrong. Break it into the effect and what is left over.
Derivation: the observed gap is the ATT plus a selection bias **Step 1: rewrite the observed means with potential outcomes.** Among the treated, the observed outcome is $Y(1)$. Among the untreated, it is $Y(0)$. So $$ \begin{aligned} \mathbb E[Y\mid T=1]-\mathbb E[Y\mid T=0] \\ &=\mathbb E[Y(1)\mid T=1] \\ &\quad-\mathbb E[Y(0)\mid T=0]. \end{aligned} $$ **Step 2: add and subtract the missing piece.** The quantity $\mathbb E[Y(0)\mid T=1]$ is what the treated would have spent without the app. We never observe it, but adding and subtracting it changes nothing: $$ \begin{aligned} \mathbb E[Y(1)\mid T=1]-\mathbb E[Y(0)\mid T=0] \\ &=\mathbb E[Y(1)\mid T=1] \\ &\quad-\mathbb E[Y(0)\mid T=1] \\ &\quad+\mathbb E[Y(0)\mid T=1] \\ &\quad-\mathbb E[Y(0)\mid T=0]. \end{aligned} $$ **Step 3: name the two halves.** The first difference is the ATT. The second compares two groups of people on the same outcome, the spend without the app: $$ \begin{aligned} \text{what you see}&=\mathbb E[Y\mid T=1]-\mathbb E[Y\mid T=0] \\ &=\underbrace{\mathbb E[Y(1)\mid T=1]-\mathbb E[Y(0)\mid T=1]}_{\text{ATT: what the app did}} \\ &\quad+\underbrace{\mathbb E[Y(0)\mid T=1]-\mathbb E[Y(0)\mid T=0]}_{\text{selection bias}}. \end{aligned} $$ **Step 4: read the bias.** Selection bias is the difference in spend that the two groups would have had *without any app*. It is zero when who installs the app is unrelated to how much they would spend anyway, which is what a coin flip guarantees and what self-selection does not. When keen customers install the app and keen customers spend more regardless, the bias is positive and the observed gap overstates the effect. In the figure the bias is the copper segment, and it grows from zero as the handle moves right. **Step 5: check the figure's numbers.** In the figure the app adds 5 dollars for everyone, so the ATT is 5 at every setting. At the starting setting the observed gap is about 12.3 dollars and the bias is about 7.3, and $5+7.3=12.3$.
So there are two ways forward. Randomise who gets the treatment, which makes the bias zero by construction. Or, when that was not possible, say what made the groups differ and correct for it. The graph is how we say it. ## Drawing the assumptions: causal graphs **Teaser.** The shop's data has a customer's tenure, whether they installed the app, whether they opened a support ticket afterwards, and their spend. Which of these should an analysis hold fixed? A **causal graph** draws each variable as a node and each direct cause as an arrow. It does not need numbers; it records which variables you believe cause which. Three small shapes are enough, and each tells you something different about what to hold fixed. Call the treatment $T$, the outcome $Y$ and the third variable $Z$. - **A common cause (fork): $T \leftarrow Z \rightarrow Y$.** Tenure makes customers both more likely to install the app and more likely to spend. It opens a path between $T$ and $Y$ that is not the effect of $T$. Comparing customers *with the same $Z$* closes it. Such a $Z$ is a **confounder**. - **A middle step (chain): $T \rightarrow Z \rightarrow Y$.** If the app raises the number of visits and visits raise spend, then visits carry part of the effect. Holding visits fixed removes the very thing we want to measure. - **A common effect (collider): $T \rightarrow Z \leftarrow Y$.** If both installing the app and spending a lot make a customer likely to open a support ticket, then among customers who opened one, the two will look related even when they are not. Conditioning on a common effect *creates* a relationship. The rule that follows is the **back-door criterion**. To estimate the total effect of $T$ on $Y$, hold fixed a set of variables that blocks every path from $T$ to $Y$ that starts with an arrow *into* $T$, and contains no effect of $T$. Confounders go in; middle steps and common effects stay out.
Adjusting helps for one role of $Z$ and hurts for two. Each arrow carries its coefficient, and the true effect of $T$ on $Y$ is 1. The bars are the effect estimated without and with $Z$ held fixed. Drag the handle to change what $Z$ is.
Derivation: why the confounder inflates the raw slope and the collider bends the adjusted one Let every variable have mean zero and let the noise in each be independent with variance 1. For the first graph, $Z\to T$ with coefficient 0.8, $Z\to Y$ with coefficient 1 and $T\to Y$ with coefficient 1: $$ T=0.8\,Z+e_T,\qquad Y=1\cdot T+1\cdot Z+e_Y. $$ **Step 1: the variance of $T$.** With $\operatorname{Var}(Z)=1$, $$ \operatorname{Var}(T)=0.8^2\cdot1+1=1.64. $$ **Step 2: the covariance of $T$ and $Y$.** Substitute $Y$ and expand: $$ \begin{aligned} \operatorname{Cov}(T,Y) &=\operatorname{Cov}(T,\,T+Z+e_Y) \\ &=\operatorname{Var}(T)+\operatorname{Cov}(T,Z). \end{aligned} $$ Here $\operatorname{Cov}(T,Z)=0.8\operatorname{Var}(Z)=0.8$, so $\operatorname{Cov}(T,Y)=1.64+0.8=2.44$. **Step 3: the raw slope.** Regressing $Y$ on $T$ alone gives $$ \begin{aligned} \frac{\operatorname{Cov}(T,Y)}{\operatorname{Var}(T)}=\frac{2.44}{1.64}=1.488. \end{aligned} $$ It is the true 1 plus $\operatorname{Cov}(T,Z)\cdot 1/\operatorname{Var}(T)=0.8/1.64=0.488$. The extra is the confounder's effect on $Y$ times how strongly $Z$ moves $T$. **Step 4: hold $Z$ fixed.** Once $Z$ is in the regression, $T$ is compared across customers with the same $Z$, where only the noise $e_T$ moves $T$. The coefficient on $T$ is then the true 1. **Step 5: the collider.** Now take $T\to Y$ with coefficient 1, and let $Z=0.8\,T+0.8\,Y+e_Z$ be an effect of both. Then $\operatorname{Var}(T)=1$, $\operatorname{Var}(Y)=2$, $\operatorname{Cov}(T,Y)=1$, and $$ \begin{aligned} \operatorname{Var}(Z)&=0.8^2\cdot1+0.8^2\cdot2+2\cdot0.8\cdot0.8\cdot1+1=4.2, \\ \operatorname{Cov}(T,Z)&=0.8+0.8=1.6, \\ \operatorname{Cov}(Y,Z)&=0.8+1.6=2.4. \end{aligned} $$ **Step 6: the coefficient on $T$ with $Z$ in the regression.** For two regressors the coefficient on $T$ is $$ \begin{aligned} &\frac{\operatorname{Cov}(T,Y)\operatorname{Var}(Z)-\operatorname{Cov}(T,Z)\operatorname{Cov}(Y,Z)}{\operatorname{Var}(T)\operatorname{Var}(Z)-\operatorname{Cov}(T,Z)^2} \\ &=\frac{1\cdot4.2-1.6\cdot2.4}{1\cdot4.2-1.6^2} \\ &=\frac{0.36}{1.64}=0.22. \end{aligned} $$ The raw slope was exactly right; holding the common effect fixed pulls it from 1 down to 0.22.
A graph is a claim, not a result. It earns its place by making the claim visible enough to argue with: if a colleague thinks tenure also depends on the app, they can draw the extra arrow, and the disagreement becomes specific. ## Closing the back door with what you can see **Teaser.** If tenure is the only thing that makes customers choose the app, then among customers of the same tenure the choice is as good as random. How do we turn that into a number? Suppose the covariates $X$ capture everything that made customers choose the treatment and that also moves their outcome. Under conditional independence (assumption 3), for customers with the same $X$ the untreated are a fair stand-in for what the treated would have done without the app.
Derivation: the ATT written in terms of things we observe **Step 1: the quantity we cannot see.** The ATT needs $\mathbb E[Y(0)\mid T=1]$. Split it by the value of $X$ among the treated: $$ \begin{aligned} \mathbb E[Y(0)\mid T=1] \\ &=\mathbb E\Big[\,\mathbb E[Y(0)\mid T=1,X] \\ &\qquad\Big|\;T=1\Big]. \end{aligned} $$ **Step 2: use conditional independence.** Given $X$, treatment is unrelated to the potential outcomes, so conditioning on $T=1$ changes nothing: $$ \begin{aligned} \mathbb E[Y(0)\mid T=1,X] \\ &=\mathbb E[Y(0)\mid T=0,X]. \end{aligned} $$ **Step 3: use consistency.** Among the untreated, $Y(0)$ is the outcome we observe, so $$ \begin{aligned} \mathbb E[Y(0)\mid T=0,X] \\ &=\mathbb E[Y\mid T=0,X]. \end{aligned} $$ This is a conditional mean of observed data, computed from untreated customers alone. Overlap is what guarantees there are untreated customers at every $X$ where we need it. **Step 4: put it together.** The outer expectation is still over the *treated* customers' $X$, since the ATT is about them: $$ \begin{aligned} \htmlClass{eq-der}{\text{ATT}} \\ &=\mathbb E[Y\mid T=1] \\ &\quad-\mathbb E_{X\mid T=1}\Big[\,\mathbb E[Y\mid T=0,X]\,\Big]. \end{aligned} $$ The first term is observed. The second reads: for each treated customer, find what customers like them *who did not get the app* spend, then average over the treated. **Step 5: where the estimators differ.** The formula says what to compute; estimators differ in how they estimate $\mathbb E[Y\mid T=0,X]$. A regression of $Y$ on $X$ among the untreated assumes a shape for it. **Weighting** reweights the untreated to look like the treated. **Matching** takes the outcome of untreated customers who are close in $X$, with no shape assumed.
## Matching: borrow the customers who look the same **Teaser.** A customer who installed the app has been with the shop for two years and visits weekly. Is there a customer who did not install the app but looks just like them? Take each treated customer, find the untreated customer closest to them in $X$, and use that customer's spend as the missing $Y_i(0)$. The matching estimator of the ATT averages the differences: $$ \htmlClass{eq-der}{\widehat{\text{ATT}}}=\frac1{n_1}\sum_{i:\,T_i=1}\Big(\htmlClass{eq-obs}{Y_i}-\htmlClass{eq-obs}{Y_{j(i)}}\Big), $$ where $n_1$ is the number of treated customers and $j(i)$ is the untreated customer matched to $i$. The same control can serve several treated customers. Two practical limits come with it. If the nearest control is far away, the pair is not really a match, so a **caliper** sets the largest distance allowed and drops treated customers with no control that close. And with many covariates, "close" is hard to define in every direction at once. That is the problem the propensity score solves. The **propensity score** is the chance of being treated given the covariates, $e(x)=\Pr(T=1\mid X=x)$. The result of Rosenbaum and Rubin is that it is enough to match on this single number.
Derivation: the propensity score balances all the covariates at once **Step 1: what has to be shown.** Matching works when, among customers with the same covariates, treatment is unrelated to the potential outcomes. The claim is that the same holds among customers with the same *score*: if $(Y(0),Y(1))\perp T\mid X$, then $(Y(0),Y(1))\perp T\mid e(X)$. **Step 2: reduce to one probability.** Because $T$ is 0 or 1, independence of $T$ from the potential outcomes given $e(X)$ means that $\Pr(T=1\mid Y(0),Y(1),e(X))$ does not depend on the potential outcomes. We will show it equals $e(X)$. **Step 3: bring $X$ back in.** By the law of total probability, averaging over the covariates that share the same score, $$ \begin{aligned} \Pr\big(T=1\mid Y(0),Y(1),e(X)\big) \\ &=\mathbb E\Big[\Pr\big(T=1\mid Y(0),Y(1),X\big) \\ &\qquad\Big|\;Y(0),Y(1),e(X)\Big]. \end{aligned} $$ **Step 4: use conditional independence.** Given $X$, treatment does not depend on the potential outcomes, so the inner probability is just the chance of treatment given $X$, which is $e(X)$: $$ =\mathbb E\Big[e(X)\;\Big|\;Y(0),Y(1),e(X)\Big]=e(X). $$ **Step 5: conclude.** The last expression involves only the score. So among customers with the same score, treatment has the same chance whatever their potential outcomes are, which is the independence we wanted. One number replaces the whole vector $X$. In practice the score is estimated, for instance with a logistic regression, and the claim to check is that the covariates look similar between treated and matched controls.
Derivation: how much error a pair at distance $d$ adds **Step 1: write the untreated outcome as a smooth function of $X$ plus noise.** Say $Y(0)=f(X)+\varepsilon$ with $\mathbb E[\varepsilon]=0$ at every $X$. **Step 2: the error in one pair.** A treated customer $i$ at $x_i$ is paired with a control $j$ at $x_j$. The missing outcome is $f(x_i)+\varepsilon_i$ and the stand-in is $f(x_j)+\varepsilon_j$. Their difference is $$ Y_i(0)-Y_j=f(x_i)-f(x_j)+\varepsilon_i-\varepsilon_j. $$ **Step 3: split into bias and noise.** The noise terms average to zero over many pairs. What is left is $f(x_i)-f(x_j)$, which for a nearby control is approximately the slope times the gap, $$ \begin{aligned} f(x_i)-f(x_j)&\approx f'(x_i)\,(x_i-x_j). \end{aligned} $$ **Step 4: the size of the bias.** Averaged over pairs, the systematic error is at most the slope times the average distance inside a pair. In the figure below the slope is 6 dollars per unit of engagement and the average distance with no limit is about 0.15, so the bias from imperfect matching is at most about 0.9 dollars. Tightening the caliper to 0.5 brings the distance down to 0.09. **Step 5: the cost.** A tighter caliper drops the treated customers who had no close control, so the estimate describes fewer people, and with fewer pairs the noise in the average grows. This is the bias-variance trade of matching, and it is why the next figure moves in both directions.
Matching removes most of the gap, then thins out the treated. The dots are 100 customers in a shop where the app adds 5 dollars. Filled dots installed the app, hollow dots did not, and each violet line joins an app user to the non-user matched to them. Drag the handle to shrink the largest distance allowed in a pair.
Worked example: does the loyalty app pay for itself? The raw gap in this shop is 12.5 dollars a month, and almost none of it is the app: app users are the engaged customers. Matching each of the 45 app users to the nearest non-user brings the estimate to 6.3 dollars. Allowing only pairs within 0.5 engagement units drops 5 of the 45 users and gives 5.2, and the true effect in this simulated shop is 5. The estimate is about half the raw gap, and the right reading is that the app adds 5 dollars a month for the customers who install it, not 12. Whether that pays for the build depends on the margin on 5 dollars of spend across the installed base. The decision changes by more than a factor of two depending on which number was used.
Matching cannot fix what it cannot see. If two customers look identical on every column in the data but one is motivated by a coupon the data does not record, that motivation is still in the bias. Overlap is the other limit: if keen customers adopt so reliably that no comparable non-user exists, matching has nothing to borrow and the honest answer is "the data does not say." ## Beyond matching: using two models so that one can be wrong **Teaser.** Matching throws away treated customers it cannot pair, and it needs a distance. Weighting by the propensity score keeps everyone but explodes when a few scores are near 0 or 1. A regression keeps everyone but is only as good as its shape. Is there a way to use more than one of these at once? There is, and it is the estimator most often used for adjustment today. Fit two models: an **outcome model** $\mu_0(x)\approx\mathbb E[Y\mid T=0,X=x]$, which predicts what an untreated customer with these characteristics spends, and the **propensity model** $e(x)$ from the matching section. The **doubly robust** estimate of the ATT uses the outcome model to fill in the missing outcome of each treated customer, then corrects the remaining error on the untreated with propensity weights. With $p=\Pr(T=1)$, it averages $$ \begin{aligned} \htmlClass{eq-der}{\widehat{\text{ATT}}}=\frac1{np}\sum_{i=1}^n\Big[&T_i\,\big(Y_i-\mu_0(X_i)\big) \\ &-(1-T_i)\,\frac{e(X_i)}{1-e(X_i)} \\ &\quad\cdot\big(Y_i-\mu_0(X_i)\big)\Big]. \end{aligned} $$ Read it as two pieces. The first is the treated customers' spend minus the outcome model's guess of what they would have spent without the app. The second is a correction: the weighted average of the same kind of guessing error among the untreated, which is zero if the outcome model is right for them. The point of the construction is that **the estimate is right if either model is right**.
Derivation: why one correct model is enough **Step 1: the target.** Multiply the estimator through by $p$. We need to show that the expectation of the bracket equals $p\cdot\text{ATT}$: $$ \begin{aligned} \mathbb E\Big[&T\,(Y-\mu_0(X)) \\ &-(1-T)\,\tfrac{e(X)}{1-e(X)}\,(Y-\mu_0(X))\Big]=p\cdot\text{ATT}. \end{aligned} $$ Write $m_0(x)=\mathbb E[Y(0)\mid X=x]$ for the true untreated mean, and $\delta(x)=\mu_0(x)-m_0(x)$ for the error of our outcome model. **Step 2: the first term.** For a treated customer $Y=Y(1)$, so $$ \begin{aligned} \mathbb E\big[T\,(Y-\mu_0(X))\big] &=\mathbb E\big[T\,(Y(1)-m_0(X))\big] \\ &\quad-\mathbb E\big[T\,\delta(X)\big]. \end{aligned} $$ The first piece is $p\cdot\text{ATT}$, because $m_0(X)$ is what the treated would have spent without the app, averaged over the treated. The second piece, using $\mathbb E[T\mid X]=e(X)$, is $$ \mathbb E\big[T\,\delta(X)\big]=\mathbb E\big[e(X)\,\delta(X)\big]. $$ **Step 3: the second term.** For an untreated customer $Y=Y(0)$. Condition on $X$ first, use conditional independence so that $\mathbb E[Y(0)\mid X,T=0]=m_0(X)$, and use $\mathbb E[1-T\mid X]=1-e(X)$: $$ \begin{aligned} \mathbb E\Big[(1-T)\,\tfrac{e(X)}{1-e(X)}\, \\ &\qquad\cdot(Y-\mu_0(X))\Big] \\ &=\mathbb E\Big[(1-e(X))\,\tfrac{e(X)}{1-e(X)}\, \\ &\qquad\cdot(m_0(X)-\mu_0(X))\Big] \\ &=-\mathbb E\big[e(X)\,\delta(X)\big]. \end{aligned} $$ **Step 4: subtract.** The bracket's expectation is first term minus second term: $$ \begin{aligned} p\cdot\text{ATT}-\mathbb E[e\,\delta]-\big(-\mathbb E[e\,\delta]\big)=p\cdot\text{ATT}. \end{aligned} $$ The error $\delta$ of the outcome model cancels exactly, provided the weights use the correct $e(x)$. So a wrong outcome model is harmless when the propensity model is right. **Step 5: the other direction.** If the outcome model is right, $\delta=0$ and the correction in Step 3 has mean zero for *any* weights, so the estimate is $p\cdot\text{ATT}$ whatever the propensity model says. A wrong propensity model is harmless when the outcome model is right. **Step 6: what is not covered.** Both steps use conditional independence given $X$. If a confounder is missing from $X$, both models are wrong in the same way and neither protects you.
**Double machine learning** keeps this estimator and changes how the two models are fitted. They are flexible machine-learning models rather than a regression and a logistic fit, and each is trained on a part of the data other than the part it scores, so that overfitting does not leak into the estimate. The payoff is honest confidence intervals even with many covariates, as set out by [Chernozhukov and coauthors](https://arxiv.org/abs/1608.00060). A different family, the **meta-learners** such as the S-, T- and X-learner, answers a different question: not the average effect but how the effect differs between customers, which is what targeting needs. [Künzel and coauthors](https://arxiv.org/abs/1706.03461) introduced the X-learner and report that no meta-learner is best everywhere. None of this changes what the data can tell you. The doubly robust estimate, double machine learning and the meta-learners all rest on the same assumption as matching: the measured columns capture why customers chose the treatment. They improve how flexibly you adjust and how honest the intervals are. They do not recover a confounder you did not measure, which is why the next section starts from a different assumption. ## When the confounder is hidden but the past is visible: difference-in-differences **Teaser.** Free shipping was switched on in three cities in March. Orders per customer rose from 3.0 to 3.9 a month. Was it the shipping? Matching needs the reasons for treatment to be in the data. Often they are not: the three cities may have been chosen because they were growing, or because they are larger, and no column records why. What we do have is time. We saw the treated cities before the change and after it, and we saw other cities over the same two periods. That gives three comparisons, and two of them are wrong in a way worth seeing. | orders per customer per month | before | after | change | | --- | --- | --- | --- | | three treated cities | 3.0 | 3.9 | +0.9 | | other cities | 2.6 | 3.1 | +0.5 | - **After minus before in the treated cities: 0.9.** This mixes the effect of free shipping with everything else that changed between the periods, such as the season. Customers everywhere ordered 0.5 more. - **Treated minus other cities, after: 0.8.** This mixes the effect with the fact that the treated cities already ordered 0.4 more before the change. - **The change in the treated minus the change in the others: $0.9-0.5=0.4$.** The season is removed by the second subtraction and the standing gap between the cities is removed by the first. This is the **difference-in-differences**.
Derivation: the double difference is the ATT when trends are parallel **Step 1: the estimand.** The treatment starts between periods 0 (before) and 1 (after). The ATT is the effect in the treated group in period 1: $$ \begin{aligned} \text{ATT}&=\mathbb E[Y_1(1)-Y_1(0) \\ &\qquad\mid T=1]. \end{aligned} $$ **Step 2: what we observe, treated group.** In period 0 nobody is treated yet, so $\mathbb E[Y_0\mid T=1]=\mathbb E[Y_0(0)\mid T=1]$. In period 1 the treated group shows its treated outcome, $\mathbb E[Y_1\mid T=1]=\mathbb E[Y_1(1)\mid T=1]$. **Step 3: what we observe, comparison group.** Both periods are untreated: $\mathbb E[Y_t\mid T=0]=\mathbb E[Y_t(0)\mid T=0]$ for $t=0,1$. **Step 4: the missing piece.** The ATT needs $\mathbb E[Y_1(0)\mid T=1]$, the treated group's outcome in period 1 without the treatment, which is never seen. Write it as its own starting point plus its own change: $$ \begin{aligned} \mathbb E[Y_1(0)\mid T=1] \\ &=\mathbb E[Y_0(0)\mid T=1] \\ &\quad+\Big(\mathbb E[Y_1(0)\mid T=1] \\ &\qquad-\mathbb E[Y_0(0)\mid T=1]\Big). \end{aligned} $$ The first term is observed. The bracket is the change the treated group would have had without the treatment. **Step 5: borrow the change (parallel trends).** Assumption 5 says that change equals the comparison group's change, which is observed: $$ \begin{aligned} \mathbb E[Y_1(0)\mid T=1] \\ &\quad-\mathbb E[Y_0(0)\mid T=1] \\ &=\mathbb E[Y_1\mid T=0]-\mathbb E[Y_0\mid T=0]. \end{aligned} $$ **Step 6: substitute into the ATT.** $$ \begin{aligned} \htmlClass{eq-der}{\text{ATT}} \\ &=\Big(\mathbb E[Y_1\mid T=1] \\ &\qquad-\mathbb E[Y_0\mid T=1]\Big) \\ &\quad-\Big(\mathbb E[Y_1\mid T=0] \\ &\qquad-\mathbb E[Y_0\mid T=0]\Big). \end{aligned} $$ Every term on the right is an average of observed data. **Step 7: check against the table.** $(3.9-3.0)-(3.1-2.6)=0.9-0.5=0.4$. **Step 8: the same number from a regression.** Regress the outcome on a treated-group dummy $G$, an after dummy $P$ and their product: $$ Y=\alpha+\gamma\,G+\delta\,P+\tau\,(G\cdot P)+\text{noise}. $$ The four group means are $\alpha$ (others, before), $\alpha+\gamma$ (treated, before), $\alpha+\delta$ (others, after) and $\alpha+\gamma+\delta+\tau$ (treated, after). The double difference of those four means is $\tau$, so the coefficient on the product is the estimate. The regression form is the one that extends to covariates, many periods and standard errors.
One subtraction removes the standing gap, the other removes the shared change. The copper dashes are where the treated cities would have gone without free shipping, parallel to the other cities. Drag the handle to change what happens to every city: only the double difference stays at the true 0.4.
Worked example: was it the free shipping? Part of the 0.9 increase was the season: every city ordered 0.5 more. The other 0.4 is the estimate of what free shipping added, per customer per month. It rests on the claim that the three cities would have gone up by 0.5 as well, as the others did. The next section is about how to test that.
## Checking the borrowed trend The assumption behind difference-in-differences is about something that did not happen, the treated cities' path without the change, so it cannot be tested directly. What can be tested is its implication for the past. If the treated and untreated cities moved in parallel after the change, they most likely moved in parallel before it. Plot the difference between the two groups in every period, relative to the last period before the change, and look at the periods before the change. They should hover around zero with no slope. With no extra drift the dots wander a little around zero and the estimate is close to the true 0.4. With extra drift of 0.03 per period the pre-change dots rise, and the estimate drifts to about 0.59: the drift is counted as the effect. The bias is the drift times the distance between the average pre-period and the average post-period, so a small slope matters more when the post-period is long. A flat pre-period does not prove parallel trends after the change, and a few things can break the design even when it looks clean. A shock that hits only the treated cities at the same time as the treatment, such as a competitor leaving, looks exactly like the effect. If treatment was rolled out to different cities at different dates, the simple two-group regression can mix comparisons between cities that were treated early and cities treated late, and newer estimators are built for that case. Customers moving between cities change who is in each group. None of these is visible in the table. ## A playbook for a causal question Both cases follow the same five steps. The mental model is the first sentence below. **Every causal estimate is a number we saw minus a number we imagined, and the design is how we defend the imagined number.** A randomised test imagines the missing outcome from a coin flip. Matching imagines it from customers who look the same. Difference-in-differences imagines it from the treated group's own past plus the change everyone else had. 1. **State the question as a contrast.** Who is the effect for (the treated, or everyone), against what (nothing, or the old version), and measured when? "Did the app work?" becomes "How much more did customers who installed the app spend in the following month than they would have without it?" 2. **Draw the graph.** Who decided the treatment, and what else drives both the decision and the outcome? Mark each variable as a confounder (hold it fixed), a middle step (leave it alone for a total effect) or a common effect (never condition on it). Add time if the same units were seen before. 3. **Choose the design by the assumption you can defend.** If you can randomise, randomise. If the reasons for treatment are measured, match or weight. If they are not, but you have a before and after and an untreated group, use difference-in-differences. If everyone was treated at once there is no group to borrow from, and none of these applies. 4. **Check what the assumption implies.** For matching: overlap, and whether the covariates look alike between treated and matched controls. For difference-in-differences: the pre-period gap, and the same estimate on an outcome the treatment should not move. 5. **Report the effect with its range, who it is for, and what would overturn it.** Say how large an unmeasured confounder, or a difference in trends, would have to be to flip the decision. | design | what it borrows | assumes | check | breaks when | | --- | --- | --- | --- | --- | | randomised test | a coin | the coin was fair and followed | balance on pre-treatment variables | people leave, or the arms interact | | matching or weighting | customers with the same $X$ | nothing unmeasured drove both choice and outcome | overlap and balance | a hidden reason for choosing exists | | difference-in-differences | the untreated group's change | parallel trends | flat pre-period gap | a shock hits only the treated |
Worked example: the whole site gets a price change at once Everyone was treated on the same day, so there are no untreated customers to match and no untreated group whose change can be borrowed. The question has to be answered another way: by a holdout kept back for the purpose, by treating a similar market as the comparison, or by modelling the series before the change and projecting it forward with a stated uncertainty. The playbook's value here is to say, in step 3, that the standard tools do not apply, rather than to pick the closest one.
## What the story cannot tell you Matching rests on the claim that the measured columns capture everything that made customers choose the treatment, and difference-in-differences on the claim that the groups would have moved together. Both are about the world without the treatment, and the data cannot confirm either. The checks above find the failures that leave a trace in the past; they cannot find a reason that left none. The graph in the first place was written by someone, and a missing arrow is a hidden confounder. The numbers in the figures come from simple shops. Effects that differ between customers, units that influence each other, treatments that start at different dates, outcomes that are counts or have long tails, and the standard errors of matched estimates all need more care than this essay gave them. The opening questions have honest answers with their conditions. The loyalty app raised monthly spend for the customers who installed it by roughly 5 dollars, not 12, provided that tenure and visit frequency capture why they installed it. Free shipping raised orders by about 0.4 per customer per month in the treated cities, provided that those cities would have followed the others. ## Further reading - Rubin — *Estimating causal effects of treatments in randomized and nonrandomized studies*, 1974. The potential-outcomes framework used throughout; I have not checked the citation details here. - Rosenbaum and Rubin — *The central role of the propensity score in observational studies for causal effects*, 1983. The balancing result behind the matching section; I have not checked the citation details here. - Pearl — *Causality*, 2009. The graph rules, including the back-door criterion; I have not checked the edition here. - Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey and Robins — [*Double/debiased machine learning for treatment and structural parameters*](https://arxiv.org/abs/1608.00060), Econometrics Journal, 2018. I confirmed the title, journal and the ATE and ATT coverage from the abstract; I did not read the proofs. - Künzel, Sekhon, Bickel and Yu — [*Metalearners for estimating heterogeneous treatment effects using machine learning*](https://arxiv.org/abs/1706.03461), PNAS, 2019. I confirmed the title, venue and the X-learner claim from the abstract; I did not read the rest. The doubly robust derivation above is my own write-up, not taken from either paper. - Hernán and Robins — *Causal Inference: What If*. A free book that develops the same ideas; I have not checked the chapters here. - Card and Krueger — *Minimum wages and employment*, 1994. A well-known difference-in-differences study of one state with a policy change and a neighbour without; I have not read it.