# Endogeneity in Media Mix Models: Why a Budget Is Not an Experiment A media mix model fitted on two years of weekly data says that every extra dollar of advertising brings back 2.95 dollars of sales. The shop keeps 40 cents of each dollar of sales, so the model says to spend more. A holdout test of the same advertising says 1.20. At 1.20, every extra advertising dollar loses the shop 52 cents. Both numbers were computed correctly. This post derives where the gap comes from, why more weeks of data do not close it, and which kinds of data do. The route has six steps. A least-squares slope answers a different question from the one the owner asked, and one identity says by how much. The first way the two come apart is a budget that follows demand. A longer history makes the wrong answer more certain, not less. Two more ways, spend that reacts to sales and spend that is recorded badly, turn out to be the same identity. Controls remove only the part of the problem you can see. Variation in spend that demand did not choose removes the rest. One colour code runs through the text and the figures: blue is what we observe, copper is a hidden quantity such as the true effect of spend, and violet is what we derive from the data. ## Assumptions 1. **Sales respond to spend in a straight line.** Weekly sales are $y=\alpha+\beta x+u$, where $x$ is the week's advertising spend and $u$ is everything else that moves sales that week. The number the owner wants is $\beta$: dollars of sales per extra dollar of spend, with everything else fixed. Saturation and carry-over are left out on purpose, so that the only thing that can go wrong is the thing under study. 2. **No memory, independent weeks.** This week's sales depend on this week's spend and nothing from earlier weeks. 3. **A demand pulse.** Some weeks demand runs high on its own: the holiday, a launch, a heat wave. The pulse $d$ sits inside $u$, and it adds $\gamma d$ to sales whatever the shop spends. The rest of $u$ is noise $\varepsilon$ that has nothing to do with spend, so $u=\gamma d+\varepsilon$. 4. **The planner sees the pulse coming.** The budget is set with the same forecast, so spend and the pulse move together with correlation $\rho$. 5. **The analyst's table has spend and sales and nothing else.** The pulse is never recorded. 6. **The shop.** Spend and sales are in thousands of dollars a week. Spend averages 10 with a standard deviation $\sigma_x=2$. The pulse has standard deviation $\sigma_d=5$ and $\gamma=1$. The noise has standard deviation $\sigma_\varepsilon=3$. The true effect is $\beta=1.2$, and the shop keeps 40 cents of each dollar of sales. The history is 104 weeks. 7. **Normal and independent.** The pulse, the noise and the other variables are jointly normal, and weeks are independent. The spread formulas use this; the identity of the first section does not. Assumptions 3 to 5 carry the weight. If any one of them fails, the problem below does not arise. ## The fitted slope answers a different question **Teaser.** The marketing lead asks: if we add a thousand dollars next week, how many more dollars of sales? The coefficient on spend in the model looks like the answer. It is the answer to a different question. There are two questions that sound alike. The first is about **intervention**: if the shop chose a different spend and the world stayed as it was, how would sales change? That is $\beta$, and it is what a budget decision needs. The second is about **observation**: among the weeks we have, how much higher were sales in the weeks with more spend? A least-squares fit answers that one. The two agree only if the weeks with more spend differ from the other weeks in nothing but the spend. The gap between them is one identity. Write $b$ for the slope that a least-squares fit of sales on spend finds. Then $$ \htmlClass{eq-der}{b}\;=\;\htmlClass{eq-hid}{\beta}\;+\;\frac{\operatorname{Cov}(x,u)}{\operatorname{Var}(x)} . $$ The fit finds the effect plus a second term. The second term is the covariance between spend and everything else that moves sales, divided by the variance of spend. It is zero when spend is unrelated to $u$. **Endogeneity** is the name for the case where it is not: spend is tied to something in the error term, and the fit credits that something to spend.
Derivation: the least-squares slope is the effect plus a covariance term The claim is that the slope a least-squares fit of $y$ on $x$ finds equals $\beta+\operatorname{Cov}(x,u)/\operatorname{Var}(x)$ exactly, whatever the data. **Step 1: what least squares asks for.** Pick the intercept $a$ and slope $b$ that make the average squared miss as small as possible, $$ S(a,b)=\mathbb E\big[(y-a-bx)^2\big]. $$ **Step 2: the intercept.** At the minimum the derivative with respect to $a$ is zero: $$ \frac{\partial S}{\partial a}=-2\,\mathbb E\big[y-a-bx\big]=0 \quad\Longrightarrow\quad a=\mathbb E[y]-b\,\mathbb E[x]. $$ **Step 3: the slope.** Set the derivative with respect to $b$ to zero and put in the intercept just found: $$ \frac{\partial S}{\partial b}=-2\,\mathbb E\big[x\,(y-a-bx)\big]=0 \quad\Longrightarrow\quad \mathbb E\big[(x-\mathbb E x)\,\big((y-\mathbb E y)-b\,(x-\mathbb E x)\big)\big]=0 . $$ Expanding the bracket gives $\operatorname{Cov}(x,y)-b\operatorname{Var}(x)=0$, so $$ b=\frac{\operatorname{Cov}(x,y)}{\operatorname{Var}(x)} . $$ **Step 4: put in the world.** In the shop $y=\alpha+\beta x+u$. A constant has no covariance with anything, so $$ \operatorname{Cov}(x,y)=\beta\operatorname{Cov}(x,x)+\operatorname{Cov}(x,u)=\beta\operatorname{Var}(x)+\operatorname{Cov}(x,u). $$ **Step 5: divide.** $$ b=\frac{\beta\operatorname{Var}(x)+\operatorname{Cov}(x,u)}{\operatorname{Var}(x)}=\beta+\frac{\operatorname{Cov}(x,u)}{\operatorname{Var}(x)} . $$ **Step 6: it holds in the sample too.** Replace every expectation above by an average over the 104 weeks. The algebra never used anything but the definitions of covariance and variance, so the slope found on the weeks equals $\beta$ plus the sample covariance of spend and $u$ over the sample variance of spend, exactly, with no approximation. **Check.** If the weeks were assigned spend by a coin flip, $\operatorname{Cov}(x,u)=0$ and $b=\beta$. This is why a randomised test works, and why a budget chosen by a person does not have to.
## A budget that follows demand credits the ads with the demand **Teaser.** In November the planner raises the budget because the holiday is coming. Sales rise in November too. How much of the rise belongs to the ads? The model cannot say, because both rose for the same reason. Use assumptions 3 and 4: the pulse adds $\gamma d$ to sales, and spend has correlation $\rho$ with the pulse. Then everything in $u$ that is related to spend is the pulse, and the second term of the identity is worked out in a few lines (derivation below): $$ \htmlClass{eq-der}{b}-\htmlClass{eq-hid}{\beta}\;=\;\frac{\gamma\,\rho\,\sigma_d}{\sigma_x}. $$ Three things must be true for the bias to exist, and each is a lever. The pulse must move sales ($\gamma\neq0$). The planner must follow it ($\rho\neq0$). And the analyst must not have it in the table. With the shop's numbers the bias is $1\times0.7\times5/2=1.75$ when the correlation is $\rho=0.7$, so the fit reports $1.2+1.75=2.95$ for an effect of 1.2.
The line fits the weeks and misses the effect. Each dot is a week, in thousands of dollars. The violet line is the slope a regression on spend alone finds. The copper dashes show what an extra dollar of spend does with the demand pulse held fixed: 1.2. Drag the handle to change how closely the planner's budget follows the pulse.
At the left end of the strip the budget ignores the pulse and the two lines coincide. Moving right does not change the effect of spend at all: the copper line never moves. Only the way the weeks are spread across the plane changes, and the violet line tilts to follow the spread.
Derivation: how large the bias is The claim is that if spend and the pulse have correlation $\rho$, the slope is off by $\gamma\rho\sigma_d/\sigma_x$. **Step 1: what is in $u$.** By assumption 3, $u=\gamma d+\varepsilon$, and the noise $\varepsilon$ is unrelated to spend. **Step 2: the covariance.** Covariance is linear in each argument, so $$ \operatorname{Cov}(x,u)=\gamma\operatorname{Cov}(x,d)+\operatorname{Cov}(x,\varepsilon)=\gamma\operatorname{Cov}(x,d)+0 . $$ **Step 3: the covariance of spend and pulse from their correlation.** The correlation is the covariance divided by the two standard deviations, $\rho=\operatorname{Cov}(x,d)/(\sigma_x\sigma_d)$. So $$ \operatorname{Cov}(x,u)=\gamma\,\rho\,\sigma_x\,\sigma_d . $$ **Step 4: divide by the variance of spend.** With $\operatorname{Var}(x)=\sigma_x^2$, $$ \frac{\operatorname{Cov}(x,u)}{\operatorname{Var}(x)}=\frac{\gamma\,\rho\,\sigma_x\sigma_d}{\sigma_x^2}=\frac{\gamma\,\rho\,\sigma_d}{\sigma_x} . $$ **Step 5: read the three levers.** The bias grows in a straight line with $\rho$. It grows with $\sigma_d$, the size of the swings that the planner follows. It shrinks with $\sigma_x$, because the more the budget moves for its own reasons, the less any one cause for the movement matters. **Check.** At $\rho=0$ the bias is zero. At $\rho=0.7$ it is $1\times0.7\times5/2=1.75$. At $\rho=0.3$ it is 0.75, and the fit gives 1.95. These are the slopes of the line in the figure at those settings.
Worked example: spend two thousand dollars more a week for a year? An extra dollar of spend must bring back $1/0.4=2.50$ dollars of sales to earn back its cost. The fit says 2.95, so each extra dollar earns $0.4\times2.95-1=+0.18$ dollars of profit. The plan is to add 2 thousand dollars a week for 52 weeks, which is 104 thousand dollars in all. The model predicts $2.95\times104=306.8$ thousand dollars of extra sales and a profit of $0.4\times306.8-104=+18.7$ thousand dollars. At the true effect, the same plan brings $1.2\times104=124.8$ thousand dollars of sales, and the profit is $0.4\times124.8-104=-54.1$ thousand dollars. The model recommends a plan that loses 54 thousand dollars, and it does so with a fit that matches the data weeks perfectly well.
## More weeks make the wrong answer more certain **Teaser.** Two years of weekly data left some doubt. The shop waits another three years and refits. Surely that settles it? A larger sample shrinks the noise in the fitted slope, and nothing else. The slope settles on $b=\beta+\text{bias}$, and the bias does not depend on how many weeks there are. So the interval closes around the wrong number. The figure shows the spread of the fitted slope for the planner who follows the pulse closely (violet) and for one whose budget ignores it (grey dashes).
More weeks narrow the interval around the wrong number. The curves are the large-sample spread of the fitted slope in the shop, with the budget following the pulse closely (violet) and with a budget that ignores it (grey dashes). The copper line is the true effect. Drag the handle to change the number of weeks.
Two further facts are in the figure, and the second is the stranger. At 104 weeks the usual 95% interval around the violet curve runs from about 2.50 to 3.40. The true value, 1.2, is nowhere near it. Even at 26 weeks the chance that such an interval catches the truth is only a few percent. And the violet curve is narrower than the grey one: 0.229 against 0.286 at 104 weeks. The confounded fit is the more precise of the two, because the pulse is a source of swings in sales that the slope on spend partly absorbs. A tight interval is evidence about the noise in the data. It says nothing about whether the slope is the one you wanted. The 95% in the usual interval is a promise that holds only when $\operatorname{Cov}(x,u)=0$. Even a mild link breaks it. With $\rho=0.3$, the bias is 0.75, and the interval catches the truth about 7 times in 10 with 26 weeks, about 1 in 4 with 104 weeks and essentially never with 520.
Derivation: the fit settles on the wrong number and the interval closes around it The claim is that the fitted slope is approximately normal with mean $\beta+\text{bias}$ and variance $\sigma_r^2/(n\sigma_x^2)$, so it concentrates on the wrong value as the number of weeks $n$ grows. The weeks are independent and the variables are jointly normal in this shop. **Step 1: write the data around the line the fit is aiming at.** Let $b=\beta+\text{bias}$ be the population slope from the first section, and define the miss $r=y-a-bx$ with $a$ the matching intercept. By how $a$ and $b$ were found, $\mathbb E[r]=0$ and $\operatorname{Cov}(x,r)=0$. In this shop $r$ is also independent of $x$. So every week obeys $y=a+bx+r$ exactly. **Step 2: the sample slope around its target.** Apply Steps 5 and 6 of the first derivation to this equation, with $r$ in place of $u$: the slope found on the sample is the target plus the sample covariance of $x$ and $r$ over the sample variance of $x$. On $n$ weeks that is $$ \hat b=b+\frac{\sum_i (x_i-\bar x)\,r_i}{\sum_i (x_i-\bar x)^2}. $$ **Step 3: the top.** Each term $(x_i-\bar x)\,r_i$ has mean zero. Since $x$ and $r$ are independent, its variance is $\sigma_x^2\sigma_r^2$. The $n$ terms are independent, so their sum has variance $n\,\sigma_x^2\sigma_r^2$. **Step 4: the bottom.** $\sum_i (x_i-\bar x)^2$ is close to $n\sigma_x^2$ once $n$ is large. **Step 5: put them together.** The error of the slope is a sum with variance $n\sigma_x^2\sigma_r^2$ divided by a number close to $n\sigma_x^2$. By the central limit theorem the sum is close to normal, so $$ \hat b\;\approx\;\text{Normal}\!\left(b,\ \frac{\sigma_r^2}{n\,\sigma_x^2}\right). $$ The variance goes to zero like $1/n$, and the centre stays at $b$. The fit converges to $b$, not to $\beta$. **Step 6: the size of the miss.** From $r=u-\text{bias}\times(x-\mathbb Ex)$ plus a constant, and $\operatorname{Cov}(x,u)=\text{bias}\times\sigma_x^2$, $$ \sigma_r^2=\operatorname{Var}(u)-2\,\text{bias}\operatorname{Cov}(x,u)+\text{bias}^2\sigma_x^2=\operatorname{Var}(u)-\text{bias}^2\sigma_x^2 . $$ The confounded fit therefore has a smaller miss than the clean one. In the shop, $\operatorname{Var}(u)=25+9=34$. At $\rho=0.7$, $\sigma_r^2=34-1.75^2\times4=21.75$, so the spread of the slope at 104 weeks is $\sqrt{21.75/(104\times4)}=0.229$. At $\rho=0$ it is $\sqrt{34/416}=0.286$. **Step 7: how often the interval catches the truth.** The usual interval is the fitted slope plus or minus $1.96$ times its spread. It contains $\beta$ when $|\hat b-\beta|<1.96\,\text{se}$. With $\hat b$ centred at $\beta+\text{bias}$, that probability is $$ \Phi\!\left(1.96-\frac{\text{bias}}{\text{se}}\right)-\Phi\!\left(-1.96-\frac{\text{bias}}{\text{se}}\right), $$ where $\Phi$ is the normal cumulative distribution. With no bias it is $\Phi(1.96)-\Phi(-1.96)=0.95$. With bias $0.75$ and 104 weeks, $\text{se}=\sqrt{(34-0.5625\times4)/416}=0.276$, the ratio is 2.72, and the chance is $\Phi(-0.76)-\Phi(-4.68)=0.23$. **Check.** At $\rho=0$ the formula gives 0.95 at every $n$. In simulated two-year histories of this shop the interval caught the truth about 22% of the time at $\rho=0.3$, and 95% of the time at $\rho=0$.
## Reacting to sales and recording badly are the same covariance **Teaser.** The finance team sets the advertising budget at ten cents per dollar of revenue. And the ad-server count of what ran differs from the invoice. Does either matter? Both are other ways for spend to be tied to something in $u$, and the identity of the first section covers them without change. **Spend that reacts to sales.** Suppose spend is set by a rule that looks at sales: $x=\kappa y+v$, with $\kappa=0.1$ for "ten cents per dollar", and $v$ the part of spend that the rule leaves to judgement. Now sales move spend as well as spend moving sales. A good week for any reason raises $u$, which raises $y$, which raises $x$. So $\operatorname{Cov}(x,u)>0$ even if nobody looks at the pulse directly: the rule reads sales, and sales contain it. As a variation on the shop, take $\operatorname{Var}(u)=34$ as before and $\operatorname{Var}(v)=3$ for the part of spend the rule leaves open, which makes spend a little more variable than in the other sections. Then $$ b-\beta=\frac{\kappa\,\operatorname{Var}(u)\,(1-\beta\kappa)}{\kappa^2\operatorname{Var}(u)+\operatorname{Var}(v)}=\frac{0.1\times34\times0.88}{0.34+3}=0.90, $$ and the fit reports 2.10 for an effect of 1.2.
Derivation: spend that reacts to sales The claim is that if sales are $y=\beta x+u$ and spend follows $x=\kappa y+v$ with $u$ and $v$ unrelated, the fitted slope is off by $\kappa\operatorname{Var}(u)(1-\beta\kappa)/(\kappa^2\operatorname{Var}(u)+\operatorname{Var}(v))$. **Step 1: solve the two equations for each week.** Put the second into the first: $$ y=\beta(\kappa y+v)+u \quad\Longrightarrow\quad y=\frac{\beta v+u}{1-\beta\kappa}. $$ **Step 2: spend in terms of the two causes.** $$ x=\kappa y+v=\frac{\kappa\beta v+\kappa u+v-\beta\kappa v}{1-\beta\kappa}=\frac{\kappa u+v}{1-\beta\kappa}. $$ **Step 3: covariance with the error.** Since $u$ and $v$ are unrelated, $$ \operatorname{Cov}(x,u)=\frac{\kappa\operatorname{Var}(u)}{1-\beta\kappa}. $$ **Step 4: variance of spend.** $$ \operatorname{Var}(x)=\frac{\kappa^2\operatorname{Var}(u)+\operatorname{Var}(v)}{(1-\beta\kappa)^2}. $$ **Step 5: divide.** The bias is the covariance over the variance: $$ \frac{\operatorname{Cov}(x,u)}{\operatorname{Var}(x)}=\frac{\kappa\operatorname{Var}(u)\,(1-\beta\kappa)}{\kappa^2\operatorname{Var}(u)+\operatorname{Var}(v)} . $$ **Check.** At $\kappa=0$ the rule ignores sales and the bias is zero. With $\kappa=0.1$ the numerator is $0.1\times34\times0.88=2.992$ and the denominator is $3.34$, which gives 0.896.
**Spend that is recorded badly.** Suppose the table holds $x^\text{obs}=x+m$, where $m$ is an error unrelated to everything else: an ad-server count that differs from the invoice, say. (Spend booked in the wrong week is a different and worse case: the error is then tied to the true spend, and with independent weeks the slope collapses toward zero.) Then the fit shrinks toward zero instead of away from it. If real spend has variance $\sigma_x^2=4$ and the recording error has standard deviation 1.5, the fitted slope keeps a share $4/(4+2.25)=0.64$ of the true effect. With no pulse at all the fit reports $0.64\times1.2=0.77$.
Derivation: spend recorded with error shrinks the slope The claim is that if the table holds $x^\text{obs}=x+m$ with the error $m$ unrelated to $x$ and to $u$, the fitted slope is $\beta\,\sigma_x^2/(\sigma_x^2+\sigma_m^2)$ when $x$ itself is unrelated to $u$. **Step 1: the covariance with sales.** The error is unrelated to sales, so $$ \operatorname{Cov}(x^\text{obs},y)=\operatorname{Cov}(x,y)=\beta\sigma_x^2 . $$ **Step 2: the variance of what was recorded.** $$ \operatorname{Var}(x^\text{obs})=\sigma_x^2+\sigma_m^2 . $$ **Step 3: divide.** $$ b=\frac{\beta\sigma_x^2}{\sigma_x^2+\sigma_m^2}. $$ **Step 4: see it as the same identity.** Rewrite sales in terms of what was recorded: $y=\alpha+\beta x^\text{obs}+(u-\beta m)$. The new error is $u-\beta m$, and it is tied to the recorded spend through $m$: $$ \operatorname{Cov}(x^\text{obs},u-\beta m)=-\beta\sigma_m^2 . $$ The identity then gives $b=\beta-\beta\sigma_m^2/(\sigma_x^2+\sigma_m^2)=\beta\sigma_x^2/(\sigma_x^2+\sigma_m^2)$, the same answer by the other route. **Check.** With $\sigma_m=0$ the share kept is 1. With $\sigma_m=1.5$ it is $4/6.25=0.64$, and with $\beta=1.2$ the fit is $0.768$.
Three stories, one equation. In each, the spend that the model sees is tied to something in $u$. What differs is the sign: a budget that follows demand and a rule that follows sales push the slope up, and a recording error pulls it down. When both act at once, they partly cancel. With the pulse at $\rho=0.7$ and the recording error above, the fit reads $2.95\times0.64=1.89$, closer to the truth than 2.95 but still too high, and nobody can tell from the table how the two net out. ## Controls remove only what you can see **Teaser.** The analyst adds a holiday calendar and a launch flag to the model. Together they explain half of the swings in the pulse. Is half the bias gone? No. With the budget following the pulse at $\rho=0.7$, seeing half of the pulse's variation removes one third of the bias, not one half. Let $q$ be the share of the pulse's variance that the analyst's controls capture. After adding them, the bias that remains is $$ \htmlClass{eq-der}{\text{bias}(q)}\;=\;\text{bias}(0)\times\frac{1-q}{1-\rho^2 q}. $$ At $q=0.5$ the factor is $0.5/0.755=0.66$. The fit moves from 2.95 to 2.36, still far from 1.2.
Controls remove less of the bias than the share they capture. The violet curve is the slope the regression finds when the analyst's controls capture a share of the pulse's variation. The grey dashes show what it would be if the bias fell in proportion. The copper line is the true effect. Drag the handle to change the share captured.
The reason is in the denominator. Once the seen part of the pulse is accounted for, the model learns about spend only from the part of the spend that the seen pulse does not explain. That leftover spend is a smaller quantity, and what it is tied to is the unseen part of the pulse. The bias shrinks by the share of the pulse you removed, but it is spread over less spend variation to learn from, so it shrinks by less.
Derivation: what is left of the bias after a control The claim is that if the analyst's table includes a variable $d_o$ that carries a share $q$ of the variance of the pulse, the slope on spend is off by $\text{bias}(0)\,(1-q)/(1-\rho^2q)$. **Step 1: split the pulse.** Write the pulse as the seen part plus the unseen part, $d=d_o+d_h$, with the two parts unrelated, $\operatorname{Var}(d_o)=q\sigma_d^2$ and $\operatorname{Var}(d_h)=(1-q)\sigma_d^2$. The planner followed the whole pulse, so spend contains $\rho\sigma_x d/\sigma_d$ plus a part unrelated to the pulse. **Step 2: what adding $d_o$ does to the slope on spend.** Fit $y$ on spend and $d_o$ together. Let $\tilde x$ be what is left of spend after removing its best straight-line fit on $d_o$, that is $\tilde x=x-\theta d_o$ plus a constant, where $\theta=\operatorname{Cov}(x,d_o)/\operatorname{Var}(d_o)$. The fitted equation is $y=a+bx+c\,d_o+e$ with the miss $e$ unrelated to $x$ and $d_o$. Substituting $x=\tilde x+\theta d_o$: $$ y=a+b\tilde x+(c+b\theta)\,d_o+e . $$ Here $e$ is unrelated to both $\tilde x$ and $d_o$, and $\tilde x$ is unrelated to $d_o$ by construction. These are the conditions that define a least-squares fit of $y$ on $\tilde x$ and $d_o$, and because $\tilde x$ and $d_o$ are unrelated, the coefficient on $\tilde x$ in that fit is $\operatorname{Cov}(\tilde x,y)/\operatorname{Var}(\tilde x)$. So the slope on spend is $$ b=\frac{\operatorname{Cov}(\tilde x,y)}{\operatorname{Var}(\tilde x)} . $$ **Step 3: the leftover spend.** The spend's covariance with the seen part is $\operatorname{Cov}(x,d_o)=\rho\sigma_x\,q\sigma_d$, so $\theta=\rho\sigma_x/\sigma_d$. Removing the fitted part leaves $$ \operatorname{Var}(\tilde x)=\sigma_x^2-\theta^2 q\sigma_d^2=\sigma_x^2\,(1-\rho^2q). $$ **Step 4: the covariance of leftover spend with $u$.** Since $\tilde x$ is unrelated to $d_o$, only the unseen part of the pulse matters. And since $d_h$ is unrelated to $d_o$, $\operatorname{Cov}(\tilde x,d_h)=\operatorname{Cov}(x,d_h)=\rho\sigma_x(1-q)\sigma_d$. So $$ \operatorname{Cov}(\tilde x,u)=\gamma\,\rho\,\sigma_x\,(1-q)\,\sigma_d . $$ **Step 5: divide.** Since $\operatorname{Cov}(\tilde x,x)=\operatorname{Var}(\tilde x)$, the covariance in Step 2 splits as $\operatorname{Cov}(\tilde x,y)=\beta\operatorname{Var}(\tilde x)+\operatorname{Cov}(\tilde x,u)$, so the slope is $\beta$ plus $\operatorname{Cov}(\tilde x,u)/\operatorname{Var}(\tilde x)$. The part that is not $\beta$ is $$ \frac{\operatorname{Cov}(\tilde x,u)}{\operatorname{Var}(\tilde x)}=\frac{\gamma\rho\sigma_d}{\sigma_x}\times\frac{1-q}{1-\rho^2q}=\text{bias}(0)\times\frac{1-q}{1-\rho^2q}. $$ **Check.** At $q=0$ nothing is controlled and the full bias returns. At $q=1$ the controls capture the whole pulse and the bias is zero. At $\rho=0.7$ and $q=0.5$ the remaining bias is $1.75\times0.5/0.755=1.16$, so the fit is 2.36. At $q=0.9$ it is $1.75\times0.1/0.559=0.31$, and the fit is 1.51.
The [causal inference post](causal-derived) draws this as closing a back door; matching and difference-in-differences there are the alternatives when the pulse is hidden but a pre-period exists. In this shop, controls move the answer the right way, and the result above is for this shop: the control is a component of the pulse that the planner followed. A control tied only to spend and not to sales can make the bias worse, so the lesson is to add what you know drives demand, not whatever is on hand. The figure says how much to trust a partly controlled fit: a model that controls for the calendar and the launches is better than one that does not, and it still carries a bias that nothing in the table can measure. ## A nudge the pulse did not choose recovers the effect **Teaser.** The radio station runs half-price weeks on a schedule of its own. In those weeks the shop buys more airtime. Is that a nuisance in the data or a help? A help. A week with a half-price offer has more spend for a reason that has nothing to do with the pulse: the station chose the week, not the planner. Compare the average sales of the half-price weeks with the others. If the pulse is no different in the two groups, any difference in sales has to have come through spend. Divide that difference by the difference in spend and what remains is the effect of spend alone. Call the half-price indicator $z$. It is called an **instrument** when two things hold: it moves spend (relevance), and it is unrelated to everything else that moves sales (exogeneity). In symbols, the effect is $$ \htmlClass{eq-hid}{\beta}\;=\;\frac{\operatorname{Cov}(z,y)}{\operatorname{Cov}(z,x)} . $$
Derivation: the ratio of two covariances is the effect The claim is that if $z$ moves spend and is unrelated to $u$, then $\operatorname{Cov}(z,y)/\operatorname{Cov}(z,x)=\beta$. **Step 1: the covariance of the instrument with sales.** Put in $y=\alpha+\beta x+u$: $$ \operatorname{Cov}(z,y)=\beta\operatorname{Cov}(z,x)+\operatorname{Cov}(z,u). $$ **Step 2: use exogeneity.** The instrument is unrelated to everything else that moves sales, so $\operatorname{Cov}(z,u)=0$ and $$ \operatorname{Cov}(z,y)=\beta\operatorname{Cov}(z,x). $$ **Step 3: divide by the effect of $z$ on spend.** This needs $\operatorname{Cov}(z,x)\neq0$, which is relevance: $$ \beta=\frac{\operatorname{Cov}(z,y)}{\operatorname{Cov}(z,x)} . $$ **Step 4: the spread of the estimate.** On $n$ weeks the ratio of sample covariances differs from $\beta$ by $\sum(z_i-\bar z)\,u_i\big/\sum(z_i-\bar z)(x_i-\bar x)$. The top has variance $n\sigma_z^2\operatorname{Var}(u)$, and the bottom is close to $n\operatorname{Cov}(z,x)$. So the spread is $$ \text{se}_\text{IV}\approx\frac{\sqrt{\operatorname{Var}(u)}\;\sigma_z}{\sqrt n\,|\operatorname{Cov}(z,x)|}=\frac{\sqrt{\operatorname{Var}(u)}}{\sqrt n\;\sigma_x\sqrt s}, $$ where $s=\operatorname{Cov}(z,x)^2/(\sigma_z^2\sigma_x^2)$ is the share of the variance of spend that the instrument explains. **Step 5: compare with the plain fit.** The plain fit has spread $\sigma_r/(\sqrt n\,\sigma_x)$ from the previous section. The instrument's spread is larger by the factor $\sqrt{\operatorname{Var}(u)/\sigma_r^2}\,/\sqrt s$. It is larger for two reasons. The instrument learns only from the part of spend it explains, which is why $s$ is in the denominator. And it cannot use the part of $u$ that the plain fit explains through spend, so all of $\operatorname{Var}(u)$ counts as noise, not just $\sigma_r^2$. **Check.** With $\operatorname{Var}(u)=34$, 104 weeks and $s=0.45$, the spread is $\sqrt{34/(104\times4\times0.45)}=0.43$. With $s=0.1$ it is 0.90, and with $s=0.02$ it is 2.02. The plain fit's is 0.229.
The instrument is right in large samples and wide when weak. The grey dashes are the plain fit, tight and centred on the wrong value. The violet curve is the large-sample spread of the instrument estimate, centred on the true effect (copper), with a spread that depends on how much of the variation in spend the instrument explains. Both are for 104 weeks. Drag the handle to change that share.
This is the price of honest variation. The plain fit is precise and wrong. The instrument is right in large samples and wide, and it gets wider as the share of spend it explains shrinks. Weak instruments also fail in a second way that the figure's bells do not show. In simulated two-year histories of this shop, an instrument that explained 2% of spend gave a typical estimate of 1.58 and sometimes wild ones, pulled back toward the plain fit's 2.95. At 10% and at 45% the typical estimate was within a few hundredths of 1.2.
Worked example: the half-price weeks In half of the weeks the station's offer lifts the shop's spend by 1.5 thousand dollars, and the offer is unrelated to the pulse. The offer explains $0.5\times0.5\times1.5^2/4=0.14$ of the variance of spend. Compare the two groups of weeks: spend is higher by 1.5 on average, and sales are higher by $1.2\times1.5=1.8$, since the pulse is no different between the groups and the whole sales difference comes through spend. The ratio is $1.8/1.5=1.2$, the true effect, while the plain fit on the same weeks says 2.95. The price is the spread. With $s=0.14$ and 104 weeks it is $\sqrt{34/(104\times4\times0.14)}=0.76$, so a typical interval is about 1.2 plus or minus 1.5: from below zero to about 2.7, which straddles the break-even 2.50. The data from the offer cannot say whether more advertising pays, but it also does not say, wrongly and precisely, that it does.
The best instrument is one the shop sets itself. If a coin flip decides which weeks, or which regions, get the extra spend or none, then $\operatorname{Cov}(z,u)=0$ by construction rather than by argument. That is a holdout or a lift test, and it is the same ratio as above. It costs something, since some weeks run at a spend the planner would not have chosen. What it buys is a point on the response curve that no regression on the planner's own choices can provide. ## What the data cannot tell you The first limit is the hardest. (The [MMM post](mmm-derived) ends its short section on budgets with the same remedy; this essay is its derivation.) The bias formula needs $\gamma$ and $\rho$, the two things that are missing from the table by assumption. No amount of data on spend and sales alone reveals whether the budget followed demand, and the fit gives no warning. It looks better as the budget follows the pulse more closely: the typical miss around the line falls from 5.8 thousand dollars at $\rho=0$ to 3.7 at $\rho=0.9$, so the dots in the first figure tighten as the line gets more wrong. The only way to learn how the budget was set is to ask the people who set it, and what signals they used. An instrument's exogeneity is an argument, not a test. The data can show that the half-price weeks moved spend, which is relevance. They cannot show that the offer was unrelated to the shop's own demand, or that it did not change sales in some way other than through spend. Those claims rest on knowing how the station chose its weeks. The shop is a straight line with independent weeks and one channel. Real models have saturation and carry-over, and many channels that move together. The identity of the first section is for the slope of a straight line; I have not derived its analogue for the curves here. The spread formulas are large-sample approximations for normal data; real weekly sales are autocorrelated, which is a reason to trust the intervals less. An experiment tests one range of spend, which may not be the range the next decision needs. ## Back to the 2.95 The model's 2.95 is a correct answer to "how much higher were sales in the weeks that had more spend?" It is not an answer to "what would an extra dollar bring?", and the owner asked the second. Under the assumptions of this shop, if the budget followed demand and demand lifted sales, the fitted slope is biased upward. Recording error pulls the other way, and nothing in the table says which wins. Variation that demand did not choose, a nudge the station set or a holdout the shop set, gives an estimate that does not carry the demand bias, with a wider interval. An honest report gives both numbers with their conditions, and a calibration of the model against the experiment before anyone moves a budget. ## Further reading - D. Chan and M. Perry — [*Challenges and Opportunities in Media Mix Modeling*](https://research.google/pubs/pub45998/), Google, 2017. I read the full text. Its section on seasonality describes advertisers raising and lowering spend with their understanding of demand, which is the first route above; it calls the problem selection bias and omitted variable bias rather than endogeneity. - B. R. Gordon, F. Zettelmeyer, N. Bhargava and D. Chapsky — *A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook*, Marketing Science 38(2), 2019, pp. 193–225. I confirmed the citation and the headline finding from an index page: common observational methods often failed to recover the lift that randomised experiments measured. I did not read the paper. - J. Bound, D. A. Jaeger and R. M. Baker — *Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak*, Journal of the American Statistical Association 90(430), 1995, pp. 443–450. I confirmed from an abstract page that weak instruments pull estimates toward the plain fit; the simulation in this essay shows the same pattern. - J. D. Angrist and J.-S. Pischke — *Mostly Harmless Econometrics*, Princeton University Press, 2009. Chapter 4 covers instrumental variables. I checked the publisher page, not the text. - M. A. Hernán and J. M. Robins — [*Causal Inference: What If*](https://miguelhernan.org/whatifbook), Chapman & Hall/CRC. A free book. I did not open the PDF.