Why Your Finest Predictive Mannequin Provides the Incorrect Therapy Impact

A query that sounds simple however isn’t

the incorrect purpose. Let’s say we wish to estimate the impact of an publicity on an end result, e.g., air air pollution on hospital admissions or a drug versus time to restoration. We now have observational information, one publicity variable, and a pile of candidate covariates. Some are confounders. Some are noise. Now the most important query is — which of them go within the mannequin?

The standard reflex palms the selection to a variety rule. Run Stepwise routines, evaluate BIC values, and even cross-validate Lasso. Full Bayesian averaging proceeds over candidate specs.

Nevertheless, the issue with these approaches is that each considered one of these instruments optimizes predictive match. A mannequin can predict the end result superbly whereas being systematically incorrect in regards to the impact we care about.

This situation was the topic of a 2012 Biometrics paper by Chi Wang, Giovanni Parmigiani, and Francesca Dominici, who known as the underlying situation adjustment uncertainty. It’s the uncertainty about which variables to regulate for, they usually proposed a repair known as Bayesian Adjustment for Confounding (BAC) [1]. It additionally has a well-known frequentist twin that most individuals don’t realise is identical perception: post-double-selection and double machine studying [3][4].

On this publish, I’ll display three issues. I’ll present you the failure mode in runnable Python, construct the repair, after which lastly present you the repair’s personal sharp edge: the case the place “modify for something that predicts publicity” is precisely the incorrect recommendation and makes your estimate worse.

When you have been studying about propensity scores, subclassification, and matching, that is the query these strategies assume you will have already answered: how do you determine, from information, which covariates a legitimate adjustment wants?

An excellent predictor and a confounder aren’t the identical

Why does a specification tuned for prediction produce a flawed impact estimate? As a result of the 2 duties reward distinct traits. Prediction favors a covariate that accounts for end result variance. Impact estimation favors a covariate tied to each publicity and end result.

Now, the issue arises when a confounder is strongly related to the publicity however solely weakly with the end result. To a predictive criterion, that variable appears to be like almost ineffective as a result of it explains little end result variance. Additionally, a lot of what it explains is already captured by the (correlated) publicity. In sensible finite samples, BIC might choose dropping it. However dropping it adjustments what the publicity coefficient means. The coefficient now absorbs the confounder’s path, and the “impact” is now truly contaminated.

Wang and coauthors illustrate the purpose with 5 candidate covariates. U1, U2, and U3 every hyperlink to each X and Y. U4 predicts Y alone. U5 is noise.

Now take into account all the argument lowered to 2 t-statistics. On the simulated information under, the variable U2 exhibits a t-statistic of 1.2 inside the end result mannequin. That yields p equal to 0.23. It seems indistinguishable from noise. Contained in the publicity mannequin, the identical variable exhibits a t-statistic of 31.5. Similar variable, identical information set. One mannequin calls it irrelevant. The opposite calls it central.

Each routine we usually invoke examines solely the primary column. This end result follows from arithmetic. Omitted variable bias from excluding U2 equals:

It’s the product of U2’s coefficient within the end result equation occasions its coefficient within the regression of X on U2 (i.e., delta). The primary issue stays small. That’s precisely why BIC stays unmoved. The second issue grows giant as a result of U2 tracks publicity intently. Their product contaminates the goal estimate. No end result match rule ever inspects the second issue. Two portions. Match guidelines register just one.BIC stays choice constant. With ample observations, it could retain U2. The current problem is finite pattern. The asymptotic disappearance of bias provides little solace when the info set at hand comprises roughly 1,000 rows. The Code under follows the sample already described.

The failure mode, in Python

Let’s simulate the setup in Determine 1 and match two end result fashions, i.e., the totally adjusted one and one which drops U2.

import numpy as np, pandas as pd
import statsmodels.api as sm

rng = np.random.default_rng(11)
n, M = 1000, 5

U = rng.regular(measurement=(n, M))                              # candidates U1..U5
X = U[:,0] + U[:,1] + 0.1*U[:,2] + rng.regular(measurement=n)    # publicity
Y = 0.1*X + U[:,0] + 0.1*U[:,1] + U[:,2] + U[:,3] 
    + rng.regular(measurement=n)                                 # true impact = 0.1

cols = [f"U{j+1}" for j in range(M)]
df = pd.DataFrame(U, columns=cols); df["X"] = X; df["Y"] = Y

def fit_outcome(consists of):
    feats = ["X"] + [c for c, inc in zip(cols, includes) if inc]
    return sm.OLS(df["Y"], sm.add_constant(df[feats])).match()

full  = fit_outcome((1,1,1,1,0))   # adjusts for U1..U4
noU2  = fit_outcome((1,0,1,1,0))   # silently drops confounder U2

The outcomes (true impact is 0.1):

adjusts for U1,U2,U3,U4   β̂ = 0.144   95% CI (0.083, 0.206)   BIC = 2907.6
drops U2                  β̂ = 0.171   95% CI (0.127, 0.214)   BIC = 2902.1

Right here, three issues occurred:

1. The mannequin lacking a real confounder has the higher BIC. By the standard logic, it’s the mannequin we’re going to select.
2. Its confidence interval excludes the true worth. We’d report a major impact of ~0.17 with confidence and truly be fully incorrect.
3. The totally adjusted mannequin, penalized for carrying a “ineffective” predictor, is the one whose interval truly covers the reality.

The 2 fashions match the info equally properly. Nevertheless, they estimate completely different parameters. Solely considered one of them is the causal amount we needed.

Bayesian Mannequin Averaging has the identical blind spot

Suppose we don’t belief any single chosen mannequin and determine to common them. Bayesian mannequin averaging (BMA) weights every mannequin by its posterior chance and blends the estimates.

Right here, the catch is that with the standard flat prior over fashions, the posterior weights are pushed by marginal probability. Once more, it’s the identical criterion that simply betrayed our estimates. Run by all 2⁵ = 32 end result fashions, apply the standard BIC weights, and watch the place the chance settles.

BMA assigns roughly 90 p.c of its mass to the model that drops U2. Therefore, we are able to conclude that averaging doesn’t assist when the weights themselves level on the incorrect goal. Wang et al. noticed precisely the identical sample of their paper and located that BMA’s 95% intervals coated the reality much less typically than the nominal degree would recommend. [1].

The BAC concept: ask the publicity mannequin who the confounders are

A confounder, by definition, should predict the publicity in addition to the end result. This implies we should always not choose covariates utilizing solely the end result mannequin; as a substitute, we should always match a second mannequin for the publicity and let that mannequin veto the end result mannequin’s choices.

BAC formalizes this with two linked variable-selection issues:

• an end result mannequin: Y as a operate of X and candidate covariates,
• an publicity mannequin: X as a operate of the identical candidates,

These two are joined by a dependence parameter ω, which represents the prior odds {that a} covariate enters the end result mannequin on condition that it’s already within the publicity mannequin. At ω = 1, the fashions decouple, and we get better plain BMA. As ω → ∞, any covariate that the publicity mannequin desires is pressured into the end result mannequin, which supplies us the complete confounding adjustment [1].

If this logic reminds you of propensity scores, that’s completely intuitive. The publicity mannequin is a propensity mannequin, and the textbook recommendation from the causal inference literature consists of covariates associated to therapy project when establishing our adjustment [2]. That is precisely the instinct BAC encodes as a previous. The essential distinction is that BAC doesn’t pressure us to decide to a single hand‑picked specification. As a substitute, it captures the uncertainty in that choice and propagates it into the ultimate inference.

A minimal, enumerated model of BAC is proven under. Real BAC makes use of MCMC, however with solely 5 covariates we are able to brute‑pressure each doable mannequin mixture.

from itertools import product

def bic_weights(bics):
    b = np.array(bics)
    w = np.exp(-0.5 * (b - b.min()))
    return w / w.sum()

# All 32 end result fashions
out_models, out_bics, betas = [], [], []
for inc in product([0,1], repeat=M):
    m = fit_outcome(inc)
    out_models.append(inc); out_bics.append(m.bic)
    betas.append(m.params["X"])
w_bma = bic_weights(out_bics)

# All 32 publicity fashions -> per-covariate inclusion chance
exp_bics, exp_models = [], []
for inc in product([0,1], repeat=M):
    feats = [c for c, i in zip(cols, inc) if i]
    Xm = sm.add_constant(df[feats]) if feats else np.ones((n,1))
    exp_bics.append(sm.OLS(df["X"], Xm).match().bic)
    exp_models.append(inc)
w_exp = bic_weights(exp_bics)
incl_prob = [sum(w for w, inc in zip(w_exp, exp_models) if inc[j])
             for j in vary(M)]

# BAC-style prior with omega -> infinity:
# end result fashions should comprise each covariate the publicity mannequin calls for
required = [p > 0.5 for p in incl_prob]
okay = np.array([all(inc[j] for j in vary(M) if required[j])
               for inc in out_models])
w_bac = w_bma * okay
w_bac = w_bac / w_bac.sum()

print("publicity inclusion probs:", np.spherical(incl_prob, 3))
print("BMA estimate:", np.dot(w_bma, betas))
print("BAC estimate:", np.dot(w_bac, betas))

The printed output exhibits that the publicity mannequin accurately recovers the confounder construction:

publicity inclusion probs: [1.    1.    1.    0.031 0.032]
BMA estimate: 0.169
BAC estimate: 0.144

The inclusion chances for U1, U2, and U3 are all roughly one, whereas U4 and U5 have chances very near zero. When the end result‑mannequin weights are conditioned on this information, the therapy impact estimate strikes from 0.169 all the way down to 0.144. Determine 2 exhibits how the mannequin weights flip decisively towards the totally adjusted specs.

A single dataset might be a fortunate draw, so I repeated the entire experiment 300 occasions.

This bias is just not random noise. It’s structural. The prediction‑pushed weights systematically favour below‑adjusted fashions, and the ultimate estimates inherit that systematic slant.

The catch: an instrument appears to be like precisely like a confounder

That is the half that the majority write‑ups of BAC skip, and it explains why the ω knob truly exists. The rule “modify for something that predicts the publicity” is just not a really protected recommendation. Let’s take into account a variable Z that predicts X strongly however has no path to Y in any respect besides by X. That variable is an instrument, and an publicity mannequin can not inform it aside from a real confounder.

Now let’s see what occurs after we obey the publicity mannequin in a setting the place UC is an unmeasured confounder (as there at all times is in actual information) and Z is a pure instrument:

Uc = rng.regular(measurement=n)                  # unmeasured confounder
Z  = rng.regular(measurement=n)                  # instrument: impacts X solely
X  = Uc + Z + rng.regular(measurement=n)
Y  = 0.1*X + Uc + rng.regular(measurement=n)     # Z does NOT seem right here

Over 500 replications with a real impact of 0.1, the outcomes are putting:

no adjustment      β̂ = 0.434    (bias +0.334)
modify for Z       β̂ = 0.601    (bias +0.501)

Adjusting for Z made issues 50% worse. The estimate didn’t turn out to be much less environment friendly but it surely grew to become extra biased. This phenomenon is called bias amplification or Z‑bias [5][6]. The instinct works as follows: conditioning on Z strips out the a part of X’s variation that was clear (the instrument‑pushed half). This leaves behind a residual X that’s extra dominated by the unmeasured confounder than earlier than. So, we have now mainly thrown away our good variation and stored the contaminated variation, so the residual confounding bought concentrated.

At $\gamma =3$, ignoring $Z$ leaves a bias of $+0.09$, whereas adjusting for Z leaves $+0.50$. And we can not detect this distinction from an publicity mannequin alone. In a simulation the place Z is a pure instrument and C is a real confounder, each coming into X identically, we receive the next:

publicity mannequin t-statistics:   Z = 32.2    C = 33.6

These numbers are statistically indistinguishable. Nevertheless, causally they’re reverse. No quantity of knowledge will separate them, as a result of the distinction lies within the DAG, not within the covariance matrix. That is exactly why BAC’s $\omega$ is a dial and never a swap. The $\omega \to \mathrm{\infty }$ setting is the aggressive model that forces in each publicity predictor. A finite $\omega$ softens the rule for publicity. The selection of $\omega$ is a causal assumption we’re making, not a parameter we are able to estimate from the info. For those who take one factor from this publish, take this sentence.

The identical concept, 3 ways:

BAC dates from 2012, however the core perception was independently rebuilt in econometrics. It will definitely grew to become Double Machine Studying, and is now used below varied completely different names

1. Put up‑double‑choice (Belloni, Chernozhukov & Hansen, 2014 [3]: The process is sort of easy. We run a lasso of Y on the covariates and run a lasso of X on the covariates. Then take the union of the chosen variables, and refit OLS on that union. The lasso‑on‑X step exists for precisely the explanation BAC’s publicity mannequin exists i.e., to rescue confounders which are weak on Y however robust on X. The influence is actual. Utilizing the theoretically motivated plug‑in penalty from the BCH paper on the identical simulation, the end result lasso drops U2​ in 100 out of 300 datasets (33% of the time). Single‑choice (lasso on Y solely) yields a imply estimate of 0.112 with bias $+0.012$, whereas double‑choice (the union) yields a imply of 0.101 with bias $+0.001$ and 95% confidence interval protection of 0.94. A 3rd of the time, the end result lasso throws away a real confounder however the publicity lasso at all times catches it, and the interval protection returns to nominal.
2. Double/debiased machine studying (Chernozhukov et al., 2018) [4]: This is identical two‑mannequin construction generalised. We predict Y from the covariates and X from the covariates. Then we take the residuals of each, and regress residual‑on‑residual. Cross‑becoming is added to get rid of the overfitting bias. Now the 2 nuisance fashions may be gradient boosting or random forests as a substitute of lassos, and the impact estimate stays ​constant. A sanity examine on the identical information‑producing course of confirms that DML lands at 0.101 i.e., the identical place that BAC and double choice attain.
3. Consequence‑adaptive lasso (Shortreed & Ertefaie, 2017) [7]: Right here is the counter‑punch, which exists exactly due to the bias‑amplification drawback described above. Consequence‑adaptive lasso builds the propensity mannequin however penalises every covariate by how weakly it pertains to the end result. It intentionally down-weights variables that predict publicity and nothing else. That is the precise reverse intuition from BAC with $\omega \to \mathrm{\infty }$. It’s a guess that pure devices are the larger hazard; BAC’s guess is that dropped confounders are. Each bets are defensible.

So what’s the conclusion?

1. We have to cease letting predictive standards choose our adjustment set. BIC, AIC, lasso paths, and stepwise p-values all rating fashions on end result match, and none of them is aware of what a confounder is. A mannequin can win on all of those metrics and nonetheless hand us a biased impact.
2. Second, we have to match the publicity mannequin even when we by no means run BAC. It prices us 4 traces of code, and it’s the solely diagnostic in our toolkit that may see the best‑hand column of that t-statistic desk. When a covariate could be very distinguished within the publicity mannequin however silent within the end result mannequin, we have now simply discovered the variable that our choice process is about to throw away.
3. However, additionally we should always not blindly obey the publicity mannequin. Earlier than we embrace a powerful publicity‑predictor, ask the query the info can not reply: may this variable have an effect on Y by any path aside from X? If the reply is sure, it’s a confounder and we should always embrace it. If the reply is genuinely no, it’s an instrument, and adjusting for it’ll amplify no matter unmeasured confounding we have now left. That is the step the place we have now to assume, and no estimator will do it for us.
4. The asymmetry between below‑adjustment and over‑adjustment ought to form our selections. Over-adjustment for variables which are merely prognostic often prices effectivity, whereas adjusting for devices can truly enhance bias. [1].

Lastly, we have to deal with adjustment uncertainty as actual uncertainty. If our confidence interval is conditional on one hand‑chosen mannequin, it understates what we have no idea.

The uncomfortable abstract is that the instruments most of us attain for first have been constructed for a special job. Prediction asks, “Which mannequin explains Y?” Causal inference asks, “Which mannequin isolates the impact of X on Y?” These questions have completely different solutions. Understanding that’s a lot of the battle, and understanding that the second query carries an assumption that no quantity of knowledge will resolve for us.

References

[1] C. Wang, G. Parmigiani and F. Dominici, Bayesian Impact Estimation Accounting for Adjustment Uncertainty (2012), Biometrics, 68(3), 661–671.

[2] A. Gelman, J. Hill and A. Vehtari, Regression and Different Tales (2020), Cambridge College Press, Ch. 20.

[3] A. Belloni, V. Chernozhukov and C. Hansen, Inference on Therapy Results after Choice amongst Excessive-Dimensional Controls (2014), Evaluate of Financial Research, 81(2), 608–650.

[4] V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey and J. Robins, Double/Debiased Machine Studying for Therapy and Structural Parameters (2018), The Econometrics Journal, 21(1), C1–C68.

[5] J. Pearl, Invited Commentary: Understanding Bias Amplification (2011), American Journal of Epidemiology, 174(11), 1223–1227.

[6] J. A. Myers, J. A. Rassen, J. J. Gagne, Ok. F. Huybrechts, S. Schneeweiss, Ok. J. Rothman, M. M. Joffe and R. J. Glynn, Results of Adjusting for Instrumental Variables on Bias and Precision of Impact Estimates (2011), American Journal of Epidemiology, 174(11), 1213–1222.

[7] S. M. Shortreed and A. Ertefaie, Consequence-Adaptive Lasso: Variable Choice for Causal Inference (2017), Biometrics, 73(4), 1111–1122.