Why Unsupported Predictions Need Special Attention
Outcome regression estimates an outcome for each treatment level at specified covariate values, then combines those predictions to estimate a causal contrast. This is useful when treatment groups differ on measured covariates. But if one treatment group has no observations in a region of the covariate space, its outcome regression has no direct data there. A predicted counterfactual in that region is then an extrapolation: it depends on how the fitted model extends beyond the covariate values that informed it.
Extrapolation is not automatically a computational error. A regression program can return a prediction wherever its equation is defined. The key question is whether that prediction is supported by the design and by assumptions that can be defended. A smooth-looking fitted curve does not create observations in an empty region, and a precise standard error from a specified model does not establish that its form is correct there.
For clarity, consider the average treatment effect (ATE), \(E\{Y(1)-Y(0)\}\), where \(Y(a)\) is the potential outcome under treatment level \(a\). Under consistency, conditional exchangeability \(\{Y(1),Y(0)\}\perp A\mid X\), and positivity \(0<P(A=a\mid X=x)<1\) at covariate values in the target population, the identifying formula is
Positivity requires that both treatment levels have a positive probability at the relevant covariate values. The earlier material on positivity and support explains why this condition is needed for identification. If a target region has no observations at one treatment level, the observed data do not identify that treatment-specific conditional mean there without additional assumptions. A regression model may still produce a value, but that value comes from extending the model, not from direct empirical support in that region.
What the Regression Model Adds
In practice, analysts fit a model for \(E(Y\mid A,X)\), perhaps with treatment-by-covariate interactions, nonlinear terms, or other structure, and then predict under both treatment levels. For a target sample with covariate values \(X_i\), a standardized estimate often takes the form
This calculation is meaningful only after specifying the population represented by the \(X_i\) and the assumptions used to obtain each prediction. If \(X_i\) is unsupported among treated observations, \(\widehat m_1(X_i)\) is model-dependent; if it is unsupported among controls, \(\widehat m_0(X_i)\) is model-dependent. The uncertainty calculation may reflect sampling uncertainty under the fitted model while failing to capture the additional uncertainty from choosing an incorrect functional form.
A useful way to state the distinction is: the causal estimand may be defined for the full target population, but the observed data may not identify it there. A model can supply an estimate for that estimand only by adding restrictions on how outcomes behave across covariate space. These restrictions may be reasonable, but they should be explicit and examined rather than hidden inside a prediction routine.
Worked Example: Two Models, Different Unsupported Predictions
Suppose a hypothetical study has control observations only at \(x=0,2,4\), with mean outcomes \(10,14,18\), respectively. Treated observations include \(x=6\), where the treated mean outcome is \(30\). The target includes people with \(x=6\), but there are no controls at that value. Compare two possible control outcome functions.
The linear control model fitting the observed means is \(m_0(x)=10+2x\). At \(x=0,2,4\), it predicts \(10,14,18\), and at \(x=6\) it predicts
The treated prediction at \(x=6\) is \(30\), so the model-based treatment contrast there is \(30-22=8\) outcome units.
Now consider the family of control curves \(m_0(x)=10+2x+c\,x(x-2)(x-4)\), where \(c\) is a constant. The added term equals zero at each observed control value \(0,2,4\). Thus every value of \(c\) gives exactly the same fitted control means at the observed points. At \(x=6\), however, the added term is \(c(6)(4)(2)=48c\). If \(c=0.05\), then
Both models fit the given control observations exactly, yet they imply different control outcomes and different treatment contrasts at the unsupported value. The observed data in this example cannot choose between these extensions. The numerical contrast at \(x=6\) is therefore not determined by the observed control outcomes alone; it depends on the assumed shape of \(m_0(x)\).
Why the Difference Is Not Merely a Large Standard Error
A short justification makes the identification issue precise. Suppose controls are observed only at covariate values in a set \(S_0\), and the target includes \(x^\ast\notin S_0\). The observed control data inform \(m_0(x)\) on \(S_0\). If two candidate functions \(g_0\) and \(h_0\) agree for every \(x\in S_0\), then they imply the same control predictions at all observed control covariate values. But they may differ at \(x^\ast\):
Consequently, the observed data cannot distinguish the two functions at \(x^\ast\) merely by fitting the data in \(S_0\). If the target estimand averages predictions at \(x^\ast\), those functions can give different values for the estimand. Identification there requires restrictions that rule out some candidate functions—for example, a specified linear relationship that is assumed to continue beyond \(S_0\). Since there are no observations in that region to check that continuation, this is an additional modeling assumption, not a conclusion established by the observed fit.
This argument also explains why ordinary confidence intervals can be misleading if interpreted too broadly. A model-based interval may quantify uncertainty under a chosen regression form. It does not necessarily account for the possibility that another form, equally compatible with the observed data, predicts a substantially different value where support is absent.
Worked Example: An Empty Treatment-by-Subgroup Cell
In a hypothetical population, half the people are classified as healthy and half as having a chronic condition. Among controls, the mean outcome is 6 for healthy people and 3 for people with the condition. Among treated people, there are healthy observations with mean outcome 8, but no treated observations with the condition. An additive regression model uses the observed healthy treatment difference to predict the missing cell.
The observed treatment difference among healthy people is \(8-6=2\) outcome units. If the model assumes this difference is the same in both health groups, it predicts a treated mean of \(3+2=5\) for people with the condition. Standardizing equally over the two groups gives predicted treated mean \((8+5)/2=6.5\), control mean \((6+3)/2=4.5\), and an estimated average difference of \(6.5-4.5=2\) outcome units.
That calculation relies on the additive model’s no-interaction assumption. To see the dependence explicitly, let \(\gamma\) be the extra treatment difference in the chronic-condition group beyond the observed difference of 2 among healthy people. The missing treated mean would then be \(3+2+\gamma=5+\gamma\), and the equally weighted average contrast would be
No treated observations in the chronic-condition group provide information about \(\gamma\). The additive prediction sets \(\gamma=0\); it does not estimate that restriction from the empty cell. Therefore, the reported value of 2 is a model-based extrapolation to that subgroup, not a contrast directly supported by treatment comparisons in both groups.
Diagnosing Extrapolation
Diagnostics should ask two distinct questions: where is the design supported? and how much does the prediction depend on the fitted model? Neither question can be answered by looking at a regression coefficient table alone.
- Check support by treatment level. Plot or tabulate covariates separately for \(A=0\) and \(A=1\). For continuous covariates, compare ranges and distributions; for categorical covariates, identify treatment-by-category cells with no observations. In multiple dimensions, check whether target covariate combinations are represented, not merely whether each covariate separately has overlapping ranges.
- Mark predictions outside the observed design. For each target observation, record whether its covariates fall outside the treatment-specific range or region of observed covariates. A nearest-neighbor distance within each treatment group can help flag target points far from any observation in that group. The distance scale and flagging rule should be stated; there is no universal cutoff that turns a distance into proof of support.
- Inspect fitted values alongside covariates. Flag predictions that require substantial extension beyond the observed covariate range, and report how many target observations receive such predictions. Predicted outcome values far outside observed outcomes can be a warning, but outcome-range checks alone do not diagnose covariate support. Valid predictions can exceed observed outcomes, and a plausible-looking prediction can still be unsupported.
- Check sensitivity to model form. Compare defensible specifications, such as linear and nonlinear terms or models with and without treatment-by-covariate interactions. Focus on how predictions and the target contrast change specifically in flagged regions. Similar estimates across a few models are reassuring but do not prove that all relevant assumptions hold.
Worked Example: A Range and Fitted-Value Check
Consider a hypothetical one-covariate study. Controls are observed for \(x\) from 0 to 3, treated observations for \(x\) from 2 to 5, and the target contains six equally weighted observations at \(x=0,1,2,3,4,5\). Suppose fitted equations are \(\widehat m_0(x)=10+2x\) and \(\widehat m_1(x)=12+3x\).
The treatment-specific observed ranges are \([0,3]\) for controls and \([2,5]\) for treated people. Their intersection is \([2,3]\). Of the six target values, only 2 and 3 are in the intersection, so \(4/6=0.667\), or 66.7%, lie outside the region represented in both groups. At \(x=5\), for example, the control prediction is
even though no controls have \(x>3\). At \(x=0\), the treated prediction is \(\widehat m_1(0)=12+3(0)=12\), although no treated observations have \(x<2\). These are clear treatment-specific range flags.
The predictions can also be compared with within-group nearest-observation distances. At \(x=5\), the closest control covariate value is 3, so the distance is \(|5-3|=2\); the closest treated value is 5, so its distance is 0. At \(x=0\), the corresponding distances are 0 for controls and \(|0-2|=2\) for treated observations. These calculations show which treatment-specific prediction relies on extension beyond observed covariates. The range and distance checks flag dependence; they do not establish that the linear predictions are wrong or provide a data-driven correction for the unsupported values.
Alternatives, Limitations, and a Common Pitfall
When unsupported regions matter, one alternative is to define a target population restricted to covariate values with support in both treatment groups and estimate effects for that population. This avoids asking the outcome model to predict a missing treatment arm in those regions, but it changes the target population and therefore the causal question. The earlier material on common-support restrictions and trimming addresses how such a restriction is defined and interpreted. Another option is to retain the original target and present outcome-regression estimates as explicitly dependent on a stated functional form, accompanied by sensitivity analyses. Neither approach creates information where the design has none.
Outcome regression is not the only causal estimation strategy. Matching or weighting can also depend on overlap and support: changing the estimation method does not make an unsupported counterfactual empirically observed. The relevant comparison is not “regression versus a method with no assumptions,” but which population is being targeted, what support is available, and what assumptions the chosen procedure adds.
Common pitfall: “The regression software returned predictions for every person, so the full-population effect is estimated.” This confuses a defined prediction with an identified causal quantity. A model equation can return a number outside the covariate region used to fit it, but the observed data cannot verify the model’s extension there. The correction is to identify unsupported predictions, state the restrictions that generate them, and either justify and assess those restrictions or report an estimate for a clearly defined supported population.
Support diagnostics also have limitations. In finite samples, an observed range is only a rough description of multivariate support, and a nearest-neighbor distance depends on scaling and the covariates included. Conversely, a large observed sample does not repair a genuine structural absence of one treatment level at a covariate combination. Diagnostics reveal where caution is needed; they do not prove exchangeability, validate the outcome model, or replace the causal assumptions required for interpretation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Controls are observed only for \(x\leq 4\), but a target person has \(x=6\). What makes a control outcome prediction for this person an extrapolation?
- Two outcome models fit all observed control data equally well but predict different control outcomes at an unsupported covariate value. What does this imply about the prediction?
- In a target population, 30% of observations lie outside the region represented in both treatment groups. What should be reported about a standardized estimate for the full target?
- Why is checking whether predicted outcomes fall within the observed outcome range not a sufficient support diagnostic?
Solutions. (1) The target value \(x=6\) lies outside the covariate values represented among controls, so the model extends beyond the control data to supply the counterfactual prediction. (2) The observed control data do not distinguish the models at that value; the prediction and any contrast using it depend on the assumed functional form. (3) Report that 30% of target observations receive predictions outside shared support, identify which treatment-specific predictions are unsupported, and state the assumptions or target-population change used to interpret the estimate. (4) Outcome values can lie outside an observed outcome range even when covariates are supported, and plausible in-range outcomes can still be predicted at unsupported covariate values. Support is principally a design question about covariate representation by treatment group.