Why Check the Reference Model First?
A delta adjustment represents a specified departure from a reference model for outcomes that are missing. The adjustment can be interpreted meaningfully only if the reference model itself is a reasonable starting point. If its predictions are systematically wrong among observed participants, applying a sensitivity shift does not repair that misspecification; it shifts a flawed baseline.
The practical task here is to assess the outcome model using outcomes that were observed. These checks can reveal inadequate fit, poor calibration, omitted covariate structure, or weak predictive performance. They cannot, by themselves, show that the model correctly describes outcomes for participants whose outcomes are missing. That distinction is fundamental: diagnostics evaluate observable implications, while the relationship between observed and unobserved outcomes requires assumptions or sensitivity analysis.
For example, a continuous-outcome reference model might specify \(E(Y\mid X,R=1)=m_0(X)\). A delta adjustment may then define a sensitivity departure from that reference for missing outcomes. The precise form of the departure depends on the chosen delta parameterization; the important point here is that \(m_0(X)\) must first be assessed as a model for observed outcomes.
A useful working assumption is that the chosen covariates and model form adequately capture the outcome relationship in the observed data over the covariate region where predictions will be needed. Using that model as a baseline for missing outcomes additionally requires a substantive extrapolation assumption: conditional on the specified covariates, the reference outcome law is a meaningful starting point for those missing outcomes. Observed-data diagnostics can examine the first assumption, but cannot verify the second.
What to Examine in the Observed Data
Begin by checking whether the model includes variables that are important for the outcome and for differences in who has an observed outcome. In a randomized study, treatment arm is often important to retain in the outcome model even if the treatment effect is not the principal target: omitting it can make predictions systematically wrong within arms. Baseline predictors, design variables, and clinically motivated effect modifiers may also matter. Covariate selection should be guided by the scientific design and model purpose, not only by which terms happen to be statistically significant in one sample.
Then examine whether the chosen functional form is plausible. A straight-line term for a continuous covariate may miss curvature; an additive model may miss a meaningful interaction. Residual plots can reveal patterns that a single overall fit statistic conceals. For observed records, define the residual as observed minus fitted outcome:
Residuals centered near zero overall are not sufficient evidence of adequacy. Plot them against fitted values and important covariates, and examine them by treatment arm or other scientifically relevant subgroups. Curvature, changing spread, clusters, or subgroup-specific offsets can indicate that the model's mean or variance structure is inadequate. For a model intended to predict a full outcome distribution, also assess whether the residual distribution and predictive uncertainty are represented reasonably; checking only the conditional mean is not enough.
Finally, evaluate predictions on data not used to fit the model when feasible. A training-data fit is usually optimistic because the model was estimated to describe those same observations. A held-out sample or cross-validation can estimate predictive performance for new observed cases drawn from a comparable setting. Define validation errors with their sign made explicit:
A positive mean validation error means observed outcomes exceed predictions on average, so predictions understate outcomes. A negative mean means predictions overstate outcomes. Mean absolute error summarizes typical absolute discrepancy, while root mean squared error gives relatively greater weight to large errors. No single metric diagnoses every problem: pair summary measures with calibration and residual-pattern checks.
Worked Checks Before Applying Delta
Worked Example: Reading the Sign of Validation Error
Suppose a hypothetical model is fitted on a training sample and evaluated on five held-out participants. Their observed outcomes are 4, 6, 7, 8, and 10 points; their predicted outcomes are 5, 5, 7, 9, and 8 points. These held-out outcomes were not used to fit the model.
The validation errors, calculated as observed minus predicted, are \(-1,1,0,-1,2\) points. Thus,
The mean absolute error is
and the root mean squared error is
On these five held-out records, predictions understate outcomes by 0.2 points on average under this sign convention, and the average absolute error is 1 point. The positive mean error does not mean predictions are too high. With only five cases, these summaries are descriptive rather than precise estimates of performance; inspect individual errors and obtain a larger, representative validation sample if possible.
Worked Example: Why a Design Covariate Can Matter
Consider hypothetical observed outcomes in two treatment arms, measured at baseline covariate values \(X=0\) and \(X=1\). The two observed outcomes in each arm-covariate cell are shown below.
| Arm | \(X\) | Observed outcomes | Cell mean |
|---|---|---|---|
| Control | 0 | 4, 6 | 5 |
| Control | 1 | 6, 8 | 7 |
| Treatment | 0 | 6, 8 | 7 |
| Treatment | 1 | 8, 10 | 9 |
Let \(A=1\) for treatment and \(A=0\) for control. A model with an intercept, \(X\), and \(A\) fits these cell means as
For instance, the predicted mean for treatment at \(X=1\) is \(5+2(1)+2(1)=9\), matching that cell's mean. If the arm term is omitted, the pooled mean at \(X=0\) is \((4+6+6+8)/4=6\), and at \(X=1\) it is \((6+8+8+10)/4=8\). The reduced model therefore predicts 6 points for both arms at \(X=0\), and 8 points for both at \(X=1\). It underpredicts the treatment-arm cell means by 1 point and overpredicts the control-arm cell means by 1 point.
This example does not prove that an arm effect will always be present. It shows why an apparently adequate pooled prediction can hide systematic errors within design groups. In real data, examine arm-specific residuals and validation performance; account for the random variation in cell means rather than treating small-sample differences as certain.
Worked Example: A Residual Pattern Reveals Curvature
Suppose four hypothetical observed records have covariate values \(X=0,1,2,3\) and outcomes \(Y=0,1,4,9\). Fit a straight-line model to these four points. The means are \(\bar{x}=1.5\) and \(\bar{y}=3.5\). The least-squares slope is
The fitted values are \(-1,2,5,8\), so the observed-minus-fitted residuals are \(1,-1,-1,1\). Their average is zero, but the signs follow a pattern: positive at the ends and negative in the middle. The linear fit misses curvature. In this small illustration, adding a quadratic term gives \(Y=X^2\), which fits all four points exactly. That perfect fit is not evidence that the quadratic model will predict new observations perfectly; it is a cue to assess scientifically plausible curvature and validate the candidate form on data not used to estimate it.
Validation, Assumptions, and Comparisons
A validation result is informative only for the prediction task it represents. In ordinary \(K\)-fold cross-validation, the observed records are divided into \(K\) groups; each group is predicted by a model fitted without that group, and the held-out errors are combined. All steps that use outcome information to choose or fit the model should be repeated within each training fold. If model selection is performed using the full data before cross-validation, the held-out performance can still be optimistic.
Validation also depends on comparability. If held-out participants have covariate values unlike those for whom predictions will be needed, their errors may not describe performance in the relevant region. Compare covariate distributions and observed-data coverage, and be cautious when predictions require extrapolation beyond the observed range. If data are clustered or longitudinal, the split should respect the independent prediction unit: for example, observations from one participant should not be divided between training and validation when the target is a new participant.
The reference-model assessment has several complementary approaches:
| Approach | What it helps assess | Important limitation |
|---|---|---|
| Residual plots and subgroup summaries | Patterns, calibration problems, and uneven fit in observed data | Do not establish performance for unobserved outcomes |
| Held-out validation or cross-validation | Prediction error for comparable observed cases not used to fit each model | Can be unstable in small samples or unrepresentative covariate regions |
| Alternative plausible model specifications | Sensitivity of predictions to covariate terms and functional form | A better in-sample fit alone does not guarantee better prediction |
For a short justification of why held-out assessment matters, consider fitting a model to a particular training sample. Its fitted values are chosen using that sample's outcomes, so residuals on that sample reflect both underlying prediction error and adaptation of the fitted model to those outcomes. When evaluated on independent held-out outcomes from the same prediction setting, the fitted model has not adapted to their particular outcome values. The resulting errors therefore provide a more direct assessment of prediction for new comparable cases, although sampling variability remains.
When comparing candidate models, use the scientific context, observed residual patterns, and validation performance together. A more complex model can capture curvature or interactions but may estimate them unreliably; a simpler model may be more stable but systematically miss structure. If predictive distributions are used as the reference, also assess distributional calibration, not merely the mean prediction. No diagnostic should be treated as a test that certifies the model.
Common Pitfalls and Limits
- Reversing the error sign. With error defined as observed minus predicted, a positive average means predictions are too low on average, not too high.
- Equating good observed-data fit with a verified missing-outcome model. The outcomes needed to check fit for \(R=0\) are, by definition, unavailable. Good fit among \(R=1\) records does not identify the outcome distribution among those with missing outcomes.
- Using an overall average to conceal subgroup errors. Opposite errors across treatment arms or covariate groups can cancel. Inspect relevant subgroups as well as overall summaries.
- Trusting training fit alone. A model can fit its estimation data well and still predict new observations poorly. Use held-out assessment when feasible.
- Confusing a delta adjustment with model repair. Delta addresses a specified sensitivity departure from a reference. It should not be used to compensate for an observed-data model that misses important outcome structure.
The central limitation is unavoidable: the observed data can challenge the reference model where outcomes are observed, but cannot determine whether its conditional outcome distribution is the right baseline for missing outcomes. State this limitation explicitly and keep predictive model uncertainty separate from the untestable assumptions represented by the sensitivity analysis.
Check Your Understanding
Answer each question, then compare with the solutions below.
- If validation errors are defined as observed minus predicted and their mean is \(-0.4\) units, what is the direction and size of average prediction error?
- Why can residuals with an overall mean of zero still indicate a poor reference model?
- A model has lower training RMSE than a competitor but higher cross-validated RMSE. Which result is more relevant to predicting new comparable observed cases, and why?
- What can good residual and validation diagnostics among observed outcomes establish, and what can they not establish about missing outcomes?
Solutions. (1) Predicted values exceed observed values by 0.4 units on average, because \(Y-\widehat{Y}\) is negative on average. (2) Positive and negative residuals can cancel; a pattern by covariate, fitted value, or subgroup can reveal systematic misspecification despite a zero overall mean. (3) Cross-validated RMSE is more relevant because each evaluated outcome was held out from the fit used to predict it; the higher value indicates worse estimated predictive performance for comparable held-out records. (4) They can identify observable lack of fit and assess prediction for relevant observed cases. They cannot verify that the reference outcome distribution applies to cases with missing outcomes; that extrapolation remains an assumption addressed through sensitivity analysis.