Sensitivity Checks for Trial Regression Models

Why Check a Trial Regression Model?

A randomized trial can have a prespecified primary regression model and still require judgment about how well its adjustment describes the baseline data. For example, a straight-line adjustment for baseline outcome may be a useful primary choice, but the relationship between baseline and follow-up outcome may be curved. A prespecified set of baseline covariates may also be compared with another reasonable set. Sensitivity checks show whether the estimated treatment contrast depends substantially on such choices.

The purpose is not to search for whichever model gives the smallest p-value. The primary model remains the main analysis; alternatives assess how dependent its conclusion is on modeling choices. A result is more reassuring when reasonable alternatives give similar treatment estimates. A noticeable change calls for transparent reporting and careful interpretation, not automatic replacement of the primary model.

Definition: A sensitivity check for a trial regression model compares the treatment contrast from a prespecified primary model with contrasts from reasonable alternative models, while keeping the outcome, treatment definition, analysis population, and target contrast as consistent as possible. The alternatives examine specified modeling choices, such as the functional form of a baseline predictor or the set of baseline covariates.

Set Up Comparable Models

Let \(Y_i\) be the follow-up outcome for participant \(i\), \(A_i\) indicate randomized assignment (\(A_i=1\) for the intervention and \(A_i=0\) for control), \(Y_{0i}\) be the baseline outcome, and \(Z_i\) represent other prespecified baseline covariates. A common primary ANCOVA model is

Y_i=\beta_0+\beta_A A_i+\beta_1Y_{0i}+\boldsymbol{\gamma}^{T}\mathbf{Z}_i+\varepsilon_i.

With no treatment-by-baseline interaction, \(\beta_A\) is the model's adjusted intervention-minus-control contrast in mean follow-up outcome, in outcome units, for participants with the same modeled baseline covariates. This model assumes the conditional mean is adequately represented by the included terms, including a linear baseline-outcome relationship as written. For conventional model-based standard errors, it also assumes independent errors with constant variance and an appropriate error distribution for the chosen inference; robust standard errors can reduce reliance on the constant-variance assumption. The trial's randomization and analysis by assignment support a causal interpretation of the assignment contrast, but do not make every regression specification equally appropriate.

A sensitivity model might replace \(Y_{0i}\) with a prespecified curved function, such as \(Y_{0i}\) and \(Y_{0i}^2\), or change the additional baseline covariate set. The treatment indicator and its coding should remain the same. Covariates used for adjustment should be measured before randomization; adjusting for post-treatment variables can distort the intended comparison. If an alternative includes treatment interactions, the coefficient of \(A_i\) alone may no longer represent the same overall contrast. In that case, compute and compare the same target contrast, rather than comparing unlike coefficients.

For a fair comparison, fit the primary and alternative models to the same participants whenever possible. State the models, the reason each alternative is reasonable, and the contrast being compared. A difference in estimates can arise from the model form, finite-sample chance imbalances, or both. It is not by itself evidence that the randomized groups were no longer comparable.

Preserve the primary analysis.
Identify the prespecified model and its treatment contrast before interpreting alternatives.

Choose plausible alternatives.
Vary a stated baseline functional form or baseline adjustment set; avoid trying many models simply to find a favorable result.

Compare estimates in context.
Report the treatment contrasts, their units and uncertainty, and whether their differences matter on the outcome scale.

Worked Comparisons

Worked Example: Checking a Linear Baseline Adjustment

Suppose a hypothetical randomized trial compares an intervention with control on a symptom score at follow-up. Higher scores indicate worse symptoms. The prespecified ANCOVA adjusts linearly for baseline score and reports an adjusted intervention-minus-control estimate of \(-4.8\) points, with \(SE=1.9\). A prespecified sensitivity model adds a squared baseline-score term and estimates \(-4.6\) points, with \(SE=2.0\). Assume the trial team decided in advance that a change smaller than 1 point would not be important to its interpretation.

The change in the point estimate is

\widehat{\beta}_{A,\mathrm{quadratic}}- \widehat{\beta}_{A,\mathrm{linear}} =(-4.6)-(-4.8)=0.2\text{ points}.

The change is 0.2 points, smaller than the 1-point threshold. Both estimates favor the intervention, and the estimated contrast is similar under these two baseline forms. For an approximate 95% interval using 1.96 standard errors, the linear model gives

-4.8\pm1.96(1.9)=[-8.524,\,-1.076]\text{ points}.

The quadratic model gives

-4.6\pm1.96(2.0)=[-8.520,\,-0.680]\text{ points}.

These intervals describe uncertainty for each model's contrast; they are not an interval for the difference between the two estimates. The comparison supports stability of the point estimate under this particular alternative, not proof that the baseline relationship is exactly linear or quadratic. The interpretation also relies on the models being fit to the same participants and the stated contrast being comparable.

Worked Example: Changing the Baseline Covariate Set

In a hypothetical trial of a rehabilitation program, the outcome is walking distance in meters after treatment. The primary model adjusts for assigned group, baseline walking distance, and baseline age. A prespecified alternative also adjusts for baseline mobility score. The primary model estimates an intervention-minus-control contrast of \(12.4\) meters; the alternative estimates \(11.7\) meters. Suppose the team has specified 3 meters as a change large enough to affect the interpretation.

The estimate changes by

11.7-12.4=-0.7\text{ meters}.

The absolute change is \(0.7\) meters, below the 3-meter benchmark. In these models, adding the baseline mobility score makes the estimated advantage slightly smaller, but not by an amount the team had defined as consequential. The estimate remains an adjusted comparison by randomized assignment; it is not the effect of receiving the program regardless of assignment.

This comparison is informative only if the added score is a baseline measure and the alternative was selected for a sound reason, such as being clinically relevant or prognostic. It would be poor practice to add or remove covariates solely because their individual p-values cross a threshold. Randomization provides the basis for the treatment comparison; baseline adjustment can improve precision and account for chance imbalances, but data-driven covariate selection can make the analysis harder to interpret.

Worked Example: Recognizing a Material Change

A hypothetical randomized trial measures a quality-of-life score at follow-up, where higher values are better. Its primary model adjusts for the baseline score and two prespecified baseline covariates, estimating an intervention-minus-control contrast of \(5.6\) points. A reasonable alternative baseline adjustment set gives an estimate of \(2.1\) points. The trial team had specified 2 points as the smallest difference that would change its interpretation.

The difference between estimates is

2.1-5.6=-3.5\text{ points},\qquad |-3.5|=3.5\text{ points}.

Since \(3.5\) is greater than the prespecified 2-point benchmark, the treatment estimate is sensitive to this adjustment choice in a practically relevant sense. Both point estimates are positive, but their magnitudes differ enough that the result should not be summarized simply as “robust.” The analysis report should show both estimates, explain which baseline covariates differ, give their uncertainty, and retain the prespecified primary model as primary.

The shift does not establish which model is correct. It motivates checking that the alternative is scientifically defensible, that both models use the same participants and outcome definition, and that neither includes a post-treatment variable. If the models imply different target contrasts, the comparison must be revised to put them on a common scale and estimand before drawing a sensitivity conclusion.

How to Judge the Comparison

A sensitivity check is about the size and direction of the treatment estimate, not merely whether one model has a p-value below \(\alpha\) and another does not. A small change in standard error can move a p-value across a conventional threshold even when the effect estimate barely changes. Conversely, estimates can differ meaningfully while both p-values lie on the same side of a threshold. Report point estimates and uncertainty, and assess their differences against a context-specific benchmark chosen before examining results when possible.

Comparing confidence intervals by eye is not a formal test of the difference between model estimates. The models use the same participants and outcomes, so their estimates are generally correlated. If the scientific question requires an inferential interval or test for the difference itself, its uncertainty must account for that dependence. For many applied sensitivity checks, a clear descriptive comparison against a prespecified practical threshold is more useful than a significance test of model-to-model differences.

A Short Justification for Keeping the Contrast Fixed

Suppose two candidate models include the same treatment indicator and estimate a constant treatment contrast, but use different baseline functions \(f_1(Y_{0i})\) and \(f_2(Y_{0i})\). Their fitted mean structures can be written as

E(Y_i\mid A_i,Y_{0i},\mathbf{Z}_i) =\beta_0+\beta_A A_i+f(Y_{0i})+\boldsymbol{\gamma}^{T}\mathbf{Z}_i.

For either model, compare the fitted conditional means at the same baseline values, once with assignment \(A_i=1\) and once with \(A_i=0\). The baseline terms are identical in the two predictions and cancel:

\{\beta_0+\beta_A+f(Y_{0i})+\boldsymbol{\gamma}^{T}\mathbf{Z}_i\} -\{\beta_0+f(Y_{0i})+\boldsymbol{\gamma}^{T}\mathbf{Z}_i\} =\beta_A.

Thus, both models express an intervention-minus-control contrast in the same outcome units, even though their fitted baseline relationships differ. Their estimated \(\beta_A\) values may still differ because the fitted regression weights and residual patterns change. This cancellation does not hold for a model in which the treatment effect varies with baseline; then a single treatment coefficient is not the contrast at every baseline value.

Common Mistakes and Reporting

Choosing the model with the smallest p-value. This makes the reported result depend on outcome-guided model searching. Keep the prespecified model primary and explain alternatives as sensitivity checks.
Calling any estimate change a failure of randomization. Randomization balances groups in expectation, not perfectly in every realized trial. Different adjustments can yield different estimates in finite samples.
Treating a nonsignificant covariate as automatically unnecessary. Covariate inclusion should follow the analysis plan and scientific rationale, not a series of individual significance tests.
Adjusting for a post-treatment variable. A measure affected by assignment is not a baseline adjustment variable; including it can change or bias the intended treatment comparison.
Comparing unlike coefficients. If model terms change what the treatment coefficient means, calculate the same target contrast from each model before comparing.

A concise report names the primary model and each planned alternative, identifies the baseline terms changed, states whether the same participants were analyzed, and presents treatment estimates with standard errors or confidence intervals in outcome units. It then describes whether the differences are small or important relative to a justified benchmark. This lets readers distinguish a stable conclusion from one that depends on a particular regression specification.

Check Your Understanding

Use the definitions and comparisons above to answer these questions.

Why should a sensitivity model generally retain the same treatment coding and target contrast as the primary model?
A quadratic-baseline model changes an estimate from 3.0 to 2.8 outcome units. What additional information would help determine whether that change matters?
Why is it not enough to conclude that an estimate is robust because the p-value is below \(\alpha\) in both models?
What is wrong with adding a post-treatment measure as an alternative baseline covariate?
If a model includes a treatment-by-baseline interaction, why might its treatment coefficient not be comparable with the primary model's treatment coefficient?