Worked Comparison of Stepwise and Full-Model Strategies

Why Compare Strategies on the Same Data?

Covariate model building can look objective when summarized as a list of selected predictors. Yet that list depends on the strategy: forward selection adds terms to a small model, backward elimination removes terms from a larger one, and a full-model test asks whether a set of terms contributes jointly. These are different questions, so their answers need not match. A worked comparison makes the decisions—and the assumptions behind them—visible.

Consider a hypothetical NLME analysis of concentration-time profiles from 120 participants, with repeated observations per participant. The structural model, random-effects structure, residual model, and observations are held fixed throughout. Three prespecified covariate terms—weight, age, and albumin—are candidates for the same model parameter. Weight and albumin have a sample correlation of 0.78. The fit results below are hypothetical maximum-likelihood results; they are not from a real study.

Let the objective function value be \(OFV=-2\ell\), where \(\ell\) is the maximized log likelihood. Smaller OFV means a better maximized likelihood. All rows use the same data and the same structural, random-effects, and residual specifications; only the listed fixed covariate terms differ.

Assumptions Behind the Comparisons

For nested models, the likelihood-ratio statistic is the OFV of the reduced model minus the OFV of the larger model. Under the null hypothesis that the added coefficients are zero, this statistic is approximately chi-square distributed, with degrees of freedom equal to the number of added parameters. This reference distribution requires a regular likelihood problem: the models must be nested, parameters must be identifiable, and the null values must not be on a problematic boundary. For these examples, each covariate contributes one fixed-effect coefficient, so a one-term comparison has \(df=1\).

The approximation also relies on adequate information for the large-sample likelihood result. It may be unreliable with a small or sparse sample, weak identification, convergence to different local optima, or a poorly specified likelihood. A fair comparison therefore requires refitting every model to the same observations, using the same likelihood and all the same non-covariate model components. Use maximum likelihood when comparing models that differ in fixed effects; restricted maximum likelihood values are not directly comparable across different fixed-effect specifications.

The worked strategies use different prespecified thresholds to illustrate their consequences: forward selection enters a term only when its likelihood-ratio test has \(p<0.05\), whereas backward elimination retains a term when its test has \(p<0.10\). These thresholds are choices for this example, not universal defaults. A full-model test uses the joint null that all three candidate coefficients equal zero. As discussed in the earlier material on multiplicity and winner’s curse, repeated data-driven testing affects how nominal p-values should be viewed.

Forward Selection and Backward Elimination

Full-Model Test and Parameter Uncertainty

A full-model test asks a joint question. Its null hypothesis is \(H_0:\beta_W=\beta_A=\beta_{Alb}=0\), where each \(\beta\) denotes the coefficient for its prespecified covariate term. Its alternative is that at least one coefficient is nonzero. The test compares the full model directly with the base model, so the degrees of freedom are three.

The distinction between a joint test and the stepwise tests matters. In this example, the joint test is significant even though forward selection did not add age or albumin after weight at its \(0.05\) threshold. A joint test can accumulate evidence across several terms; it is not a substitute for an individual test of each coefficient or a guarantee that a particular selection algorithm will retain a term.

Likelihood-ratio tests compare fit, while coefficient standard errors describe uncertainty in estimated coefficients conditional on a specified model. Both are useful, but neither alone resolves uncertainty about which model should have been selected. The following hypothetical full-model estimates illustrate how to report coefficient uncertainty. The coefficients are on their prespecified model scale; their numerical units and clinical interpretation depend on that specification.

Why Correlated Candidates Complicate the Picture

Weight and albumin have correlation 0.78 in the hypothetical sample. Correlated candidates can carry overlapping information, so an estimate conditional on both may be less precise than an estimate from a model containing only one. Their coefficients can also change when the other term is included: each conditional estimate describes its contribution after accounting for the other included terms, not an isolated effect in the data.

The fit table makes this issue concrete. Weight alone improves the OFV over the base model by 9.0 units. But when weight and age are already present, adding albumin improves the OFV by only \(988.2-987.8=0.4\); when weight and albumin are present, adding age improves it by \(988.5-987.8=0.7\). These small conditional improvements do not contradict the full-model joint result. They answer different comparisons, and correlated candidates can make it difficult to assign shared information uniquely.

A useful derivation clarifies why AIC can lead to a different decision from a fixed p-value rule. AIC is \(AIC=-2\ell+2K=OFV+2K\), where \(K\) is the number of estimated parameters. When one coefficient is added, \(K\) increases by one. Thus:

Therefore, AIC favors the larger model when \(\Delta OFV>2\). For adding age to the weight model, \(\Delta OFV=2.8\), so \(AIC_{\text{larger}}-AIC_{\text{smaller}}=-2.8+2=-0.8\): AIC favors including age, even though its one-degree-of-freedom test has \(p\approx0.094\). AIC and a \(0.05\) test rule are different decision criteria; agreement is not required.

Diagnostics, Alternatives, and Limitations

Before interpreting a selection result, check whether the comparisons are credible. Confirm that every fit used the same observations and model components, that the models are genuinely nested for the stated test, and that the likelihood maximization converged adequately. Review parameter estimates and standard errors for instability, unexpected changes, or signs of weak identification. Examine the residual and prediction diagnostics appropriate to the NLME model; an improved OFV does not rescue a systematically inadequate structural or residual model.

A failed fit, a boundary estimate, a large standard error, or substantial sensitivity to initial values can make nominal likelihood-ratio p-values misleading. Sparse covariate ranges and correlated predictors can also make it hard to separate conditional effects. Do not treat a non-significant test as proof of no effect, or a selected coefficient as automatically stable. The earlier material on influence diagnostics and bootstrap stability describes related ways to assess fragility; here the essential point is that one selected model does not capture all uncertainty from the search.

The main alternatives answer different questions. A full-model likelihood-ratio test assesses the candidate set jointly, while sequential LRT selection asks whether additions or deletions meet chosen thresholds along a path. AIC balances fit and parameter count rather than applying a single p-value cutoff. Other approaches, including penalized or Bayesian selection, use different principles and should not be read as equivalent versions of stepwise testing. Whichever strategy is used, prespecify the candidate terms and decision rules where possible, report the search path, and interpret estimates in the context of the final model and its uncertainty.

• Mistake: “The full-model test is significant, so all three covariates are individually important.” Correction: The joint test only gives evidence that at least one candidate contributes under the model; individual conditional tests address different hypotheses.
• Mistake: “Forward selection omitted age, so age has no effect.” Correction: The term did not meet the selected entry rule at that step; correlated candidates, the threshold, and the path can affect that decision.
• Mistake: Comparing OFVs from fits with different observations or different random-effects structures. Correction: Such differences confound the covariate comparison; keep the data and non-covariate specifications fixed.