Why Reconsider Effects After Adding Covariates?
Forward inclusion asks whether an excluded covariate improves the current model. But each addition changes the model against which the remaining effects are assessed. A covariate that looked useful earlier may contribute little once another covariate is present. Bidirectional stepwise search responds to this by alternating forward additions with backward checks of effects already included.
This is not the same as fitting a single covariate-rich model and removing effects once. The model is refitted as the search proceeds, and an effect’s support is assessed conditionally on the other effects currently in the model. A change in support is not automatically proof that the earlier effect was spurious: correlated covariates can share information, and the result depends on the candidate set, the test criteria, and the path through the models.
The Model Comparisons That Drive the Search
Suppose the current NLME model has log likelihood \(\ell_0\), and a proposed model adds a covariate effect while leaving the response model, analysis records, and random-effects structure unchanged. If the models are nested and fitted by maximum likelihood, the likelihood-ratio statistic is
where \(\ell_1\) is the maximized log likelihood of the larger model. Under the null hypothesis that the added fixed-effect parameter or parameters are zero, and under the usual regularity conditions, \(D\) is approximately distributed as a chi-square random variable with degrees of freedom equal to the number of added parameters. The p-value is the upper-tail probability from that reference distribution. These conditions include a well-defined nested comparison, adequate model fitting, and a null value that is not on a problematic parameter-space boundary.
For a backward test, the larger model is the current model and the smaller model omits the effect under review. The statistic is still twice the log-likelihood difference, larger minus smaller. Each comparison must use the same data and likelihood basis. A failed fit, a changed analysis population, or a changed random-effects structure can make the difference uninterpretable as the intended covariate test.
A Practical Search Cycle
Before fitting, specify the candidate effects and how each is represented. A categorical covariate may add several parameters and should be treated as one effect when the scientific question concerns the factor as a whole. Preserve any required model hierarchy—for example, do not retain an interaction while omitting its required main effects. Use the entry and removal rules chosen for the analysis; those rules need not be identical.
Begin with the prespecified base model, including its response, fixed effects, random effects, and analysis records.
Fit one model per eligible excluded effect. Compare each with the current model and add an effect that meets the entry rule, typically choosing the strongest qualifying candidate according to the prespecified criterion.
After an addition, compare the current model with models omitting each removable effect in turn. Remove an effect that fails the retention rule, refit, and reassess remaining included effects.
Return to forward testing after backward changes. Stop when no eligible excluded effect qualifies for addition and no included effect qualifies for removal.
A practical implementation should define how it handles ties, failed fits, and hierarchy constraints before searching. If a removal makes an excluded effect eligible again, the entry and removal rules should be coordinated so the procedure does not cycle between the same models.
Worked Example: One Forward Addition and a Backward Check
Consider a hypothetical search with a base model log likelihood of \(-120.0\). Two eligible one-parameter covariates, A and B, are tested using maximum likelihood on the same records and with the same random-effects structure. For this illustration, use an entry and retention cutoff of \(0.05\). The fitted log likelihoods are hypothetical.
Adding A gives \(\ell_A=-116.0\), so
With 1 degree of freedom, the chi-square upper-tail p-value is approximately \(0.0047\). Adding B alone gives \(\ell_B=-117.0\), so \(D_B=2[-117.0-(-120.0)]=6.00\), with \(df=1\) and \(p\approx0.0143\). Both meet the entry rule, and A has the smaller p-value, so add A first.
With A in the model, adding B gives \(\ell_{AB}=-113.8\). The conditional comparison is
with \(df=1\) and \(p\approx0.0360\). B qualifies for addition. Now reconsider A by comparing the two-effect model with the B-only model:
This has \(df=1\) and \(p\approx0.0114\), so A remains. If another eligible covariate C gave a conditional statistic \(D_{C\mid AB}=0.80\), its \(df=1\) p-value would be about \(0.371\), so it would not qualify. With no other additions or removals, the search stops at the model containing A and B.
The comparison for A is not its original test against the base model. It asks whether A contributes after B is already included. Here the evidence for A is weaker than in its first test, but it still meets the stated rule.
How an Earlier Effect Can Lose Support
A later covariate can explain variation that was previously attributed to an earlier effect. This is especially plausible when covariates are associated or encode overlapping aspects of the same physiology. The relevant backward test asks whether the earlier effect adds enough information after the later effect is present. Its likelihood-ratio statistic can therefore be smaller than the earlier forward statistic.
The following hypothetical fits isolate that reconsideration. Suppose effect A has three parameters and B has one. The base model has log likelihood \(-100.0\); the A-only model has \(-95.9\); the B-only model has \(-96.5\); and the model with both has \(-93.8\). These comparisons are nested fits on the same data with the same random-effects structure.
Worked Example: Reassessing a Multi-Parameter Effect
Against the base model, adding A gives \(D_A=2[-95.9-(-100.0)]=8.20\), with \(df=3\) and \(p\approx0.042\). In the model that already contains A, adding B gives
with \(df=1\) and \(p\approx0.040\). If B meets the entry rule, it is added. Reconsider A by comparing the both-effects model with the B-only model:
Here \(df=3\), and the chi-square upper-tail p-value is approximately \(0.145\). Under an illustrative removal rule of \(p>0.10\), A would be removed, while B would remain if its conditional test meets the retention rule. The results show how A’s conditional support can be lower than its initial support. They do not establish that B caused the change or that A has no biological relevance.
This table illustrates the backward calculation, not a complete forward path: if both effects had been eligible from the outset, the initial candidate tests would also have to be considered in deciding which to add first. In an actual search, record the full sequence and apply the rules to the eligible candidates at each step.
Degrees of Freedom and a Common Mistake
The degrees of freedom depend on how many parameters the tested effect adds, not on the number of covariate names in the model. For example, a factor represented by two indicator parameters contributes two parameters to its joint test. The test must compare the model with the entire factor to the model without it; testing only one indicator does not answer the same question.
Worked Example: Using the Correct Reference Distribution
Suppose adding a hypothetical three-level factor requires two parameters, and the larger model’s log likelihood is \(-77.2\) compared with \(-80.0\) for the model without the factor. Then
The factor test has \(df=2\), so \(p=P(\chi^2_2\ge5.60)\approx0.0608\). At a \(0.05\) cutoff this comparison does not meet the criterion. Treating the factor as a one-parameter effect would instead use \(df=1\) and give a different p-value; that would be the wrong test because two parameters were added. The conclusion is about this joint model comparison, not about whether the factor is clinically important.
Diagnostics, Alternatives, and Limits
A search is only as trustworthy as its model fits and comparisons. Check that all compared fits converged to usable solutions, that the likelihoods are finite, and that parameter estimates and standard errors are plausible. A small change in the optimizer’s starting values that produces a different likelihood can alter a stepwise decision. Confirm that the dataset, response definition, likelihood method, and random-effects structure did not change across candidate fits.
Also inspect whether selected effects have adequate covariate support and whether collinearity makes their estimates unstable. A change in a p-value after another effect enters can be a sign of shared information, but it can also arise from sparse covariate patterns, weak identification, or numerical difficulty. Review the estimated effects and relevant model diagnostics; do not use the selection p-values as a substitute for checking model adequacy.
Bidirectional search combines advantages of forward inclusion and backward elimination: it can begin with a manageable model while allowing earlier additions to be reconsidered. Forward-only search never removes an entered effect; backward-only search requires fitting a sufficiently rich starting model and does not revisit effects excluded at the outset. A prespecified full model may be preferable when scientific questions require simultaneous conditional estimates, while a small hypothesis-driven model may be preferable when the candidate set is already narrow. These approaches answer different modeling objectives; no search guarantees the uniquely correct covariate model.
Stepwise selection has important limitations. Its path depends on the candidate set and eligibility rules; repeated testing makes ordinary p-values after selection poor summaries of uncertainty; and modest changes in data or criteria may change the selected model. Treat the result as a model-building aid, document all tested comparisons, and assess whether the final model is scientifically interpretable and diagnostically adequate. Do not interpret inclusion as causal evidence.
Common Pitfalls
- Testing an earlier effect only once. Its initial test is against the base model; after other effects enter, its relevant test is conditional on them. Refit and reconsider it.
- Comparing fits that changed in more than one way. If the random-effects structure or analysis records also change, the likelihood difference cannot be attributed solely to the candidate covariate.
- Using the wrong degrees of freedom. Count the parameters added by the effect as represented in the model.
- Calling a selected effect “proved.” Selection indicates that a comparison met a chosen rule; it does not establish causality, clinical importance, or stability in new data.
- Ignoring failed or unstable optimization. Do not interpret a likelihood-ratio statistic from a fit that did not produce a reliable maximum.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A current model has log likelihood \(-50.0\). Adding a one-parameter effect gives \(-47.5\). Calculate the likelihood-ratio statistic and state the degrees of freedom.
- Why should an effect entered earlier be retested after another covariate is added?
- A factor adds three parameters. Which degrees of freedom should be used for its joint likelihood-ratio test?
- Name two checks to make before interpreting a covariate comparison in an NLME search.
Solutions. (1) \(D=2[-47.5-(-50.0)]=5.00\), with \(df=1\). (2) The new model changes the conditional comparison: the earlier effect must be tested against the model containing the later covariate, since the effects may share information. (3) Use \(df=3\), because the effect adds three parameters. (4) Examples include confirming both fits converged reliably and checking that they use the same records, likelihood basis, and random-effects structure.