Why the Search Path Matters
A stepwise covariate search does not usually compare every possible model and then choose an overall best model. Instead, it moves through a sequence of models, adding or removing effects according to specified tests and thresholds. Each move changes the model against which the next candidate is evaluated. Consequently, the same dataset can lead to different final models if the search begins elsewhere or evaluates candidates in a different way.
This is called path dependence. It does not mean that the likelihood calculations are arbitrary. It means that the search procedure makes decisions using local comparisons, and those comparisons depend on the current model. A reported final model therefore reflects both the data and the search path used to reach it.
The issue is most visible when covariates carry overlapping information. A covariate may improve fit when considered alone but add little after a correlated covariate is already in the model. Conversely, a covariate that looks weak by itself may become useful after another effect is included. A test at one step is conditional on the model at that step; it is not a permanent score attached to the covariate.
Conditional Likelihood-Ratio Tests
Let \(M_0\) be a model without a candidate covariate, and let \(M_1\) be the nested model that adds it. Write \(\ell(M)\) for the maximized log-likelihood under model \(M\), using the same observations and likelihood specification in both fits. The likelihood-ratio statistic is
Under the null hypothesis that the added parameters are zero, and under the usual regularity conditions, \(G^2\) is approximately distributed as a chi-square random variable with degrees of freedom equal to the difference in the models’ numbers of estimated parameters. For example, adding one continuous covariate effect commonly gives \(df=1\); adding a categorical effect with three levels commonly gives \(df=2\), if it is represented by two parameters.
For the usual likelihood-ratio interpretation, the models must be nested and fitted to the same data with compatible likelihoods. Estimates must be obtained successfully, and the null parameter value must satisfy the regularity conditions for the chi-square approximation. Boundary hypotheses, such as testing whether a variance component is zero, can violate the usual reference distribution; such a test should not automatically be treated as an ordinary fixed-effect covariate test.
In a stepwise search, the relevant comparison changes with the current model. If \(A\) and \(B\) are candidate covariates, the statistic for adding \(A\) to a model already containing \(B\) is based on \(M_{AB}\) versus \(M_B\). It is not the same comparison as adding \(A\) to the null model \(M_0\).
Worked Example: Calculating Marginal and Conditional Tests
Suppose a hypothetical dataset is fitted using four nested models. \(M_0\) has neither covariate \(A\) nor \(B\); \(M_A\) and \(M_B\) each include one; \(M_{AB}\) includes both. All fits use the same observations and the same likelihood.
| Model | Maximized log-likelihood |
|---|---|
| \(M_0\) | -100.0 |
| \(M_A\) | -95.5 |
| \(M_B\) | -95.0 |
| \(M_{AB}\) | -94.4 |
Assume each covariate contributes one parameter, the tests use \(df=1\), and the prespecified inclusion threshold is \(G^2=3.84\). From the null model, adding \(A\) gives \(G^2=2[-95.5-(-100.0)]=2(4.5)=9.0\). Adding \(B\) gives \(G^2=2[-95.0-(-100.0)]=2(5.0)=10.0\). Both exceed 3.84.
After \(A\) is included, adding \(B\) gives \(G^2=2[-94.4-(-95.5)]=2(1.1)=2.2\), which is below 3.84. After \(B\) is included, adding \(A\) gives \(G^2=2[-94.4-(-95.0)]=2(0.6)=1.2\), also below 3.84. Thus, each covariate improves the null model, but neither passes this conditional threshold once the other is present. The arithmetic is consistent with the nesting: every larger model has a log-likelihood at least as high as its smaller nested model.
Candidate Ordering and Starting Models
Candidate ordering matters only when the search rule permits the order of evaluation to affect which move is made. For instance, a forward algorithm may scan a prespecified list and add the first candidate that passes the threshold, then restart its scan. If more than one candidate passes at the same step, reversing the list can change the first addition. By contrast, an algorithm that evaluates all eligible candidates and always adds the one with the largest statistic has a specified ranking rule; merely rearranging a list should not change its choice, except in a tie or implementation-dependent case.
Starting models create another route to path dependence. Forward inclusion starts with a limited model and evaluates additions. Backward elimination starts with a covariate-rich model and evaluates deletions. They need not make equivalent decisions: the forward test of \(A\) may be marginal, while the backward test of removing \(A\) is conditional on the other effects in the full model.
Worked Example: Candidate Order and Search Direction
Use the hypothetical log-likelihoods from the previous example, with a threshold of 3.84. Consider a forward procedure that scans candidates in list order, adds the first one whose \(df=1\) likelihood-ratio statistic exceeds the threshold, and then tests remaining candidates conditional on that addition.
If the list is \(A,B\), the first test gives \(G^2_A=9.0\), so \(A\) is added. The next test is \(B\) given \(A\): \(G^2_{B\mid A}=2.2\). Because \(2.2<3.84\), the procedure stops at \(M_A\). If the list is \(B,A\), the first test gives \(G^2_B=10.0\), so \(B\) is added. The test for \(A\) given \(B\) is \(G^2_{A\mid B}=1.2\), so this path stops at \(M_B\). These are two outcomes of the same explicitly order-sensitive rule, not two choices that both follow a smaller-statistic-first rule.
Now start instead from the full model \(M_{AB}\) and use backward elimination that removes the effect with the smaller deletion statistic. Removing \(A\) compares \(M_{AB}\) with \(M_B\), giving \(1.2\); removing \(B\) compares \(M_{AB}\) with \(M_A\), giving \(2.2\). The rule therefore removes \(A\), leaving \(M_B\). This backward path and the forward path with \(A\) first finish at different models. Every comparison uses the same fitted likelihood table and the same threshold.
The example also shows why the search rule must be stated precisely. “Add a significant covariate” is incomplete if several candidates pass: does the procedure add the first in a list, the one with the largest statistic, or all eligible effects? Likewise, a backward rule must say how it chooses among several removable effects. A deterministic rule applied to the same starting model and same inputs should give the same path; unexplained differences across runs suggest ties, numerical behavior, implementation differences, or undocumented choices.
Correlated Effects and Substitution
With correlated covariates, two effects may provide similar information about the outcome or a model parameter. When tested separately, each may capture much of that shared information. Once one is included, the second is judged by its additional contribution beyond the first, which may be small. This does not establish that the omitted covariate has no scientific relevance. It indicates that this dataset and model do not clearly separate its contribution from that of the included covariate under the chosen test.
Worked Example: Correlated Covariates Give Different Endpoints
In a hypothetical NLME analysis of 150 participants, renal function and age are strongly correlated. Suppose the fitted log-likelihoods for the null model, the two single-covariate models, and the joint model are respectively \(-250.0\), \(-245.8\) for age, \(-246.2\) for renal function, and \(-245.0\) for both. Assume one parameter per covariate and a \(df=1\) threshold of 3.84.
At the null model, the age statistic is \(2[-245.8-(-250.0)]=8.4\), and the renal-function statistic is \(2[-246.2-(-250.0)]=7.6\). Both pass. If age is added first, the conditional renal-function statistic is \(2[-245.0-(-245.8)]=1.6\), so the search stops with age. If renal function is added first, the conditional age statistic is \(2[-245.0-(-246.2)]=2.4\), so that search stops with renal function. Both conditional values are below 3.84.
This pattern is compatible with overlapping information, though the likelihood table alone does not prove why the effects overlap. It would be incorrect to conclude that age is universally “the true covariate” because its marginal statistic is larger, or that renal function has no relationship because it fails the conditional test after age. The endpoint depends on the path, and scientific interpretation requires considering the covariate definitions, model, and evidence beyond one selected model.
Recognizing and Handling Unstable Paths
Path dependence matters because a selected model can appear definite even when modest procedural choices change which covariate it contains. Useful diagnostics include rerunning the search from the null and a scientifically defensible richer model, recording the order in which candidates are evaluated, and tabulating both marginal and conditional statistics for competing effects. Also check that every comparison uses the same dataset, likelihood, and parameterization and that all fits converged to plausible solutions. Numerical failures can mimic selection instability.
A practical record of the search should include the starting model, candidate list and ordering, inclusion and removal thresholds, test degrees of freedom, the statistic at each decision, and the final model. If reasonable paths produce different models, report that dependence rather than presenting one endpoint as uniquely dictated by the data. This is evidence that model selection is uncertain or that the search rule is consequential; it is not, by itself, a formal test that any one model is correct.
Alternative approaches answer related but different questions. A prespecified model avoids choosing among candidates through a data-driven sequence, though its scientific specification still needs justification. A full model examines effects conditional on all included covariates, but its estimates and tests can be imprecise when effects overlap. Exhaustive comparison of a defined candidate set avoids dependence on a single stepwise route, but the number of models can grow rapidly and the selection process still requires interpretation. Penalized or Bayesian approaches use different selection or shrinkage rules; they do not make correlated information disappear.
No search strategy removes the underlying limitations of finite data, model misspecification, or weak information to distinguish correlated effects. Stepwise thresholds are decision rules, not guarantees of a scientifically unique covariate model. The earlier material on choosing between correlated covariates addresses the interpretation of that competition; here the central point is that the route through candidate models can determine which competitor remains.
A Compact Audit of a Stepwise Result
Record the starting model, candidate order, rules for choosing among eligible moves, thresholds, and degrees of freedom.
Check nesting, identical observations and likelihood definitions, successful fitting, and the appropriate reference distribution.
Check whether a different defensible start or candidate sequence changes the endpoint, especially for correlated covariates.
Report the model selected by the stated procedure and note material path sensitivity instead of implying that the endpoint is uniquely determined.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In one sentence, define path dependence in a stepwise covariate search.
- A nested model has log-likelihood \(-42.0\), and the model adding one covariate has log-likelihood \(-39.8\). Calculate the likelihood-ratio statistic and compare it with a threshold of 3.84.
- Why can a covariate pass its test from the null model but fail after another covariate is included?
- A forward algorithm always chooses the largest eligible statistic after evaluating all candidates. Should reversing the candidate list alone change its choice when the statistics are unequal? Explain.
- What should be reported if starting the search from the null and from a full model leads to different final covariates?
Solutions. (1) The search is path-dependent if changing the starting model, candidate-evaluation order, or allowed sequence can change its final selected model on the same data. (2) \(G^2=2[-39.8-(-42.0)]=2(2.2)=4.4\), which exceeds 3.84, so the added one-parameter effect passes that stated threshold, subject to the likelihood-ratio assumptions. (3) The first test is marginal, while the later one is conditional on the included covariate; overlapping information can make the additional improvement smaller. (4) No. With unequal statistics and the stated largest-statistic rule, list order alone should not change the selection; ties or implementation details may require an explicit rule. (5) Report the starting models, the applicable search rules, and the different endpoints, and describe the result as path-sensitive rather than uniquely determined by the data.