Choosing a Search Strategy
Covariate model building is not just a choice of which variables to include. It is also a choice of how to explore a candidate set: where the search begins, which models are feasible to fit, and what evidence will guide decisions. In a nonlinear mixed-effects (NLME) model, those choices matter because a model may be computationally difficult to fit, and a large number of observations within a few individuals does not necessarily provide strong information about between-subject covariate effects.
The exploratory plots and candidate assessments described in the earlier material on covariate candidate assessment can help prioritize plausible effects. The next decision is strategic: choose a search approach that fits the scientific purpose, the number and structure of candidates, the information in the study, and the ability to estimate the proposed model. No search algorithm can make a weakly supported or poorly specified model reliable.
For a fair comparison, the strategies should target the same scientific question and use compatible response, data, random-effects, and residual-error assumptions. A model comparison is difficult to interpret if the models also differ in how the outcome or random effects are defined. The candidate set should likewise be settled before searching: the number of candidate effects, their parameterization, and any scientific restrictions affect both computation and interpretation.
What Candidate Count Implies for Computation
Suppose there are \(K\) candidate covariate effects, and initially treat each effect as a single inclusion-or-exclusion decision. An exhaustive search over every subset would consider \(2^K\) models, including the model with no candidate effects. This grows rapidly. Forward and backward strategies search a sequence of nested models instead, but they may still require many fits because they evaluate multiple possible changes at each step.
In a forward search that could add all \(K\) effects, the maximum number of candidate additions evaluated across the successive steps is \(K+(K-1)+\cdots+1=K(K+1)/2\). A baseline fit is also needed. In a backward search, the corresponding maximum number of single-effect removals considered is also \(K(K+1)/2\), in addition to fitting the initial full model. These are counts of candidate model evaluations under this simple setup, not guarantees about run time: difficult NLME fits can take very different amounts of computation. A full-model joint comparison can require only a small number of model fits, but it depends on successfully estimating the full model. Shrinkage-based screening may also require repeated fits, for example when tuning a screening procedure or checking stability.
Worked Example: Comparing Search Workloads
Suppose a hypothetical candidate set contains \(K=8\) single-parameter covariate effects. Compare the maximum number of models involved in exhaustive search and in a complete forward search.
Exhaustive search. There are \(2^K=2^8=256\) possible subsets, including the no-covariate model.
Forward search. The maximum number of additions considered is \(8+7+6+5+4+3+2+1=36\). Including one baseline model gives up to \(36+1=37\) model fits in this simplified count.
Thus, exhaustive search considers 256 subsets, while the stated forward-search path evaluates at most 37 fits under these assumptions. This arithmetic illustrates a computational difference, not a promise that forward selection will find the scientifically best model. A multi-parameter covariate, failed fits, or repeated runs to assess stability can change the practical workload.
How the Strategies Differ
| Strategy | Starting point | Potential advantage | Key limitation |
|---|---|---|---|
| Forward selection | A simpler model, often with no candidate effects | Does not require fitting the full candidate model first | Can miss an effect that is weak alone but informative alongside another effect |
| Backward elimination | A model containing the candidate effects | Begins by considering the candidate set jointly | Requires the initial rich model to be estimable and adequately supported |
| Full-model testing | A model containing the candidate effects | Evaluates the candidate set jointly without a sequence of single-effect decisions | Can be unstable with many or correlated effects; a joint result may not identify a useful final model |
| Shrinkage-based screening | A fitted model or screening procedure that shrinks or ranks effects | Can prioritize a large set for follow-up | Screening rankings are not, by themselves, evidence that a covariate should be retained |
Forward selection is attractive when the full model may be too large or too difficult to fit. Its sequence can, however, depend on the order in which effects become useful: an effect that does not appear important at an early step may not be reconsidered in the same way as in a joint model. Backward elimination and full-model testing start from a richer model, so they can reflect relationships among candidates more directly if that model is estimable. Their shared disadvantage is that the model may be over-parameterized for the available information.
A full-model test asks whether a set of effects, considered together, contributes evidence against a model without that set. It does not automatically answer which individual effect is scientifically important, whether the model is stable, or whether the model is useful for the intended prediction or dosing decision. Shrinkage-based screening is different again: it is often best viewed as a way to reduce or prioritize a candidate set before more direct model evaluation, rather than as a final model-selection rule.
Sample Size and Information in an NLME Study
For covariate effects that explain between-subject differences, the number of independent individuals and the distribution of their covariate values are central to the information available. Repeated measurements can improve estimation of an individual's response pattern, but they are not equivalent to adding the same number of new individuals with independent covariate values. A study with many records but few individuals, little covariate variation, or poor covariate overlap may provide limited information for a complex covariate model.
There is no universal subjects-per-covariate rule that determines whether a particular NLME model is safe to fit. Support, collinearity, model complexity, the number of parameters used to represent each covariate, and the quality of the repeated data all matter. A multi-level categorical covariate, for example, can require several parameters rather than one. The relevant question is whether the data can identify and estimate the effects with adequate stability for the scientific purpose—not simply whether the record count looks large.
Worked Example: Many Records, Limited Subject-Level Support
Consider a hypothetical study with 48 individuals and 20 concentration records per individual, for \(48\times20=960\) records. Investigators are considering eight candidate effects on a population parameter. One candidate is a three-level categorical covariate, represented by two parameters; the other seven each use one parameter. How should the study size inform strategy choice?
The candidate effects require \(2+7=9\) parameters under this coding, not eight. Although there are 960 records, there are only 48 independent individuals supplying between-subject covariate information. The record count is \(960/48=20\) observations per individual; it does not turn the study into 960 independent individuals for those covariate effects.
The arithmetic does not establish that nine parameters are estimable or not estimable. Instead, it shows why the full model deserves careful feasibility checks: nine effects’ parameters must be distinguished using the available subjects and covariate support. If fitting all effects together is unstable, a narrower, scientifically justified candidate set or a staged screening plan may be more defensible than automatic backward elimination. The decision should also consider the covariate distributions and overlap, not just these counts.
Assumptions, Inference, and Diagnostics
Likelihood-based comparisons require comparable models fitted to the same data under compatible likelihood specifications. For a likelihood-ratio comparison of nested models, the usual chi-square reference distribution also relies on regularity conditions, including identifiable parameters and appropriate model regularity. A covariate coefficient fixed at zero is typically an interior null value in its parameter space, provided the coefficient is identifiable and the other regularity conditions hold. In contrast, testing a variance component equal to zero can violate the boundary condition because variances cannot be negative. The reference distribution then may not be the ordinary chi-square distribution.
Selection also affects interpretation. If the same data are repeatedly used to choose a covariate and to report its apparent evidence or effect size, the resulting summaries do not behave like results from a single pre-specified test. A selected model can make effects look more compelling than they would in an independent evaluation. The strategy should therefore be described as part of the analysis plan, and inferential claims should acknowledge the selection process.
Warning signs include failure of the rich model to converge reliably, implausible or poorly determined parameter estimates, high sensitivity to starting values, and large changes in selected effects after small changes to the candidate set. A failed fit is not evidence that a covariate has no effect. It may indicate inadequate support, excessive model complexity, or another estimation problem. Likewise, a stable-looking ranking from a screening method does not prove that the ranking is reproducible.
Worked Example: Interpreting a Nested-Model Comparison
In a hypothetical analysis, a reduced model has log likelihood \(-128.4\), while a nested model adds one identifiable fixed covariate coefficient and has log likelihood \(-125.9\). Assume both models use the same observations and otherwise compatible specifications. Illustrate the likelihood-ratio statistic and the conditions for interpreting it.
The statistic is $$ D=2\{\ell_{\text{larger}}-\ell_{\text{reduced}}\} =2\{-125.9-(-128.4)\} =2(2.5)=5.0. $$
Under the null hypothesis that the added coefficient is zero, and assuming the usual regularity conditions, the reference distribution is approximately chi-square with one degree of freedom. The upper-tail probability is \(P(\chi^2_1\geq5.0)\approx0.0253\). This is evidence against that null under the stated model and reference assumptions; it is not the probability that the null hypothesis is true.
Here the null value zero for the fixed coefficient is ordinarily interior to its parameter space. That fact alone is not enough: the coefficient must also be identifiable and the remaining conditions for the comparison must be reasonable. This calculation illustrates one possible comparison within model building; it does not make a data-driven selection result equivalent to a pre-specified confirmatory test.
Matching the Strategy to the Scientific Purpose
The strategy should follow the objective established for the covariate model. If the purpose is to evaluate a small set of strong, pre-specified scientific hypotheses, a focused comparison may be preferable to searching a broad candidate list. If the aim is to reduce a large exploratory set, a screening stage may help organize follow-up work, but the final model still needs substantive and statistical evaluation. If the full candidate model is stable and the question concerns the candidates jointly, full-model testing may be appropriate. When the full model is not feasible, forward search may be a practical alternative, with its path dependence stated clearly.
These approaches are not interchangeable guarantees of a uniquely correct model. Search methods can produce different selected covariates because they evaluate different subsets and rely on different decisions. Scientific plausibility, covariate support, model adequacy, and the intended use of the model should remain central. A model chosen for explaining a parameter may not be the best model for prediction, and a computationally convenient path does not itself establish scientific relevance.
Common Pitfalls
- Counting records as independent subjects. Repeated observations are valuable, but do not automatically supply independent between-subject covariate information.
- Assuming a full-model approach is always more rigorous. A rich model can be poorly supported or difficult to estimate; joint evaluation does not remove those problems.
- Treating a screening rank as a final decision. A ranking prioritizes candidates; it does not by itself establish a stable effect or a suitable final model.
- Calling zero a boundary for an ordinary fixed covariate coefficient. Zero is typically interior for an identifiable coefficient. A zero variance component is a common boundary case and can require different likelihood-ratio reference theory.
- Reporting selected evidence as if the model had been pre-specified. Reusing the data to search and then report unqualified evidence can understate the uncertainty created by selection.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- With five single-parameter candidates, how many subsets are in an exhaustive search, including the model with no candidates?
- Why might 1,200 records from 60 individuals not provide the same information about between-subject covariate effects as 1,200 independent individuals?
- What is one important feasibility requirement for backward elimination or full-model testing?
- Is a zero fixed covariate coefficient typically a boundary value? What different issue can arise when testing a zero variance component?
Solutions. (1) \(2^5=32\) subsets. (2) The repeated records are clustered within 60 individuals; the between-subject covariate information depends importantly on those individuals and their covariate support, not simply the total record count. (3) The initial rich model must be estimable with adequate support; otherwise, unstable or failed fits can undermine the search. (4) Zero is typically an interior value for an identifiable fixed coefficient, subject to the other regularity conditions. A variance component constrained to be nonnegative is at its boundary when it equals zero, so the ordinary chi-square reference distribution may not apply.