Why Covariate-Selection Methods Can Disagree
Covariate selection rarely produces a single, unambiguous verdict. A likelihood-ratio test may favor adding a covariate, a screen based on individual empirical Bayes estimates may not, a bootstrap may select the effect in only some resamples, and cross-validation may show little predictive gain. These results are not necessarily contradictory: each method measures a different feature of the evidence.
The likelihood comparison asks whether the observed data support a better fit under a specified larger model, subject to assumptions about the models and test. A shrinkage-based screen asks whether estimated individual effects show a pattern associated with the covariate, but those estimates may be pulled toward the population mean. Bootstrap selection frequency describes how often a prespecified selection procedure chooses the effect under resampling. Predictive validation asks whether the model predicts held-out observations better. None of these quantities alone answers every scientific question.
What Each Source of Evidence Says
For a likelihood-ratio test, let \(M_0\) be a reduced model and \(M_1\) a larger model that contains all the parameters in \(M_0\) plus the covariate effect being tested. The models must be fitted to the same data using comparable likelihoods. The statistic is twice the difference in maximized log likelihoods. Under the null hypothesis, and when the usual regularity conditions hold, it is approximately chi-squared with degrees of freedom equal to the number of added parameters. For one added parameter, the reference distribution has \(df=1\). This approximation can be unreliable for nonregular problems, weak identification, or parameters on a boundary; the model and test conditions therefore matter.
A small p-value is evidence against the reduced model under the test assumptions; it is not the probability that the covariate has no effect. Also, testing many candidate effects can make a nominal p-value overstate the strength of the evidence from the overall search. Earlier material on multiplicity and winner’s curse is especially relevant when the covariate was selected after examining many alternatives.
A shrinkage-based screen commonly examines whether individual parameter estimates or empirical Bayes estimates appear related to a covariate. In an NLME model, individual estimates are informed by both the subject’s data and the population distribution. When individual information is weak, estimates are pulled toward the population value. Thus, weak association in shrunken estimates is not equivalent to a formal test showing that the population covariate effect is zero. Conversely, a visible pattern in estimates does not by itself establish a reliable population effect.
A bootstrap selection frequency is the proportion of bootstrap data sets in which a stated procedure selects the covariate. It reflects sensitivity to resampling and to the selection algorithm, including its thresholds and candidate set. It is not a posterior probability that the effect is true, nor does a high frequency guarantee clinical importance. Bootstrap resampling should respect the independent sampling units: for repeated observations within subjects, resample subjects and retain each selected subject’s observation structure rather than treating records as independent.
Cross-validation evaluates performance on data not used to fit the corresponding model. Its result depends on the prediction target, the loss function, the split or fold scheme, and how the entire model-building procedure is repeated. A small average improvement may be practically negligible, while variation across folds may make the direction uncertain. Predictive validation is not a test of whether a covariate is biologically real; it assesses prediction for the validation setting and target.
A Structured Adjudication Process
Decide whether the model is intended primarily to describe population differences, support interpretation of a plausible effect, or predict outcomes for a defined target population. These objectives can favor different decisions.
Check that the likelihood models use the same data and are nested for the likelihood-ratio test. Examine covariate range, subgroup counts, missingness, influential subjects, and whether the proposed effect is identifiable from the observed design.
Record the likelihood statistic and its conditions, the shrinkage screen’s limitations, the bootstrap procedure and frequency, and the validation metric with its uncertainty and practical scale.
Ask whether the direction and plausible size make sense, whether the effect matters for decisions, and what harm follows from including a weak effect or omitting a credible one.
Choose a model that fits the stated objective, but document competing evidence, sensitivity analyses, and any limitation. If conclusions depend strongly on one analysis choice, report that instability rather than disguising it.
Worked Examples
Worked Example: A Significant Likelihood Test but Little Predictive Gain
Hypothetical setting. A covariate \(C\), measured in kg, is added as one fixed effect to a population clearance model. The reduced model has maximized log likelihood \(-812.4\), and the larger model has maximized log likelihood \(-809.6\). Both models use the same subjects and observations, and the larger model differs only by the one prespecified covariate coefficient. Assume the coefficient is an ordinary interior parameter and the likelihood-ratio approximation is appropriate.
The likelihood-ratio statistic is
With one added parameter, \(df=1\), giving \(p\approx0.018\). At a prespecified \(\alpha=0.05\), this is evidence against the reduced model under the test assumptions. It does not establish that the effect is clinically important.
Suppose five-fold cross-validation gives a 0.2% reduction in prediction error for the larger model, with a fold-to-fold standard error of 1.1 percentage points. The average improvement is small relative to its uncertainty and may not matter for the intended prediction task. The two findings address different questions: the likelihood test detects evidence of improved in-sample fit, whereas cross-validation estimates out-of-sample predictive performance.
If prediction is the primary objective and a 0.2% improvement is immaterial, the reduced model may be preferred for parsimony, while the likelihood evidence and plausible effect are reported as unresolved. If the aim is population description and the covariate is scientifically justified, retaining it may be defensible, with the caveat that predictive benefit was not demonstrated. The decision should not be made by declaring one method universally superior.
Worked Example: A Weak Shrinkage Screen but Stable, Supported Effect
Hypothetical setting. Investigators examine a continuous covariate \(C\) as a potential predictor of an individual random effect. Individual empirical Bayes estimates show little visible trend with \(C\), but the diagnostic reports 65% shrinkage for that random effect. A prespecified one-parameter likelihood comparison gives \(LR=4.20\), with \(df=1\) and \(p\approx0.040\). In 500 subject-level bootstrap resamples, the same selection procedure retains the covariate in 390 resamples.
The bootstrap frequency is
or 78.0%. This indicates that the procedure selects the covariate in a substantial fraction of these resamples; it is not a 78% probability that the effect is real. The likelihood result is nominally significant at \(\alpha=0.05\), provided its nesting and regularity assumptions hold.
The weak empirical Bayes trend should be interpreted in light of 65% shrinkage: individual estimates carry limited subject-specific information and are strongly pulled toward the population mean. It is therefore not decisive evidence against an effect. Suppose, in addition, that the covariate has a prespecified mechanistic rationale and adequate coverage across the observed range. The combined evidence supports considering the effect in the population model, while the modest bootstrap frequency short of near-universal selection signals some instability. The appropriate report distinguishes the population-level likelihood evidence from the weak individual-estimate screen and avoids claiming certainty.
Worked Example: Frequent Selection with Weak Support and Worse Validation
Hypothetical setting. A binary covariate identifies a subgroup containing 8 of 240 subjects. A stepwise procedure selects it in 430 of 500 bootstrap resamples, so its selection frequency is
or 86.0%. The procedure repeatedly selects the effect, but the small subgroup means the estimated contrast may depend heavily on a few subjects. Further, suppose that removing the most influential subgroup subject changes the estimated covariate effect by 55%, and cross-validation shows a 3.0% increase in prediction error when the covariate is included, with a standard error of 1.0 percentage point.
The validation result favors the model without the covariate for the stated predictive target: the larger model’s error is higher by three estimated standard errors. The bootstrap frequency does not overturn that result. A frequently selected effect can arise because a particular selection algorithm repeatedly exploits a pattern in the resampled data; selection frequency does not measure subgroup representativeness or guarantee external predictive value.
With only 8 subgroup members, substantial influence from one subject, and no strong scientific justification supplied, a defensible predictive model would omit the effect or treat it as exploratory. The report should note the sparse support and the instability, and should not generalize the estimated subgroup contrast confidently. If the effect is important scientifically, collecting more subgroup data may be more appropriate than treating selection frequency as a resolution.
Why the Results Differ: A Short Justification
The likelihood and validation criteria differ because fitting a larger nested model cannot reduce its maximized in-sample likelihood: the reduced model is available as the special case in which the added coefficient is zero. Therefore, the observed likelihood improvement is nonnegative. But this fitting advantage does not guarantee better prediction on new observations. The larger model estimates an additional parameter, and that estimate can reflect sampling noise. Validation evaluates predictions on observations not used for that fit, so the apparent in-sample gain may shrink or reverse.
The inequality justifies why the likelihood difference is nonnegative for correctly nested models; it does not establish that the added effect is scientifically real or predictively useful. The likelihood-ratio test calibrates the gain under a null model and regularity assumptions. Bootstrap frequency instead records the behavior of a selection rule under resampling. A shrinkage screen reflects the quality of individual estimates. Cross-validation targets out-of-sample loss. Their disagreement is therefore expected whenever the signal is weak, the sample is limited, individual estimates are noisy, or the target differs.
Comparison, Diagnostics, and Limitations
| Evidence source | Primary question | Important caution |
|---|---|---|
| Likelihood-ratio test | Does a specified larger nested model improve fit enough to challenge the reduced model? | Requires comparable likelihoods, valid nesting, and an adequate reference distribution; does not measure clinical importance. |
| Shrinkage-based screen | Do estimated individual effects appear associated with the covariate? | Strong shrinkage can obscure patterns; an apparent pattern is not itself a formal population-effect test. |
| Bootstrap selection frequency | How often does this specified procedure select the effect across resampled data sets? | Depends on the resampling unit, algorithm, thresholds, and candidate set; it is not a probability that the effect is true. |
| Predictive validation | Does the model improve predictions for the defined validation target and loss? | Depends on the target, folds, metric, and repetition of the full selection procedure; it does not test biological truth. |
Several checks help identify when a result should be treated cautiously:
- Unequal analysis data: If candidate models use different observations, the likelihood comparison may reflect changed data rather than only the covariate.
- Weak support: Sparse categories, narrow covariate ranges, or little overlap can make an effect difficult to identify and sensitive to a few observations.
- Unstable fitting: Convergence problems, implausible parameter estimates, or large changes under reasonable starting values undermine a nominal test result.
- Misleading resampling: Resampling individual records instead of subjects when records are clustered can exaggerate the apparent amount of independent information.
- Validation leakage: Selecting covariates using all observations before cross-validation lets information from held-out data influence selection. The selection procedure must be repeated within each training fold to estimate the predictive performance of the procedure.
- Unclear objective: A model chosen for prediction need not be the model that best summarizes a plausible population relationship. State which goal governs the decision.
No adjudication rule removes uncertainty. A covariate can be scientifically plausible but weakly supported by the available design; it can improve fit without materially improving prediction; or it can be predictively useful without a convincing causal interpretation. The model should be presented as an analysis choice supported by specified evidence, not as an automatic verdict delivered by a single threshold.
Common Mistake
Wrong reasoning: “Three methods selected the covariate and one did not, so the majority vote proves it belongs in the model.” This treats different targets as equivalent and ignores the quality of the evidence and the consequences of the decision.
Correction: Identify what each result measures, check its assumptions and data support, and decide which objective governs model choice. Report meaningful disagreement. For example, a credible likelihood result can support a population effect even when noisy empirical Bayes estimates show little trend, while poor validation can still argue against using that effect in a prediction-focused model.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A one-parameter nested covariate model has \(LR=3.84\), with valid regularity conditions. What is the approximate p-value, and what does it mean at \(\alpha=0.05\)?
- A covariate is selected in 72 of 100 subject-level bootstrap resamples. State the selection frequency and one conclusion it does not justify.
- A shrinkage-based screen shows little association, but shrinkage is high. Why is that screen not decisive evidence against a population covariate effect?
- A model improves cross-validated prediction slightly, but the uncertainty in the improvement is larger than its mean. What should be reported, and why does a significant likelihood test not settle the prediction question?
Solutions. (1) For \(df=1\), \(LR=3.84\) gives \(p\approx0.050\). It is approximately at the 0.05 threshold; because the rounded statistic is borderline, the unrounded p-value and prespecified decision rule should be used. This is evidence against the reduced model under the test assumptions, not the probability that the null is true. (2) The frequency is \(72/100=0.720\), or 72.0%. It does not justify saying there is a 72% probability the effect is real. (3) High shrinkage means individual estimates are strongly pulled toward the population mean, so their weak apparent association may reflect limited individual information rather than absence of a population relationship. (4) Report the estimated predictive improvement and its uncertainty, noting that the evidence for a practically reliable gain is weak. A likelihood test concerns in-sample model fit under its assumptions, whereas cross-validation evaluates prediction on held-out data.