Ensuring Fair Comparisons During Covariate Selection

Why Fair Comparisons Matter

Covariate selection asks whether a candidate model describes the data better in a way that justifies its added complexity. A numerical difference between two objective functions does not answer that question by itself. The difference is interpretable only when it comes from the model change of interest, rather than from a change in which observations were analyzed, how the likelihood was evaluated, or whether one fit stopped at a poorer optimum.

This issue is especially important in nonlinear mixed-effects models. A model’s likelihood depends on its fixed effects, random-effects distribution and covariance parameters, residual model, and the individual data records included in the analysis. Optimization is also more demanding than in many simple regression models. A fair comparison therefore requires more than running two fits and subtracting their reported objective values.

What Must Be Held Constant

For subject \(i\), let \(y_i\) denote the observed data and let \(f_i(y_i\mid\theta)\) be its marginal likelihood contribution after accounting for the subject-specific random effects. The total log likelihood is \( \ell(\theta)=\sum_{i=1}^{N}\log f_i(y_i\mid\theta) \). Here \(\theta\) collects all estimated parameters, including population parameters, random-effects covariance parameters, and residual parameters. In practice, a software package may evaluate or approximate this likelihood using a specified estimation method. The method and its relevant settings must be consistent across the candidate fits.

• Same records and likelihood contributions. Use the same subjects and observations, with the same handling of missing data, censoring, event records, and likelihood contributions. A covariate model that silently removes records with missing covariate values is not directly comparable by a likelihood-ratio test with a model fitted to more records.
• Same likelihood and estimation settings. Keep the estimation method, likelihood approximation, data handling, and relevant numerical settings consistent. If the software reports an objective such as \(-2\ell\), verify that it represents the same criterion in both fits.
• Same non-covariate structure. Keep the structural model, random-effects structure, residual model, and other covariate relationships fixed when the intended question concerns one covariate change. If several features change, the comparison cannot attribute the difference to just one of them.
• Successful optimization of each model. Check convergence messages and available diagnostics. A converged status is useful but not sufficient: inspect gradients, parameter estimates, standard errors, and whether estimated covariance matrices are admissible. Implausible estimates or unstable results across reasonable initial values can signal that the reported fit is not a reliable optimum.

The parameters must be re-estimated in every candidate model. Suppose a covariate is removed. It is not fair to hold the other population and variance parameters at their values from the larger model: those values were estimated under a different model, and the best-fitting values may change when the covariate is absent. The same principle applies when a covariate is added. Refit the full candidate model rather than evaluating it with old parameter estimates.

Likelihood-Ratio Comparisons and Their Conditions

For nested models fitted by maximum likelihood to the same data under the same likelihood evaluation, let \(\ell_R\) be the maximized log likelihood for the reduced model and \(\ell_F\) the maximized log likelihood for the full model. The likelihood-ratio statistic is \( \Lambda=2(\ell_F-\ell_R) \). Equivalently, if the reported objective is \(-2\ell\), subtract the full-model objective from the reduced-model objective. Under the null hypothesis that the added parameters are zero, and under regularity conditions, \(\Lambda\) is approximately distributed as chi-square with degrees of freedom equal to the number of additional free parameters.

The comparison requires nested models: every parameter configuration allowed by the reduced model must also be allowed by the full model. The chi-square approximation also relies on regularity conditions, including an interior null value and an identifiable model. A covariate effect tested at zero often satisfies this condition, but tests involving variance components at a boundary, or poorly identified parameters, may not. The standard chi-square reference distribution should not be used automatically in such cases.

A likelihood-ratio test addresses a specified nested comparison; it does not establish that the selected covariate model is uniquely correct or that the covariate has a causal effect. Its p-value is calculated under the null model and the test assumptions. For an ongoing stepwise search, it is also conditional on the comparison being made; it does not account automatically for all the searches already performed.

Re-estimation and Optimization Checks

Even when the data and settings match, a comparison can fail if one model is not adequately optimized. A failed or shallow optimization may make the log likelihood too low, exaggerating or concealing the difference between candidate models. Use consistent convergence criteria and estimation settings, and retain enough fit information to verify what the software actually estimated.

If a model does not converge, do not treat its reported objective as an ordinary comparable maximum. Review the model specification and data, use sensible alternative initial values where appropriate, and refit under the same settings. If estimates change substantially across starts, or a parameter is weakly identified, the fit may not support a dependable selection decision. Changes in objective-function evaluation caused by a different estimation method or approximation also make a raw subtraction difficult to interpret as a likelihood-ratio statistic.

Information Criteria and Other Comparisons

Not every useful comparison is a likelihood-ratio test. For models fitted to the same data with comparable likelihood definitions, the Akaike information criterion is \( \mathrm{AIC}=-2\ell+2k \), where \(k\) is the number of estimated parameters. AIC trades fit against a parameter-count penalty and can compare nonnested models. Lower AIC indicates the preferred trade-off among the candidates considered; it is not a significance test and does not prove that the selected model is true. The same fairness requirements about data, likelihood evaluation, and reliable optimization still apply.

Wald statistics, screening approaches, and visual or predictive assessments answer different questions and may help assess a candidate covariate from other perspectives. They do not make an unfair likelihood comparison fair. In particular, a Wald test from one fitted model is not a substitute for refitting and evaluating both models when the intended evidence is a likelihood difference.

Common Pitfalls and Limitations

• Comparing fits on different records. A lower objective may simply reflect a different collection of likelihood contributions. Align the analysis set before comparing.
• Reusing parameter estimates. Holding other parameters fixed after changing the covariate model fails to compare the best-fitting versions of the candidates. Re-estimate every free parameter in each model.
• Trusting a convergence label alone. A nominally converged fit can still have poor gradients, implausible estimates, or an invalid covariance estimate. Inspect diagnostics and investigate sensitivity to starting values.
• Subtracting incompatible objective values. Different estimation methods or likelihood approximations may yield different reported criteria. Confirm that the objective has the same meaning and settings in both fits.
• Using the chi-square reference automatically. The standard LRT approximation depends on nesting and regularity. Boundary parameters or weak identification can invalidate the usual reference distribution.
• Interpreting a selected model as definitive. A valid comparison supports a limited statement about the candidate models and data. It does not establish causal effect, clinical relevance, or certainty that the selected model is best among all possible models.

Fair comparisons reduce avoidable artifacts in covariate selection, but they do not eliminate uncertainty from model search or guarantee that the likelihood approximation is adequate. The conclusion should therefore describe the particular models, data, and estimation conditions used.