Why Control the Change?
An NLME model-development decision is easier to understand when a comparison answers a specific question: what changed, and what happened when it changed? If one candidate adds a covariate, another changes the residual-error model, and a third changes both, a better fit for the third candidate does not reveal which change mattered. It may be the covariate, the error model, their combination, or differences in the data or fitting procedure.
The purpose of a controlled comparison is not to guarantee that one model is true or that a numerical fit measure settles the decision. It is to make the evidence attributable and reviewable. A traceable candidate-run registry supports this work by recording each run; the comparison adds an explicit account of which runs form a contrast and why.
In NLME work, a candidate may differ in structural model, random-effects structure, residual-error model, covariate terms, or another planned specification. These are distinct model features, even though they can affect one another. A comparison should state the reference candidate, the changed feature, and the outcome being assessed. For example: “Relative to candidate A, candidate B adds a proportional residual-error component; both use the same analysis dataset, likelihood, and estimation settings.”
Conditions for a Meaningful Contrast
Before interpreting a difference, check that it is not simply a difference in the comparison setup. For likelihood-based comparisons, both runs should use the same observations, response scale, likelihood definition, and treatment of missing or censored data. Fit measures such as the objective function are not automatically comparable across different data sets or likelihood definitions. Likewise, a run that did not converge reliably cannot serve as sound evidence that its model specification fits worse.
When the fitting method is stochastic or sensitive to initial values, identical settings do not guarantee identical numerical results. Record settings that can affect the result, use suitable initial estimates, and consider whether repeated or otherwise checked fits are needed to assess numerical stability. A change in optimizer settings may be necessary to fit a model, but then the comparison no longer isolates only a scientific model change unless both candidates are also refitted under aligned conditions.
A useful comparison plan identifies:
- Reference: the candidate against which the change is evaluated.
- Single change: the feature changed, stated narrowly enough to distinguish it from other modifications.
- Held-constant conditions: data version, analysis population, response definition, likelihood, and fitting settings.
- Evidence to review: fit summaries and relevant parameter estimates or predictions, chosen before interpreting the result.
- Decision and limitation: what the comparison supports, and what it does not establish.
Worked Example: Reading a One-Change Sequence
Suppose three hypothetical runs use the same 240 observations, the same likelihood and estimation settings, and each fit is checked as numerically reliable. Candidate A has a baseline structural model with an exponential residual-error model. Candidate B adds a prespecified weight covariate to clearance, leaving the other features unchanged. Candidate C adds a second prespecified covariate to B. Their reported objective function values (OFVs, with lower values indicating a larger maximized likelihood under this common likelihood) are 1,200, 1,192, and 1,189.
Contrast A to B. The change is the weight covariate alone. The OFV difference is \(1200-1192=8\). Under the stated conditions, B has an OFV 8 units lower than A. This describes improved likelihood fit for this change; by itself it does not establish that the covariate is scientifically important or that B is the preferred final model.
Contrast B to C. The second covariate is the only specified change. The OFV difference is \(1192-1189=3\), so C has an OFV 3 units lower than B. The effect of adding the second covariate is therefore assessed against B, not by comparing C with A and attributing the full difference of 11 to the second covariate.
The sequence supports two separate descriptive statements: adding the weight covariate changed the OFV by \(-8\), and then adding the second covariate changed it by \(-3\). It does not show that the two changes have independent effects in all model contexts; the second comparison is conditional on the first change already being present.
Why Bundled Changes Obscure Attribution
A bundled comparison changes two or more features at once. If the fit improves, the observed difference belongs to the whole bundle, not to any one feature. If the fit worsens, one beneficial change may have been offset by another harmful change. Thus, a bundle can be useful for comparing complete candidate specifications, but it is weak evidence for deciding which component to retain.
This is a question of attribution, not a claim that model features act independently. In fact, features may interact: the contribution of a covariate may depend on the structural model, or a random-effects change may alter the usefulness of a residual-error change. A one-change-at-a-time sequence gives interpretable conditional contrasts, but cannot by itself rule out such interactions.
Worked Example: Two Changes in One Candidate
Consider hypothetical candidates fit to the same data with aligned settings. Candidate A has OFV 1,200. Candidate B adds a maturation covariate and changes the residual-error model; its OFV is 1,188. The observed difference is \(1200-1188=12\), a reduction of 12 OFV units for the combined changes.
It would be incorrect to report that “the maturation covariate improved the OFV by 12.” The residual-error change was made at the same time, so the available contrast does not separate their contributions.
To investigate attribution, fit two additional hypothetical candidates: C adds only the maturation covariate to A and has OFV 1,195; D changes only the residual-error model from A and has OFV 1,198. The isolated contrasts are \(1200-1195=5\) for the covariate and \(1200-1198=2\) for the error model. These results show that each change, when made alone in A, lowered the OFV. They do not imply that their separate reductions must add to 7 or predict the bundled result of 1,188. The combined candidate’s reduction is 12, indicating that the joint result differs from that simple sum; further investigation may be needed to understand the interaction or other fitting behavior.
These numbers are descriptive, not proof that either feature is correct. The comparison still needs substantive assessment and reliable fits. The key improvement is that the additional runs make the effects of each change in the reference context visible.
Organizing and Reporting the Comparison
A clear comparison can be represented as a sequence of parent-and-child candidates. Each child names its parent and records the single intended change. If a change is rejected, retain the result and the reason; do not silently overwrite the candidate or its settings. This makes it possible to distinguish the planned contrast from later revisions.
For a pair of nested likelihood models fit to the same data under regular conditions, an objective-function difference may also be used in a formal likelihood-ratio procedure. That is a different inferential step: it requires the relevant nesting and regularity assumptions, and any parameter on a boundary or other irregularity needs special care. For non-nested candidates, an ordinary likelihood-ratio test is not justified merely because both models produce OFVs. AIC or predictive assessment may be more suitable, but those approaches answer different questions and still require appropriately comparable data and evaluation procedures.
For example, if OFV is defined as \(-2\) times the maximized log likelihood and the likelihood is the same, AIC is \(\mathrm{AIC}=\mathrm{OFV}+2k\), where \(k\) is the number of estimated parameters. A lower OFV alone need not mean a lower AIC, because a more complex candidate pays a parameter-count penalty. Nor does an AIC difference identify which individual model feature produced a difference in predictive adequacy. Select the comparison measure to match the decision question, and report it alongside the specification contrast rather than as a substitute for one.
Worked Example: Fit Improvement Versus Complexity
Suppose two hypothetical candidates use the same observations and likelihood. Candidate A has OFV 1,200 and estimates \(k=8\) parameters. Candidate B adds one model term, has OFV 1,196, and estimates \(k=9\) parameters. Assume all other estimation and data conditions are aligned.
For A, \(\mathrm{AIC}=1200+2(8)=1216\). For B, \(\mathrm{AIC}=1196+2(9)=1214\). The OFV falls by \(1200-1196=4\), while AIC falls by \(1216-1214=2\). Thus, in this example both measures favor B numerically, but by different amounts because AIC includes a complexity penalty. This calculation does not establish that B is scientifically preferable; it illustrates why the stated criterion and parameter count matter.
Diagnostics, Alternatives, and Limits
Signs that a comparison may not support its intended conclusion include different observation counts, a changed response transformation, inconsistent likelihood definitions, materially different fitting settings, parameter estimates at implausible values, or unreliable numerical fits. A mismatch can make the contrast uninterpretable even when the reported fit statistic looks favorable. Revisit the candidate specifications and run records before attributing the difference to a model change.
There are several legitimate comparison strategies, each with a different purpose:
| Approach | What it clarifies | Main limitation |
|---|---|---|
| One-change-at-a-time sequence | Conditional effect of a specified change from a defined parent | May miss interactions or depend on the chosen sequence |
| Small factorial set of candidates | Separate changes and their joint behavior across combinations | Requires more runs and careful interpretation |
| Whole-specification comparison | Which complete candidate performs better under a chosen criterion | Does not isolate the contribution of each feature |
| Predictive evaluation | Performance for predictions under a defined evaluation design | Depends on the evaluation target and can be noisy |
A factorial set is particularly useful when interaction is plausible. For two changes, compare the baseline, each change alone, and both together. Such a set can show whether the combined result is unlike what the separate contrasts suggest, but it does not automatically explain the mechanism. Conversely, if the decision concerns only which of two complete prespecified models to use, a whole-specification comparison may be appropriate; be explicit that it evaluates the package of choices, not a single component.
The common mistake is to say, “Candidate B fits better, therefore the covariate caused the improvement,” when B also changed the error model or fitting conditions. The correction is to name the full contrast and either restrict the conclusion to the combined candidate or fit controlled contrasts that isolate the feature of interest. Even a well-controlled comparison supports a limited conclusion: it describes evidence under the particular data, model context, and fitting procedure, not a universal effect of that feature.
Name the reference candidate and the single change whose contribution is being examined.
Check data, response, likelihood, and estimation settings; record any unavoidable difference.
Do not interpret numerical fit differences as model evidence until the fits are sufficiently reliable for the comparison.
Report the change, the evidence, and the limits; use additional candidates if multiple changes were bundled or interaction is plausible.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Candidate P and candidate Q have different analysis observation counts. Can their OFVs be treated as a controlled comparison of a covariate addition? Explain.
- A baseline has OFV 850. A candidate that changes both the random-effects structure and residual-error model has OFV 842. What conclusion is supported, and what conclusion is not?
- Why might an OFV reduction from adding a parameter not be enough to prefer the larger model?
- What extra candidates would help investigate two changes when their combination appears to improve fit?
Solutions. (1) No: the observation sets differ, so the OFVs do not isolate the covariate change and may not be directly comparable. (2) The combined candidate has an OFV 8 units lower under the reported comparison; the result cannot identify which of the two changes contributed. (3) Fit improvement alone does not account for complexity or establish scientific or predictive value; use a criterion matched to the decision and assess the model in context. (4) Fit each change alone from the same baseline, in addition to the combined candidate, so the individual conditional contrasts and joint result can be compared.