Why Reproducible Reporting Matters
A statement such as “renal function was retained in the final model” does not reveal how that conclusion was reached. Another analyst needs to know which renal marker was tested, which model parameter it affected, how the marker was represented, what statistic and threshold governed the decision, and whether the model fits were acceptable. Without those details, a reported covariate model is difficult to reproduce and its decisions are difficult to audit.
Reporting is not a substitute for a sound search strategy or a well-specified population PK model. It is the record that connects the stated strategy to the actual fitted models and final decisions. The central principle is simple: report each decision as a reproducible comparison, not merely as a list of covariates that survived selection.
State the Decision Rule Before Reporting Decisions
A report should identify the candidate covariates and the model components to which they were allowed to contribute. For a continuous covariate, this includes the tested functional form and any centering or scaling convention. For a categorical covariate, report its coding and reference category. Also state whether a covariate was tested on a typical value, between-subject variability, or another model component; these are different hypotheses and should not be presented as interchangeable tests.
For a nested-model likelihood-ratio test, the objective function value is commonly \(OFV=-2\ell\), where \(\ell\) is the maximized log likelihood. The test statistic is the decrease in OFV from the reduced model to the expanded model. Under the null hypothesis and appropriate regularity conditions, it is compared with a chi-square distribution whose degrees of freedom equal the difference in the number of freely estimated parameters.
For the usual likelihood-ratio interpretation, the models must be nested, fitted to the same observations using the same likelihood and compatible data handling, and optimized to their maxima. The null parameter values must satisfy the regularity conditions for the chi-square approximation. These conditions can fail, for example, when testing a variance component constrained to be nonnegative, so the ordinary chi-square reference distribution should not be applied automatically to such a test. In that setting, the report should identify and justify the appropriate reference distribution or calibration method.
The decision rule must say whether the threshold applies to a p-value, an OFV difference, or another prespecified criterion. It must also state whether equality passes. In the examples below, a candidate passes only when \(p<\alpha\). A report should not silently switch thresholds or criteria partway through a search.
Worked Example: Reporting a Single Candidate Test
Worked Example: Reporting a Single Candidate Test
Suppose a hypothetical analysis tests whether albumin is associated with a typical-value parameter. The reduced model has \(OFV=1012.00\); the expanded model adds one albumin coefficient and has \(OFV=1006.10\). Both fits use the same observations and likelihood, and the expanded model differs only by that one freely estimated coefficient. Assume both models converged successfully and the chi-square approximation is justified.
The statistic and degrees of freedom are
The p-value is \(P(\chi^2_1\geq5.90)=0.01514\). If the prespecified inclusion rule is \(p<0.010\), then \(0.01514>0.010\), so the albumin effect does not pass this rule. This conclusion is about the specified test and threshold; it does not establish that albumin has no association.
A reproducible report would state the tested parameter, albumin coding and scaling, both OFVs, the one-degree-of-freedom comparison, the p-value, the \(0.010\) rule, and the decision. If the covariate was tested after another effect had already been included, the report must identify that conditional model rather than label it simply as “the base model.”
Record the Search Order and Conditional Comparisons
In a sequential procedure, a candidate’s test depends on the model in which it is tested. After an effect is added, later candidates are assessed conditional on that updated model. The report therefore needs a traceable sequence: starting model, candidate order, model at each test, decision, updated model, and stopping rule. Listing only the final retained effects loses this information.
A search plan should also clarify whether candidate order was prespecified, determined by a screening result, or chosen during analysis. If a candidate is skipped, not estimable, or removed after a diagnostic, record the reason. State any forward-inclusion or backward-removal thresholds separately if they differ, and specify whether previously included effects are retested after later additions.
Worked Example: Documenting a Sequential Search
Consider a hypothetical forward search with a prespecified inclusion rule \(p<0.010\). The starting model has \(OFV=1200.00\). The order is eGFR followed by a second renal marker, and each candidate is tested on the current model. Assume the comparisons are nested one-parameter tests on the same records, with successful convergence and a justified chi-square reference distribution.
Step 1: eGFR. Adding eGFR gives \(OFV=1192.00\), so
Because \(0.004678<0.010\), eGFR passes the rule and is included. The updated model, with \(OFV=1192.00\), becomes the reduced model for the next comparison.
Step 2: second renal marker. Adding the second marker to that updated model gives \(OFV=1187.50\). Thus
Since \(0.03389>0.010\), the second marker does not pass the inclusion rule. The report should record that this was a conditional test after eGFR had entered, not an independent test against the original starting model. Under the stated rule, the search stops with eGFR included and the second marker excluded. The conditional result does not show that the second marker would have the same test result in a different model.
Make Fit Checks Part of the Decision Record
A numerical test is meaningful only if the fitted models are suitable for comparison. Report the estimation method and software sufficiently precisely to repeat the fit, including software and version, relevant estimation settings, data version, and any non-default options. Record whether each fit converged, whether the optimizer reached a stable solution, and whether parameter estimates or gradients raised concerns. A “converged” label alone may not establish that the fit is reliable.
For each model comparison, verify that the same observations and likelihood contributions were used. A change in the number of records, likelihood method, or data exclusions can make an OFV difference unsuitable as the stated nested-model test. Inspect whether estimates are finite and plausible, whether standard errors or covariance estimates are available when required, and whether the expanded model has introduced boundary estimates or other warning signs. If a fit fails, do not report its OFV as if it were a valid comparison; record the failure and how it affected the prespecified procedure.
These checks do not guarantee that the structural model or covariate relationship is correct. They establish that the reported numerical decision was based on a fit that can reasonably support the stated comparison. A report should distinguish a candidate that failed the statistical threshold from one that could not be evaluated reliably.
Worked Example: Reporting a Multi-Parameter Candidate
Suppose a hypothetical three-level categorical covariate is added to a reduced model with \(OFV=900.00\). The expanded model has \(OFV=893.00\), and the coding adds two freely estimated parameters. Assume the models are nested, use the same data and likelihood, converge appropriately, and meet the conditions for the chi-square approximation.
The comparison is
For a chi-square variable with two degrees of freedom, the upper-tail probability is \(e^{-x/2}\). Therefore,
At a prespecified threshold of \(\alpha=0.050\), the omnibus test passes because \(0.03020<0.050\). The report should give the two added parameters and \(df=2\), not describe this as a one-parameter test. This omnibus decision concerns the joint addition; it does not, by itself, identify which individual coefficient is responsible for the improvement.
Sensitivity Analyses and Alternative Approaches
A useful sensitivity analysis repeats a defined part of the selection procedure under a plausible alternative, such as a different structural model, functional form, candidate order, or threshold. For every alternative, report what changed and what stayed fixed. Give the comparison within each fitted model framework; do not subtract an OFV from one structural model from an OFV in a different structural model and interpret that difference as the candidate’s likelihood-ratio statistic.
Worked Example: Reporting a Structural-Model Sensitivity Check
Return to the hypothetical albumin test. In the primary structural model, the reduced and expanded OFVs are 1012.00 and 1006.10, respectively. In a prespecified alternative structural model, they are 980.00 and 974.10. Assume each pair is fitted to the same observations within that pair, is nested, converges adequately, and adds the same one albumin coefficient. The decision threshold remains \(p<0.010\).
For the alternative model,
The albumin effect does not pass the \(0.010\) rule in either model. This supports the limited statement that the decision was unchanged in this specified sensitivity check. It does not show that the models are equivalent or that the effect estimate is identical. The two base-model OFVs must not be compared with each other to form the albumin test: each likelihood-ratio statistic comes from its own reduced-expanded pair.
A sequential likelihood-ratio search is only one approach. A prespecified set of tests can make the hypotheses and testing plan clearer, while simulation-based evaluation can examine operating characteristics of a search procedure. Bootstrap stability, cross-validation, and model averaging address different aspects of uncertainty and predictive performance; they do not turn a selected model into a prespecified test. When these approaches are used, report their resampling or validation design and their own decision summaries rather than presenting them as interchangeable p-value thresholds.
Selection also creates limitations that reporting cannot remove. Testing many candidates can make nominal p-values difficult to interpret as if there had been one isolated test, and the final selected set can be sensitive to data, model structure, and search order. The course material on covariate p-values, bootstrap stability, cross-validation, and model averaging addresses those issues in more detail. Here, the reporting task is to make the procedure and its limits visible.
What a Complete Report Should Contain
A compact table can make the decision path auditable, provided its columns are defined and the accompanying text specifies the overall procedure. Useful fields include candidate and tested model component, coding or functional form, reduced-model identifier and OFV, expanded-model identifier and OFV, \(\Delta OFV\), \(df\), p-value, threshold, decision, and fit status. Include an explicit reason for any candidate that was not tested or whose fit was rejected.
- Before the search: state the candidate list, model components, coding, order, thresholds, and stopping or retesting rules.
- During the search: identify the current model at each comparison, report the statistic and fit checks, and record the decision and updated model.
- After the search: list the final model and distinguish the primary analysis from sensitivity results. Summarize deviations from the plan.
- For reproducibility: identify data and code versions, software and estimation settings, and any exclusions or changes that could affect model comparisons.
A common mistake is to report only “the covariate was significant at \(p<0.05\)” or to list only the final covariates. That wording omits what was tested, against which model, and under what search process; it can also imply a single prespecified test when the result came from sequential selection. Correct the record by reporting the actual comparison and its place in the search, then state the decision strictly under the declared rule.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A report gives only “the covariate was retained.” Name four additional details needed to reconstruct the decision.
- A nested one-parameter comparison has \(OFV_{\text{reduced}}=700.0\) and \(OFV_{\text{expanded}}=695.0\). Calculate \(\Delta OFV\) and state the degrees of freedom.
- Why should a report identify the model on which a candidate was tested after another covariate entered?
- Two models use different records because one fit excluded observations. Why is their OFV difference not automatically the stated likelihood-ratio test?
- In a sensitivity model, a candidate has \(p=0.01514\) under a strict rule \(p<0.010\). State the decision and one limitation of comparing it with the primary analysis.
Solutions. (1) Suitable details include the candidate and tested parameter, covariate coding, reduced and expanded model identities and OFVs, test statistic and degrees of freedom, threshold, search order, and convergence status. (2) \(\Delta OFV=700.0-695.0=5.0\), with \(df=1\) because one freely estimated parameter was added. (3) Sequential tests are conditional on the current model, so the comparison and result may change when earlier effects are included. (4) The usual comparison assumes the models use the same observations and likelihood; a change in records changes the basis of the likelihood comparison. (5) The candidate does not pass because \(0.01514>0.010\). A sensitivity result is conditional on its specified alternative model and does not prove the models are equivalent or that the effect is absent.