What a Transparent Model-Based Test Must Show
A model-based hypothesis test compares specified explanations of biological data. Its conclusion is only as interpretable as the models and assumptions behind it. In a quantitative systems pharmacology (QSP) analysis, readers need to know not only which model scored better, but also what mechanisms differed, what was held fixed, how parameters were calibrated, and whether the fit is adequate for the intended comparison.
The aim is not to list every line of code in the main report. It is to provide enough information to reconstruct the comparison, identify its conditional assumptions, and judge whether the result answers the stated biological question. A reproducible report also makes clear what the result does not establish: relative support among the specified candidates does not show that the favored mechanism is uniquely true or that untested mechanisms are implausible.
Define the Candidates and the Shared Basis of Comparison
State each candidate in terms of its biological structure. For example, one model may include receptor-mediated internalization but omit target degradation; another may include degradation as well. Say which reactions, feedback links, compartments, or species differ. If two candidates have the same structure but different parameter values, describe them as parameter variants rather than presenting them as distinct mechanisms.
Then specify what is common to the comparison. A reader should be able to identify the same observations, time points, units, covariates, initial conditions, and measurement definitions used for every candidate. State the observation model—the assumed relation between model predictions and recorded data—and any handling of missing, censored, or excluded observations. Also report whether the candidates share the same error structure and fitting procedure. If they do not, explain why the chosen comparison remains appropriate.
Calibration details should identify the parameters estimated for each candidate, their units and constraints, which values were fixed, and the fitting algorithm or search strategy. Report whether multiple starting values or other checks were used to look for a better optimum. Give the objective function or likelihood being optimized. These details matter because a poor local fit can make a model appear less compatible with the data than it would be after adequate calibration.
Worked Example: Reporting a Small Information-Criterion Comparison
Suppose a hypothetical QSP analysis compares two models using the same 12 independent subject-level biomarker summaries. Model A represents internalization without target degradation; Model B adds target degradation. Both use the same Gaussian observation-error assumption, the same endpoint and data, and the same calibration procedure. For illustration, the fitted residuals are 12 values: Model A has six residuals of \(+1\) and six of \(-1\) normalized biomarker units; Model B has six of \(+\sqrt{0.5}\) and six of \(-\sqrt{0.5}\) normalized units. Thus, their residual sums of squares are \(SSE_A=12\) and \(SSE_B=6\), respectively.
For Gaussian errors with an estimated residual variance, use the small-sample corrected Akaike information criterion \(AIC_c=n\ln(SSE/n)+2k+2k(k+1)/(n-k-1)\), omitting constants common to both candidates. Here \(k\) counts all estimated parameters, including the residual variance. Suppose Model A estimates three mechanistic parameters plus that variance (\(k_A=4\)); Model B estimates four mechanistic parameters plus the variance (\(k_B=5\)). Both satisfy \(n>k+1\), as required for this correction.
For Model A, substitution gives \(AIC_{c,A}=12\ln(12/12)+2(4)+2(4)(5)/(12-4-1)=0+8+40/7=13.71\). For Model B, \(AIC_{c,B}=12\ln(6/12)+2(5)+2(5)(6)/(12-5-1)=-8.318+10+10=11.68\). The difference from the lower value is \(\Delta AIC_{c,A}=13.71-11.68=2.03\); Model B has the lower AICc in this comparison.
For a compact description of relative information-criterion support, the normalized values are proportional to \(\exp(-\Delta AIC_c/2)\). Model A’s relative value is \(\exp(-2.03/2)=0.362\), and Model B’s is 1. The normalized shares are therefore \(0.362/(1+0.362)=0.266\) for A and \(1/(1+0.362)=0.734\) for B. These are criterion-based relative support measures for this candidate set, not probabilities that the respective biological mechanisms are true. The modest AICc difference warrants a cautious conclusion, not a declaration that Model B is established.
State the Comparison Criterion and Its Conditions
A comparison criterion answers a specific question under specific conditions. For information criteria, report the criterion, the numerical values for each model, the difference from the best value, and the convention used for counting estimated parameters. AICc is appropriate only when its underlying likelihood formulation and finite-sample correction are applicable; for the Gaussian residual-sum-of-squares form above, the data and error assumptions must be common across candidates. The correction is undefined if \(n-k-1\) is not positive.
A likelihood-ratio test is a different option when one candidate is nested within another: the restricted model is obtained from the larger model by fixing parameters or imposing constraints. Let \(\ell_0\) and \(\ell_1\) be the maximized log-likelihoods under the restricted null model and larger alternative, respectively. The statistic is \(2(\ell_1-\ell_0)\). Under regularity conditions, its null distribution is approximately chi-squared with degrees of freedom equal to the number of independent restrictions. The conditions include adequate model specification, identifiable parameters, and restrictions that are not problematic boundary constraints. Mechanistic models can violate these conditions, so a nominal chi-squared p-value may then be unreliable.
A p-value from this test is the probability, assuming the null model and test assumptions hold, of obtaining a likelihood-ratio statistic at least as large as the observed one. It is not the probability that the null mechanism is true. Information criteria, in contrast, rank candidate models using a fit–complexity trade-off; their differences are not p-values and do not directly test a null hypothesis.
Worked Example: Reporting a Nested Likelihood-Ratio Test
In a separate hypothetical comparison, a restricted model omits one feedback parameter, while the larger model estimates it. Assume the models use the same observations and likelihood, the restricted model is nested in the larger one, and the added parameter is identifiable and lies in the interior of its allowable range under the null. Suppose their maximized log-likelihoods are \(\ell_0=-18.4\) and \(\ell_1=-15.9\).
The statistic is \(2(\ell_1-\ell_0)=2[-15.9-(-18.4)]=2(2.5)=5.00\). There is one restriction, so \(df=1\). Under the stated regularity assumptions, the p-value is \(P(\chi^2_1\geq 5.00)\approx0.0253\). At a pre-specified \(\alpha=0.05\), this result is evidence against the restricted model in favor of the larger model, conditional on the likelihood and regularity assumptions. It does not show that the added feedback is the true biological explanation. If the parameter were constrained to be nonnegative and the null placed it at zero, the usual chi-squared reference distribution might not apply; the report would need to address that boundary condition rather than presenting the nominal p-value without qualification.
Make Calibration Evidence and Diagnostics Visible
A numerical ranking is not a substitute for showing whether candidates reproduce the data. Report observed and predicted quantities on interpretable scales, with units, and use plots or tables that reveal time-course and dose-dependent discrepancies. Residual summaries should be compatible with the stated observation model. For example, under an independent Gaussian error model, systematic residual patterns across time may indicate missing dynamics, while a spread that increases with the predicted response may suggest that constant-variance errors are inadequate.
Diagnostics should be relevant to the model and design. Check that predicted trajectories are biologically and numerically plausible, that solutions are stable over the simulated range, and that the fit is not driven by a few influential observations. If the model uses repeated measurements, report how within-subject dependence is represented; treating correlated measurements as independent can exaggerate the amount of evidence. Also report calibration failures, excluded points, and material deviations from the planned procedure. A candidate can have the best score among those tested and still fit poorly in absolute terms.
Worked Example: Reporting Parameter Uncertainty and Fit Checks
Consider a hypothetical target-turnover model fitted to normalized biomarker measurements. The estimated degradation rate is \(0.42\ \mathrm{day}^{-1}\), with an approximate standard error of \(0.08\ \mathrm{day}^{-1}\). Under a local normal approximation, a 95% interval is estimate \(\pm1.96\) standard errors: \(0.42\pm1.96(0.08)=0.42\pm0.1568=(0.263,0.577)\ \mathrm{day}^{-1}\).
A transparent report would identify this as an approximate interval from the stated calibration procedure, not as a 95% probability that the fixed parameter lies in this particular interval. It should say whether the approximation is credible for the nonlinear model—for example, whether the fit is locally well behaved and the estimate is not close to a parameter boundary. A strongly asymmetric or boundary-limited uncertainty distribution would make a symmetric normal interval questionable.
Suppose, as a separate visual check, the observed normalized biomarker values at 0, 2, 4, and 8 hours are 100, 72, 52, and 31, while fitted predictions are 100, 75, 54, and 29. The observed-minus-predicted residuals are \(0,-3,-2,\) and \(2\) normalized units. Reporting these values shows the direction and scale of the discrepancies, but four points alone cannot establish that the observation model is adequate. The report should pair such summaries with diagnostics across all observations and explain any visible pattern.
Uncertainty, Sensitivity, and Limits of the Conclusion
A complete report separates several kinds of uncertainty. Parameter uncertainty concerns estimates within a candidate model. Model-selection uncertainty concerns how the ranking might change across plausible candidates, assumptions, or data choices. Measurement and design limitations concern the information available in the observations. These are not interchangeable: a precise parameter estimate conditional on one model does not imply certainty about which mechanism generated the data.
Where feasible, report whether the conclusion is sensitive to reasonable changes in the observation-error assumptions, parameter constraints, or influential data. Give the alternative specification and the resulting change in the comparison, rather than saying only that results were “robust.” A sensitivity analysis does not prove that all relevant assumptions have been covered; it shows how the conclusion behaves under the variations actually examined.
Explain the scope of the candidate set and the evidence. Mechanisms that were not encoded cannot be ruled out by ranking the models that were encoded. Sparse or indirect measurements may leave structural alternatives difficult to distinguish even when each model calibrates well. A useful conclusion therefore names the winning or non-winning candidate, the criterion and evidence, and the conditions that qualify that conclusion.
Common Reporting Pitfalls
- Calling the best-fitting model “the true mechanism.” A ranking only compares the specified candidates under the stated assumptions. The correction is to describe relative support and acknowledge untested alternatives.
- Reporting a score without its ingredients. An AICc value is difficult to reproduce without the likelihood or residual definition, sample size, and parameter-count convention. Report those items and show the comparison calculation or enough information to reconstruct it.
- Treating an information-criterion difference as a p-value. AICc differences are not tail probabilities from a null distribution. Name the criterion correctly and reserve p-values for a specified test with justified assumptions.
- Ignoring calibration quality. The lowest score does not guarantee an adequate fit. Show diagnostics and identify patterns that could invalidate the assumed error model or expose missing dynamics.
- Presenting uncertainty without its conditioning assumptions. State which model, calibration method, and approximation produced an interval, and describe limitations such as boundaries or weak identification.
Check Your Understanding
Use the reporting principles above to answer each question. Short solutions follow the questions.
- Two mechanistic candidates use different observation-error models. What should a report disclose before presenting a score-based comparison?
- A larger nested model has a higher maximized likelihood, and the added parameter is constrained to be nonnegative and equals zero under the null. Why might a standard chi-squared likelihood-ratio p-value be inappropriate?
- An AICc comparison favors one candidate by 1.2 units. What can and cannot be concluded?
- Why should a report give both a parameter interval and model-comparison uncertainty when both are relevant?
Solutions. (1) State each error model, the likelihood or criterion used, and why the comparison is valid despite the difference; explain any assumptions that differ across candidates. (2) The null value is on the boundary of the allowed parameter space, violating a standard regularity condition for the usual chi-squared reference distribution; the nominal p-value may therefore be wrong. (3) The stated criterion gives a modest relative preference to that candidate within the specified set and assumptions. It does not provide a p-value, prove the favored mechanism true, or rule out untested models. (4) The parameter interval describes uncertainty conditional on a model and estimation procedure, while model-comparison uncertainty concerns which candidate is supported; one does not replace the other.