Information Criteria for Competing QSP Models

Why Compare Calibrated Mechanisms with Information Criteria?

Two QSP models can embody different biological mechanisms yet both describe the available observations reasonably well. A more flexible model may fit those observations better simply because it has more adjustable parameters. Information criteria combine a model’s maximized likelihood with a penalty for parameter count, providing a summary for comparing fit and complexity.

These criteria are useful only when the comparison is defined carefully. Candidate models should be fitted to the same observations using the same outcome definitions and compatible likelihood and error assumptions. If one model is fitted to a different data subset, or uses a different likelihood scale, its score is not directly comparable. As emphasized in the earlier material on shared assumptions for fair model comparisons, the comparison is only as interpretable as the setup that makes it fair.

Here, \(\hat{\theta}\) denotes the parameter estimates at the likelihood maximum. The count \(k\) includes every freely estimated parameter contributing to that likelihood, not just the parameters with direct biological labels. For example, an estimated residual variance usually counts; a parameter fixed in advance does not. The appropriate count can be less straightforward in hierarchical or highly constrained models, so it should be documented rather than guessed.

The quantity \(n\) is the number of independent observational units relevant to the likelihood, not automatically the number of rows in a data file. If repeated measurements within a person or experimental unit are dependent, counting every measurement as independent can overstate the information in the data. The likelihood must represent the dependence, and the meaning of \(n\) for a criterion must be chosen consistently with that likelihood. There is no general rule that makes an arbitrary effective sample size a substitute for specifying the observation model.

What the Penalties Do

All three criteria start with \(-2\ell(\hat{\theta})\), which is smaller when the maximized likelihood is larger. AIC adds \(2k\); AICc adds a further penalty that increases as the sample size becomes small relative to the number of parameters; BIC adds \(k\ln(n)\). For a fixed dataset and likelihood, a lower score is preferred by that criterion. The scores themselves depend on the likelihood scale, so comparisons should use differences between models rather than treating a raw score as an absolute measure of model quality.

The penalty does not require that a complex model fit worse. It recognizes that a model with more estimated parameters has more opportunity to fit the particular data used for calibration. The criteria differ in the trade-off they make: AIC is motivated by estimating expected predictive discrepancy, whereas BIC uses a penalty arising from a large-sample approximation to model evidence. Neither penalty measures the biological plausibility of a mechanism directly.

A Short Justification for the AIC Penalty

AIC is designed to estimate relative expected predictive discrepancy, commonly expressed using Kullback–Leibler divergence. The fitted likelihood is evaluated on the same data used to estimate the parameters, so it tends to look better than the likelihood would on new data. In a regular, identifiable model with an interior parameter value, a large-sample quadratic approximation explains the approximate size of this optimism.

Let \(I\) be the per-observation Fisher information matrix, assumed nonsingular. Under the usual large-sample approximation, the covariance of the maximum-likelihood estimate is approximately \(I^{-1}/n\), and the curvature of the negative log-likelihood for \(n\) observations is approximately \(nI\). The expected quadratic loss from estimation error is therefore approximately

on the log-likelihood scale, where \(I_k\) is the \(k\)-by-\(k\) identity matrix. Comparing fitted performance on the training data with expected performance on new data introduces this optimism twice, giving an approximate difference of \(k\) log-likelihood units, or \(2k\) deviance units. This motivates the AIC penalty. The argument relies on regularity and an adequate likelihood approximation; it does not guarantee that AIC is unbiased in a small, weakly identified, or structurally irregular QSP model.

AICc adds a finite-sample correction to AIC. Its stated formula is the conventional correction used in settings where the usual AIC approximation is expected to be too optimistic in small samples. It is not a universal exact correction for every nonlinear, hierarchical, or non-identifiable model. In particular, its denominator makes the warning visible: as \(n\) approaches \(k+1\), the correction becomes very large, and for \(n\leq k+1\) the formula is undefined.

Comparing Candidate Mechanisms

For BIC, the common large-sample motivation can be seen from a Laplace approximation to the integrated likelihood. With a regular, identifiable, fixed-dimensional model and a prior density that is smooth and positive near the maximum, the log marginal likelihood is approximately \(\ell(\hat{\theta})-\frac{k}{2}\ln(n)+O(1)\). Multiplying by \(-2\) gives \(-2\ell(\hat{\theta})+k\ln(n)\), apart from terms of order one that BIC omits. This explains its penalty, but does not make BIC a direct measure of biological truth. Its approximation can be poor when regularity conditions fail, and the result is sensitive to the assumed model set and likelihood.

Assumptions, Diagnostics, and Limits

Before interpreting a ranking, check the foundations of the calculation. The candidate models should use the same data and compatible likelihood definitions; likelihood constants that vary between models must not be dropped selectively. Each fit should have converged, and the reported log-likelihood should correspond to the stated estimates. The parameter count should include estimated nuisance parameters and reflect constraints actually imposed. If these checks fail, a small numerical difference between criteria is not meaningful.

In QSP applications, several issues deserve particular attention:

• Dependence and repeated observations. Measurements from the same subject, study, or experimental unit may be correlated. An independence likelihood can overstate information and distort both fit and the sample-size penalty. Use an observation model that represents the relevant dependence rather than counting correlated rows as independent evidence.
• Non-identifiability and weak identification. If several parameter combinations produce nearly the same predictions, the likelihood surface may be flat or poorly approximated by a quadratic. The nominal parameter count can then be a poor guide to effective flexibility, and the regularity arguments behind standard penalties may not apply.
• Boundary estimates or singular structures. Parameters on boundaries and mechanisms whose parameters become unidentifiable under a simpler model can violate regularity assumptions. The earlier material on boundary parameters explains why ordinary asymptotic arguments can fail at such boundaries; information criteria do not automatically repair that failure.
• Different observation assumptions. If mechanisms use different error distributions, weighting schemes, or treatment of missing observations, a score difference may reflect those choices as well as the biological structures. Make the comparison’s assumptions explicit.
• Small effective information relative to complexity. AICc can become much larger than AIC, or undefined, when \(n\) is small relative to \(k\). This is a signal to scrutinize the comparison, not a reason to force a numerical result.

Information criteria are relative summaries among the candidate models supplied to them. If every candidate omits an important biological process, the lowest score still selects only the least disfavored member of that set. A lower score does not prove causation, establish that a pathway is present, or quantify the probability that a mechanism is true. Differences in scores also do not replace examination of whether the fitted models reproduce the relevant observations or make biologically discriminating predictions.

Common Pitfall: Reading a Score as Biological Evidence

A tempting but incorrect conclusion is: “Model B has the lower AIC, so Model B is the true mechanism.” The correction is that a lower AIC indicates a lower estimated expected predictive discrepancy relative to the other fitted candidates, under the model and data assumptions used. It is not a probability that the mechanism is true, and it does not rule out mechanisms absent from the comparison. If the difference is small, the ranking itself may be sensitive to the criterion, the likelihood, or the sampled data.

A second mistake is to compare scores calculated from different observations or incompatible likelihoods as though the difference reflected mechanism alone. The correction is to define the comparison first: same observations, same outcome scale, consistently specified likelihood, and documented parameter counts. If those conditions cannot be met, report the scores with the differences in setup rather than presenting a single model ranking as decisive.