Why a Mechanism Ranking Depends on How Data Are Observed
A QSP mechanism does not predict an assay result directly unless the observation process has also been specified. A model may predict a latent biological quantity, such as receptor abundance, while an assay reports fluorescence or a bounded signal related to that quantity through a calibration curve. Likewise, residuals may be treated as normally distributed with constant variance, or the assay may be more plausibly described by proportional error or a heavy-tailed distribution. These choices affect the likelihood and can affect which mechanism ranks first.
The question here is not whether a mechanism is “true” because it has the best score under one specification. It is whether the relative support for competing mechanisms persists across a defensible set of observation and error assumptions. A ranking that changes is not automatically useless: the change identifies which assumptions carry the conclusion and may indicate what evidence is needed to resolve it.
Specify What Is Being Varied—and What Is Held Fixed
A useful sensitivity analysis starts with candidate mechanisms and a shared dataset. For each mechanism, the model links parameters and inputs to a latent biological trajectory. An observation mapping then links that trajectory to the measured endpoint, and an error model describes deviations between predicted and observed measurements. The ranking depends on all three components.
For a likelihood-based comparison, one common criterion is the Akaike information criterion, \(AIC=-2\ell+2k\), where \(\ell\) is the maximized log likelihood and \(k\) is the number of estimated parameters counted under the chosen specification. Lower AIC indicates a better balance of fit and parameter count among the models being compared. To interpret a change in ranking, compare mechanisms within each specification. In general, do not treat raw log likelihoods from different error families as directly comparable evidence: their densities may have different forms and scales.
The analysis requires several conditions. The same observations and endpoint must be used across the compared fits; each likelihood must be defined on the scale of the observed data; and all parameters must be refitted for each specification. The candidate specifications should be scientifically plausible, not selected after seeing which one produces a preferred mechanism. If observations are correlated, or have different known precision, the corresponding structure should be represented consistently rather than ignored. These requirements build on the earlier material on comparing models with correlated and heterogeneous data and on information criteria.
A practical sensitivity set might include a normal residual model, a heavy-tailed alternative, plausible additive and proportional assay-error models, and alternative calibrated observation mappings. Vary one element at a time first, so the source of any ranking change is visible. Combined specifications can then be examined when there is a scientific reason to consider them.
Why Changing the Error Model Can Change the Ranking
Consider a continuous endpoint \(y_i\) with model-predicted mean \(\mu_i\). Under independent normal residuals with common standard deviation \(\sigma\), the log likelihood is
When \(\sigma\) is fixed and the candidate mechanisms have the same parameter count, a smaller sum of squared residuals gives a larger likelihood and lower AIC. This ranking rewards close fits but can be strongly influenced by a few large residuals. A heavy-tailed model assigns less relative penalty to extreme residuals, so the mechanism that fits the bulk of the data best may rank higher even if another mechanism fits one extreme observation more closely.
This is not a license to choose a heavy-tailed distribution whenever it improves a preferred model’s score. The residual distribution is an assumption about measurement deviations, and should be supported by assay knowledge, residual diagnostics, or a prespecified sensitivity plan. The same is true of error variance and observation mapping.
Worked Example: Normal Versus Heavy-Tailed Residuals
Suppose eight hypothetical biomarker observations are fitted by two mechanisms, \(M_A\) and \(M_B\). The fitted models use the same endpoint and observations. Under a normal-error specification, each mechanism has three estimated parameters. Under a Student-\(t\) specification, each has four because the degrees of freedom are estimated. Assume all fits converged and the reported maximized log likelihoods are the results of refitting under the stated error model.
Normal errors. The hypothetical fitted log likelihoods are \(\ell_A=-15.2\) and \(\ell_B=-14.7\). Thus:
The normal-error specification ranks \(M_B\) ahead of \(M_A\), by an AIC difference of \(36.4-35.4=1.0\).
Student-\(t\) errors. Suppose refitting gives \(\ell_A=-13.0\) and \(\ell_B=-13.8\). Then:
Under the heavy-tailed specification, \(M_A\) ranks ahead by \(35.6-34.0=1.6\) AIC units. The ordering has reversed. The correct conclusion is not that either distribution proves a mechanism. Rather, this hypothetical comparison shows that the mechanism preference is sensitive to how extreme residuals are weighted. The next step is to examine whether the heavy-tailed error model is scientifically and diagnostically credible, and whether the data contain observations whose influence explains the reversal.
Assay-Error Structure Is Part of the Comparison
An assay-error model describes how measurement variability changes with the magnitude of the signal. With additive error, the standard deviation is approximately constant on the measurement scale. With proportional error, variability tends to increase with the signal, often making a coefficient of variation a more stable description than an absolute standard deviation. Some assays require a combined structure, such as a constant component plus a signal-dependent component.
The error model must match the scale on which the likelihood is written. For example, a lognormal model describes a distribution for a positive measurement on its original scale, even though its log is normally distributed. Its likelihood must include the density transformation appropriate to the observed measurement. Comparing a correctly specified likelihood with one that omits required scale terms can produce misleading scores.
Worked Example: Additive Versus Proportional Assay Error
Consider two hypothetical QSP mechanisms fitted to the same ten positive assay results. Calibration data suggest that assay variation might be roughly constant in absolute units at low signals but could increase with signal magnitude over the observed range. Suppose the following AIC values result after refitting each mechanism under each error assumption. The likelihoods are defined on the same observed assay scale and include all required density terms; both mechanisms have the same parameter count within each specification.
| Error specification | \(AIC\) for \(M_C\) | \(AIC\) for \(M_D\) | Preferred mechanism |
|---|---|---|---|
| Additive Gaussian error | 42.6 | 45.0 | \(M_C\), by 2.4 AIC units |
| Proportional lognormal error | 47.8 | 43.2 | \(M_D\), by 4.6 AIC units |
Under additive error, \(M_C\) has the lower AIC because \(45.0-42.6=2.4\). Under proportional error, \(M_D\) has the lower AIC because \(47.8-43.2=4.6\). The ranking changes when the variance structure changes. This result makes the mechanism comparison conditional on the assay-error assumption; it does not justify choosing the assumption that yields the desired biological story.
A useful diagnostic is to examine residual magnitude against predicted signal. A fan-shaped increase in residual spread can be inconsistent with constant additive variance, while approximately stable relative spread may support proportional error. Such patterns are clues, not automatic proof: replicate-assay information and calibration studies can provide stronger evidence about the measurement process.
Observation Mappings Can Also Reverse a Ranking
An observation mapping translates latent model output into the scale actually recorded. Examples include a linear calibration, a saturable assay response, a thresholded response, or a transformation from concentration to a bounded signal. If competing mechanisms predict different latent trajectories, a nonlinear mapping can alter how their differences appear in the measured endpoint. The mapping should come from assay knowledge or be treated explicitly as an uncertain modeling choice; it is not merely a cosmetic conversion.
The comparison must use predicted values on the observed scale. Comparing a measured signal directly with a latent concentration, for example, is not a valid fit assessment unless those quantities have the same meaning and units. If mapping parameters are estimated, they must be counted consistently in the model comparison and refitted as appropriate. If the mapping has been calibrated independently and is held fixed, that calibration assumption should be stated.
Worked Example: A Direct Versus Saturable Observation Mapping
Four hypothetical assay responses are \(0.20,\ 0.38,\ 0.55,\) and \(0.67\) assay units. Two mechanisms, \(M_F\) and \(M_I\), are compared using independent Gaussian errors with known \(\sigma=0.10\) assay units. For clarity, assume the mechanisms have equal parameter counts and each mapping below is prespecified from calibration, so the AIC difference is determined by the residual sums of squares. The predictions shown are already on the assay-response scale.
With a direct mapping, suppose \(M_F\) predicts \(0.18,\ 0.35,\ 0.57,\ 0.70\), while \(M_I\) predicts \(0.24,\ 0.42,\ 0.50,\ 0.62\). Their residual sums of squares are:
For fixed known \(\sigma\), the AIC difference in favor of the model with lower \(SSE\) is the difference in \(SSE/\sigma^2\). Here \(0.0082-0.0026=0.0056\), and \(0.0056/(0.10)^2=0.56\). Thus the direct mapping ranks \(M_F\) ahead by \(0.56\) AIC units.
Now suppose a saturable mapping is considered, and its calibrated predictions are \(0.22,\ 0.39,\ 0.52,\ 0.60\) for \(M_F\), and \(0.19,\ 0.36,\ 0.54,\ 0.69\) for \(M_I\). Then:
The saturable mapping ranks \(M_I\) ahead: \((0.0063-0.0010)/(0.10)^2=0.53\) AIC units. The ordering has reversed, but both separations are small in this illustration. The appropriate interpretation is that neither ranking is decisive and that the apparent preference depends on the observation mapping. Calibration uncertainty or additional measurements of the latent quantity could be important for resolving the comparison.
A Structured Robustness Assessment
A sensitivity analysis is most interpretable when it records both the specification and the resulting comparison. A compact table can show the winning mechanism, the score difference, and whether the ranking changed from the reference specification. In a larger analysis, the same information can be displayed as a ranking matrix, with mechanisms as rows and observation/error specifications as columns.
Choose residual distributions, assay-error structures, and observation mappings using assay knowledge, calibration data, and diagnostic evidence. State which choices are primary and which are sensitivity alternatives.
Use the same observations and endpoint, optimize parameters under each specification, and check convergence. Do not reuse a parameter estimate from one likelihood as though it were fitted under another.
Apply the same stated criterion and parameter-counting rules. Record the preferred mechanism and the size of the score separation, not just the winner.
Identify which observations, variance assumptions, or mapping features drive the change. Check residual diagnostics, parameter estimates, and whether the alternative specification remains scientifically credible.
State whether the ranking was stable across the examined assumptions or changed under specified alternatives. Do not convert stability within this set into a claim that the mechanism is proven.
Diagnostics, Alternatives, and Limits
Warning signs include optimizer failures for one mechanism, estimates pushed to parameter boundaries, implausible assay-error parameters, or residual patterns that remain systematic under every candidate error model. These can make a score comparison unreliable or expose a misspecified observation process. Check that likelihood calculations use the same observations and a valid density on the measured scale. Inspect residuals by predicted value, time, dose, and assay batch when those dimensions are relevant. A ranking that depends on one extreme observation deserves particular scrutiny, especially if a heavy-tailed alternative changes the ordering.
A single preferred specification is simpler to report but hides dependence on assumptions. A robustness grid makes that dependence visible, at the cost of requiring justified alternatives and more computation. Predictive cross-validation can assess performance on held-out observations, but it answers a predictive question and does not by itself identify which observation model is biologically correct. Bayesian model averaging, covered in the earlier material on Bayesian model averaging across mechanisms, represents uncertainty across mechanisms under a specified probabilistic setup; it does not remove sensitivity to the likelihood or observation mapping. These approaches complement rather than replace explicit checks of observation and error assumptions.
Robustness is always relative to the alternatives examined. If every specification shares the same incorrect calibration curve, a stable ranking can still be misleading. Conversely, a ranking reversal does not mean that the data contain no information: it can show precisely which measurement assumptions determine the inference. Report the specifications examined, the criterion, the ranking and its separation under each, and the main limitations.
Common Pitfall
Correction: The residual distribution and observation mapping determine how predictions are connected to measured data and how discrepancies are weighted. If a scientifically plausible alternative reverses the ranking, the evidence favors no assumption-independent ordering. Report the conditional result and examine which assumption is supported by calibration, assay knowledge, and diagnostics.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two mechanisms use the same data and have equal parameter counts. Under additive error, mechanism \(M_1\) has lower AIC; under proportional error, \(M_2\) has lower AIC. What is the appropriate conclusion?
- Why should parameters be refitted when the residual distribution changes?
- An assay reports a bounded signal, but a model compares that signal directly with a predicted latent concentration. What is the problem?
- Under normal errors with fixed \(\sigma=2\) units, two equal-parameter models have \(SSE_1=20\) and \(SSE_2=28\). Which has lower AIC, and by how much?
- A ranking is unchanged across three error models, but all three use the same unvalidated observation mapping. What does the stable ranking establish, and what does it not establish?
Solutions. (1) The ranking is assumption-sensitive; report the preferred mechanism under each error assumption and assess which structure is credible. (2) The likelihood and its weighting of residuals have changed, so the parameter values that maximize fit can also change; carrying estimates over would not compare fitted models consistently. (3) The quantities are on different scales and do not have the same meaning; apply an observation mapping to predict the measured signal before calculating fit. (4) \(M_1\) has lower AIC. With equal parameter counts and fixed \(\sigma\), the difference is \((28-20)/2^2=2.0\) AIC units in favor of \(M_1\). (5) It establishes ranking stability only across those three error models conditional on the shared mapping. It does not establish robustness to mapping uncertainty or prove either mechanism.