Why Validate Across Scales?
A QSP model may predict a change in a molecular biomarker and use that change to predict a downstream physiological or clinical outcome. Validating the biomarker trajectory alone does not establish that the downstream prediction is credible. Nor does agreement in the outcome alone establish that the model represented the biomarker-to-outcome link correctly. Cross-scale validation examines whether the predicted changes at both scales correspond in direction, magnitude, and timing.
For example, a model might accurately predict a sustained reduction in an inflammatory cytokine but incorrectly predict a rapid improvement in lung function. The biomarker prediction can be good while the cross-scale prediction fails because the outcome response is smaller, delayed, or absent. The reverse can also occur: an outcome trajectory may be predicted well even though the biomarker trajectory is not. These are distinct findings and should be reported separately.
The validation data should represent the prediction claim being evaluated. In particular, the model predictions should be fixed before examining the validation outcomes, and biomarker and outcome observations should be aligned to the same participants or populations, treatment conditions, baseline definitions, and observation times as far as the data allow. Population-level validation provides useful context, but averages can conceal whether biomarker and outcome changes co-occur within the same units.
Set Up a Comparable Validation
Before comparing predictions with observations, specify what each quantity means. State the biomarker and outcome, their measurement scales and units, the population and treatment context, the baseline reference, and the times at which predictions and observations are compared. A biomarker may be reported as a concentration, a percentage of baseline, or a transformed assay value; these are not interchangeable without a defined conversion. Likewise, higher values of a clinical endpoint may indicate either improvement or worsening, depending on the endpoint.
For a value \(B(t)\), define its change from baseline as \(\Delta B(t)=B(t)-B(0)\); define \(\Delta Y(t)\) similarly. If percent change is used, calculate it relative to the same baseline definition for predictions and observations. The sign of a change has a numerical meaning, but its clinical interpretation depends on the endpoint. A negative change in symptom score may indicate improvement, while a negative change in lung function may indicate worsening.
A useful comparison retains the paired trajectories. At each time, compare the predicted and observed biomarker, then compare the predicted and observed outcome. Also inspect whether the observed biomarker and outcome changes occur in the relationship and temporal order implied by the model. A discrepancy in the outcome conditional on a well-predicted biomarker is particularly informative: it points to a weakness in the downstream link, its timing, or another part of the model, rather than automatically implicating the biomarker response.
Worked Example: Biomarker Agreement, Outcome Mismatch
Consider a hypothetical treatment group with a baseline cytokine concentration of 100 pg/mL and a baseline physiological impairment score of 50 points, where lower impairment is better. At week 4, the model predicts a cytokine concentration of 60 pg/mL and an impairment score of 40 points. Validation observations are 62 pg/mL and 49 points, respectively. Assume the measures are comparable at baseline and week 4 and the model predictions were fixed before validation data were examined.
For the biomarker, the predicted change is \(60-100=-40\) pg/mL and the observed change is \(62-100=-38\) pg/mL. The prediction error in change is \(-38-(-40)=2\) pg/mL. Thus the model closely predicts the cytokine reduction in this example.
For the impairment score, the predicted change is \(40-50=-10\) points and the observed change is \(49-50=-1\) point. The prediction error in change is \(-1-(-10)=9\) points. The observed outcome improved by only 1 point, compared with the predicted 10-point improvement. Because lower impairment is better, the sign is consistent with a small improvement, but the predicted magnitude is not supported.
The conclusion is not that the whole model is validated because the biomarker matched, nor that the biomarker response was absent. The biomarker prediction was close while the downstream outcome prediction was substantially too large at week 4. This pattern is compatible with a mis-specified biomarker-to-outcome relationship, an unrepresented delay, or other mechanisms; these data alone do not identify which explanation is correct.
Timing Can Change the Apparent Link
A downstream physiological response often accumulates or relaxes more slowly than its upstream biomarker. Comparing biomarker and outcome values at the same instant can therefore give a misleading impression of the model’s intended relationship. A simple first-order response illustrates this point. Let \(x(t)\) be a normalized biomarker-driven signal and \(z(t)\) a normalized downstream response, with larger values defined as larger predicted responses. Suppose the model specifies
Here \(k\) has units of inverse time. The downstream response moves toward the current input \(x(t)\), but does not jump immediately to it. If \(x(t)=x_i\) is constant over an interval of length \(\Delta t\), the solution over that interval is
This result follows by rewriting the differential equation as \(dz/dt+kz=kx_i\). Multiplying by the integrating factor \(e^{kt}\) gives \(d(e^{kt}z)/dt=kx_i e^{kt}\). Integrating over the interval and rearranging yields the expression above. Since \(0<e^{-k\Delta t}<1\), the new response is a weighted average of its previous value and the current input. This derivation justifies a specific lagged-response prediction; it does not establish that this model form is appropriate for every biomarker-outcome pair.
Worked Example: Checking a Predicted Lagged Response
Suppose a hypothetical model predicts the normalized inputs \(x_1=0.40\), \(x_2=0.80\), and \(x_3=0.80\) at three successive weekly intervals. Let \(z_0=0\) and \(k=0.50\) per week. For each one-week interval, \(e^{-k\Delta t}=e^{-0.5}=0.6065\), so the input weight is \(1-0.6065=0.3935\).
At week 1, the predicted outcome response is \(z_1=0.6065(0)+0.3935(0.40)=0.1574\). At week 2, it is \(z_2=0.6065(0.1574)+0.3935(0.80)=0.4103\), to four significant figures. At week 3, it is \(z_3=0.6065(0.4103)+0.3935(0.80)=0.5636\).
Suppose validation observations of the normalized downstream response are 0.16, 0.41, and 0.30 at those weeks, with the same normalization and direction. The first two observations are close to predictions. At week 3 the prediction error is \(0.30-0.5636=-0.2636\), so the observed response is substantially below the predicted response. The biomarker-driven input remained at 0.80, and the model predicted further accumulation; the observed response instead fell. That late divergence is a cross-scale and temporal mismatch. It should not be dismissed by noting that the week 1 and week 2 values agree.
Compare the Joint Pattern, Not Just a Correlation
A correlation between biomarker change and outcome change is not a complete validation test. Correlation measures how two quantities vary together in a dataset; it does not show whether the model predicted their absolute changes correctly, whether the time ordering matches, or whether the relationship is causal. A high correlation can coexist with systematically overpredicted outcomes. Conversely, a genuine lag can make same-time correlation weak even when the model’s dynamic relationship is plausible.
A direct validation comparison begins with the model’s paired predictions and the corresponding observations. If there are repeated observations, examine the trajectories over time, not only a single summary. If there are multiple individuals or groups, inspect whether the paired patterns are preserved across them. Summarizing changes can help, but should not erase informative timing or heterogeneity. For example, an average biomarker decrease in one subset and an average outcome improvement in a different subset do not demonstrate that the individual-level link was predicted correctly.
The following sequence makes the reasoning explicit. It complements residual diagnostics and structural-discrepancy assessment rather than replacing them.
State which biomarker change is predicted to precede or accompany which downstream response, in which population and treatment context, and over what time horizon.
Match baseline, units, endpoint direction, sampling times, and aggregation level. Record missing or non-comparable measurements rather than silently treating them as matched.
Compare predicted and observed biomarker changes and predicted and observed outcome changes. Agreement on one scale does not compensate for disagreement on the other.
Ask whether the observed downstream response has the predicted direction, size, delay, and persistence given the observed biomarker trajectory. Identify when divergence begins.
Report the cross-scale pattern and plausible sources of discrepancy without converting association into a causal claim or declaring broader validity than was tested.
Worked Example: Opposite Outcome Direction Despite Biomarker Change
In a hypothetical study, a model predicts that a biomarker will fall from 20 to 12 assay units by day 14 and that a downstream symptom score will fall from 30 to 24 points, where lower scores indicate fewer symptoms. The validation observations at day 14 are a biomarker value of 13 assay units and a symptom score of 33 points. Assume the same assay scale, baseline definition, and day-14 measurement schedule were used for predictions and observations.
The biomarker changes are \(\Delta B_{\text{pred}}=12-20=-8\) assay units and \(\Delta B_{\text{obs}}=13-20=-7\) assay units. The observed biomarker change is close in direction and magnitude to the prediction. The symptom changes are \(\Delta Y_{\text{pred}}=24-30=-6\) points and \(\Delta Y_{\text{obs}}=33-30=3\) points. The outcome prediction error in change is \(3-(-6)=9\) points.
The observed symptom score increased, indicating worsening, whereas the model predicted a decrease, indicating improvement. This is a directional cross-scale failure at day 14, not merely an error in the size of an otherwise correctly directed response. It does not prove that the biomarker is unrelated to symptoms: the biomarker might have an unobserved delay, an opposing pathway might dominate, or the endpoint may be noisy. Those explanations require further evidence and should not be substituted for the validation result.
Diagnostics, Alternatives, and Limitations
When the scales disagree, first check comparability and implementation. Confirm units and transformations, baseline selection, endpoint direction, sample timing, and whether model and observed values refer to the same population or aggregation level. A mismatch caused by a percentage-versus-concentration error is not evidence against the biological link. Conversely, once such issues are ruled out, a consistent late or directionally reversed discrepancy should not be hidden by reporting only a favorable biomarker fit.
Next examine where the mismatch occurs. A discrepancy that grows over time may indicate an incorrect response time, duration, or feedback representation. A discrepancy present at every time may indicate a wrong relationship magnitude or an unrepresented component. Different patterns across groups may indicate that a population mean conceals heterogeneity. These are diagnostic interpretations, not unique diagnoses: multiple model structures can produce similar observed discrepancies.
| Approach | What it can reveal | Main limitation |
|---|---|---|
| Joint trajectory comparison | Agreement and divergence in direction, magnitude, and timing at both scales | Requires comparable repeated measurements and can be difficult to summarize across many individuals |
| Change-change summary | Whether biomarker and outcome changes vary together over a defined period | Can hide delay, trajectory shape, and population heterogeneity; correlation alone does not test prediction accuracy |
| Outcome-only validation | Whether the model predicts the downstream endpoint in the tested setting | Does not establish that the predicted biomarker response or biomarker-to-outcome link is correct |
| Biomarker-only validation | Whether the model predicts the measured biomarker trajectory | Does not establish downstream physiological or clinical performance |
Cross-scale checks also have practical limits. Biomarker and outcome measurements may be collected at different times, have different measurement error, or be available only for subsets of participants. A weak observed relationship may reflect these data limitations as well as model error. Moreover, a successful test in one dose, population, assay, or time horizon supports only that tested prediction context; it does not automatically establish performance elsewhere. The detailed residual patterns and validation context should accompany any summary judgment.
Common Pitfall
Wrong reasoning: “The biomarker decreased as predicted, and the outcome improved on average, so the model’s biomarker-to-outcome link is validated.” This conclusion can be wrong if the outcome improvement was too small, occurred at the wrong time, or occurred in different individuals from those with the biomarker decrease.
Correction: Compare the predicted and observed paired changes at both scales, with aligned baselines and times. Ask whether the outcome pattern is the one predicted conditional on the biomarker trajectory, and report discrepancies explicitly. Even close correspondence supports a prediction claim, not by itself a causal claim that the biomarker change produced the outcome change.
Check Your Understanding
For each question, give the stated calculation or interpretation. Solutions are included below.
- A model predicts a biomarker change of \(-5\) units and an outcome change of \(-4\) points. Observed changes are \(-4.5\) units and \(+1\) point. If lower outcome values are better, characterize the cross-scale result.
- In the first-order response model, let \(k=1\) per day, \(\Delta t=1\) day, \(z(t_i)=0.20\), and \(x_i=0.60\). Calculate the predicted response at the end of the interval using \(e^{-1}=0.3679\).
- Why can a high correlation between biomarker changes and outcome changes fail to validate the model’s link?
- If biomarker and outcome averages agree with predictions but are calculated from different subsets of participants, what important validation claim remains unsupported?
Solutions. (1) The biomarker change is close: its change error is \(-4.5-(-5)=0.5\) units. The outcome change is directionally discordant: the model predicts improvement, while the observed \(+1\)-point change indicates worsening; its change error is \(1-(-4)=5\) points. The link is not supported at the outcome scale. (2) \(z(t_i+1)=0.3679(0.20)+(1-0.3679)(0.60)=0.07358+0.37926=0.4528\). (3) Correlation does not establish accurate predicted magnitudes or timing, and it does not show causation. (4) The paired, within-unit correspondence is untested: agreement in separate averages does not show that the predicted biomarker and outcome changes co-occur in the same participants.