Why laboratory and assay differences matter
A QSP model may predict a biological quantity, such as a cytokine concentration or receptor occupancy. An external laboratory does not observe that quantity directly: it observes a signal produced by a particular assay, instrument, protocol, and sample-handling process. Two laboratories can therefore report different numbers for the same underlying biology. Conversely, similar-looking assay results need not mean that the underlying biological effects agree.
External validation across laboratories asks two related but distinct questions: does the model predict the biological quantity in the new setting, and does the scientific conclusion drawn from that quantity replicate there? The first requires a defensible connection between predicted biology and observed assay values. The second requires comparing conclusions on a scale that is meaningful for the biological question. Neither question is answered reliably by comparing raw numbers from unlike assays.
The essential discipline is to keep the model prediction separate from the measurement process. Calibration can translate assay results to a biological scale, but calibration must be supported by appropriate reference data. It is not a license to adjust the QSP prediction until it matches the validation observations.
Representing the observation process
The affine form is useful when the assay response is approximately linear over the relevant range. It is not a universal law. An assay might instead have a nonlinear response \(y_{il}=g_l(z_i)+\epsilon_{il}\), or report a value censored below a detection limit. The function \(g_l\) must reflect assay behavior in the range used for validation. If the assay saturates, different high biological values may produce nearly indistinguishable signals, so inversion of the assay response may be unreliable.
Calibration should use suitable standards or shared reference samples that cover the relevant range. Calibration samples should be distinct from the biological validation observations, or at least handled so that the validation outcomes are not used to tune the mapping. When possible, split aliquots from the same biological samples can help isolate laboratory or assay differences: variation between measurements of an identical sample is evidence about the measurement process, not a biological difference between samples.
A comparison also depends on experimental comparability. Treatment, dose, sampling time, population, sample processing, and endpoint definition should be sufficiently aligned with the prediction claim. If these differ, a disagreement may reflect a different biological context rather than laboratory measurement alone. Calibration cannot repair a mismatch in the question being asked.
Calibrating before comparing
Suppose the affine observation model has been estimated from reference samples. If \(b_l\ne 0\), a measured value can be mapped back to the biological scale by inverting the fitted relationship. For \(b_l>0\), this gives:
A useful residual on the measurement scale is \(r_{il}=y_{il}-(\widehat{a}_l+\widehat{b}_l\widehat{z}_i)\), where \(\widehat{z}_i\) is the QSP prediction. Alternatively, compare the calibrated value \(\widehat{z}_{il}\) with \(\widehat{z}_i\). These are not identical analyses: the former keeps the residual in assay units, while the latter puts it on the biological scale. State which scale is used and give its units.
Calibration is itself uncertain. A slope and offset estimated from a small set of standards are not known exactly, and that uncertainty contributes to uncertainty in calibrated values and residuals. Treating the estimated mapping as exact can make an apparent discrepancy seem more precise than the evidence warrants. If measurement error and model predictive uncertainty are independent, their variances contribute to the variance of a prediction-versus-observation residual:
Here \(\widehat{Y}_{il}\) is the prediction after applying the laboratory observation model. If calibration uncertainty is material, it must also be included. If the two terms are correlated, a covariance term is needed; independence should not be assumed merely for convenience.
Worked Example: Correcting a Scale Difference With Reference Samples
A QSP model predicts concentrations of 10, 20, and 30 ng/mL in three hypothetical validation conditions. A second laboratory's assay is calibrated using separate reference samples. The reference results support the relationship \(y=2+0.8z\), where \(z\) is concentration in ng/mL and \(y\) is the reported assay value. The laboratory reports 10.1, 18.2, and 25.8 assay units for the three validation conditions. Compare the predictions with the observations on the biological scale.
The calibration equation is used to translate the model predictions to the laboratory's measurement scale: \(2+0.8(10)=10.0\), \(2+0.8(20)=18.0\), and \(2+0.8(30)=26.0\) assay units. The raw residuals, observed minus predicted on that scale, are therefore \(10.1-10.0=0.1\), \(18.2-18.0=0.2\), and \(25.8-26.0=-0.2\) assay units.
For a biological-scale comparison, invert the calibration equation. The three calibrated observations are \((10.1-2)/0.8=10.125\), \((18.2-2)/0.8=20.25\), and \((25.8-2)/0.8=29.75\) ng/mL. Their differences from the QSP predictions are \(0.125\), \(0.25\), and \(-0.25\) ng/mL. The root mean squared difference is:
These hypothetical observations are close to the predictions on the calibrated scale. The calculation does not establish that the predictions are accurate in general: it describes these three conditions and does not include uncertainty in calibration or prediction. It also depends on the calibration relationship being valid over 10–30 ng/mL.
Check whether the biological conclusion replicates
Sometimes the central claim concerns an effect rather than an absolute concentration. For example, the claim may be that treatment reduces a biomarker relative to control. A change in assay units between laboratories need not undermine that conclusion. But the effect measure must be chosen deliberately. A multiplicative change in assay scale cancels in a ratio, while an additive offset generally does not. Ratios are therefore not automatically comparable when assays have important offsets.
For positive measurements, a treatment-to-control ratio is often interpretable, and its logarithm converts multiplicative effects into differences. With independent treatment and control groups, an approximate standard error for the log ratio of sample means can be calculated using the delta method. If the standard deviations and sample sizes are \(s_T,n_T\) and \(s_C,n_C\), respectively, then:
This approximation assumes positive means, independent groups, and enough regularity for the sample means to be approximately normal on the log scale. It is not suitable without modification for paired observations, very small samples, or data dominated by zeros or detection-limit censoring. A confidence interval for the log ratio can be exponentiated to express the interval for the ratio. A confidence interval that excludes 1 indicates evidence of a nonzero ratio-scale effect at the corresponding two-sided level under the model assumptions; it does not establish that two laboratories have identical effect sizes.
Worked Example: Comparing a Treatment Effect Across Assays
In two hypothetical laboratories, treatment and control groups are independent, each with 25 samples. Laboratory A reports control mean 100 units and standard deviation 20, and treatment mean 70 units and standard deviation 17.5. Laboratory B uses a different assay and reports control mean 5.0 units and standard deviation 1.0, and treatment mean 4.0 units and standard deviation 1.0. Assess whether the direction of the effect replicates using treatment-to-control ratios.
For Laboratory A, the ratio is \(70/100=0.700\), a 30.0% lower treatment mean than control. The standard error of its log ratio is:
The log ratio is \(\log(0.700)\approx -0.3567\). Using 1.96 for an approximate 95% interval gives \(-0.3567\pm1.96(0.06403)\), or approximately \([-0.4822,-0.2312]\). Exponentiating the endpoints gives a ratio interval of approximately \([0.617,0.794]\).
For Laboratory B, the ratio is \(4.0/5.0=0.800\), a 20.0% lower treatment mean. Its standard error is:
The log ratio is \(\log(0.800)\approx-0.2231\). The approximate 95% log-scale interval is \(-0.2231\pm1.96(0.06403)\), or \([-0.3486,-0.0976]\). Exponentiating gives a ratio interval of approximately \([0.706,0.907]\). Both intervals are below 1, so both laboratories show evidence of a treatment-associated reduction under the stated assumptions. The direction of the biological conclusion replicates, although the estimated reductions differ. This comparison does not prove that the effect sizes are equal, and the raw means of 70 and 4.0 are not meaningfully comparable across these assay units.
Nonlinear response and failure of calibration
A calibration relationship must match the assay response. A linear model can mislead when the assay responds nonlinearly, and an invertible-looking equation is not enough if small signal errors correspond to large uncertainty in the biological quantity. Examine calibration residuals across the range, assess whether standards cover the validation values, and check for saturation, lower limits of quantification, batch shifts, and systematic changes with sample handling.
Worked Example: Using a Log-Response Assay
A hypothetical assay has a calibrated response \(y=5+10\log_{10}(z)\), where \(z\) is concentration in ng/mL and \(y\) is signal in assay units. The QSP prediction is 20 ng/mL, and the validation assay reports 18.2 units. Convert the observation to the biological scale and compare it with the prediction.
Rearrange the calibration equation: \(\log_{10}(z)=(y-5)/10\), so \(z=10^{(y-5)/10}\). Substituting the observed signal gives \(z=10^{(18.2-5)/10}=10^{1.32}\approx20.89\) ng/mL. The calibrated difference is \(20.89-20=0.89\) ng/mL, or about \(100(0.89/20)=4.45\%\) above the prediction.
This comparison is valid only if the log-response calibration is supported around 20 ng/mL and the measured signal is within the assay's usable range. If the assay were saturated near 18.2 units, small differences in signal might map to substantial differences in concentration; the apparent precision of 20.89 ng/mL would then be unjustified.
Deriving the residual comparison and diagnosing problems
The rationale for mapping predictions to the assay scale follows directly from the observation model. Suppose the model prediction is the relevant biological mean \(z_i\), the laboratory response is \(a_l+b_lz_i\), and the measurement error has mean zero. Then:
If the QSP prediction is correct for that condition, replacing \(z_i\) by \(\widehat{z}_i\) gives the expected assay result \(\widehat{Y}_{il}=a_l+b_l\widehat{z}_i\). Therefore:
When \(z_i=\widehat{z}_i\), the expected residual is zero. If the residual has a consistent direction across conditions, possible explanations include a biological prediction error, an incorrect calibration offset or slope, or an unmodeled laboratory effect. A residual pattern alone does not identify which explanation is correct. Independent reference samples and assay diagnostics are needed to distinguish them.
Useful diagnostics include plotting calibration residuals against reference concentration, checking replicate precision, and plotting validation residuals against predicted value, time, dose, and laboratory. A trend with predicted value may suggest a scale or nonlinear-response problem; a consistent laboratory shift may suggest an offset or batch effect. Large scatter without a pattern may reflect measurement variability, biological variability, or both. These checks should inform interpretation, not become an informal procedure for repeatedly modifying the model until the validation data look favorable.
Choosing a comparison and recognizing its limits
| Approach | Useful when | Main limitation |
|---|---|---|
| Compare raw assay values | Assays use the same validated scale and compatible procedures | Misleading when units, offsets, or response functions differ |
| Calibrate to a shared biological scale | Reliable standards or reference samples cover the relevant range | Calibration uncertainty and model misspecification must be considered |
| Compare relative effects, such as ratios | The biological claim is about proportional change and the assay scale supports that measure | Additive offsets, censoring, or incompatible conditions can distort the comparison |
No approach removes every limitation. Reference materials may not behave like biological samples; calibration can drift across batches; and a model may predict a population quantity while a laboratory reports a sample-level measurement. A replication in the direction of an effect is also weaker than agreement in magnitude and time course. State which aspect replicated and what remains uncertain. When laboratories use different populations or protocols, interpret differences as evidence about the combined biological and measurement setting rather than attributing them automatically to assay performance.
Common mistake: treating an assay discrepancy as a model failure
A common mistake is to compare a predicted concentration directly with a reported signal from a different assay and conclude that the model is wrong because the numbers differ. The reasoning fails because the values may be on different scales and may include different offsets, nonlinear response, or detection limits. The correction is to identify the biological quantity being predicted, specify and justify the assay observation model, and then compare on a stated scale with calibration and measurement uncertainty in view. The opposite mistake is also possible: using a flexible calibration fitted to the validation outcomes to make any prediction appear compatible. Calibration must be independently supported, and biological disagreement should not be erased by post hoc adjustment.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A second assay follows \(y=3+2z\). A model predicts \(z=8\), and the assay reports \(y=19\). What is the calibrated biological value and its difference from the prediction?
- Why can an additive assay offset make treatment-to-control ratios misleading?
- Two laboratories report different raw values, but after independently supported calibration both show a treatment reduction. What conclusion can be stated, and what stronger claim is not established?
- Name two diagnostics that could help distinguish calibration problems from a biological prediction discrepancy.
Solutions. (1) Invert the calibration: \(z=(19-3)/2=8\), so the difference is \(8-8=0\) biological units. (2) An offset is added to both group measurements and does not generally cancel in their ratio; the resulting ratio can depend on the assay's offset as well as the biological effect. (3) The direction of the reduction replicates on the calibrated scale. This alone does not establish identical effect magnitudes or prove that the model is quantitatively accurate in every condition. (4) Examples include plotting calibration residuals against reference concentration, checking replicate precision, and plotting validation residuals against prediction or laboratory.