Why Assay and Protocol Differences Matter
An external validation compares predictions from a locked model with observations collected in a new setting. A discrepancy is not automatically evidence that the model’s pharmacokinetic structure is wrong. The observed concentration can also differ because the assay measures on a shifted scale, the administered dose differs from the recorded dose, a sample is collected at a different time than documented, or handling changes the amount of analyte recovered. These causes matter because they can create systematic prediction errors even when the model adequately describes concentrations under the conditions it was designed to represent.
The first task is therefore to establish what was predicted and what was actually measured. The model prediction should be on a clearly specified concentration scale and tied to a specified dose, sampling time, route, and other relevant conditions. The assay result should be interpreted in light of its calibration and sample handling. This is a targeted audit of potential discrepancies, not permission to alter the locked model until its predictions agree with the validation data.
Representing Measurement and Protocol Effects
A simple assay model makes the scale issue explicit. Suppose the concentration on the intended reference scale is \(C\), while a new assay reports \(Y\). Over the range relevant to validation, a useful first-order representation is
where \(a\) is an additive offset in concentration units, \(b\) is a dimensionless proportional calibration factor, and \(\varepsilon\) represents remaining measurement error. If predictions are on the reference scale and the underlying concentration equals \(C_{\text{pred}}\), the expected residual is
An offset can therefore produce residuals of similar size across concentrations, while a proportional difference tends to produce larger discrepancies at higher concentrations. In practice, both effects may be present. The linear relationship is an approximation to be checked over the observed concentration range; it should not be extrapolated beyond the calibration evidence.
Protocol deviations act differently: they may change the concentration before it is measured. Let \(C_{\text{act}}\) denote the true concentration under the conditions that actually occurred. Then
Adding and subtracting \(bC_{\text{pred}}\) gives a useful decomposition:
The first two terms describe the assay-scale difference at the predicted concentration. The third describes a difference in the underlying concentration caused, for example, by the actual dose or sampling time. This decomposition is a way to organize evidence, not a guarantee that each component can be estimated separately from the residuals alone.
Worked Example: An Assay With Offset and Proportional Bias
Suppose a hypothetical validation assay is related to the reference concentration scale by \(Y=1.2+0.90C+\varepsilon\), where concentrations are in mg/L. For three samples, the locked predictions on the reference scale are 10, 20, and 40 mg/L. Assume for this calculation that the underlying concentrations equal those predictions and set \(\varepsilon=0\).
For the first sample, the expected reported result is \(1.2+0.90(10)=10.2\) mg/L, so \(e=10.2-10=0.2\) mg/L. For the second, it is \(1.2+0.90(20)=19.2\) mg/L, so \(e=19.2-20=-0.8\) mg/L. For the third, it is \(1.2+0.90(40)=37.2\) mg/L, so \(e=37.2-40=-2.8\) mg/L.
The same result follows from \(E(e\mid C_{\text{pred}})=1.2+(0.90-1)C_{\text{pred}}=1.2-0.10C_{\text{pred}}\). Thus an additive positive offset and a proportional downward shift partly cancel at low concentrations, while the proportional term dominates at higher concentrations. This pattern could look like concentration-dependent model error, but these calculations show that the assay relationship alone can produce it.
Dose Implementation and Sampling-Time Differences
A dose discrepancy changes the input to the pharmacokinetic process. Under linear pharmacokinetics, with the same route, bioavailability, dose timing, and other conditions, concentrations are proportional to dose. If a prediction is based on a documented dose \(D_{\text{rec}}\) but the administered dose was \(D_{\text{act}}\), the corresponding concentration prediction can be scaled by \(D_{\text{act}}/D_{\text{rec}}\). This is only justified when dose proportionality applies; it is not a general correction for nonlinear pharmacokinetics, a different route, altered bioavailability, or a different dosing history.
Worked Example: A Dose Smaller Than the Recorded Dose
In a hypothetical validation record, the model prediction for a patient receiving a documented 500 mg dose is 8.0 mg/L at one sampling time and 4.0 mg/L at another. Administration records later show that only 450 mg was given. Assume linear, dose-proportional pharmacokinetics and that route, timing, and all other relevant conditions match.
The dose ratio is \(450/500=0.900\). The corresponding predictions for the administered dose are \(8.0(0.900)=7.20\) mg/L and \(4.0(0.900)=3.60\) mg/L. If the observed concentrations are 7.1 and 3.7 mg/L, the residuals against the original predictions are \(7.1-8.0=-0.9\) mg/L and \(3.7-4.0=-0.3\) mg/L. Against the dose-adjusted predictions they are \(7.1-7.2=-0.1\) mg/L and \(3.7-3.6=0.1\) mg/L.
The revised comparison is a diagnostic using verified dose information; it does not change the locked model parameters. It also does not establish that dose recording explains every discrepancy. The conclusion depends on the stated dose-proportionality assumption and on verifying that no other administration conditions changed.
Sampling-time discrepancies can also create residuals even when the dose is correct. In a terminal elimination phase described by \(C(t)=C_0e^{-kt}\), a delay \(\Delta t\) changes concentration by the factor \(e^{-k\Delta t}\). This approximation is appropriate only when the exponential terminal-phase description applies over the time interval considered. Near absorption, around a changing infusion, or in a multicompartment phase, the concentration-time curve may not follow this simple relationship.
Worked Example: A Sample Collected Later Than Recorded
Suppose a hypothetical model predicts 12.0 mg/L at a nominal sampling time of 8 hours after a dose. The sample was actually collected at 9 hours. Assume the relevant concentration-time segment follows terminal elimination with \(k=0.150\) per hour.
The time difference is \(\Delta t=1\) hour. The concentration expected at the later time is \(12.0e^{-0.150(1)}=12.0(0.8607)=10.33\) mg/L. The expected difference from the nominal-time prediction is \(10.33-12.0=-1.67\) mg/L, or about \(-13.9\%\) of 12.0 mg/L. If the observed result is 10.4 mg/L, its residual against the nominal-time prediction is \(10.4-12.0=-1.6\) mg/L, close to the timing-based expectation.
This calculation does not show that the model is valid at all times or that the recorded time is wrong in every case. It shows why sample collection time must be checked before interpreting a residual as structural failure. If the exact time is known, predictions should be evaluated at that time according to the prespecified validation procedure.
Sample Handling and Analyte Recovery
Handling can alter the concentration available for measurement. For example, degradation, adsorption to a container, or an inappropriate storage condition may reduce recovery. A simple multiplicative representation is \(Y=rC+\varepsilon\), where \(r\) is the fraction recovered. If \(r=1\), this mechanism creates no systematic shift; if \(r<1\), it tends to produce lower measured concentrations. More complicated or concentration-dependent effects require a more appropriate analytical model and supporting assay evidence.
Worked Example: Reduced Recovery After Handling
Consider a hypothetical sample with a locked-model prediction of 6.0 mg/L. A handling investigation suggests that the sample would retain 85% of its analyte under the storage condition used. Assume, for illustration, a constant recovery fraction \(r=0.85\), no additive assay offset, and no other source of difference.
The expected measured concentration is \(0.85(6.0)=5.10\) mg/L, giving an expected residual of \(5.10-6.0=-0.90\) mg/L. If the observed value is 5.2 mg/L, the residual against the original prediction is \(5.2-6.0=-0.8\) mg/L. That is compatible with the illustrative recovery effect, although one observation cannot confirm it. Independent evidence about storage conditions and assay recovery is needed to support the explanation.
A Practical Investigation
A disciplined investigation first checks whether the predictions and observations refer to comparable conditions, then quantifies plausible discrepancies using information external to the residual pattern. The sequence below helps keep an operational explanation separate from a post hoc adjustment that merely makes predictions fit better.
Check concentration units, analyte definition, matrix, assay calibration, and whether results were transformed or converted before comparison. Record any justified mapping between the validation assay and the reference scale.
Use source records to check actual dose, route, administration time, sample collection time, processing, storage, and relevant protocol deviations. Distinguish verified facts from assumptions or uncertain recollections.
Use an assay comparison, an appropriate dose-scaling relationship, a concentration-time relationship, or recovery evidence. State the assumptions and calculate the expected difference in the same units as the residual.
Show the original locked-model comparison and, where appropriate, a clearly labelled diagnostic comparison using verified conditions or a justified measurement mapping. Explain which comparison answers the prespecified validation question.
A measured discrepancy is persuasive as an explanation only when the proposed mechanism is supported independently and its expected size and direction are compatible with the observed errors. For example, a positive assay offset should not be invoked to explain a consistent negative difference without additional effects that could reverse the direction. If timing, assay, and dose all differ, their contributions may interact, and a one-factor correction may be inadequate.
Alternatives, Diagnostics, and Limitations
The main alternatives are to preserve the original comparison, make a justified harmonization or sensitivity comparison, or conclude that the cause of the discrepancy remains unresolved. These answer different questions. The original comparison describes performance under the external study’s observed measurement and implementation conditions. A diagnostic comparison asks what the discrepancies might look like after accounting for a documented difference. Neither should be presented as if it were the other.
| Approach | What it can establish | Main caution |
|---|---|---|
| Original locked-prediction comparison | Performance as observed under the external study conditions. | Discrepancies may combine model, assay, and protocol effects. |
| Assay or protocol sensitivity comparison | Whether a documented, quantitatively plausible difference could explain some of the error. | Depends on the correctness of the correction model and the supporting records. |
| Model updating or refitting | How a changed model performs after parameters or structure are altered. | It no longer measures validation of the original locked model. |
Useful diagnostics include residuals examined against concentration, assay batch, sample time, dose, and handling history, provided those data are available and the comparisons are interpreted with appropriate uncertainty. A nearly constant shift may be compatible with an assay offset; an error increasing with concentration may be compatible with proportional assay bias; and lower values after a documented dose shortfall may be compatible with dose proportionality. Such patterns are clues, not unique signatures. Several mechanisms can produce similar residual patterns.
The strongest limitation is often identifiability: if assay batch, site, handling practice, and patient group coincide, their effects may be impossible to separate from the available data. Predictions and residuals alone generally cannot determine which mechanism caused an error. Sparse records, uncertain times, unknown actual doses, or a lack of cross-assay samples further restrict attribution. In these situations, report the uncertainty, retain the original validation result, and avoid claiming that the model has passed or failed for a cause the data cannot resolve.
Common Mistake
Wrong reasoning: “The external residuals are mostly negative, so the model overpredicts concentration and its elimination structure must be wrong.” This conclusion skips the possibility that the assay reads low, the administered doses were smaller than recorded, samples were collected later than documented, or analyte recovery was reduced.
Correction: First check whether a verified measurement or protocol difference predicts a residual of the observed direction and approximate size. Then distinguish the original external performance result from any sensitivity comparison. If those checks do not explain the pattern, model-related explanations remain possible, but the residual pattern alone does not identify a specific structural cause.
Check Your Understanding
For each question, consider what can be concluded from the stated information and what assumptions would still need support.
- An assay follows \(Y=0.5+1.10C\), with concentrations in mg/L. If the predicted concentration is 10 mg/L and the underlying concentration equals that prediction, what residual is expected when measurement error is zero?
- A 300 mg dose is recorded, but 270 mg was administered. Under dose-proportional pharmacokinetics, a prediction based on 300 mg is 5.0 mg/L. What is the dose-adjusted diagnostic prediction?
- A sample is collected 2 hours after its nominal time during a terminal phase with \(k=0.10\) per hour. What fraction of the nominal-time concentration is expected at the actual time under the exponential model?
- Residuals are low in one assay batch and normal in another, but all patients in the low-residual batch came from a single site. Can the residual pattern alone establish that assay bias caused the difference?
Solutions. (1) The expected measured value is \(0.5+1.10(10)=11.5\) mg/L, so the residual is \(11.5-10=1.5\) mg/L. (2) The dose ratio is \(270/300=0.900\), giving \(5.0(0.900)=4.50\) mg/L, assuming the other dose-proportionality conditions hold. (3) The fraction is \(e^{-k\Delta t}=e^{-0.10(2)}=e^{-0.20}=0.8187\), or about 81.9% of the nominal-time concentration. (4) No. Assay batch and site are confounded in this description, so the pattern is compatible with assay bias but cannot distinguish it from site-related differences or other mechanisms without additional evidence.