Why Independent Studies Can Disagree
A PBPK model can make predictions before data are unblinded and perform acceptably in one independent validation study, yet show appreciable error in another. That disagreement is important evidence to explain, not a reason to select the more favorable result. Study populations, clinical protocols, assays, and the conditions represented in the predictions may differ in ways that change observed exposure, prediction error, or both.
The goal is to compare the studies systematically while keeping their original validation outcomes intact. A diagnostic comparison can identify plausible reasons for discordance, but it does not establish that a proposed reason caused the difference. Nor does it turn the studies into a single validation result.
“Independent” here means that each study’s validation data were not used to fit or recalibrate the predictions being evaluated in that study. Preserve that status by keeping the locked predictions, observed data, endpoints, and prespecified predictive criteria identifiable for each study. A post hoc comparison may help interpret the evidence, but it must not be presented as though its findings were prespecified or as though changing the model retroactively improves the original validation result.
Use a Consistent Comparison
Start with the same endpoint and ratio direction in every study. For an individual or matched prediction-observation pair, define the ratio as predicted divided by observed. For positive values, a ratio above 1 means the prediction is higher than the observation; a ratio below 1 means it is lower. State the endpoint and units, such as AUC in mg·h/L, so that the comparison is interpretable.
The geometric mean is useful because prediction errors are naturally multiplicative: ratios of 2 and 0.5 are equally far from 1 on the log scale. It does not, by itself, show the spread of individual errors, explain their causes, or prove that a study meets a predictive criterion. Apply the criterion set before results were examined, and report any relevant individual-level or profile-level diagnostics as well as the summary.
Before comparing outcomes, check that the studies refer to comparable prediction questions. Confirm the dose, formulation, route, endpoint definition, sampling schedule, participant eligibility, and prediction conditions. Also check whether the observed values came from comparable assay methods and whether missing samples or protocol deviations could have changed the evaluable data. These checks build on the earlier material on external validation, predictive error, assay noise, below-quantification samples, and protocol deviations; they do not replace each study’s original analysis.
Worked Example: Two Studies Have Different Prediction Errors
Suppose two hypothetical independent studies evaluate the same locked PBPK model’s AUC predictions, in mg·h/L. Study A has three predicted-to-observed ratios: 0.80, 1.00, and 1.25. Study B has three ratios, each equal to 1.44. For illustration, assume the prespecified acceptance criteria require a geometric mean ratio from 0.80 to 1.25 and each individual ratio from 0.50 to 2.00, inclusive.
For Study A, the product of the ratios is \(0.80\times1.00\times1.25=1.00\), so \(G_A=(1.00)^{1/3}=1.00\). All three individual ratios are within 0.50 to 2.00, and the geometric mean is within 0.80 to 1.25. By these hypothetical criteria, Study A meets both requirements.
For Study B, the geometric mean is \(G_B=(1.44\times1.44\times1.44)^{1/3}=1.44\). Each individual ratio also equals 1.44 and is within 0.50 to 2.00, but the geometric mean exceeds the upper limit of 1.25. Study B therefore does not meet the stated criteria.
The ratio convention determines the direction: \(1.44>1\), so the model overpredicts observed AUC in Study B by a factor of 1.44 on these ratios. The correct report is that Study A meets the illustrative criteria while Study B does not, with Study B showing overprediction. The discrepancy is a prompt for a structured comparison, not permission to omit Study B or to redefine the criterion after seeing it.
Compare Conditions Before Proposing Explanations
A practical reconciliation begins with a side-by-side inventory. Record what each study actually evaluated, rather than relying on broad labels such as “healthy adults” or “standard protocol.” Relevant differences may include the distributions of body size, age, organ function, genotype, or concomitant medications; dose timing, meal status, formulation, sampling times, and adherence; assay method, laboratory, calibration, and quantification limits; and the conditions supplied to the model when predictions were generated. Differences matter only insofar as they are relevant to the prediction question and can plausibly affect observed values, predicted values, or the evaluable sample.
| Comparison domain | Questions to document | Interpretive caution |
|---|---|---|
| Population | Do eligibility rules or covariate distributions differ? Are important subgroups represented in both studies? | A difference in composition can explain a summary contrast without showing that population membership caused it. |
| Protocol | Were dose, formulation, route, meal status, timing, and sampling procedures comparable? | A protocol difference can change the prediction question, not merely add noise. |
| Assay | Were methods, calibration, laboratories, and quantification procedures comparable? | Do not treat an unverified assay discrepancy as a known correction factor. |
| Prediction conditions | Were the same model version and parameter choices used? Did the inputs match the conditions actually observed? | Distinguish a difference in model inputs from a difference in observed data or population. |
| Evaluation | Were endpoints, evaluable samples, exclusions, and prespecified criteria handled consistently? | Changed inclusion rules or endpoints can make summaries non-comparable. |
The comparison should also examine the error pattern, not only the aggregate. A shifted geometric mean can arise from consistent error across participants, a few large errors, or different subgroup composition. Observed-predicted plots, time-structured residuals, and endpoint-specific summaries can help distinguish these patterns. A summary agreement for AUC does not imply that the full concentration-time profile agrees.
Worked Example: Checking Whether Population Mix Could Matter
Consider two hypothetical studies with the same protocol and assay. Suppose their subgroup-specific geometric mean predicted-to-observed ratios are 1.00 in a reference subgroup and 1.40 in a high-covariate subgroup. Assume Study A is 80% reference and 20% high-covariate, whereas Study B is 20% reference and 80% high-covariate. For this calculation, assume each subgroup’s error pattern is the same in both studies and that the stated proportions are the proportions among evaluated participants.
On the log scale, a mixture with subgroup proportions \(w_k\) has a composition-weighted geometric mean \(G=\exp\{\sum_k w_k\log(G_k)\}\), provided the subgroup-specific summaries and weights describe the same evaluated population. Thus, \(G_A=\exp\{0.80\log(1.00)+0.20\log(1.40)\}=1.40^{0.20}\approx1.0696\).
For Study B, \(G_B=\exp\{0.20\log(1.00)+0.80\log(1.40)\}=1.40^{0.80}\approx1.3088\). The difference arises in this calculation from the differing subgroup weights, even though the assumed subgroup-specific ratios are the same. Study B’s ratio is about 1.309, indicating average overprediction on the ratio scale in this hypothetical mixture.
This calculation shows how composition can contribute to discordant summaries; it does not prove that composition explains a real study difference. The subgroup definitions, weights, measurements, and prediction conditions must be verified. If the relevant subgroups are not observed in both studies, or if their error patterns differ, this simple comparison may not identify the source of discordance.
Assay and Protocol Differences Need Evidence
An assay difference is a plausible explanation only when there is evidence about its direction and magnitude. For example, a bridge experiment using matched samples can assess whether two assays show a systematic proportional difference across the relevant concentration range. A single assumed scale factor is not adequate if the relationship varies with concentration, has an important additive component, or is uncertain. Report the original assay-based validation result regardless of whether a bridge supports an additional harmonized analysis.
Worked Example: Interpreting a Hypothetical Assay Bridge
Suppose Study B reports an observed AUC of 100 mg·h/L using assay B, while the locked model predicted 144 mg·h/L. The original predicted-to-observed ratio is \(144/100=1.44\), indicating overprediction relative to the value measured by assay B. A separate hypothetical bridge study of matched samples suggests that assay B reads 0.80 times the reference-assay value across the relevant range. Assume, only for this illustration, that this proportional relationship is supported and appropriate for the AUC measurement.
The implied reference-scale observed AUC is \(100/0.80=125\) mg·h/L. The corresponding exploratory ratio is \(144/125=1.152\). It is lower than the original ratio because the bridged reference-scale observation is higher than the assay B observation.
The original result remains \(1.44\) for the assay and data used in Study B. The value \(1.152\) is a diagnostic, harmonized comparison conditional on the bridge assumption; it does not replace the original outcome or establish that the assay difference caused the study discordance. A sound report identifies the assay, bridge evidence, assumptions, and both ratios.
Protocol differences require similar care. A different meal condition, sampling schedule, or formulation may mean the studies tested different prediction conditions. Check whether the prediction file actually represented the condition used in the study and whether the endpoint can be compared at the available sampling times. Do not attribute a discrepancy to “protocol noise” when the protocol changes the scientific question or when the evidence cannot separate that explanation from other differences.
Justifying the Composition Comparison
The weighted geometric-mean calculation follows directly from the log definition. If subgroup \(k\) has geometric mean ratio \(G_k\) and proportion \(w_k\), with \(\sum_k w_k=1\), the average log ratio across the mixture is the weighted average of subgroup average log ratios. Exponentiating returns the mixture’s geometric mean on the ratio scale:
This identity requires compatible subgroup definitions and positive ratios, and it describes a composition calculation when the subgroup log-error distributions are represented by the subgroup summaries. It is not a causal decomposition. A standardized comparison can be useful when both studies provide data in the same subgroups: apply common weights to each study’s subgroup log ratios and compare the resulting summaries. If subgroup support is absent, standardization can require extrapolation and should not be presented as an observed validation result.
Confirm the locked predictions, observations, endpoint, ratio direction, evaluable samples, and criteria for each study.
Compare populations, protocols, assays, prediction inputs, and deviations using documented data rather than assumptions.
Compare participant-level ratios, profiles, subgroups, and endpoint summaries to see whether the disagreement is broad or concentrated.
Use justified bridge information or common subgroup weights where the data support them; label post hoc analyses as exploratory.
Present each study’s original outcome and relevant context, then state which explanations are supported, uncertain, or not assessable.
Alternatives, Diagnostics, and Limits
The main alternative to a study-by-study comparison is to pool results into one summary. Pooling may be useful for a clearly defined common target, but it can conceal a study with systematic error, give a misleading average when protocols differ, or imply common conditions that were not present. A pooled estimate should not replace the individual independent validation outcomes. When studies are comparable, a common-weight subgroup summary can answer a specific descriptive question, but it still needs the original study results beside it.
Look for signs that the reconciliation is failing: inconsistent endpoint definitions; ratio directions that differ across analyses; predictions that were not actually locked; unexplained exclusions; assay comparisons outside the relevant range; subgroup summaries based on very few observations; or explanations that depend on unobserved covariates. Also distinguish imprecision from disagreement: small studies can have unstable summaries, so report sample sizes and uncertainty where available. A difference between point estimates alone does not establish a statistically meaningful difference.
No single comparison can prove why studies differ. Study conditions may change together, records may be incomplete, and the available data may not distinguish population effects from protocol or assay effects. A careful conclusion therefore separates observation from explanation: state what each study found, describe the documented differences, and qualify any proposed mechanism according to its evidence. Keep exploratory adjustments visibly separate from original results.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A predicted-to-observed AUC ratio is 0.72. Which value is larger, prediction or observation, and what is the direction of model error?
- Two studies have the same subgroup-specific ratios but different proportions in those subgroups. Can their overall geometric mean ratios differ? What assumption is needed to calculate a composition-weighted comparison?
- A bridge suggests one assay reads lower than another. Should the bridged ratio replace the original validation ratio? Explain.
- Why can pooling two discordant study summaries conceal an important validation result?
Solutions. 1. The prediction is 0.72 times the observation, so it is lower; the model underpredicts AUC. 2. Yes. Their weighted averages of subgroup log ratios can differ when subgroup proportions differ. The subgroup definitions, weights, and summaries must be comparable and describe the evaluated populations. 3. No. Retain the original ratio for the assay used in validation and report the bridged value separately as a diagnostic conditional on the bridge evidence and assumptions. 4. A pooled summary may average away a systematic error in one study or mask differences in population and protocol, so both study-specific outcomes must remain visible.