Evidence Is Strong Only in Relation to a Claim
A qualification package may contain code reviews, mechanistic rationale, clinical comparisons, and simulation results, yet still fail to support a particular decision. The key question is not whether the package is large, but whether its evidence supports the claims required for the model’s intended use. A model supported for predicting a short-term biomarker response in adults at studied doses is not automatically supported for predicting a long-term clinical outcome in a different population.
The earlier material on intended-use claims and traceability matrices provides the starting point: write down what the model is being asked to do, and connect each claim to its supporting evidence. This tutorial adds a critical appraisal of those links. For each claim, ask whether its evidence is relevant to the claim, reliable enough to trust, and sufficient to support the conclusion.
These are distinct judgments. A carefully conducted experiment can be reliable but irrelevant to a clinical prediction if it measures a different biological process. A highly relevant study can be unreliable if its endpoint is poorly measured or its analysis cannot be checked. Several relevant, reliable pieces of evidence may still be insufficient if they leave a decision-critical uncertainty unresolved.
Appraise the Claim, Not the Evidence in Isolation
Begin by expressing the claim in a form that can be checked. Identify the predicted quantity, population, intervention or dose, time point, and intended decision. Then list the evidence that bears on it and the assumptions needed to connect that evidence to the claim. A general statement such as “the model is validated” is too broad to appraise; a bounded statement such as “the model predicts the average 12-week biomarker response in adults within the studied dose range” can be examined.
For each evidence item, consider its fit to the claim, its quality, and its limitations. Relevance includes whether the experimental system, population, endpoint, and range match the intended use. Reliability includes study conduct, measurement quality, analysis transparency, and whether results can be independently checked. Sufficiency concerns the combined support: how much uncertainty remains, whether important alternative explanations remain plausible, and whether the evidence covers the consequences of the intended decision.
A practical review keeps these dimensions separate rather than summing them into one “strength” number. An ordinal rating can help organize discussion, but it is not a probability that a claim is true and is not a universal regulatory threshold. For example, a team may use ratings from 0 to 3, where 0 means absent, 1 weak, 2 partial, and 3 strong, provided it defines the anchors before applying them and records the rationale. The ratings expose uneven support; they do not replace scientific judgment.
| Question | Evidence is more supportive when... | Warning sign |
|---|---|---|
| Relevant? | The evidence matches the claimed population, endpoint, intervention, time horizon, and use. | The evidence concerns a different context and the bridge to the claim is unstated. |
| Reliable? | Methods, data provenance, measurements, and analyses are sufficiently transparent and checkable. | Results depend on opaque processing, weak measurement, or unexamined assumptions. |
| Sufficient? | The evidence collectively addresses important uncertainties and decision consequences. | A central link is unsupported, or remaining uncertainty could reverse the decision. |
Why Evidence Cannot Simply Be Counted
Suppose a decision claim depends on a chain of links: the intervention changes a biological process, that change alters the modelled endpoint, and the endpoint predicts a decision-relevant outcome. Support for the first link does not by itself establish the last. Likewise, many studies of one link do not necessarily compensate for no evidence on another link. The reasoning is structural: if the claim requires all links, a missing critical link leaves the conclusion unsupported even when other links are well documented.
This does not mean every link requires a separate clinical study. Mechanistic knowledge, experimental observations, clinical data, and model-based reasoning can jointly support a bridge. The reviewer should make the bridge explicit and examine its assumptions. Evidence can support a narrower claim than the one proposed; narrowing the claim is sometimes more defensible than collecting additional data.
Worked Example: A Biomarker Claim Within the Studied Range
Hypothetical case. A QSP model is intended to predict the average 12-week biomarker change in adults receiving doses from 10 to 30 mg. The qualification claim is explicitly limited to that population, endpoint, time point, and dose range. Two clinical studies measured the same biomarker at 12 weeks in adults; one included 10 and 20 mg, and the other included 20 and 30 mg. Both report their methods and uncertainty, and the model predictions are compared with the observed results.
Using a pre-agreed 0–3 appraisal rubric, the reviewer assigns relevance 3 because the studies match the stated population, endpoint, time, and dose range; reliability 2 because the analyses are transparent but one study has incomplete reporting of a measurement procedure; and sufficiency 2 because the dose range is covered but evidence at the boundaries is limited. The ratings are \(3, 2, 2\), respectively. Their sum is \(3+2+2=7\), but that total is not interpreted as a pass score: the separate ratings show that the main residual concern is sufficiency, not relevance.
Suppose the team also ranks gaps using a defined planning aid: priority \(=\) impact \(\times\) uncertainty \(\times\) decision sensitivity, with each factor rated from 1 to 5. The limited boundary-dose evidence is assigned impact 4, uncertainty 3, and decision sensitivity 4. Its priority is \(4\times3\times4=48\), out of a possible 125. This value is a relative planning rank, not a probability or a standard qualification criterion.
The evidence supports a bounded claim more strongly than a broad claim about all doses or long-term outcomes. The appropriate conclusion is that support is plausible for average 12-week biomarker response within 10–30 mg, with a material boundary-dose limitation requiring qualification or follow-up. It would be unjustified to infer long-term clinical benefit from these studies alone.
Locate and Describe the Gap
A gap is not simply “more data would be useful.” It is a missing or weak connection between evidence and a claim. Describe it precisely: which claim is affected, what evidence is absent or limited, why that matters to the intended decision, and what would reduce the uncertainty. Distinguish a lack of evidence from evidence against the claim. If results conflict, the evidence is not absent; the conflict itself needs explanation and may limit the supported conclusion.
A useful gap statement names the boundary of the evidence. Examples include: the model has not been evaluated for the subgroup that drives the decision; the only supporting data concern a surrogate endpoint rather than the claimed clinical outcome; or the evidence covers observed doses but not the proposed extrapolation. This specificity helps select a proportionate response, such as narrowing the claim, obtaining additional data, testing a sensitive assumption, or documenting why the uncertainty is acceptable for a particular decision.
For planning, a team may rank gaps using the transparent heuristic introduced above:
Here \(I\) is the consequence if the claim is wrong, \(U\) is the degree of unresolved uncertainty, and \(S\) is how sensitive the intended decision is to that uncertainty. If each is rated from 1 to 5, the product ranges from 1 to 125. The multiplication makes a gap rank higher when it is consequential, uncertain, and decision-sensitive. The scale is an organizing device only: changing the anchors can change rankings, and the score does not quantify the chance that the model is wrong.
Worked Example: A Subgroup Claim With an Uncovered Population
Hypothetical case. A model is proposed to support dose selection for adults overall and for a subgroup with reduced renal function. The available clinical dataset includes 240 adults, but only 8 have reduced renal function. The overall comparison is relevant to the overall claim; it does not automatically provide adequate evidence for the subgroup claim. The subgroup sample is small, and no external evidence is provided to establish that the model’s renal-function mechanism is reliable in this population.
On the 0–3 rubric, assign relevance 2: the subgroup is present, but scarcely represented. Assign reliability 2: the overall dataset is well documented, while subgroup-specific estimates are necessarily fragile. Assign sufficiency 1: there is too little subgroup-specific support for a confident dose-selection claim. The ratings remain separate; \(2+2+1=5\) is not a calibrated evidence score and cannot be compared mechanically with another package’s total.
For gap planning, rate the consequence of a wrong subgroup dose recommendation as \(I=5\), unresolved uncertainty as \(U=4\), and decision sensitivity as \(S=5\). Then \(5\times4\times5=100\) on the 1–125 planning scale. This high rank indicates that the subgroup gap deserves attention before relying on the model for that decision. It does not establish that the recommendation is wrong.
The defensible action is to restrict the dose-selection claim to the adequately represented population, or to obtain and assess evidence that specifically addresses reduced renal function. Reporting the overall result as if it established the subgroup result would overstate what the evidence supports.
Compare Evidence Strategies and Diagnose Weak Support
Different evidence sources answer different questions. Mechanistic experiments can support biological plausibility but may not establish clinical predictive performance. Clinical observations can be directly relevant to patient outcomes but may not identify which model mechanism is credible. Simulation studies can explore implications of stated assumptions, but repeating simulations does not independently confirm those assumptions. Independent review or replication can improve confidence in methods and implementation, yet it cannot make an evidence source relevant to an unstudied population or endpoint.
A broad evidence package is often more informative than reliance on one source, but breadth is not sufficient by itself. The sources should address complementary links in the claim, and their limitations should be stated. Where evidence sources disagree, do not select only the favorable result; investigate whether the difference follows from populations, endpoints, assumptions, or analysis choices. The qualification conclusion should reflect the unresolved disagreement.
Several signs indicate that an appraisal may be going wrong:
- The claim is vague, so reviewers cannot determine which evidence is relevant.
- Evidence ratings are added or averaged, allowing a strong rating in one dimension to conceal an absent critical link.
- Evidence quantity is treated as evidence quality or as proof of generalizability.
- Uncertainty is described but not connected to the intended decision or its consequences.
- A limitation is recorded without stating which claim it weakens or what action follows.
- A result from one population, dose range, endpoint, or time horizon is generalized without an explicit supporting argument.
These checks are not a substitute for technical assessment of model implementation or predictive performance. They help establish what the existing evidence can and cannot support. A claim may be well supported by evidence that is relevant, reliable, and sufficient for its stated scope while remaining unsupported outside that scope.
Worked Example: Prioritizing a Time-Horizon Gap
Hypothetical case. A QSP model has evidence comparing predicted and observed biomarker changes through 12 weeks. The proposed decision, however, concerns whether treatment should continue for 1 year. The model’s long-term prediction depends on an assumed persistence of treatment effect, and no year-long observations or other evidence supporting that persistence are available.
The 12-week data are relevant to the short-term biomarker claim, but only partially relevant to the 1-year continuation claim. Assign the long-term claim relevance 1, reliability 2 for the available short-term evidence and its documented analysis, and sufficiency 1 because the key persistence assumption is untested. The short-term evidence is not discarded; its scope is bounded at 12 weeks.
For planning, assign impact \(I=4\), uncertainty \(U=5\), and decision sensitivity \(S=4\), because the continuation decision depends substantially on the long-term effect. The gap priority is \(4\times5\times4=80\). This rank is lower than the subgroup example’s 100 under the stated ratings, so the subgroup gap would be addressed first if resources allow only one immediate activity. The ranking is conditional on the team’s explicit ratings, not an objective ordering guaranteed by the formula.
A suitable conclusion is that the evidence supports short-term biomarker predictions, but not the 1-year continuation claim without additional support for persistence. The gap could be addressed with relevant longer-term evidence or by narrowing the intended use to the supported time horizon.
Limitations and Practical Conclusions
No checklist can determine sufficiency independently of context. A low-consequence exploratory use may tolerate uncertainty that would be unacceptable for a high-consequence treatment decision. Similarly, “reliable” is not an absolute label: the required level of scrutiny depends on the claim and the evidence’s role. Reviewers should state the basis for judgments, consider plausible alternative explanations, and record which limitations affect the conclusion.
Ratings and gap-priority products can make discussion more consistent, but they may create false precision. They depend on chosen anchors, can hide disagreement among reviewers, and should not be presented as inferential statistics. Use them to make reasoning visible, then retain the underlying evidence, rationale, and decision implications. Where expert judgments differ, document the disagreement rather than forcing a single score.
A defensible qualification conclusion is therefore bounded: state which claim is supported, by what evidence, under what conditions, and with what remaining limitations. State separately which claims are not supported and what additional testing or evidence would address those gaps. This makes uncertainty actionable without treating a qualification judgment as a universal certificate for every possible model use.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A well-run cell experiment supports a proposed mechanism but measures no clinical endpoint. Which evidence dimension is strongest, and what limitation remains for a clinical prediction claim?
- A model matches average outcomes in the overall study population. Only a few participants belong to a subgroup used in a treatment decision. Why does the overall match not establish the subgroup claim?
- Using the planning aid, calculate priority when impact is 3, uncertainty is 4, and decision sensitivity is 2. What does the result mean?
- Two studies are directly relevant but report conflicting outcomes. Is this an evidence gap or evidence against the claim? What should the appraisal do?
- A model is supported for a 12-week endpoint, but the proposed use concerns a 1-year decision. Give two defensible responses to this gap.
Solutions. (1) The experiment may provide reliable and relevant mechanistic evidence, but it does not directly establish a clinical endpoint prediction; the bridge from mechanism to clinical outcome remains to be supported. (2) The subgroup is inadequately represented, so uncertainty and possible differences specific to that subgroup remain; an overall result cannot substitute for subgroup evidence without a justified bridge. (3) \(3\times4\times2=24\). This is a relative planning rank under the stated ratings, not a probability or evidence threshold. (4) It is conflicting evidence, not missing evidence. Examine differences in population, endpoint, methods, and assumptions, and reflect unresolved disagreement in the conclusion. (5) Narrow the intended use to the supported 12-week horizon, or seek relevant longer-term evidence that addresses the assumptions needed for the 1-year claim.