Why Traceability Matters
Qualification is not a single property that a model either possesses or lacks. It is a reasoned judgment about whether a particular model, with a specified implementation and evidence base, is adequate for a defined use. The intended-use claim sets the question; a traceability matrix makes visible how each supporting item relates to that question.
The earlier material on intended-use claims for QSP model qualification establishes that a claim should identify the population, intervention or exposure, outcome, time horizon, and decision context relevant to its use. A matrix does not replace that specification. It preserves the connection between each such claim and the data, implementation checks, validation analyses, criteria, and limitations that bear on it.
The matrix is a documentation and reasoning tool, not a scoring system. A filled cell does not establish that evidence is adequate, and an empty cell does not by itself determine whether a model is unusable. The point is to make the basis and boundaries of each qualification judgment inspectable.
Define the Links Before Filling the Matrix
A claim is a bounded statement about what the model is intended to support. Evidence should be linked to the claim it addresses, rather than collected in a general inventory and assumed to apply everywhere. One item may support more than one claim, and one claim will usually require several kinds of evidence. Keep the links explicit in both cases.
Use the categories consistently. Data identifies the source, population, variables, and relevant time points. Verification tests address whether the implemented equations, code, and data handling behave as specified. Validation analyses compare model predictions with relevant observations for the claim’s intended use. Acceptance criteria state in advance how results will be judged. Unresolved limitations record restrictions or uncertainties that remain relevant to interpreting the evidence.
Verification and validation are related but distinct. A program may execute its specified calculations correctly without its predictions being adequate for a clinical use; conversely, a close match to some observations does not establish that the implementation is free of errors. A matrix should therefore not place a validation comparison in the verification column, or treat a code check as evidence of clinical predictive performance.
Give items stable identifiers, such as C-01 for a claim, D-01 for a data set, V-01 for a verification test, and A-01 for a validation analysis. Record the model version and analysis version to which each item applies. That way, a reviewer can follow a link to the actual protocol, dataset description, test output, or analysis report rather than relying on a vague label such as “model checked.”
A Practical Matrix Structure
A useful row represents a claim, with linked evidence IDs in the relevant columns. If a claim needs several datasets or analyses, list each one separately or use a clearly indexed list; do not compress distinct evidence into an unexplained phrase. Include a status field if it helps track whether an item is planned, completed, or not applicable, but keep status separate from the evidence description and the conclusion about adequacy.
| Field | What to record | Traceability question |
|---|---|---|
| Claim ID and wording | Bounded intended use, including population, intervention, outcome, horizon, and decision context as applicable | What specific use is being supported? |
| Data | Data ID, source, population, measurements, time points, and role in calibration or evaluation | Which observations inform or test this claim? |
| Verification tests | Test ID, implementation feature checked, reference or expected behavior, and recorded result | How was correct implementation checked? |
| Validation analyses | Analysis ID, prediction being compared, observed quantity, method, and result | How were predictions compared with observations relevant to the claim? |
| Acceptance criteria | Criterion ID, threshold or decision rule, scope, and whether it was prespecified | What result counts as meeting the stated requirement? |
| Unresolved limitations | Limitation ID, affected claim, and the boundary or uncertainty it introduces | What remains untested, uncertain, or outside scope? |
Acceptance criteria should be operational, not merely aspirational. “Adequate prediction” is not a criterion until the relevant predicted quantity, comparison, tolerance, and scope are stated. Criteria should be chosen for the decision context before the result is interpreted. There is no universal numerical threshold that qualifies every QSP model.
Worked Example: Linking a Clinical Prediction Claim to Validation Evidence
Consider a hypothetical QSP model intended to predict mean change in HbA1c, in percentage points, at 24 weeks for adults with type 2 diabetes receiving a specified dose. The claim is C-01. Five hypothetical external cohort summaries are used for an illustrative validation analysis; they are not real study results. The predicted and observed mean changes are:
| Cohort | Predicted change | Observed change | Prediction minus observation |
|---|---|---|---|
| 1 | -0.8 | -0.7 | -0.1 |
| 2 | -1.0 | -0.9 | -0.1 |
| 3 | -0.9 | -0.8 | -0.1 |
| 4 | -1.2 | -1.0 | -0.2 |
| 5 | -0.7 | -0.6 | -0.1 |
Suppose the prespecified illustrative criteria are an absolute mean bias no greater than 0.15 percentage points and a mean absolute error no greater than 0.25 percentage points. Define each error as \(e_i=\text{prediction}_i-\text{observation}_i\). Then:
The absolute mean bias is \(0.12\) percentage points, below \(0.15\); the mean absolute error is also \(0.12\) percentage points, below \(0.25\). Both stated criteria are met by these illustrative calculations. The negative bias means predictions were, on average, 0.12 percentage points more negative than the observations. This result alone does not establish qualification: the matrix must also show which data and analysis were used, what verification evidence applies, and what limitations constrain the claim.
A traceability row could link C-01 to data D-01 (the five specified cohort summaries), analysis A-01 (the comparison and error calculations above), and criteria AC-01 (the two prespecified thresholds). It should separately reference the verification test IDs relevant to the model version and record applicable limitations, such as the small illustrative set and any population or dose-range boundaries. This preserves the distinction between a numerical result and the scope of the claim.
Completeness as a Traceability Check
The matrix can be viewed as a set of typed links. Let \(C\) be the set of intended-use claims and \(E\) the set of evidence and decision-rule items. A link is not merely “related”; it has a role, such as data supporting an analysis, a test checking an implementation feature, or a criterion judging a result. The role matters because one item cannot silently substitute for another.
This condition follows from the purpose of the matrix. If a claim has no linked validation analysis, a reviewer cannot trace how predictions for that use were compared with observations. If no criterion is stated, the reviewer cannot tell what result was intended to count as acceptable. If a limitation is omitted, the claim may appear broader than the actual evidence permits. Thus completeness requires checking each claim across the relevant evidence categories, not simply counting populated cells.
Assign a stable ID and record the population, intervention, outcome, horizon, and decision context that define the use.
Give each data source, verification test, validation analysis, criterion, and limitation a retrievable ID and version.
For each claim, state what each item contributes; do not use a broad label such as “supporting evidence” when the role is different.
Confirm that the listed evidence applies to the model version and claim scope, and that each criterion and unresolved limitation is explicit.
Worked Example: Recording Verification Evidence Separately
Suppose claim C-02 concerns whether a QSP implementation correctly maps a specified input dose and time schedule into the model’s dosing-input record. A hypothetical verification test V-04 uses a predefined test case with expected dose entries of 50 mg at 0 hours and 50 mg at 24 hours. The model’s recorded entries are exactly 50 mg at 0 hours and 50 mg at 24 hours. The test criterion is exact agreement for these recorded values and times, and the test passes.
The matrix should identify V-04 as a verification test, state the test case and expected behavior, record the observed output, and link the result to the tested software version. This is evidence that this particular input-mapping behavior matches its specification. It is not validation evidence that the model predicts a clinical outcome accurately, nor does it establish that every possible dosing schedule is implemented correctly.
If C-02 is itself an implementation claim, V-04 directly addresses it within the stated test scope. If the broader intended-use claim is a clinical prediction under that schedule, V-04 is one relevant verification link, but the matrix must also identify relevant data and validation analyses for the clinical prediction. The type of claim determines what the test result can support.
Worked Example: Making a Limitation Visible in the Claim Row
Suppose claim C-03 concerns predictions for a hypothetical treatment dose of 20 mg. The available validation data cover doses from 5 mg through 10 mg, and no observations are available at 20 mg. The matrix can still document C-03, but it should not imply that the available validation data directly test the 20 mg prediction.
Record the available data IDs and their dose range, identify any validation analysis performed on those data, and state the unresolved limitation explicitly: “No validation observations at 20 mg; the claim extends beyond the dose range represented in the validation data.” If there is a prespecified criterion for the 20 mg claim, list it, but do not mark the claim as meeting it on the basis of observations at lower doses alone.
This record does not decide whether the claim is ultimately acceptable. It makes the evidence boundary visible so that the qualification decision can be made with the limitation in view. A matrix that omits this limitation would make it difficult to distinguish direct evidence from an extrapolative use.
Diagnostics, Alternatives, and Limits
Several signs suggest that a matrix is not functioning as a traceability tool:
- Many claims point to the same broad phrase, such as “validated against clinical data,” without dataset IDs, analyses, or results.
- Acceptance criteria appear only after results are known, or use undefined terms such as “good agreement.”
- Verification and validation are combined into one column, making it unclear whether an item tests implementation or predictive performance.
- Limitations appear in a general report but are not linked to the claims they affect.
- Evidence is linked to an old model version, or its population and conditions do not match the claim’s stated scope.
- A populated table is treated as proof of adequacy, although it only demonstrates that entries have been recorded.
A matrix table is often easier to review than a narrative because rows support claim-by-claim checks and columns expose missing categories. A narrative can explain context and rationale more naturally, but cross-references can become difficult to follow. An evidence graph can represent many-to-many links clearly, especially for a large model, but it still needs a readable key and retrievable item records. These formats are alternatives for organizing the same traceability relationships; none removes the need to define claim scope and evidence roles.
A matrix also has limits. It cannot determine whether a dataset is representative, whether an analysis is statistically or scientifically appropriate, or whether a criterion is justified for a particular decision. Those require substantive review of the linked material. Nor should every conceivable model property be forced into every claim row: include items relevant to the claim, and explain a non-applicable category where its absence could otherwise be misunderstood.
Check Your Understanding
Answer each question, then compare your response with the brief solutions below.
- A row lists a validation analysis but no dataset ID. What traceability information is missing?
- A model passes a test that checks whether its output file is written in the specified format. Does that establish clinical predictive validity? Why or why not?
- A prespecified criterion is an absolute mean prediction bias of at most 0.20 percentage points. For hypothetical errors of 0.10, -0.20, and 0.10 percentage points, calculate the mean bias and state whether the criterion is met.
- What should the matrix say when a claim covers a dose for which no validation observations are available?
- Why should a matrix distinguish a documented limitation from an acceptance criterion?
Solutions. (1) Identify the data source used in the analysis, including a retrievable data ID and the relevant population and measurements; otherwise the comparison cannot be traced to its observations. (2) No. The test is verification evidence about output formatting. It does not compare clinical predictions with observations and therefore does not establish predictive validity. (3) The mean bias is \((0.10-0.20+0.10)/3=0.00\) percentage points. Its absolute value is \(0.00\), which is at most \(0.20\), so this criterion is met for these errors. (4) State the dose range represented in the available data and record that the claim has no direct validation observations at the specified dose; do not present lower-dose observations as direct validation at that dose. (5) A criterion is a rule for judging a result, whereas a limitation describes a boundary or uncertainty that remains. One does not substitute for the other.