Why Define the Review Before Starting?
An independent model review is useful only when its conclusion is clear about what was reviewed, for what purpose, and against which criteria. Without those boundaries, reviewers may spend time repeating the original modeling work while leaving important risks unchecked. A vague conclusion such as “the model was reviewed” can also be mistaken for a broad assurance that the model, data, implementation, and intended use are all correct.
The scope is the agreement that prevents those mismatches. It identifies the specific analysis and decision context under review, the questions the reviewer will address, the work that is out of scope, and the evidence needed to reach a conclusion. The goal is not to make the review as large as possible. It is to make the chosen level of checking adequate for the risks and the decision it supports.
Set the Boundary and the Intended Assurance
Begin by naming the review object precisely enough that two people would select the same analysis. A useful object description identifies the model or analysis version, the run or result set in question, and the decision context. If any of these are ambiguous, a reviewer may assess a different object from the one the requester intends.
Next, translate the request into review questions. “Review the model” is not a review question. Questions such as “Are the specified model components represented consistently in the analysis implementation?” or “Are the reported conclusions supported by the designated outputs?” are more actionable. The review question sets the boundary: it tells the reviewer what claim to examine, rather than inviting a second attempt at model development.
A scope should also state the level of assurance sought. A focused review may assess a small number of high-consequence claims. A broader review may cover more categories of claims, but still need not repeat every modeling decision or computation. Complete independent re-performance is a different and more resource-intensive approach; it is not an automatic feature of an independent review.
| Scope element | Question to answer | Example of a bounded statement |
|---|---|---|
| Object | Which analysis is being reviewed? | The specified population analysis and its designated final run. |
| Purpose | Which decision or claim will the review inform? | Whether the analysis is ready to inform a dose-selection discussion. |
| Review questions | Which claims will be checked? | Whether selected high-impact model features are implemented as specified. |
| Exclusions | What is not being assessed? | New model development and a full independent re-analysis. |
| Criteria | What evidence supports a pass or a finding? | All pre-defined critical checks pass, with no unresolved critical discrepancy. |
Exclusions are not a way to conceal risk. They make clear which claims the review cannot support. If an excluded area is necessary to answer the decision question, the scope should change or the conclusion should explicitly remain limited. A finding that the defined checks passed is not equivalent to a finding that all possible checks would pass.
Worked Example: Turning a Broad Request Into a Bounded Scope
A hypothetical team asks for an independent review of a population pharmacokinetic analysis intended to inform a dose-selection discussion. The request does not specify which model or what “review” means. The reviewer and requester agree on the following boundary.
Object and purpose. The object is the designated final analysis run for the specified model version. The review is intended to inform whether the analysis is ready for the planned dose-selection discussion, not to select a new model.
Questions included. The reviewer will assess a defined set of high-impact implementation and output-consistency claims, and will check that the stated interpretation is consistent with the designated results. This defines questions to examine without asking the reviewer to rebuild the model or repeat the original model-development process.
Exclusions. New structural model development, an independent full re-analysis, and claims about uses beyond the planned dose-selection discussion are excluded. If a review finding raises concern in one of those areas, it will be recorded as a limitation or trigger a recommendation for additional work; the finding will not be treated as evidence that the excluded area is satisfactory.
The result is a bounded assurance statement: the review can support a conclusion about the specified questions and analysis object, but not a general claim that the model is valid for every population, use, or decision.
Prioritize Review Effort by Risk
When time is limited, review questions should not all receive equal attention by default. A practical prioritization starts with two considerations: how plausible it is that a particular problem exists, and how consequential it would be if it did. The review plan then gives more attention to issues that are both plausible and consequential, while ensuring that the stated scope still covers its required checks.
A simple score can help make this reasoning visible. Rate likelihood and consequence on pre-defined ordinal scales, for example from 1 (low) to 5 (high), and calculate their product. This is a ranking aid, not a probability, a statistical estimate, or a universal threshold. The ratings should be explained in context; multiplying ordinal ratings does not turn them into precise measurements of risk.
Here \(R_i\) is the priority score for review question \(i\), \(L_i\) is its likelihood rating, and \(C_i\) is its consequence rating. The logic is related to expected loss: if likelihood were a probability and consequence were a loss measured on a meaningful scale, their product would represent expected loss. In a practical ordinal scheme, however, the inputs do not meet those measurement conditions. Use the score to prompt and document judgment, not to claim a quantitatively calibrated risk.
Worked Example: Allocating Attention Across Review Questions
For a hypothetical review, the team rates three candidate questions on scales from 1 to 5. A rating of 5 means the highest likelihood or consequence category in the team’s defined scale. The ratings are planning judgments, not measured frequencies.
| Review question | Likelihood \(L_i\) | Consequence \(C_i\) | Score \(R_i\) |
|---|---|---|---|
| Could a high-impact model feature differ from its specification? | 3 | 5 | 15 |
| Could a stated interpretation overreach the designated results? | 3 | 4 | 12 |
| Could a minor presentation detail be unclear? | 2 | 2 | 4 |
For the first question, \(R_1=3\times5=15\). For the second, \(R_2=3\times4=12\). For the third, \(R_3=2\times2=4\). The ordering is therefore 15, 12, then 4. The first question receives the most review attention because a discrepancy is judged both plausible and consequential. The presentation issue remains within scope if it matters to interpretation, but it should not displace higher-priority checks merely because it is easy to inspect.
The score does not mean the first issue is 15 times as risky as the third, nor does it give the probability that a discrepancy will occur. If stakeholders disagree about a rating, record the reasoning and resolve the judgment before relying on the ranking. A high-consequence issue should not be omitted solely because its likelihood rating is low.
Write Acceptance Criteria Before Evaluating Results
Acceptance criteria define how the reviewer will classify the evidence for each in-scope question. They should be stated before the review conclusions are known. A criterion is useful when it identifies what must be true, what evidence will be considered, and how a discrepancy or missing evidence affects the conclusion.
Not every check needs the same criterion. A critical check might require no unresolved discrepancy. A less consequential check might permit a documented limitation if it does not affect the review’s stated purpose. Whatever the rule, define it in advance rather than deciding after seeing a result that a particular issue is unimportant.
Keep three outcomes distinct:
- Pass: the specified evidence supports the criterion for the defined review question.
- Fail: the evidence shows that the criterion is not met, or a discrepancy exceeds the pre-defined tolerance.
- Not assessed or unresolved: adequate evidence is unavailable, ambiguous, or insufficient to decide. This is not a pass.
Worked Example: Applying Pre-Defined Acceptance Criteria
A hypothetical review plan lists 12 checks: 4 critical checks and 8 supporting checks. The criteria state that every critical check must pass, no critical discrepancy may remain unresolved, and at least 7 of the 8 supporting checks must pass. Any failed or unassessed critical check prevents an overall “meets criteria” conclusion.
After review, 3 critical checks pass and 1 critical check fails. All 8 supporting checks pass. The supporting-check requirement is met because \(8\geq7\), but the critical-check requirement is not met because only \(3\) of \(4\) critical checks pass.
The overall criteria are not met: the critical failure is decisive under the stated rule, even though all supporting checks pass. The conclusion should identify the failed check and its consequence for the review purpose. It should not describe the analysis as accepted, and it should not imply that the review assessed areas outside the defined 12 checks.
Justifying the Scope and Its Limits
A review plan can be represented as a set of questions \(Q\), each with an importance weight \(w_i\) and a defined criterion. The weights make the intended priority explicit, with higher values assigned to questions judged more consequential. If a summary coverage measure is useful, define it as the weight of questions meeting criteria divided by the total weight of in-scope questions:
In this expression, \(I_i=1\) when question \(i\) meets its criterion and \(I_i=0\) otherwise. If the weights are \(5,3,2\) and only the first two questions meet criteria, weighted coverage is \((5+3)/(5+3+2)=8/10=0.80\). This summary can describe the proportion of assigned review weight that met criteria; it cannot replace critical-check rules. A single critical failure may prevent acceptance even when weighted coverage is high.
This distinction follows directly from the purpose of the measure. A weighted average can summarize several results, but averaging does not erase a non-negotiable condition. Therefore, specify both any summary measure and any mandatory criteria. The scope is adequate only if its set of questions addresses the decision’s material risks and the resulting criteria support the exact conclusion being sought.
Diagnostics, Alternatives, and Common Pitfalls
Several signs indicate that a scope is not working as intended. Reviewers repeatedly ask what version or claim they should assess; time is consumed by low-impact issues while high-impact questions remain open; different reviewers apply inconsistent pass rules; or the final conclusion is broader than the checks performed. These are signals to clarify the boundary, adjust priorities, or narrow the conclusion—not reasons to silently expand the work.
A checklist-only approach can improve consistency and prevent omissions, but a checklist is not a risk assessment unless its items and priority reflect the review purpose. At the other extreme, repeating the entire analysis may provide broader comparison evidence but requires substantially more effort and answers a different question from a bounded independent review. A risk-focused review offers a deliberate middle ground: it targets important claims while documenting what was not assessed.
A common mistake is to treat “independent” as meaning “repeat everything.” That reasoning confuses independence of judgment with duplication of work. The correction is to specify independent review questions, evidence, and criteria; repeat or extend work only when the agreed scope calls for it or a finding makes additional checking necessary. Another mistake is to call an unassessed item “acceptable” because no problem was observed. The correction is to classify it as unassessed and limit the conclusion accordingly.
A scope also has limitations. Risk ratings depend on informed judgment, and a low score may reflect incomplete understanding rather than low actual risk. Acceptance criteria can be too permissive, too vague, or mismatched to the decision. Finally, a bounded review cannot provide assurance about excluded questions. These limitations should be addressed through transparent rationale, escalation rules, and careful wording of the conclusion.
Specify the analysis, version or run, and decision context so the reviewer and requester share the same target.
State the claims to assess and the areas deliberately excluded; do not substitute an open-ended request to “check the model.”
Document likelihood and consequence judgments, identify mandatory high-priority checks, and allocate effort accordingly.
Define pass, fail, and unassessed outcomes before review, including how critical discrepancies affect the overall conclusion.
Report only what the defined checks support and identify exclusions, unresolved items, and any need for additional review.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why is “review the model” an inadequate scope statement?
- A review question has likelihood rating 4 and consequence rating 2 on the agreed ordinal scales. Calculate its priority score and state what the score does not mean.
- A plan requires all 3 critical checks to pass. Two pass and one is not assessed. Can the review meet its critical-check criterion?
- Why should an excluded area be named explicitly rather than assumed to be outside the review?
- When is a full independent re-performance different from a bounded independent review?
Solutions. (1) It does not identify the object, purpose, questions, exclusions, evidence, or criteria, so the reviewer and requester may have different expectations. (2) \(4\times2=8\). This is a relative planning score from ordinal ratings, not a probability or a calibrated quantity of risk. (3) No. The criterion requires all 3 critical checks to pass, and an unassessed check is not a pass. (4) Explicit exclusions prevent a bounded conclusion from being interpreted as assurance about unexamined areas; if an excluded area is necessary to the decision, the scope must change or the conclusion remain limited. (5) Re-performance attempts a broader independent recreation or repetition of the analysis, whereas a bounded review assesses specified claims against defined evidence without automatically repeating all original work.