Acceptance Criteria and Failure Rules for Qualification

Why Qualification Needs Prespecified Rules

Validation evidence does not determine its own meaning. A prediction error that is acceptable for exploratory prioritization may be unacceptable if it could change a dose-selection decision. Likewise, an apparently good average can conceal a failure in a population or outcome that matters to the intended use. Qualification therefore requires criteria that are chosen before validation results are examined, with a clear rule for combining them.

The claim being assessed should identify the model version, population, intervention or regimen, outcomes, prediction times, and decision context. It should also specify the domain in which the qualification claim applies. The earlier material on defining the domain of applicability is relevant here: performance outside the evaluated domain is not established by passing criteria within it.

A defensible plan distinguishes the claim, the performance measure, the threshold, the analysis set, and the consequence of meeting or missing the threshold. Thresholds are not universal properties of a model class; they depend on the scientific use, the measurement scale, and the consequences of prediction error. They must be justified in context rather than selected because a particular model happens to meet them.

Specify Quantitative Criteria in Context

For observed values \(y_i\) and corresponding predictions \(\hat{y}_i\), the signed prediction error is \(e_i=\hat{y}_i-y_i\). Positive values indicate overprediction and negative values indicate underprediction. For \(n\) validation outcomes, common summaries include mean error, mean absolute error (MAE), and root mean squared error (RMSE):

Mean error describes signed bias, while MAE and RMSE describe error magnitude. RMSE gives relatively greater influence to large errors because errors are squared. Each is expressed in the units of the outcome. A relative error can aid comparisons across scales, but its denominator and handling of values near zero must be specified; dividing by an observed value near zero can produce unstable percentages.

A threshold-based criterion can be more directly interpretable than an average. For example, a plan might require a specified proportion of predictions to be within a prespecified tolerance. The tolerance must have a meaningful scale and a justified connection to the use of the model. If an endpoint is critical, an overall average should not substitute for a rule on that endpoint or its relevant subgroup.

Coverage Criteria and Their Uncertainty

A prediction interval is an interval produced by the model for a future outcome under a specified prediction procedure. Its nominal level, such as 90%, is a model-based target under the assumptions of that procedure; it does not guarantee that exactly 90% of a finite validation sample will fall inside. Empirical coverage is the proportion of observed validation outcomes lying within their corresponding prespecified prediction intervals.

If the validation cases are independent and share a common probability \(q\) of being covered, each \(I_i\) is a Bernoulli variable with mean \(q\) and variance \(q(1-q)\). Therefore,

This shows why empirical coverage is noisy when the sample is small. The derivation relies on independence and a common coverage probability; repeated observations from the same participant or markedly different subgroups may not satisfy these assumptions. A plan should define the unit of analysis, the interval construction, the coverage target, and any subgroup-specific checks in advance. Coverage alone is also insufficient: very broad intervals can achieve high coverage while providing little useful predictive precision.

Combine Criteria Without Hiding Critical Failures

Qualification is usually better represented by a set of gates than by a single composite score. A plan may include an overall error limit, a coverage target, requirements for important subgroups or prediction times, and qualitative conditions such as correct direction of response. Each gate should name the data to which it applies and whether it is mandatory. A critical failure rule should state the failure condition precisely enough that assessors can apply it consistently.

For example, a critical rule could identify a maximum tolerable underprediction for an outcome whose low predicted value might lead to a harmful decision. Another could specify that a predicted response must have the correct direction across a prespecified dose contrast. These are not generic thresholds: their rationale and operational definition belong in the qualification plan. Any qualitative assessment should use an explicit rubric, define who evaluates it, and record the evidence supporting the judgment.

A practical decision rule distinguishes three outcomes. A pass requires all mandatory criteria to be met and no critical failure. A fail occurs when a prespecified critical failure is observed or a mandatory criterion fails under the stated rule. An inconclusive result applies when the evidence is insufficient to decide—for example, when a required subgroup has too few evaluable cases or a key measurement is unavailable—and no definitive failure rule has been triggered. Missing information is not evidence of a pass.

Justification, Diagnostics, and Alternative Decision Styles

The thresholds should be linked to the decision the model is intended to inform, not merely to customary values. One useful justification is to identify the prediction error that would change a decision and set tolerances accordingly. For example, if a difference of 10 units in a biomarker would not affect a downstream decision but an error of 30 units could, an error tolerance should reflect that distinction. The scientific and clinical rationale, including the direction of harmful error, should be documented.

Before applying the rule, check that the validation set, prediction times, units, interval construction, exclusions, and denominators match the plan. Inspect errors by regimen, time, and relevant population as prespecified. A trend in signed errors, an isolated extreme error, unexpectedly poor interval coverage, or a pattern of missing observations can reveal a problem that an overall summary obscures. Diagnostics explain how a criterion was met or missed; they should not become an unplanned route to redefine the criteria.

There are alternatives to a gate-based rule, but each answers a different question. A single average-error threshold is simple, yet may mask subgroup or tail failures. A collection of mandatory criteria makes important requirements visible but can be difficult to satisfy when evidence is sparse. Hypothesis tests ask whether data are inconsistent with a specified statistical null; they do not, by themselves, establish that predictive error is acceptably small. Equivalence-style reasoning can assess whether performance lies within prespecified margins, but requires an explicit margin and an appropriate uncertainty rule. The qualification plan should choose an approach that matches the claim and explain how uncertainty affects the decision.

A common mistake is to set thresholds after inspecting the validation results, or to treat a favorable average as proof that every relevant use is safe. This makes the decision vulnerable to selective reporting and can erase a failure that matters to the intended use. Set criteria before unblinding or analysis, preserve a record of the rule, and report every mandatory result—including unmet criteria, unavailable evidence, and deviations. If the evidence is inconclusive, report it as inconclusive rather than converting uncertainty into a pass.