Measuring Safety-Net Performance with Near Misses

Why Safety-Net Performance Needs More Than an Alert Count

A dosing safety net may reject an out-of-range dose, flag a contraindication, or prompt a clinician to review a recommendation. Counting alerts tells us how often the system signalled, but not whether it recognized the hazards it was meant to catch, how often it signalled unnecessarily, or whether a hazardous decision was actually stopped. Those are different performance questions, and each requires a clearly defined event and denominator.

The earlier material on hard safety constraints, abstention, input validation, clinician overrides, and alert thresholds describes different safeguards and their intended actions. Here the focus is evaluation: defining indicators that distinguish detection, false alarms, interception, and near misses. These indicators measure different stages of a safety pathway. None, on its own, establishes that a patient was or was not harmed.

Define the Episode and the Reference Standard

Before calculating a rate, choose the unit being counted. It might be a dose recommendation, a prescribing decision, or a treatment episode. Specify whether repeated alerts within one episode count as one signal or several. A useful evaluation unit is often one eligible dosing decision, with each decision classified against a prespecified target hazard. If several distinct hazards can occur in one decision, either evaluate each hazard separately or define an explicit rule for classifying the episode; otherwise the counts can be ambiguous.

The reference classification must describe whether the target hazard was present, not merely whether the safety net raised an alert. For example, trained reviewers might assess whether the proposed dose violated a prespecified safety rule using the information available at the decision time. Review procedures should be defined in advance and should not use the alert itself as the sole basis for calling a case hazardous. Reviewers may be blinded to the safety-net signal where feasible. A reference assessment is still an operational standard, not an infallible truth: incomplete records, disputed clinical judgments, or missing information can lead to classification error.

The four counts partition the evaluated episodes only if each episode has one target-hazard classification and one signal classification. For this table, “signal” must also be operationally defined: a warning displayed, a blocked recommendation, or a request for review may be distinct actions. If a signal for one hazard is counted as a positive for another, report the target and rule for each analysis.

Detection and False-Alarm Indicators

Detection sensitivity asks what fraction of adjudicated hazardous episodes the safety net signals. The false-alarm rate asks what fraction of nonhazardous episodes it signals. Specificity is the complementary proportion of nonhazardous episodes without a signal. Positive predictive value (PPV) asks what fraction of signals occur in hazardous episodes. These measures answer different questions and should be reported with their denominators.

Sensitivity is conditional on a hazardous episode; the false-alarm rate and specificity are conditional on a nonhazardous episode; PPV is conditional on a signal. The denominators therefore matter. A measure such as “percent of alerts that were appropriate” is PPV only if “appropriate” is defined as detecting this particular target hazard and all relevant alerts are counted. It is not sensitivity.

A short derivation makes the distinction clear. The hazard-positive episodes are exactly the \(TP\) and \(FN\) episodes, so their total is \(TP+FN\); among them, the safety net signals in \(TP\) episodes. Thus sensitivity is \(TP/(TP+FN)\). The nonhazardous episodes are \(FP+TN\), and \(FP\) of those receive a signal, giving the false-alarm rate \(FP/(FP+TN)\). Since \(FP+TN\) is also the denominator for specificity and \(TN+FP\) is partitioned into signalled and unsignalled episodes, specificity is \(TN/(FP+TN)=1-\text{false-alarm rate}\).

PPV depends not just on the safety net’s conditional performance but also on how common the target hazard is in the evaluated episodes. If hazards are rare, even a modest false-alarm rate can produce many false alarms relative to true detections. Consequently, a PPV from one setting or period may not transfer to another with a different hazard prevalence, even if sensitivity and specificity remain similar.

Interception Is a Separate Stage

Detection is a signal or other recognition of a hazard. Interception requires a further event: the unsafe action or error is stopped or corrected before it reaches the patient. A displayed alert is not automatically an interception. The clinician may override it, the alert may arrive too late, or another process may already have corrected the decision. Conversely, a hazard may be intercepted by a clinician or a different safeguard even when the safety net under evaluation did not signal.

For a clearly defined set of adjudicated hazardous episodes, let \(I\) be the number in which the target error was actually intercepted before reaching the patient. The intercepted fraction among hazardous episodes is \(I/(TP+FN)\), provided the same episode definition and complete outcome ascertainment are used. If the question is instead “how often did a detected hazard lead to interception?”, use \(I_D/TP\), where \(I_D\) counts detected hazardous episodes that were intercepted. State which measure is reported. They are not interchangeable.

Attribution also matters. An intercepted error is not necessarily intercepted because of the safety net under study. To describe system performance, record the sequence where possible: hazard present, signal generated, signal received, action taken, and error stopped before reaching the patient. A claim that the safety net itself prevented an error is strongest when the event record supports that causal sequence. Without such evidence, report the observed interception and its context rather than asserting a counterfactual outcome.

Near Misses: Useful Events, Not a Complete Performance Score

A near miss is an error or hazardous process that is intercepted before it reaches the patient. This definition distinguishes it from a harmful event, in which the error reaches the patient and causes harm, and from a hazardous episode that reaches the patient without documented harm. Local reporting systems sometimes use “near miss” more broadly, so the operational definition should be stated whenever counts are compared.

A near-miss count is an event count, not a sensitivity estimate. A rate can be calculated by dividing near misses by a meaningful exposure denominator, such as eligible dosing decisions or patient-days, and multiplying by a stated scale. The denominator must correspond to the opportunities for the event. For example, a count per 1,000 dosing decisions is not directly comparable to a count per 1,000 patient-days unless the relationship between those exposures is understood.

Near-miss data are especially vulnerable to ascertainment differences. Some events may be logged automatically, while others depend on staff recognition and reporting. A rise in reports can reflect more hazards, better reporting, changed workflow, or a combination. A low count can indicate few hazards, missed or unreported events, or failure to retain event records. Thus a count or rate alone cannot establish that the safety net is improving or worsening.

Diagnostics, Alternatives, and Limits

A defensible performance summary checks the data structure before interpreting the numbers. Confirm that each eligible episode has a defined target hazard, an ascertainable signal status, and an outcome status where interception is being assessed. Reconcile the four classification counts to the number of episodes. Investigate missing records rather than silently treating them as true negatives. If reviewers disagree, report how disagreements were resolved and consider whether ambiguous cases materially change the results.

• Zero denominator: if no hazardous episodes were observed, sensitivity cannot be estimated from that set; it is not 100%. If no signals occurred, PPV is undefined, not zero.
• Changing alert definitions: a change in thresholds, logging, or what counts as a signal can alter the measures even when underlying hazard frequency is unchanged.
• Multiple hazards or alerts: counting alerts rather than episodes can inflate event totals. Define whether repeated signals are deduplicated and how overlapping hazards are classified.
• Imperfect adjudication: reviewer error can move cases between TP, FN, FP, and TN. Sensitivity and PPV are only as credible as the reference process and event data.
• Prevalence and case mix: PPV and near-miss counts can change when the frequency or type of hazardous opportunity changes, even if the safety net itself has not changed.

Different summaries are useful for different decisions. Sensitivity and false-alarm rate characterize detection conditional on the reference classification. PPV helps describe the yield of signals in the evaluated setting, while alert count or alerts per exposure unit helps describe operational burden. Interception fractions assess whether hazardous episodes were stopped, and near-miss rates describe recorded intercepted events relative to exposure. Reporting these together is usually more informative than selecting just one.

A common mistake is to say, “The system caught 90% of errors, so it prevented 90% of harm.” That conclusion confuses detection with interception and then treats interception as equivalent to avoided harm. The correction is to report each stage separately: the denominator of hazardous episodes for detection, the number and proportion intercepted before reaching the patient, and any observed patient outcomes. An intercepted error is valuable evidence of a functioning safety process, but it does not reveal with certainty what harm would have occurred in the counterfactual absence of interception.