Postdeployment Audits of Dosing Policy Safety

Why a Dosing Policy Needs a Postdeployment Audit

A dosing policy that performs acceptably in development can behave differently after deployment. Patient mix, clinical workflows, data capture, and the way clinicians respond to recommendations may change. A postdeployment audit is a planned review of these real-world signals, designed to detect possible safety degradation early enough to investigate and respond. It is not simply a count of adverse events: a useful audit also examines whether the policy was used as intended, when clinicians overrode it, whether risks are concentrated in particular groups, and whether required processes failed.

The audit should extend, not replace, the earlier material on Measuring Safety-Net Performance with Near Misses. Near misses can reveal hazards before harm occurs; a postdeployment audit also follows patient outcomes and implementation performance over defined periods. Neither a low adverse-event count nor a high override rate, by itself, establishes that the policy is safe or unsafe. Interpretation depends on a suitable denominator, reliable data, context, and prespecified follow-up rules.

Specify the Audit Before Looking at Results

An audit begins with a written plan. For each measure, define what counts as an event, who is eligible to contribute to the denominator, the observation window, the data source, and how missing or duplicated records will be handled. If the outcome is toxicity within 30 days of a recommendation, for example, the numerator should count eligible patients with a qualifying toxicity in that window, and the denominator should count eligible patients with sufficient follow-up to ascertain it. Counting events per recommendation when some patients receive several recommendations can overweight those patients; the unit of analysis must match the question.

Choose comparison periods or reference values that are clinically interpretable. A predeployment rate can be useful, but changes in patient mix, care, surveillance, or documentation can also change rates. A before-and-after difference is therefore a signal for review, not necessarily an effect of the policy. Where feasible, retain relevant covariates and a clearly defined comparator so that subsequent investigation can assess whether the groups and observation processes are sufficiently comparable.

Include multiple complementary measure types. Adverse outcomes assess patient impact. Overrides describe how clinicians respond to recommendations, but an override can be prudent and appropriate. Subgroup summaries can reveal unequal performance hidden by an overall average. Process measures assess whether essential steps—such as valid inputs, contraindication checks, or escalation—were completed. For each measure, distinguish a signal that prompts verification from a threshold that requires immediate protective action.

Comparing Outcome Rates and Their Uncertainty

For two independent groups with binary outcomes, let \(\hat p_1=x_1/n_1\) and \(\hat p_0=x_0/n_0\) be the observed event proportions in the postdeployment and reference groups. The estimated risk difference is \(\hat p_1-\hat p_0\); a positive value indicates a higher observed event proportion in group 1. Under independent binomial sampling, a large-sample standard error is

An approximate 95% Wald interval is the estimated difference plus or minus \(1.96\) standard errors. This approximation assumes independent observations, reasonably stable event ascertainment, and enough events and non-events for the normal approximation to be useful. It can perform poorly with sparse events or small denominators. In an operational audit, use a score-based or exact method when appropriate, and report the method used. An interval that includes zero does not demonstrate safety; it may simply be too wide to rule out a clinically important increase.

The justification for the standard error follows from the variance of a sample proportion. For independent Bernoulli outcomes, the variance of \(\hat p_i\) is \(p_i(1-p_i)/n_i\). Independence between the groups means the variance of their difference is the sum of the two variances. Replacing the unknown \(p_i\) values with their observed proportions gives the estimated variance above. The interval then uses the approximate normal sampling distribution of that difference. This derivation also shows why a small denominator or few events makes the estimate unstable.

Overrides, Subgroups, and Process Failures

An override rate is interpretable only after defining its denominator and categories. For example, count recommendations overridden per eligible recommendations, and separate overrides that move the dose upward from those that move it downward or lead to escalation. Review a sample of records to determine whether overrides reflect appropriate clinical judgment, a recommendation that is systematically unsuitable, or missing information. Treating every override as a policy failure can discourage clinicians from correcting an unsafe recommendation; treating every override as harmless can conceal systematic problems.

Subgroup analysis can identify a safety signal that an overall rate obscures. Prespecify clinically relevant groups where possible, and report each subgroup’s numerator, denominator, and uncertainty. A striking percentage based on very few patients may be unstable. Conversely, a wide interval is not evidence that the subgroup is unaffected. If many subgroups are searched after seeing the data, some will look unusual by chance. Treat exploratory patterns as leads for validation, and avoid claiming a subgroup difference merely because one subgroup estimate is statistically significant while another is not.

Process failures need their own denominators. If a required check was missed in 9 of 450 eligible decisions, the observed failure proportion is \(9/450=0.0200\), or 2.00%. That percentage is not the proportion of all patients harmed; it measures how often a specified process step failed among opportunities for that step. Severity matters: a rare failure that can permit a dangerous dose may warrant immediate review even when the overall rate is low. Audits should therefore track both frequency and potential consequence.

Triggers, Diagnostics, and Corrective Action

Before deployment, specify how often the audit will run, which measures receive routine review, and what findings trigger verification, escalation, or a policy change. A trigger can combine statistical evidence with clinical severity. For instance, a prespecified increase beyond a clinically meaningful margin may prompt formal review, while any verified event involving a potentially catastrophic process failure may prompt immediate escalation regardless of the aggregate rate. Do not select thresholds after observing a favorable or unfavorable result.

Repeated monitoring creates another risk: if many measures are checked frequently, some will cross a nominal threshold by chance. Reduce overreaction by defining a limited set of primary safety measures, using appropriate sequential monitoring methods when repeated formal testing is planned, and treating secondary signals as evidence to verify rather than as definitive proof. A control limit or action threshold should be selected for the monitoring objective and consequences of missed versus false alarms; it is not automatically a universal boundary between safe and unsafe.

When a signal appears, first check for coding changes, delayed reports, denominator shifts, duplicate records, and changes in surveillance. Then inspect the clinical details and the policy’s operational pathway: input validity, recommendation, any override or fallback, required checks, and the subsequent outcome. If the signal persists, consider proportionate actions such as targeted monitoring, workflow repair, retraining, restricting use for a defined situation, or pausing use while a serious hazard is assessed. Document the evidence, decision, owner, and date for reassessment. After an intervention, continue the same measures so the audit can determine whether the signal changed.

Common Pitfalls and Limits

• Calling a before-and-after difference a policy effect. Patient mix, clinical practice, follow-up, and event recording may have changed. Describe the observed contrast and investigate plausible alternatives before making causal claims.
• Using the wrong denominator. Events per patient, per recommendation, and per eligible safety-check opportunity answer different questions. State the unit and denominator explicitly.
• Equating no detected increase with no risk. Sparse events and small samples can produce wide intervals. Report uncertainty and whether clinically important harm remains compatible with the data.
• Reading subgroup percentages without their counts. Always show numerators and denominators; small counts can make rates unstable.
• Labeling every override a failure—or ignoring overrides. Examine direction and clinical rationale, then assess whether a recurring pattern points to an unsuitable policy or workflow.
• Changing thresholds after seeing results. Post hoc rules invite selective interpretation. Prespecify triggers and clearly label exploratory findings.