What Shadow Mode Tests
Before a dosing service is allowed to influence care, it can be run in shadow mode: the service receives data from eligible clinical encounters and produces its usual output, but that recommendation is not shown to clinicians and does not enter the care decision. The existing clinical workflow remains authoritative. The point is to observe whether the service can operate alongside that workflow under real conditions, not to test whether clinicians benefit from using its recommendations.
Shadow mode is useful because a system can pass reference test cases and still encounter practical problems when connected to real records. Required data may arrive late, fields may be absent, encounters may be duplicated, or a recommendation may finish after the relevant decision has already been made. A hidden output lets a team discover such issues without asking clinicians to rely on an unlaunched service. It also reveals whether the service fits the timing and structure of care as it currently occurs.
The distinction from a controlled pilot deployment is consequential. A pilot may deliberately expose recommendations to a limited group under defined safeguards; shadow mode does not. It also differs from offline replay of historical records: shadow mode observes current workflow and data arrival, although it still cannot establish that a recommendation would improve outcomes. The earlier material on controlled pilot deployment addresses a setting where limited clinical exposure may be planned; here, the output remains hidden.
Define the Run Before Collecting Records
A shadow run needs an operational specification. Without one, a high success percentage can be misleading: the denominator might omit encounters where the service never triggered, and a late recommendation might be counted as successful despite missing the decision point. Specify the target population, eligible encounter types, start and end dates, triggering event, expected data window, acceptable response time, and the meaning of a successful execution before examining results.
Use an eligible encounter as the basic unit of analysis, and define it in a way that can be applied consistently from clinical records. For example, eligibility might require an active medication order, a specific care setting, and a dosing decision within the service’s intended scope. Decide in advance how to handle repeat triggers for one encounter, transfers between units, and records that are later corrected. Keep the encounter identifier stable enough to connect the service record to the relevant workflow event, while limiting access to identifiable data to authorized roles.
At minimum, a useful record distinguishes whether an encounter was eligible, whether the expected input data were available, whether the service ran, whether it returned an output or a defined failure state, and when each event occurred. Preserve the service version and relevant data timestamps so that a recorded output can be interpreted in context. A record saying only “recommendation generated” is inadequate if it cannot be linked to the correct encounter or to the time at which the input data became available.
The run also needs a clear stop rule. Pause or terminate it if a configuration error causes recommendations to become visible, if the service interferes with care, or if data are processed outside the approved scope. A “hidden” label alone is not a safeguard: confirm the actual display and access behavior, and make sure the team knows how to respond if separation fails.
Measures That Describe Operational Readiness
Shadow records can support a small set of interpretable measures. Each measure should name its numerator, denominator, unit, time window, and handling of missing records. These are descriptions of the observed run, not guarantees about future performance. They can point to where the workflow or system needs attention, but they do not by themselves validate the dosing model or establish clinical benefit.
| Measure | Example definition | What it helps assess |
|---|---|---|
| Input completeness | Eligible encounters with all required inputs available by the specified cutoff, divided by all eligible encounters | Whether routine records provide the fields and timing the service needs |
| Execution success | Eligible encounters with a completed, interpretable service response, divided by all eligible encounters | Whether the service runs reliably across the observed workflow |
| On-time completion | Completed responses returned by the prespecified decision deadline, divided by eligible encounters | Whether outputs arrive when they could be operationally relevant |
| Workflow coverage | Eligible encounters with a recorded trigger and a traceable outcome, divided by eligible encounters | Whether encounters enter the shadow pathway and can be accounted for |
These definitions make a difference. For example, calculating execution success only among encounters that triggered the service excludes trigger failures and may overstate end-to-end readiness. Reporting both the eligible-encounter denominator and the number of encounters that actually triggered helps separate failures of pathway coverage from failures after execution begins. Missing or unlinked records should be reported, not silently removed.
Worked Examples: Reading the Shadow Records
Worked Example: Input Completeness and Execution Success
In a hypothetical two-week run, 120 encounters meet the prespecified eligibility definition. The required inputs are available by the cutoff for 102 encounters. The service returns a completed response for 96 encounters; the other 6 input-complete encounters fail during execution. No completed responses are recorded for the 18 encounters missing required inputs.
Input completeness. The numerator is 102 encounters with complete inputs by the cutoff; the denominator is all 120 eligible encounters:
Thus, 85.0% of eligible encounters had the required data available on time. This measure does not say that 85.0% of all patients in the service’s intended population would have complete data; it describes these eligible encounters during this run.
End-to-end execution success. Using all eligible encounters as the denominator:
For additional diagnosis, execution success among encounters with complete inputs is \(96/102=94.12\%\). The difference between 80.0% and 94.12% matters: the first includes data availability in end-to-end performance, whereas the second describes execution conditional on complete inputs. Reporting only the conditional percentage would conceal the 18 encounters that could not proceed because their inputs were unavailable.
Worked Example: Does the Output Arrive in Time?
In a separate hypothetical run, a response is considered timely if it is recorded within 5 minutes of the defined trigger. For 40 eligible encounters, 36 produce a completed response. Of those 36, 30 finish within 5 minutes. The remaining four encounters have no completed response.
Among completed responses, the timely fraction is \(30/36=0.8333\), or 83.33%. But this excludes the four encounters with no response. For a measure of on-time completion across the whole eligible workflow, use 40 as the denominator:
The 75.0% figure answers: “What proportion of eligible encounters had a completed response by the deadline?” The 83.33% figure answers a narrower question: “Among completed responses, what proportion met the deadline?” Both can be useful if labelled precisely. Neither establishes that 5 minutes is the clinically appropriate deadline; that threshold must be set from the intended workflow before the run.
Worked Example: Locating a Workflow Bottleneck
A hypothetical service is expected to trigger when a medication order is entered. Across 50 eligible encounters, records show 50 order-entry events, 45 service triggers, and 42 completed responses. The team has also defined an acceptable response window of 4 minutes; 35 of the 42 completed responses arrive within that window.
Trigger coverage is \(45/50=90.0\%\), so five eligible order-entry events did not generate a recorded trigger. Execution success after a trigger is \(42/45=93.33\%\), leaving three triggered encounters without a completed response. On-time completion among all eligible encounters is \(35/50=70.0\%\). The stepwise counts point to distinct questions: why some order events did not trigger, why some triggered runs did not complete, and why some completed responses arrived late.
This breakdown is more informative than saying “the service worked in 70% of cases.” It does not yet identify the cause of any failure. The team should inspect authorized, appropriately limited records and compare timestamps and documented statuses before attributing a cause, rather than inferring that the model itself is responsible.
Justifying the Denominator
The main interpretive safeguard is to preserve the complete eligible-encounter denominator. Let \(N\) be the number of eligible encounters, \(C\) the number with complete inputs by the cutoff, and \(R\) the number with a completed response. If a completed response requires complete inputs, then \(R\leq C\leq N\). End-to-end response coverage can be decomposed as:
provided \(C>0\) and every completed response belongs to the input-complete group. The first factor is input completeness; the second is completion conditional on input completeness. This identity follows by cancelling \(C\): \((C/N)(R/C)=R/N\). It explains why conditional execution success alone is insufficient: it describes only the second stage and leaves out failures to supply the required inputs. If the service can produce a defined response without complete inputs, define that response category separately rather than forcing the records into this decomposition.
This is descriptive accounting, not a causal analysis. A low rate might reflect data access, an incorrectly defined trigger, a service defect, or an unusual mix of encounters. The measure identifies a pattern to investigate; it does not prove which mechanism produced it.
Diagnostics, Alternatives, and Limits
Inspect records in sequence: eligibility, trigger, input availability, execution status, output recording, and elapsed time. Look for absent stages, implausible timestamp order, duplicate encounter records, and disagreement between the service log and the clinical event record. Summaries should be checked against the underlying counts. A percentage with an unexplained denominator, or a timestamp calculated from inconsistent events, can make a healthy-looking dashboard conceal a broken pathway.
A shadow run is not the only way to test a system. Offline reference cases are controlled and reproducible, making them useful for checking known inputs and expected outputs, but they do not show whether current clinical data arrive in usable form. A controlled pilot can reveal clinician interaction with recommendations, but exposes users to the output and therefore requires different safeguards. Shadow mode occupies a middle ground: it observes current workflow and real-time data flow while withholding the recommendation from care. It cannot answer how clinicians will interpret the output, whether they will act on it, or whether patient outcomes will improve.
A further limitation is representativeness. A short run, a narrow set of shifts, or an unusual mix of encounters may not reflect the workflow across all intended settings. Report the dates, sites or units, eligibility rules, and encounter counts alongside the measures. Do not treat a successful shadow period as proof that the service will remain reliable after configuration changes or broader use. The operational monitoring of a live service is a distinct activity.
Shadow mode can also create a false sense of safety if hidden recommendations are mistaken for harmless data. They may still contain sensitive information, and logs may be accessible to people who could use them informally in clinical decisions. Apply appropriate permissions and retention practices, and avoid sharing patient-specific outputs with treating clinicians during the run. If recommendations are reviewed for operational testing, separate that review from clinical decision-making and document its purpose.
A common mistake is to claim that a high shadow-mode success rate proves the dosing recommendation is accurate and safe. That conclusion does not follow. For example, a service could return a response on time for every eligible encounter while using an incorrect input mapping or an unsuitable model. Shadow-mode measures describe whether the intended pathway ran and when; they do not establish analytical validity or clinical effectiveness. Those claims require evidence designed for those questions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a run with 80 eligible encounters, 68 have complete inputs and 60 receive completed responses. What are end-to-end execution success and execution success conditional on complete inputs?
- Why can a service that completes 95% of triggered encounters still have poor end-to-end coverage?
- For 25 eligible encounters, 20 completed responses are recorded and 16 arrive by the prespecified deadline. What is on-time completion among all eligible encounters?
- Name one question shadow mode can answer that offline replay may not answer, and one question it cannot answer about clinician behavior.
- A team finds that every response is generated on time. Why is this not enough to conclude that the dosing service is clinically valid?
Solutions. (1) End-to-end success is \(60/80=75.0\%\); conditional success is \(60/68=88.24\%\). The denominators answer different questions. (2) The triggered denominator may exclude eligible encounters that failed to trigger or lacked usable inputs; report success against all eligible encounters as well. (3) On-time completion is \(16/25=64.0\%\). (4) Shadow mode can reveal whether current data and triggers reach the service at the right time; because clinicians do not see recommendations, it cannot show how they interpret or act on them. (5) Timely execution describes operational behavior, not whether the input mapping, model predictions, or resulting dose recommendations are accurate or beneficial.