From a Validated Model to a Working Hospital Service
A model can perform well in evaluation and still fail to support care if the hospital sends it incomplete data, displays its recommendation in the wrong place, or has no agreed response when an interface is unavailable. Implementation is therefore a coordinated clinical and technical workflow, not simply installation of software. The goal is a service that produces recommendations only within its intended use, makes limitations visible, and leaves prescribing authority with the responsible clinician.
Consider a hypothetical new hospital deploying a dosing service for fictional Drug Q. The hospital has selected a model that has already undergone analytical and external evaluation. The service is intended to support maintenance-dose review for adult inpatients with a specified indication; it is not intended for loading doses, pediatric patients, or patients receiving a particular excluded concomitant treatment. All names, thresholds, counts, and performance targets below are hypothetical. A local clinical and governance group would need to approve the actual scope and criteria before use.
Map the Workflow Before Connecting Systems
Start by mapping what must happen in sequence, including steps that end the process without a recommendation. A useful map names the event that starts each step, the system or person responsible, and the condition for proceeding. This makes missing handoffs visible before they become clinical problems.
The electronic health record (EHR) detects an inpatient Drug Q order or another locally approved trigger. Eligibility is checked against the model’s stated population and exclusions.
The interface retrieves required demographics, dosing history, laboratory results, and relevant clinical covariates. Each item is checked for completeness, units, and clinically appropriate timing.
Valid, in-scope inputs are sent to the model service. Missing, contradictory, stale, or out-of-scope data lead to a clear no-recommendation state rather than an unsupported dose.
The EHR receives the recommendation, its relevant context, and any cautions. The display identifies the intended patient and encounter and distinguishes a recommendation from an order.
An authorized clinician reviews the output and may accept, modify, or decline it. The clinical decision and its author are recorded in the EHR.
Operational and clinical teams review service availability, exception patterns, and pilot measures. Defined escalation routes apply when a defect or safety concern is suspected.
The order matters because errors can propagate. For example, a dose history attached to the wrong encounter could produce a mathematically valid result for the wrong clinical context. Eligibility and identity checks must therefore precede calculation, while display and documentation checks must follow it. This is a workflow-design justification, not a claim that any one check can guarantee safety.
Inventory the Data and Its Meaning
For every model input, record its source, representation, timing, and handling when it is unavailable. “Laboratory value” is not a sufficient specification: the inventory must distinguish the analyte, unit, collection time, result status, and any rule for selecting among multiple results. It should also identify which inputs are required, which are optional, and which are used only to determine eligibility.
| Data element | Source and representation | Implementation check | Failure response |
|---|---|---|---|
| Patient and encounter identifier | EHR; approved identifiers | Confirm the returned result is linked to the triggering patient and encounter | Suppress result; log and route for technical review |
| Age and body size inputs | Demographics and measured observations; specified units | Check plausible range, unit conversion, and date of measurement | Request correction or return no recommendation |
| Drug Q administration history | Medication administration record; dose and timestamp | Distinguish administered doses from orders that were held or not given | Do not infer administration from an order alone |
| Laboratory result | Laboratory system; value, unit, collection time, status | Apply the agreed result-selection and recency rules | Show the missing or stale-data reason; do not silently substitute |
| Concomitant treatment and eligibility covariates | Medication list and clinical record | Map local codes to the specified inclusion or exclusion definitions | Escalate ambiguous mapping; suppress if scope cannot be established |
The inventory should be reviewed by both the model team and the clinicians who understand how local records are created. A field may exist in the EHR yet be unsuitable for the service because it is entered late, uses a different unit, or represents an order rather than an administered dose. Record these distinctions in the interface specification and in test cases.
Worked Example: Resolving a Timing Rule
Suppose the hypothetical Drug Q service requires the most recent valid laboratory result collected within 48 hours before a recommendation request. The interface receives results of 72 units at Monday 08:00 and 68 units at Tuesday 10:00. A request arrives Wednesday 09:00. Assume both results are final, use the required unit, and the stated recency rule is inclusive of exactly 48 hours.
The Monday result is 49 hours old at the request time: from Monday 08:00 to Wednesday 08:00 is 48 hours, with one additional hour to Wednesday 09:00. It fails the rule. The Tuesday result is 23 hours old: from Tuesday 10:00 to Wednesday 09:00 is 23 hours. It is within 48 hours and is the most recent qualifying result. The service should use 68 units, retain its collection time, and make that context available for review.
If the Tuesday result were preliminary or in a different unit, it would not qualify under the assumptions. The service should then return the specified no-recommendation state rather than quietly use the older result. This example shows why recency, status, and units must be explicit data rules.
Plan Integration and Local Validation
An integration plan specifies how information travels among the EHR, interface engine, model service, and any monitoring or audit system. For each direction, document the trigger, message structure, required fields, acknowledgement behavior, timeout, retry policy, and owner. Confirm that identifiers remain linked across the round trip and that a retry cannot create duplicate recommendations. Failures should be visible to users in language that supports the next safe action, not merely as a technical code.
Before clinical use, test the complete path with synthetic or otherwise approved test records. The matrix below separates tests that concern data transport from tests of calculations and clinical presentation. Passing a model calculation test does not establish that the EHR displayed the right result for the right patient.
| Test area | Test case | Expected result | Evidence and owner |
|---|---|---|---|
| Eligibility | In-scope and excluded encounters | Only in-scope encounters proceed; exclusions show a specific reason | Recorded test cases; clinical informatics lead |
| Data mapping | Known values, units, timestamps, and missing fields | Values map correctly; invalid or absent required values stop the request | Input-output comparison; interface lead |
| Model service | Approved reference cases, including boundary cases | Returned result agrees with the independently prepared expected output within the specified tolerance | Versioned test report; model owner |
| Display and documentation | Recommendation, warning, and no-recommendation states | Correct encounter, units, context, status, and review controls are visible and recorded | Clinician usability sign-off; clinical lead |
| Failure handling | Timeout, unavailable service, duplicate message, and malformed response | No unsupported dose is shown; failure is communicated and logged as designed | Interface test log; IT operations lead |
| Access and audit | Authorized and unauthorized roles; recommendation reconstruction | Access follows policy and the record captures inputs, output, version, and user action as specified | Security and audit review; designated owners |
Worked Example: Interpreting a Validation Matrix Result
In a hypothetical pre-launch test, 40 requests are submitted: 30 eligible cases and 10 excluded cases. The service returns recommendations for 29 eligible cases, correctly returns no recommendation for one eligible case with a missing required input, and returns no recommendation for all 10 excluded cases. Of the 29 recommendations, one is linked to the wrong encounter in the display.
Eligibility handling is correct in all 10 excluded cases. The 30 eligible cases also receive an appropriate state: 29 recommendations and one justified stop for missing data. But end-to-end validation has not passed, because one recommendation is associated with the wrong encounter. The display linkage check is a critical failure even though the model outputs themselves may match expected calculations.
The team should record the defect, prevent clinical use, correct the linkage, and rerun the affected and relevant regression tests. It would be misleading to report “39 of 40 passed” as an acceptable overall result without addressing the severity of the single mismatch. Local acceptance criteria should define critical failures in advance; patient-result misidentification should not be averaged away.
Stage the Pilot and Assign Accountability
A pilot tests the implemented service under controlled conditions; it is not a substitute for prior technical testing or for the hospital’s required approvals. Specify eligible users and patients, the pilot period, support coverage, measures, and stop conditions before the first live request. A staged approach can begin with a non-interruptive or shadow phase, then move to limited clinician-facing use only after predefined review and authorization.
For a hypothetical plan, the hospital might run a two-week shadow phase followed by a four-week limited pilot on one inpatient unit. During shadow operation, outputs are not shown as clinical recommendations; the team checks data completeness, latency, exception reasons, and agreement with the approved reference process. Before clinician-facing use, the governance group reviews the results and confirms that training, support, and fallback procedures are ready.
Worked Example: Applying a Pilot Threshold
Assume the local pilot protocol specifies that at least 95% of eligible requests must have all required inputs available, and that any wrong-patient display is an immediate pause condition. In a hypothetical set of 100 eligible requests, 97 have complete inputs, while three stop because a required field is missing. One of the 97 displayed recommendations is also found to be linked to the wrong encounter.
Completeness is \(97/100=0.97\), or 97%, which exceeds the specified 95% threshold by 2 percentage points. However, the wrong-encounter finding meets the independent pause condition. The pilot therefore pauses despite meeting the completeness target. The team investigates and corrects the linkage defect, evaluates whether prior pilot records were affected, and obtains the required authorization before resuming. Passing one metric does not cancel a separate safety stop rule.
Assign a named role and a backup for each responsibility. The exact job titles vary by hospital, but responsibilities should be explicit:
- Clinical service lead: defines intended use, exclusions, review expectations, training, and clinical escalation.
- Model owner: controls the approved model version, reference tests, and interpretation of model-related defects.
- Interface and EHR leads: own data mappings, display behavior, technical testing, and operational support.
- Pharmacy or dosing-service lead: coordinates daily use, exceptions, and user feedback within the approved workflow.
- Privacy, security, and quality representatives: review data access, records, and required deployment controls.
- Governance group: approves launch and scope changes, reviews pilot evidence and incidents, and documents decisions.
These assignments should be written into a responsibility record with decision authority, escalation routes, and contact coverage. Governance is not satisfied by listing a committee: someone must be accountable for deciding whether a defect requires a pause, who investigates it, and who authorizes restart. The earlier material on governance for clinical dosing decision support and change control provides the relevant governance concepts; this workshop applies them to a concrete launch.
Failure Modes, Alternatives, and Limits
Common implementation failures include treating an EHR field as clinically equivalent to the model input without checking its meaning, allowing a technical timeout to appear as if no action is needed, and training users only after a workflow has been fixed in production. Another failure is using aggregate pilot performance to conceal a critical event, such as an incorrect patient linkage. Define failure states and pause rules before the pilot, and review exceptions at a level where individual high-severity failures remain visible.
Hospitals may choose between a fully integrated EHR workflow, a separate clinician-facing application, or an initial shadow workflow. Deep EHR integration can reduce duplicate entry and place recommendations near the relevant order, but it requires careful mapping, interface testing, and release coordination. A separate application may be quicker to pilot but can introduce manual transcription, identity-matching risk, and fragmented documentation. Shadow operation reduces the risk of an unreviewed recommendation influencing care, but it cannot fully test clinician interaction or downstream order entry. The appropriate path depends on local resources and risk controls; none removes the need for end-to-end testing and accountable clinical review.
A successful pilot also has limits. It demonstrates performance under its specific site, users, patient mix, and workflow conditions; it does not guarantee performance for every ward, population, or future software version. Changes to inputs, interfaces, clinical scope, or the model require assessment under the hospital’s change-control process. Continue operational monitoring after launch and retain a reliable fallback workflow so care does not depend on service availability.
Check Your Understanding
Answer the questions, then compare your reasoning with the concise solutions.
- Why should eligibility and patient-encounter linkage be checked before a model recommendation is displayed?
- A required input is present but has an undocumented unit. What should the service do, and why?
- In a pilot, 46 of 50 eligible requests have complete required inputs. What is the completeness percentage? If the locally approved minimum is 90%, does this metric meet the threshold?
- Why is a separate dosing application not automatically safer than EHR integration?
- Name two responsibilities that should be assigned before a live pilot begins.
Solutions. (1) Errors in scope or encounter identity can make an otherwise valid calculation irrelevant or associate it with the wrong patient; these checks prevent unsafe downstream display. (2) Stop or request clarification according to the specified failure process; an unknown unit cannot be safely assumed or converted. (3) \(46/50=0.92\), or 92%; it exceeds the 90% threshold by 2 percentage points, although other stop conditions must still be assessed. (4) A separate application may require manual entry or identity matching and can fragment documentation; its risks differ rather than disappear. (5) Examples include who may pause or restart the pilot, who investigates interface defects, who responds to clinical exceptions, and who approves scope or model changes.