Why Recommendations Can Disagree
A recommendation that conflicts with a clinician’s expectation is a signal to investigate, not proof that the model is wrong or that the clinician has overlooked something. The two may have been based on different information, different times, or different assumptions. A displayed dose can also differ from the model’s internal output because of unit conversion, rounding, dose limits, or interface presentation.
The practical task is to trace the recommendation from the information available at the time to the value shown to the user. This is a focused diagnostic exercise: identify where the discrepancy enters, determine whether it changes the recommendation, and decide whether the result can be explained or requires escalation. It does not replace clinical judgment or establish that a recommendation is safe for an individual patient.
A useful representation separates five elements: the source inputs, the event times associated with those inputs, the transformations that create model-ready inputs, the model and recommendation logic used, and the assumptions displayed to the user. Let \(X\) denote source inputs, \(T\) the transformation from source data to model-ready data, \(v\) the model version, and \(C\) the configured recommendation rules. The recommendation can be represented as:
Here \(M_v\) is the model calculation for version \(v\), and \(H_v\) represents any downstream logic that turns model results into a recommendation, such as applying an authorized target or dose constraint. Displayed assumptions should be checked as a separate element: they may be used by the calculation, or they may describe its context without affecting it. A mismatch between what the display says and what the calculation actually used is important in either case.
A Disciplined Troubleshooting Sequence
Start by preserving the decision context: what recommendation was shown, when it was requested, and what expectation it conflicted with. Then compare the actual values used in the calculation with the values the clinician believes were used. Do not begin by changing several inputs or settings at once; that can make a discrepancy disappear without revealing its source.
Name the compared recommendations, their units, and the relevant decision time. Clarify whether the disagreement is about dose amount, interval, target, or another displayed result.
Compare the values the clinician intended to provide with the values received by the service. Check identifiers, missingness, duplicates, units, and whether values belong to the intended patient and treatment episode.
Determine when each measurement or treatment event occurred, when it was recorded, and whether it was available when the recommendation was requested. Check time zones and ordering where relevant.
Trace conversions, derived variables, rounding, and constraints. Confirm which model version and recommendation settings were used, and whether the displayed assumptions match them.
Using an approved test or review process, change one factor at a time and compare results. If the discrepancy remains unexplained, or a possible unsafe output is involved, follow the clinical and operational escalation process.
This sequence assumes the recommendation can be reviewed in a controlled environment and that the relevant input and configuration values can be compared. If values are unavailable, ambiguous, or not reliably linked to the decision, the cause may remain indeterminate. Do not infer a missing value or reconstruct an intended input as if it had actually been used.
Worked Example: A Weight Unit Is Misinterpreted
In a hypothetical training system for a fictitious compound, a clinician expects a recommendation based on a weight of 154 lb. The configured calculation uses 3.00 mg/kg and expects weight in kg. The recommendation display shows 462 mg. Investigate the discrepancy.
First check whether the received value and model-ready value agree with the intended units. Suppose the incoming record contains the number 154, with a source unit of lb, but the transformation passes the number through as 154 kg. If the calculation uses that value, it produces:
The arithmetic matches the displayed result, but the input is not expressed in the required unit. Convert the intended value using \(1\ \text{kg} \approx 2.20462\ \text{lb}\):
With the correctly converted weight, the calculation is:
The ratio \(462/209.6 \approx 2.20\) is consistent with the pounds-to-kilograms conversion factor. That supports a unit-transformation explanation, but does not by itself show where the error occurred. Confirm the source unit, the transformation, and the model-ready value in the controlled environment. The two calculated amounts are illustrative troubleshooting values only, not clinical dosing advice.
Replaying Inputs, Times, and Assumptions
A replay is informative only when the comparison is controlled. Hold the model version and recommendation settings fixed while changing one candidate input or transformation. Then restore the original state before testing another factor. This distinguishes a direct input effect from a version change or a display mismatch. If the recommendation is a schedule or category rather than a single number, compare its components rather than forcing the entire result into a numeric difference.
Time is part of the input state. A concentration measured before a dose is not interchangeable with one measured after it; an event recorded late may appear to have occurred in a different order from the clinical event. Check both the event time and the time the record became available to the recommendation service. A value can be accurate yet inappropriate for a decision if it was not available at the decision time.
For a numerical recommendation, a sequence of controlled replays can summarize how the output changes. Let \(R^{(0)}\) be the baseline result and \(R^{(k)}\) the result after changing one factor at step \(k\). Define the step change as:
The changes telescope:
This identity justifies describing how a chosen sequence of changes moves the result from baseline to final replay. It does not prove that each step has a unique contribution. If factors interact, the change attributed to a factor can depend on the order in which factors are changed. Record the sequence and test plausible interactions rather than presenting an order-dependent decomposition as a universal causal explanation.
Worked Example: A Record Is Available at the Wrong Time
In a hypothetical review, a clinician expects a recommendation based on a treatment event at 09:00 and a measurement collected at 10:00. The service request was made at 11:00. The displayed recommendation is 20 units, while the clinician’s expectation is 16 units. For this example, the service uses the most recent eligible measurement according to its event-time rule; the values and outputs are illustrative.
Inspection shows two records: a measurement of 12 collected at 08:00 and a measurement of 9 collected at 10:00. The 10:00 record reached the service after the 11:00 request because of a data-transfer delay. Therefore, it was not among the values available to the original calculation. A controlled replay using only the available 08:00 record returns 20 units; replaying with the 10:00 record included returns 16 units.
The arithmetic comparison is \(16-20=-4\) units, so including the later-arriving record changes the illustrative output by \(-4\) units. This explains why the clinician and service were working from different information states. It does not establish that the service should have used a record unavailable at request time, nor that either recommendation is clinically appropriate. The next question is whether the delay and the resulting information state are expected and acceptable for the intended workflow.
Check that the timestamps describe the intended events, not merely database receipt times; verify ordering and any applicable eligibility rule; and confirm whether the displayed result made its data cutoff clear. If event times or availability times cannot be established, the replay cannot reliably identify the cause.
Versions, Displays, and Assumption Mismatches
A model version is only one part of the calculation. The same model may be paired with different targets, constraints, or recommendation logic. Conversely, the display may show an assumption that did not enter the calculation, or may show a default that differs from the active setting. Compare the version and each relevant setting independently, and establish whether the displayed assumption is computational or descriptive.
Worked Example: Separating a Version Change From a Target Change
Suppose a hypothetical test case can be replayed under two model versions, A and B, and two target settings, narrow and broad. The following controlled outputs are obtained:
| Model version | Target setting | Illustrative recommendation |
|---|---|---|
| A | Narrow | 100 units |
| A | Broad | 110 units |
| B | Narrow | 105 units |
| B | Broad | 115 units |
Starting at version A with the narrow target gives 100 units. Changing only the target to broad gives \(110-100=10\) additional units. Returning to the baseline and changing only the version gives \(105-100=5\) additional units. Applying both changes gives \(115-100=15\) additional units, equal to \(10+5\) in this particular example.
This comparison shows how to separate candidate effects under the tested conditions; it does not establish that version and target effects will always add. Here, the broad-target change is also 10 units under version B (\(115-105=10\)), and the version change is 5 units under the broad target (\(115-110=5\)). Thus the two changes happen to be additive in this test. If those differences had varied across settings, that would indicate an interaction or a more complex change, requiring further review.
If the screen displayed “broad target” but the active calculation used “narrow target,” the discrepancy would include a configuration-display mismatch. If both the calculation and display used the same setting, version A versus B could still explain a difference. The useful conclusion is not “the newer version is right,” but that the tested input, setting, and version combinations reproduce specific outputs. Clinical and technical reviewers must still determine whether the combination used was intended.
Diagnostics, Alternatives, and Common Pitfalls
Useful diagnostic evidence is a reproducible result under a clearly specified state. Warning signs include implausible input magnitudes, unexpected unit labels, duplicate or missing records, inconsistent event ordering, a recommendation that changes after an unrelated display action, or a screen that names different assumptions from those confirmed in the calculation. If the output does not reproduce under the stated inputs and configuration, first check whether the replay truly matches the original decision context.
Troubleshooting has several complementary approaches. A controlled single-factor replay helps localize a computational or configuration difference. A clinician’s review helps assess whether the inputs and assumptions are clinically appropriate. Rechecking a data pipeline or interface can reveal ingestion, transformation, or display problems. Broader validation evaluates performance across cases rather than explaining one discordant recommendation. These approaches answer different questions; a successful replay does not establish clinical validity, and a population-level validation does not explain a particular input mismatch.
| Approach | Question it can answer | Important limitation |
|---|---|---|
| Controlled replay | Which tested input, transformation, version, or setting changes the output? | Results depend on faithfully reproducing the decision state. |
| Clinical review | Are the inputs, assumptions, and result appropriate for the clinical context? | Expectation alone does not identify a software or data cause. |
| Pipeline or interface check | Was information converted, transmitted, or displayed as intended? | A display check alone may not establish what the model calculated. |
| Validation across cases | Does performance meet defined criteria across a relevant set of cases? | It does not automatically resolve an individual case discrepancy. |
A common mistake is to say, “The clinician expected 16, so the system’s 20 must be wrong,” or the reverse: “The system returned 20, so that must be correct.” Neither conclusion follows from the disagreement. The correction is to compare the information and assumptions behind each result, reproduce the system output where possible, and seek the appropriate clinical review. Do not alter inputs simply to make a recommendation match an expectation.
A second pitfall is changing the input, timestamp, and model version together, then attributing the new output to one of them. This confounds the explanation. Change one factor at a time where feasible, document the tested state, and check interactions when relevant. Finally, if a possible error could affect care, use the established safety process rather than treating an explanatory replay as authorization to act on a dose.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A source value is 88 lb, but the model expects kg. What should be checked before concluding that the model calculation is wrong?
- A measurement was collected before a recommendation request but arrived after it. Why can a replay that includes this measurement fail to explain the original output?
- In a controlled replay, changing a setting shifts the recommendation from 40 to 46 units. State the observed change and one conclusion that does not follow from it.
- The interface displays a target setting that the calculation did not use. Is this necessarily a model-version problem? What should be investigated?
Solutions. (1) Verify the source unit, the transformation, and the model-ready value. Convert \(88\ \text{lb} \div 2.20462\ \text{lb/kg} \approx 39.92\ \text{kg}\); passing 88 through as kg would create a unit mismatch. (2) The original request may have used only information available at request time. Including a later-arriving record changes the information state, so that replay is not a faithful reconstruction unless the original calculation also had access to it. (3) The observed change is \(46-40=6\) units under the tested conditions. It does not prove that the setting is clinically appropriate or that it will always add 6 units in other cases. (4) No. Investigate whether the displayed setting is descriptive or computational, which setting was actually active, and whether the interface accurately reflects the calculation. A version difference is only one possible explanation.