Start With the Recommendation, Then Trace Back
A missing recommendation and an implausible recommendation are both symptoms, not diagnoses. A blank display might reflect an intentional safety block, an incomplete request, a failed model call, or an interface that did not render a valid response. An extreme displayed value might come from a model result, a unit-conversion error, stale cached content, or a formatting problem. The diagnostic task is to establish which event occurred at each handoff and where the observed behavior first diverged from the expected behavior.
Treat any unresolved recommendation as unavailable for clinical use. Follow local procedures for clinical review and incident escalation; do not infer a dose from a partially reconstructed trace. The investigation described here is a technical method for locating failures, not a substitute for clinical judgment.
A useful trace preserves both the value and its meaning: units, timestamp and time zone, patient or encounter context, and source. It also links events with stable request and response identifiers. Record the software and model versions, configuration, validation messages, and interface events relevant to the request. Avoid changing or overwriting original records while investigating.
Reconstruct the Handoffs
For each transition, compare the actual record with the expected contract. A contract states what fields are required, their permitted formats and units, how values are transformed, and what a successful or blocked response should contain. Check the path in order rather than starting with the model: the model cannot explain a value it never received, and an interface cannot explain a value that was never returned to it.
Identify the request, patient or encounter context, and approximate time. Use identifiers to distinguish repeated requests for the same person.
Check that required values came from the intended source and were available at the relevant time. Inspect units, timestamps, time zones, missingness, and conflicting records.
Compare raw and transformed values field by field. Recalculate conversions or filters independently, using the documented rules and retaining intermediate outputs.
Verify whether the model ran, which model and configuration it used, what inputs it received, and what response or error it returned.
Match the model response to the interface event. Check units, rounding, suppression rules, cached results, and whether the displayed item belongs to this request.
Test the suspected cause against the records or a controlled replay. Record supporting evidence, remaining uncertainty, and the actions taken under local procedures.
The key reasoning is a continuity check. If event \(i\) produces output \(y_i\), and the next event consumes input \(x_{i+1}\), the handoff is consistent only if the documented transformation \(g_i\) explains the relationship \(x_{i+1}=g_i(y_i)\), including units and context. If the values do not match, the discrepancy localizes the investigation to that handoff; it does not by itself establish why the discrepancy occurred.
For example, if the recorded model response is correct but the displayed result differs, the model may have executed properly while the failure lies downstream. Conversely, if the transformed input is already wrong, a model rerun with that same input will reproduce the wrong behavior without identifying the upstream error.
Worked Traces
Worked Example: A Missing Result After a Timestamp Filter
Hypothetical case. A service requires a laboratory value recorded before the request time. The source record shows a draw time of 08:15 in a clinic using UTC−5. The request was created at 13:00 UTC. The transformed-input log shows that the laboratory value was excluded, and the model response is “required input unavailable.” The interface displays no recommendation.
Check the time conversion. A time in UTC−5 is five hours behind UTC, so the source draw time converts as \(08{:}15+5\text{ hours}=13{:}15\) UTC. That is 15 minutes after the 13:00 UTC request time. Under the stated rule, the value is not eligible for that request. The transformed-input log and model response therefore agree with the configured filter; the blank display is consistent with a blocked result, not evidence that the model silently failed.
What remains to verify. Confirm the source time zone and whether the service is intended to use draw time or record-entry time. If the source time zone was incorrectly assigned, the conversion would be wrong. If the documented rule is “available by request time,” check when the result became available, rather than substituting a different timestamp without authorization. The trace locates the immediate cause of the missing output in input eligibility; it does not alone determine whether the source record or the intended rule should change.
Worked Example: An Implausible Display From a Unit Mismatch
Hypothetical test fixture, not a clinical dosing calculation. A controlled software test uses a model response of 320,000 micrograms. The interface contract requires the response to be converted to milligrams before display. The screen shows “320,000 mg.”
Check the conversion. Since \(1\text{ mg}=1{,}000\text{ micrograms}\), the expected display value is
The unit labels cancel, and the numerical result is 320 mg. The displayed number is 1,000 times that value, while the model-response log retains 320,000 micrograms. If the response identifier and request identifier match across those records, the first observed divergence is at or after the interface conversion, not in the model output.
Confirmation. In a controlled test environment, replay the same fixture and inspect the value immediately before rendering. If it remains 320,000 and is labelled as micrograms, but the screen labels it as milligrams, the conversion or unit-label handling is implicated. Do not treat the arithmetic in this test fixture as evidence that a real patient’s recommendation is correct.
Worked Example: A Valid Response Paired With a Stale Display
Hypothetical case. Two requests for the same test patient were made in one session. Request R-881 returned 180 mg with response ID A-41. Later, request R-882 returned 240 mg with response ID A-42. The server log records 240 mg for R-882, but the screen still shows 180 mg. The interface event log shows that its visible card is associated with A-41, not A-42.
Trace the identities. The model response for R-882 is 240 mg, and the interface is displaying the response linked to R-881. The arithmetic difference is \(240-180=60\) mg, but that difference is not the diagnostic evidence. The decisive mismatch is the response identity: A-41 belongs to the earlier request, while the expected current response is A-42.
The trace supports a stale-display or response-association problem. It does not establish whether caching, a delayed client update, or another interface mechanism caused it. Inspect the client event sequence and cache key, then reproduce the behavior with controlled test requests. A display that is numerically plausible can still be wrong for the current request.
Using Audit Records and Replay Carefully
Audit records are most useful when they preserve the relationship between events, not merely a final value. A timestamped entry saying “recommendation generated” is weak evidence if it cannot be linked to the request, model version, inputs, and response shown to the user. Compare identifiers, timestamps, and available input or output hashes. A hash can support a claim that two recorded payloads are identical, but it does not reveal their contents or prove that those contents were clinically appropriate.
A replay is informative only when its conditions are known. Reproducing the original result generally requires the original transformed inputs, model version, configuration, relevant software behavior, and any randomness or external dependencies. If any of these differ, a changed replay result may be expected rather than diagnostic. First preserve the original evidence; then, where authorized, replay in a controlled environment and label the replay as a new event.
When logs are incomplete, do not fill gaps with assumptions. State what is confirmed, what is only consistent with the evidence, and what cannot be determined. For example, a server record with a valid output and no corresponding interface event narrows the problem to response delivery or logging, but cannot distinguish those possibilities without further records.
Diagnostics, Alternatives, and Limits
A component-by-component trace is especially useful when the system records intermediate inputs and outputs. Other diagnostic approaches can complement it. A broad incident review can identify patterns across requests but may not localize one request’s failure. A controlled component test can isolate a conversion or rendering rule but does not prove that the same component caused the original incident. A replay can test reproducibility but depends on fidelity to the original execution context. Use the approach that answers the current question, and link its evidence back to the original request.
Warning signs include missing identifiers between adjacent records, timestamps that cannot be reconciled, unit labels without unit conversions, different model versions across logs, unexplained duplicate responses, and a displayed value that lacks a matching response record. These are reasons to investigate, not automatic proof of a particular defect. A single unusual recommendation is not enough to diagnose model performance; investigation of calibration, prediction errors, or transportability addresses different questions.
The trace can locate a technical discrepancy, but it cannot always establish its clinical significance or the correct clinical action. Records may be incomplete, clocks may differ, or a system may not retain intermediate values. Where the evidence cannot distinguish competing explanations, document that limitation and escalate rather than presenting a hypothesis as a confirmed cause.
A good incident note is concise but reproducible: identify the affected request, state the expected and observed behavior, list the records and versions examined, show the first confirmed divergence, and separate confirmed facts from unresolved possibilities. Preserve evidence and follow the applicable incident and governance procedures.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A source time is 10:20 in UTC−4. The request time is 14:00 UTC, and the service excludes inputs recorded after the request. Is the input earlier or later than the request after conversion?
- A model returns 75,000 micrograms. A display contract requires milligrams. What value should be displayed?
- The server log links response B-12, containing 210 mg, to the current request. The screen shows 160 mg linked to response B-09. What is the first discrepancy established by these facts?
- A replay returns a different result from the original run, but it used a newer model version. Does this confirm that the original run failed? Explain.
Solutions. (1) Convert the source time: \(10{:}20+4\text{ hours}=14{:}20\) UTC. It is 20 minutes later than the 14:00 UTC request, so it is excluded under the stated rule. (2) Divide by 1,000: \(75{,}000\text{ micrograms}/1{,}000=75\text{ mg}\). (3) The established mismatch is between the current response identity, B-12, and the response displayed, B-09. The facts do not yet identify the interface mechanism that caused it. (4) No. The model version differs, so the replay does not isolate the original execution. Reconstruct the original version and configuration, if possible, or report that the cause remains unresolved.