Why Test the Work, Not Just the Interface
A dosing decision-support system can calculate a recommendation correctly and still be difficult to use safely. A clinician might enter the wrong patient, overlook a needed qualification, misunderstand a displayed interval, or fail to find the information needed to act. Task-based usability testing examines what representative clinicians actually do when they use the system to complete realistic dosing work. It is evidence about interaction with the interface and workflow—not, by itself, evidence that the model is clinically valid.
The introductory material on displaying recommendations clearly and embedding dosing support in EHR workflows helps establish the intended interaction. A usability test then checks whether users can carry it out in context. Test in a controlled, simulated environment using fictional patient records and a defined safety procedure; do not create real orders or alter patient care. Keep the recommendation and data scenario fixed when the question is about usability, so a changed model output does not become an uncontrolled explanation for different user behavior.
Designing the Test
Select clinicians who reflect the intended users in relevant ways, such as clinical discipline, experience with the dosing workflow, and familiarity with the EHR environment. The aim is not necessarily to estimate a population parameter from a random sample. It is to expose interaction problems across plausible users and contexts. A small, purposively selected group can reveal important failures, but its results should not be presented as a precise estimate for every clinician or site.
Write tasks as clinical situations rather than instructions that disclose the interface path. For example: “For this fictional patient, determine the recommended dose and the monitoring information you would need before acting.” Do not say “open the dosing tab and click the recommendation,” because that tests whether users follow the prompt, not whether they can find the function. Define in advance what counts as a correct result, which information must be interpreted, and what constitutes a critical error. The earlier material on communicating uncertainty and fail-safe behavior is relevant when deciding what information must be understood and how a simulated unavailable-data condition should be handled.
Use the same patient inputs, software version, starting state, and task wording for each participant when comparing performance. If think-aloud comments are collected, explain that verbalizing may affect task time and record it consistently. Avoid coaching during a task; if a moderator must intervene for safety or to prevent an unusable session, record the intervention and whether the task result is considered assisted. Counterbalance task order where feasible, because practice on one task can make a later task easier.
Describe the clinical decision context, the fictional patient information available, and the expected task outcome.
Define correct completion, interpretation errors, navigation events, task timing, assistance, and stopping rules before observing participants.
Use the same interface state and moderator script; capture actions and comments without prompting users toward the answer.
Connect measured outcomes and observed behavior to specific interface or workflow barriers, then prioritize corrective action and retesting.
Define Measures Before the Session
A completion measure needs a task-specific success criterion. “Completed” might mean that the clinician reached the correct recommendation and identified required qualifications without moderator assistance. It should not mean merely that the user reached a screen. Report both the numerator and denominator, and state whether assisted completions count as successes.
An interpretation error occurs when a user incorrectly understands a clinically relevant displayed item, such as dose units, a range, a monitoring condition, or a reason the recommendation is qualified. Record the exact item, the user’s interpretation, and whether the error would change a decision. A wrong answer caused by ambiguous wording should not be coded as a model-calculation failure without separate evidence.
A navigation problem is an observable interaction cost or barrier, such as an unintended detour, repeated backtracking, failure to locate a required panel, or abandonment after a search. Decide which events count before the test. Time on task can help describe burden, but a longer time is not automatically a failure: a careful user may take longer, while a fast user may overlook an important qualification. Pair time with accuracy and observed behavior.
For each measure, specify the unit of analysis. If each clinician performs several tasks, task outcomes from the same clinician are related; they should not casually be treated as independent observations. Report task-level results with the number of clinicians and tasks, and show important participant-level patterns. Small usability tests are often most informative through the combination of counts, observed behavior, and error severity—not a single summary statistic.
Worked Example: Estimating Task Completion in a Small Test
In a hypothetical simulated test, 12 clinicians each attempt the same predefined task: find the recommendation for a fictional patient and state the dose in the displayed units. Nine do so correctly without assistance. The other three either give an incorrect dose or require moderator assistance, which the prespecified rule classifies as not completed. The observed completion proportion is \( \hat{p}=9/12=0.75 \), or 75%.
To describe sampling uncertainty under a simple binomial model, suppose the 12 outcomes are independent Bernoulli observations with a common underlying completion probability \(p\). A 95% Wilson score interval is calculated using \(z=1.96\):
Substituting \( \hat{p}=0.75 \), \(n=12\), and \(z^2=3.8416\), the denominator is \(1+3.8416/12=1.3201\). The center is \( (0.75+3.8416/24)/1.3201=0.6894 \). The half-width is \(1.96\sqrt{0.75(0.25)/12+3.8416/(4(12^2))}/1.3201=0.2217\). The interval is therefore approximately \(0.4677\) to \(0.9111\).
This is a 95% Wilson score interval: under the stated binomial assumptions it is constructed by inverting a 95% score test. Its actual repeated-sampling coverage is not generally exactly 95%, because binomial outcomes are discrete; coverage varies with \(p\) and \(n\). Further, purposive recruitment or dependence among participants can undermine the binomial interpretation. For this small usability test, the most direct result remains that 9 of 12 clinicians completed the task under the defined rule; the interval is not a guarantee about all clinicians or sites.
Interpretation and Navigation Need Separate Evidence
A successful click path does not establish correct interpretation, and a correct interpretation does not show that the interface is efficient to navigate. Score each construct separately, then inspect where they interact. For example, a user may find a dose quickly but miss that it is conditional on a laboratory result. That is an interpretation problem even if task time is short. Conversely, a clinician may eventually state the correct dose after repeated searching; the task may be completed, but the navigation burden remains important.
Worked Example: Coding Interpretation Errors
In a hypothetical test, 10 clinicians each review one fictional recommendation that displays a dose of 600 mg every 24 hours and a separate condition that must be checked before administration. The test script asks each clinician to state both the dose and interval and to explain the condition. The scoring rubric defines a critical interpretation error as omitting or misreading the condition. Two clinicians omit it; the other eight state it correctly. The observed critical-error proportion is \(2/10=0.20\), or 20% of participants on this task.
The numerator is two clinicians with the prespecified error, and the denominator is the 10 clinicians who attempted this interpretation task. The dose-and-interval responses should also be reported separately if they are scored. Do not combine all incorrect statements into one undifferentiated “error” count: confusing units and missing a safety condition can have different consequences and require different interface changes. Because each clinician contributes one observation here, the participant-level proportion is transparent; if several responses per clinician were counted, dependence would need to be acknowledged.
The observation supports a usability finding that this display or task context did not reliably communicate the condition to these participants. It does not establish that the underlying dose calculation is wrong, nor does it show that exactly 20% of all clinicians would make the error. Review recordings and comments to identify whether the condition was hidden, ambiguous, or overlooked, then revise the interface or task and test again.
Worked Example: Separating Task Success from Navigation Burden
Six clinicians complete a hypothetical task to locate a recommendation and its monitoring note. All six eventually report the correct information. Their times are 3.2, 4.1, 4.4, 5.0, 7.2, and 11.0 minutes. Four follow the intended route without backtracking; two make at least two unintended detours before finding the information.
Task completion is \(6/6=100\%\), while the proportion with the prespecified navigation problem is \(2/6=0.333\), or 33.3%. Sorted times are 3.2, 4.1, 4.4, 5.0, 7.2, and 11.0 minutes, so the median is the mean of the middle two: \((4.4+5.0)/2=4.7\) minutes. The range is 3.2 to 11.0 minutes.
The result is not “perfect usability” merely because everyone completed the task. The two detours and the broad time range point to possible discoverability problems. Review which screens or labels caused the detours; do not infer the cause from time alone. A practical follow-up is to revise the relevant navigation cue and retest the task with the same success and event definitions.
Diagnostics, Alternatives, and Limits
Warning signs include participants asking what a task means, moderators giving inconsistent hints, scoring disagreements, frequent missing recordings, or outcomes that vary mainly with task order. These indicate that the protocol or instrumentation may be producing unreliable evidence. Pilot the script with someone who is not in the formal test group. Have two reviewers independently code a subset of recordings, compare disagreements, and refine ambiguous coding rules before finalizing counts.
Interpret failures by tracing the observed sequence: what information was available, what the participant noticed, what action followed, and where the task diverged from the success criterion. Separate interface issues from incorrect or missing input data, system delays, or unexpected recommendation behavior. If the system output itself differs across sessions, first determine whether the test conditions were comparable; usability scores alone cannot diagnose the underlying technical cause.
Task-based observation is stronger than a post-session satisfaction survey for identifying specific barriers because it records performance in context. A survey can efficiently capture perceived confidence or burden but relies on self-report. Expert inspection or a cognitive walkthrough can find obvious design issues before recruiting clinicians, but does not show how representative users actually perform. Routine usage analytics can reveal patterns at scale, but often cannot distinguish confusion from workflow constraints or establish why a user took an action. These approaches complement rather than replace one another.
A usability test is limited by scenario realism, participant selection, small samples, artificial observation, and the particular interface version tested. A favorable result does not establish clinical effectiveness, analytical validity, or safe performance in every setting. Nor is one completion threshold universally appropriate: a task involving a safety-critical qualification warrants a different response to failure than a low-consequence navigation task. Prioritize findings by consequence, frequency within the observed test, and the plausibility of recurrence; treat observed critical errors as actionable signals even when the sample is small.
Common Pitfall
Check Your Understanding
Answer each question, then compare with the short solutions below.
- In a test of 8 clinicians, 6 complete a task unaided. What is the observed completion proportion, and what does it describe?
- Why should a study not treat four tasks performed by each of five clinicians as 20 independent clinician observations without qualification?
- Everyone reaches the correct dose, but three clinicians miss a required monitoring condition. Which measure captures this problem, and why is completion alone insufficient?
- A participant spends a long time on a task but gives a correct answer. What additional evidence should be examined before calling the task a failure?
Solutions. (1) \(6/8=0.75\), or 75%; it describes the observed fraction of these eight clinicians who met the stated unaided-completion rule. (2) Tasks from one clinician share that person’s experience and behavior, so the outcomes may be dependent; report participant and task structure rather than assuming 20 independent people. (3) Record an interpretation-error measure, with the three affected clinicians as numerator and the clinicians who attempted the interpretation as denominator; reaching a screen or stating a dose does not show that the condition was understood. (4) Examine accuracy, navigation events, requests for help, and the action sequence or comments; time by itself cannot distinguish careful work from confusion or an inefficient path.