Performance Requirements for Point-of-Care Calculations

Why Performance Is a Clinical Requirement

A dosing calculation can be numerically correct and still be operationally unsuitable if it arrives too late, cannot serve the expected number of requests, or is unavailable when clinicians need it. Performance requirements therefore belong in the service specification alongside the calculation’s inputs, outputs, and verification criteria. The requirement is not simply “fast enough”: it identifies what is measured, under which workload and system conditions, and what result counts as passing.

For a point-of-care service, performance is usually described using three related but distinct properties: response time, throughput, and availability. Response time concerns how long an individual request takes. Throughput concerns how many requests the system can complete per unit time while meeting its response-time target. Availability concerns whether a valid request can obtain a usable result when the service is expected to operate. A system may perform well on one measure and poorly on another.

Requirements should reflect the clinical workflow and the service boundary. If the clinician initiates a request from an electronic health record (EHR), a model service might need to retrieve data, execute the model, and return a recommendation. Measuring only the model’s computation time omits the time spent in those other steps. The earlier material on mapping the clinical workflow and writing testable use-case specifications provides the context for deciding which steps belong inside the performance measurement.

Define the Measures Before Setting Targets

A response-time requirement should name its start and end points. For example, it might begin when the EHR submits a complete, valid dosing request and end when the EHR receives a complete response. That is an end-to-end measure. A separate model-execution measure can help diagnose bottlenecks, but it does not replace the end-to-end measure when the clinical concern is how long the user waits.

Latency varies across requests. Reporting only the mean can conceal a slow tail: most requests might be quick while a small but important fraction takes much longer. A percentile is a threshold below which a stated proportion of observed response times falls. If the 95th percentile is 2 seconds, approximately 95% of measured requests completed in 2 seconds or less; the remaining requests took longer. Percentile targets should be paired with a measurement window, workload, and a clear rule for timeouts and failed requests.

A performance requirement is testable only when its measurement conditions are explicit. Specify, as relevant, the request mix, arrival pattern, number of concurrent users or requests, duration, system configuration, and whether dependencies are included. A service measured with a single request at a quiet time has not thereby demonstrated performance during a busy clinic period.

Throughput, Workload, and Capacity

Throughput targets need a workload definition. State the expected arrival rate, whether arrivals are steady or bursty, the request mix, and how long the target must be sustained. Also state the response-time and error-rate conditions that must hold at that rate. A high count of completed requests is not evidence of adequate capacity if response times become unacceptable or requests fail.

Capacity planning commonly includes headroom: the tested rate is set above the expected peak so that ordinary variation does not immediately push the system into overload. Headroom is a design choice, not a universal constant. It should be justified from expected demand, uncertainty, and the consequences of falling behind. A service might be required to sustain a specified peak rate for a defined period, while also passing a separate test for short bursts.

A useful relationship for interpreting concurrency is Little’s law. In a stable system over a sufficiently long interval, the average number of requests in the system equals the average arrival rate multiplied by the average time each request spends in the system. In symbols, if \(\lambda\) is the average arrival rate in requests per second, \(W\) is the average time in seconds, and \(L\) is the average number of requests in the system, then:

This relation is not a capacity guarantee. It assumes the measured system is stable over the interval and that the averages refer to the same system boundary and request population. It is useful for estimating concurrent work and for recognizing queue growth: if requests arrive faster than the system completes them for a sustained period, the backlog grows instead of reaching a stable average.

Availability: Specify the Denominator and Window

Availability is easy to misstate when the measurement rule is vague. A time-based measure can define availability as the fraction of a specified service window during which the service is usable. A request-based measure can define it as the fraction of eligible requests that receive a usable response under the stated criteria. These measures answer different questions, so a requirement should name the chosen measure and define what counts as usable.

For a time-based target, 99.9% availability means that 0.1% of the defined service window may be unavailable. In a 30-day month with continuous service required, the window contains \(30 \times 24 \times 60=43{,}200\) minutes. The corresponding unavailable time is:

The service window must be stated. If planned maintenance is excluded, that exclusion should be explicit and consistently applied; if the clinical workflow requires the service during maintenance, excluding it may hide a clinically relevant interruption. Likewise, a request-based metric needs a defined denominator. For example, it might include all valid requests submitted during the measurement period, with a usable response defined as a complete response returned within the specified timeout. Invalid requests may be excluded, but the exclusion rule should be established before testing.

Design Tests That Represent Clinical Conditions

Performance testing should represent more than an average day. A useful test plan covers the expected request mix and plausible demanding conditions, while keeping the test environment and measurement boundary clear. For a dosing service, requests may differ in model, input pattern, data-retrieval needs, and computation path. If only the simplest request type is tested, the result may not represent the workload that matters.

Diagnostics help distinguish a true performance problem from a misleading test. Compare end-to-end latency with component-level timings to locate where time is spent. Examine percentiles rather than only averages, and check whether slow requests cluster around a particular request type, dependency, or load period. A rising queue or increasing latency during a sustained-rate test suggests that the system is not keeping pace with arrivals. A test that never exercises realistic concurrency, dependency delays, or the intended model configuration cannot support claims about those conditions.

Common Mistakes, Alternatives, and Limits

• Using only average latency. The mean can hide slow requests. Report relevant percentiles and the measurement conditions.
• Timing only model execution. This omits delays between the clinical request and the returned result. Use end-to-end latency for an end-to-end workflow requirement, with component timings for diagnosis.
• Reporting a throughput number without latency or failures. Count successful completions at the target rate and require latency and error criteria to hold at that same workload.
• Calling a service available based on an undefined uptime figure. Specify the service window, denominator, success rule, timeout, and maintenance treatment.
• Extrapolating from a small or convenient test. A quiet single-request test does not establish performance under peak load, and one load-test run does not prove every production condition.

Time-based and request-based availability are alternatives, not interchangeable labels. Time-based measurement is useful when the service must be ready throughout a defined operational window, including periods with few requests. Request-based measurement directly describes the proportion of submitted eligible requests that succeed under a specified rule. Depending on the workflow, both may be valuable. Similarly, mean response time is compact and useful for some comparisons, but percentile requirements better expose a slow tail. No single metric captures every performance concern.

A performance test does not establish that the dosing recommendation is clinically appropriate; computational verification and clinical validation address different questions. Nor can a finite test guarantee performance under every future workload, dependency failure, or infrastructure change. The test report should therefore state the conditions tested and the limits of the evidence. Requirements are most useful when they are measurable, clinically motivated, and tied to conditions that can be reproduced.