Predictive Accuracy Versus Clinical Impact

Two Different Questions: Prediction and Patient Benefit

A dosing model can predict a pharmacokinetic quantity more accurately yet have no effect on patient outcomes. This is not inherently a contradiction: a prediction metric evaluates how close predictions are to observed or reference values, whereas clinical impact concerns what happens when a dosing policy is applied. Between those questions lie the actions the model recommends, the clinical response to those actions, and whether the policy is actually followed.

For example, reducing error in a predicted exposure may not change a dose if all recommendations remain on the same side of a decision threshold. Even when recommendations change, the patients affected may have nearly identical outcomes under the alternative doses, or benefits for some patients may be offset by harms for others. A claim of improved prediction therefore does not, by itself, establish improved clinical outcomes.

Definition: A prediction metric summarizes agreement between predictions and specified reference values in a defined data set. A dosing policy is a rule that maps a patient’s available information to a dosing action. The clinical impact of a policy is the difference in patient outcomes that would result from applying that policy rather than a specified alternative in a target population.

The distinction is clearest when policy value is defined in intervention terms. Let \(U^g\) denote the utility a patient would have under policy \(g\), including the consequences of the policy’s dosing recommendation and its implementation. In a target population, the value of \(g\) is \(V(g)=E[U^g]\). For two policies, the clinical-impact estimand is often the difference \(V(g_1)-V(g_0)\). This compares the outcomes that would result under each policy in the same target population; it is not generally the difference between the observed outcomes of patients who happened to receive each policy.

Why Better Predictions Need Not Change Policy Value

Suppose a model produces a prediction \(s\), and a policy converts that prediction and patient information into a dose \(d\). Predictive performance might be summarized by a loss such as mean squared error, while policy value is based on patient utility under the resulting dose. These quantities have different targets and units: prediction error may be measured in concentration-squared units, whereas utility may represent a clinically defined score or another outcome scale.

A direct implication follows when improved predictions do not change any action. If two model-based policies assign the same dose to every patient in the target population, then, under the same implementation conditions, the patient-level outcomes under those policies are the same. Their policy-value difference is consequently zero. This conclusion concerns the policies, not the accuracy metrics: the predictions can still differ substantially in accuracy.

d_{g_1}(x)=d_{g_0}(x)\ \text{for every patient }x \quad\Longrightarrow\quad U^{g_1}=U^{g_0} \quad\Longrightarrow\quad V(g_1)-V(g_0)=E[U^{g_1}-U^{g_0}]=0

The implication assumes that the policies differ only through the dose recommendation and that other aspects of implementation are held fixed. If one policy also changes monitoring, clinician review, or follow-up, identical doses alone do not guarantee identical outcomes. More generally, even changed doses do not ensure a change in utility: dose-response relationships may be flat over the relevant range, or gains and harms may offset in the target population.

Worked Example: Lower Dose-Prediction Error, Unchanged Recommendations

Consider a hypothetical validation set of four patients. The reference individualized doses are 90, 110, 90, and 110 mg. A standard model predicts 82, 118, 78, and 122 mg; an updated model predicts 88, 112, 88, and 112 mg. For a simple measure of predictive accuracy, calculate root mean squared error (RMSE), the square root of the mean squared prediction error.

For the standard model, the errors are \(-8, 8, -12,\) and \(12\) mg. Therefore,

\mathrm{RMSE}_{\mathrm{standard}} =\sqrt{\frac{(-8)^2+8^2+(-12)^2+12^2}{4}} =\sqrt{\frac{416}{4}} =\sqrt{104} \approx 10.20\ \mathrm{mg}.

For the updated model, the errors are \(-2, 2, -2,\) and \(2\) mg. Thus,

\mathrm{RMSE}_{\mathrm{updated}} =\sqrt{\frac{(-2)^2+2^2+(-2)^2+2^2}{4}} =\sqrt{\frac{16}{4}} =2.00\ \mathrm{mg}.

The updated model has lower RMSE in this hypothetical data set. Now suppose the dosing policy rounds each predicted dose to the nearest available dose in 50 mg increments, with 100 mg available. Every prediction from both models rounds to 100 mg. For example, 82 and 88 mg are closer to 100 mg than to 50 mg, and 122 and 112 mg are closer to 100 mg than to 150 mg. The two policies therefore recommend the same dose for all four patients. If their implementation is otherwise identical, this prediction improvement cannot create a policy-value difference through dosing: each patient receives the same dose under either policy. The example illustrates a possible lack of clinical impact; it does not establish that 100 mg is clinically appropriate.

Estimating Impact Requires an Outcome Comparison

A predictive validation set can describe how well a model predicts its chosen reference quantity. To estimate the clinical impact of using its recommendations, evaluation must instead address outcomes under competing policies. The central comparison is between \(E[U^{g_1}]\) and \(E[U^{g_0}]\), for policies and a target population defined in advance. Random assignment to policies can identify this contrast because, on average, assignment makes the policy groups comparable before treatment. The interpretation still depends on appropriate outcome measurement, adherence and missing-data handling, and a study population relevant to the intended use.

A non-significant estimate is not proof that the policies have identical value. The estimate may be close to zero, but its uncertainty may also permit clinically important benefit or harm. Conversely, an observed difference should be interpreted in its clinical units and against a prespecified decision-relevant effect, rather than being treated as meaningful just because a prediction metric improved. These are separate pieces of evidence.

Worked Example: Prediction Improves, but a Policy Trial Is Inconclusive

In a hypothetical evaluation, an updated model reduces RMSE for a specified exposure prediction from 6.0 to 4.0 concentration units in a validation set. In a separate randomized policy comparison, 70 of 100 patients assigned to the standard policy and 71 of 100 assigned to the updated-model policy achieve a prespecified utility outcome. Treat this outcome as binary utility, equal to 1 for achievement and 0 otherwise. The estimated difference in mean utility is

\hat{\Delta} =\bar{U}_{\mathrm{updated}}-\bar{U}_{\mathrm{standard}} =\frac{71}{100}-\frac{70}{100} =0.0100.

Under independent random assignment and the usual large-sample approximation for two proportions, the standard error is

SE(\hat{\Delta}) =\sqrt{\frac{0.71(1-0.71)}{100}+\frac{0.70(1-0.70)}{100}} =\sqrt{0.004159} \approx 0.06449.

An approximate 95% confidence interval is \(0.0100 \pm 1.96(0.06449)\), or approximately \([-0.116,\,0.136]\) on the utility-probability scale. The point estimate is a one-percentage-point advantage for the updated policy, but the interval includes zero and effects in either direction. The data do not establish an improvement, and they also do not establish no clinically relevant effect. The RMSE reduction is evidence of better prediction on its validation metric; it does not resolve the uncertainty about policy impact.

Observed Policy Use Can Be Confounded

When patients are not randomly assigned, a simple comparison of observed outcomes among policy users may not estimate the policy-value contrast. Clinicians may select a model-based policy for patients who are sicker, more complex, or otherwise different in ways that also affect utility. In that case, the conditional mean among patients observed under a policy can mix the policy’s effect with the characteristics that influenced policy use. Defining value as \(E[U\mid \text{policy used}]\) would therefore be unjustified without assumptions that identify the intervention-based quantity \(E[U^g]\).

The following hypothetical example shows how the direction of an observed association can differ from a target-population policy comparison. Assume, for illustration, that the policy-specific mean utilities within each risk group are known from a valid randomized comparison. To calculate target-population values, also assume the target population is half low-risk and half high-risk. These assumptions permit standardization to the stated population; they are not warranted merely by observing policy use.

Worked Example: Selection Reverses an Observed Comparison

Suppose the hypothetical mean utilities are as follows: among low-risk patients, the standard policy yields 0.95 and the model-based policy yields 0.95; among high-risk patients, the standard policy yields 0.50 and the model-based policy yields 0.60. In the target population, each risk group makes up one half. The standardized policy values are

V(g_{\mathrm{standard}}) =0.50(0.95)+0.50(0.50)=0.725, \qquad V(g_{\mathrm{model}}) =0.50(0.95)+0.50(0.60)=0.775.

Thus, the target-population value difference is \(0.775-0.725=0.0500\), or 0.05 utility units. Now suppose, in an observational dataset, 80% of patients using the model-based policy are high-risk, while only 20% of standard-policy users are high-risk. The observed mean utilities would be

E[U\mid \text{model used}] =0.20(0.95)+0.80(0.60)=0.670, \qquad E[U\mid \text{standard used}] =0.80(0.95)+0.20(0.50)=0.860.

The observed comparison is \(-0.190\) utility units for model-policy users, despite the assumed positive target-population policy contrast. The reversal occurs because the policy groups have different risk compositions. The arithmetic demonstrates why observed outcomes alone do not identify policy value. In an actual observational evaluation, appropriate adjustment would require adequate measurement of the factors affecting both policy selection and outcomes, as well as other identification assumptions; this example’s stipulated within-group values should not be inferred from unadjusted group averages.

Evaluation Checks, Alternatives, and Limits

A sound evaluation keeps the prediction question and the policy question distinct while connecting them explicitly. Before interpreting a reported improvement, check what the prediction target is, how the error metric is calculated, and whether its validation data represent the patients and clinical setting of intended use. Then inspect whether recommendations actually change, for whom they change, and whether the outcome evaluation compares policies in a way that supports the intended causal interpretation.

Check decision translation. Determine whether improved predictions move patients across a dosing threshold or alter a feasible recommendation after rounding and other operational rules. A lower average error may leave every action unchanged.
Check the outcome target. State which utility or clinical outcome is evaluated, its time horizon, and the direction that represents benefit. A prediction target and a patient-important outcome are not interchangeable.
Check evaluation design. Randomized policy assignment supports a direct comparison under its design assumptions. Observational comparisons require a defensible strategy to address differences between patients receiving the policies.
Check delivery and follow-up. Recommendations that are not implemented, or outcomes not measured consistently, can weaken the link between model output and observed impact. The intention-to-treat effect of policy assignment and the effect of following recommendations are distinct questions.
Check precision and clinical relevance. Report uncertainty around the policy contrast and compare it with a prespecified clinically meaningful difference. A point estimate near zero with a wide interval is inconclusive, not evidence of equivalence.

The main alternatives answer different questions. Validation metrics summarize predictive performance; they are useful for model development and comparison, but cannot alone establish patient benefit. A randomized policy evaluation directly addresses outcomes under assigned policies, though it may be costly and its findings apply most directly to the studied population and implementation. Observational evaluation may use routine-care data, but causal interpretation depends on assumptions and adjustment for relevant differences. Decision analysis can project consequences under explicit assumptions when direct evaluation is unavailable; projected utility is conditional on the model, utility specification, and input evidence, not a substitute for observed clinical impact.

No single metric captures every consequence of a dosing policy. A policy can improve average utility while worsening outcomes for some patients, or improve a short-term target while affecting other outcomes not included in that utility. Population averages also depend on the distribution of patient characteristics. The appropriate conclusion is therefore bounded: state which prediction metric improved, which policy contrast was estimated, in which population, and with what uncertainty. Do not infer a broader clinical benefit than the design and outcome support.

Common Pitfall

Common mistake: “The updated model has lower prediction error, so it improves dosing and patient outcomes.” Lower prediction error only establishes improvement on the defined prediction metric in the evaluated data. The conclusion about outcomes requires evidence that the policy changes treatment in a consequential way and improves the specified outcome compared with a relevant alternative. A second mistake is to compare outcomes among observed policy users and call the difference the policy effect. That comparison can reflect selection; policy value is defined through outcomes under intervention, such as \(E[U^g]\), and requires a design or assumptions that identify it.

Check Your Understanding

For each problem, identify what the evidence supports. Full solutions follow each question.

A revised model reduces exposure-prediction RMSE, but it recommends exactly the same doses as the current policy for every patient. Under identical implementation, what can be concluded about the value difference through dose recommendations?
A randomized policy trial estimates an updated-minus-standard utility difference of 0.02 with a 95% confidence interval from \(-0.04\) to \(0.08\). Does this show that the updated policy has no effect?
In an observational dataset, a model-based policy is used more often for high-risk patients. Why may the observed mean utility among its users fail to equal \(E[U^g]\)?
A model improves the average accuracy of predicted optimal dose, but only a small fraction of patients receive a different rounded dose. Name one further question required before claiming clinical benefit.

Solutions. (1) If the policies recommend the same dose to every patient and other implementation is identical, then their patient-level outcomes under these policies are the same; the value difference through those recommendations is zero. The RMSE improvement remains a predictive result. (2) No. The interval includes zero and both negative and positive values, so the trial is inconclusive about the direction and magnitude of impact at this precision; it does not prove equivalence or no effect. (3) High-risk status may affect utility and policy selection. The observed conditional mean can therefore reflect both risk composition and policy consequences, rather than the outcome that would result from applying the policy to the target population. (4) For example, assess whether the changed doses improve the prespecified patient outcome compared with the alternative policy, using a design or assumptions that identify that policy contrast. A different rounded dose by itself is not evidence of benefit.