Why the Source of Uncertainty Matters
A model prediction can sit close to a decision threshold, so modest uncertainty may matter more than a much larger uncertainty in a prediction far from that threshold. But uncertainty is not one undifferentiated quantity. It may come from estimated parameter values, imperfect measurements, assumptions about the model’s form, or applying the model in a different population or setting. These sources call for different evidence and different ways of describing what is uncertain.
The immediate task here is to ask whether a source could change the decision—not to calculate a comprehensive sensitivity analysis. A decision rule might classify a predicted response as adequate when it is at least a specified threshold. The context of use and rule determine which prediction and threshold matter.
These labels identify the source of a concern, not necessarily separate, independent quantities. For example, measurement error in the development data may contribute to uncertainty in estimated parameters. A model may also have both structural and transport uncertainty. The useful question is: what would have to be different for the predicted outcome to cross the decision threshold?
Set Up the Decision Before Assessing Uncertainty
Let \(Y\) be the model-predicted outcome relevant to a decision, and let \(T\) be its threshold. For a rule that classifies outcomes at or above the threshold as meeting the criterion, the decision indicator is \(D=1\) when \(Y\geq T\) and \(D=0\) otherwise. The outcome and threshold must use the same scale and units. If the rule instead concerns a probability, such as a minimum probability of target attainment, that probability—not an individual outcome threshold—is the quantity to compare with its decision threshold.
For an uncertainty source \(U\), consider the prediction \(Y=f(U)\) while holding other assumptions fixed. The source can move the prediction across the outcome threshold if plausible values or states of \(U\) produce predictions on both sides of \(T\). This is a screening test. It does not establish that the source is likely to cause a change, nor that changing the predicted outcome necessarily changes the final decision when the decision rule has additional criteria.
If a justified uncertainty distribution is available, a probability of crossing can be calculated. If only a plausible range is defensible, the appropriate conclusion is about whether that range straddles the threshold; assigning a probability would require additional assumptions. In either case, uncertainty must be stated on the decision-relevant prediction scale.
Worked Examples
Worked Example: Parameter Uncertainty Near a Criterion
Hypothetical scenario. A model predicts a dimensionless response ratio \(R\), with a decision criterion of \(T=1.00\): a prediction at least 1.00 meets the criterion. Conditional on the specified model and inputs, suppose the prediction is adequately represented by \(R\sim N(1.06,0.08^2)\). The standard deviation of 0.08 represents parameter-related predictive uncertainty; measurement, structural, and transport uncertainty are not included.
The point prediction is 1.06, which exceeds 1.00. Under the stated normal approximation, the probability that the uncertain prediction is at least the criterion is:
Thus, under this assumed distribution, about 77.3% of the parameter-related prediction distribution is at or above the criterion. The distribution also assigns probability below the criterion, so parameter uncertainty could change a classification based on whether \(R\) meets 1.00. If a separate rule required this probability to be at least 0.80, the result would not meet that rule. This calculation depends on the stated normal approximation and on excluding the other uncertainty sources; it is not a general guarantee about the model.
Worked Example: Measurement Uncertainty in a Model Input
Hypothetical scenario. A concentration input is measured as 12.0 mg/L, with an assay standard uncertainty of 1.2 mg/L. Assume the measurement error is unbiased and normally distributed, so the uncertain concentration \(C\) is represented by \(C\sim N(12.0,1.2^2)\) mg/L. For this illustration, treat the model coefficients as known and the response as \(Y=0.04C+0.20\), where \(Y\) is a dimensionless response. The decision criterion is \(Y\geq 0.65\).
At the measured concentration, the predicted response is \(0.04(12.0)+0.20=0.68\). Because the model is linear in \(C\), the mean predicted response is 0.68 and its standard deviation from measurement uncertainty is \(0.04(1.2)=0.048\). The concentration that gives exactly the criterion is:
Therefore, under the assumed measurement model, the chance that the measured-input-based prediction meets the criterion is:
The uncertainty in the input can move the predicted response across the criterion, even though the prediction calculated from the observed concentration is above it. This calculation accounts only for measurement uncertainty in \(C\); it does not include parameter uncertainty, residual variation, or possible model-form error.
Worked Example: Structural Uncertainty From Alternative Forms
Hypothetical scenario. Two plausible model forms are being considered for a dimensionless response at a dose \(D=20\) mg/kg. The threshold is 0.60. Model A predicts \(Y=0.20+0.021D\), while Model B predicts \(Y=0.20+0.80D/(30+D)\). For this comparison, treat their coefficients and the dose as fixed; the uncertainty under examination is the choice of structural form.
Model A gives \(Y=0.20+0.021(20)=0.620\), which is above the threshold. Model B gives:
which is below the threshold. Thus, the two candidate forms place the prediction on opposite sides of 0.60. This makes model structure decision-relevant at this dose. It does not tell us which form is more credible or justify averaging the predictions. That requires evidence and assumptions beyond this crossing check.
Worked Example: Transport Uncertainty in a Target Population
Hypothetical scenario. A model predicts that 0.68 of patients in the evidence-base population meet a response criterion. The decision concerns a target population, and the decision threshold for the predicted fraction is 0.60. Because the target population differs in a relevant characteristic, suppose available evidence supports only a plausible adjustment \(\delta\) between \(-0.12\) and \(+0.04\), where the target-population prediction is \(0.68+\delta\). No probability distribution for \(\delta\) is available.
At the lower end of the adjustment range, the target prediction is \(0.68-0.12=0.56\). At the upper end, it is \(0.68+0.04=0.72\). The resulting plausible range, from 0.56 to 0.72, straddles 0.60. The transport uncertainty could therefore change the threshold classification within the stated range. Because the range is not a probability distribution, it would be incorrect to report a probability that the target-population prediction exceeds 0.60 from these values alone.
A Practical Check and Its Limits
The examples use different representations of uncertainty because the evidence differs: a distribution for parameter uncertainty, a measurement-error model for an input, competing forms for structural uncertainty, and a plausible range for transport uncertainty. There is no general rule that converts all four into a single probability or a single interval. A disciplined assessment states the source, the prediction affected, the assumptions used, and whether the resulting predictions cross the relevant threshold.
Specify the predicted outcome or probability, its units or scale, the relevant threshold, and the direction that meets the criterion.
Ask whether the concern is about parameter values, measurements, model form, or application to the target population or setting.
Use a justified distribution when one is available; otherwise, state a defensible range or compare plausible forms without inventing probabilities.
Determine whether the uncertainty representation includes predictions on both sides of the threshold, and state what assumptions were held fixed.
Report which source could affect the classification, what evidence would reduce that uncertainty, and what important sources remain unassessed.
A range that lies wholly on one side of the threshold supports robustness to that particular uncertainty representation, not to every unexamined source. A range that straddles the threshold flags a decision-relevant concern; it does not by itself say which decision should be made. The final interpretation still depends on the decision rule and the evidence standard appropriate to the consequences of error.
Diagnostics, Alternatives, and Common Pitfalls
Several checks help reveal when an uncertainty assessment is inadequate. Confirm that the threshold and prediction use the same endpoint, population, time point, and units. Check that the uncertainty statement actually concerns the prediction used by the rule rather than a different model quantity. For a probability calculation, verify that the distribution and its assumptions are stated and justified. For a range, explain why its endpoints are plausible. For structural comparisons, ensure the alternatives are scientifically defensible rather than selected simply because they produce a desired result. For transport, check that the population or setting difference is named and that the evidence supports the proposed adjustment or range.
A common mistake is to say, “The point prediction is above the threshold, so the decision is secure.” That reasoning ignores whether plausible uncertainty places predictions below the threshold. The correction is to compare the uncertainty representation—not just the central prediction—with the threshold. The opposite mistake is to treat any uncertainty interval that crosses the threshold as proof that the model is unusable. Crossing signals that the source may matter; its importance depends on the decision rule, the credibility of the uncertainty representation, and the consequences of an incorrect decision.
The alternatives answer different questions. A probability distribution quantifies uncertainty when a defensible probability model is available. A plausible range can show whether crossing is possible without implying a probability. Comparing alternative model forms makes structural disagreement visible but does not automatically assign weights to those forms. A direct prediction from the evidence-base population avoids some transport assumptions but may not answer the target-population question. None of these approaches removes the need to explain what is included and omitted.
Finally, uncertainty about a model prediction is not the same as variability among future patients. A model can have a well-determined mean prediction while individual outcomes vary substantially. Conversely, a mean prediction may be uncertain even if individual variability is modest. They may both be relevant to a decision, but they are different quantities and should not be combined or described as though they were interchangeable.
Check Your Understanding
For each question, identify the uncertainty source or apply the threshold-crossing check. Solutions are provided below.
- A model prediction is 0.74, with a justified uncertainty range from 0.66 to 0.79. The decision threshold is 0.70. Can the represented uncertainty change the classification?
- A response model’s coefficients are treated as fixed, but an assay input has measurement error. What is the primary uncertainty source being assessed?
- Two defensible model forms predict 1.2 and 0.9 units, while the threshold is 1.0 unit. What can be concluded, and what cannot be concluded from these values alone?
- A model gives a target-population prediction range of 0.52 to 0.58. The threshold is 0.60. Does this range cross the threshold, and what is the scope of the resulting conclusion?
- Why is it not valid to report a probability of meeting a threshold from a plausible range alone if no probability model is specified?
Solutions. (1) Yes. The range includes predictions below 0.70 and above it, so this represented uncertainty could change the classification. (2) Measurement uncertainty, specifically uncertainty propagated from the assay input; other sources may still be present. (3) The candidate forms place predictions on opposite sides of 1.0, so structural choice could affect classification. The values alone do not establish which form is better or how to weight them. (4) No. The range is wholly below 0.60, so it does not cross the threshold; this supports robustness only to the stated transport uncertainty range, not to unassessed sources. (5) A range states plausible values but assigns no relative probabilities to them. A probability requires a probability distribution or other justified probabilistic assumptions.