Why Compare Observed and Predicted PD?
A PD model can produce predictions that track the general ordering of responses yet be systematically too high, too low, or too extreme. An observed-versus-predicted plot puts each observed response against the model prediction for the same response target. Its central question is agreement: do observed values lie near the values the model predicted, on the same measurement scale?
Before plotting, specify what each prediction represents. It may be a predicted response mean, median, or another target, conditional on defined covariates and model information. The observed and predicted quantities must refer to the same endpoint, time, transformation, and units. For example, a prediction on the log-response scale should not be plotted against an observed response on the original scale without an appropriate transformation. This plot complements residual diagnostics: it displays agreement in the response scale, while residuals display observed-minus-predicted differences or standardized versions of those differences.
Calibration is not a claim that every observation must equal its prediction. Observed PD responses vary, and predictions may be uncertain. The plot is a diagnostic summary, not proof that a model is correct. Its interpretation depends on how predictions were generated, whether observations are independent, the response error structure, and whether the data were used to fit the model.
Identity Line, Calibration Trend, and Smooths
The identity line has intercept zero and slope one. A point above it has \(Y_i>\hat{Y}_i\): the observed response is greater than predicted. A point below it has \(Y_i<\hat{Y}_i\): the observed response is less than predicted. Whether “greater” means a stronger PD effect depends on the endpoint’s direction and coding, so retain the response’s clinical meaning when interpreting the plot.
A useful descriptive summary is a regression of observed response on prediction:
Here \(a\) is the fitted intercept and \(b\) the fitted slope. The ideal reference values are \(a=0\) and \(b=1\), but estimates close to these values do not establish calibration: uncertainty, scatter, curvature, and the data used to estimate the line all matter. This regression is a summary of the observed trend, not a replacement for inspecting the points.
A smooth curve, such as a locally weighted smoother, can help reveal a non-linear pattern. Compare it with the identity line, not only with a fitted straight line. A smooth that is consistently above or below identity suggests level-dependent bias; curvature or changing departures can indicate that a single intercept-and-slope summary hides a more complex mismatch. The smoother’s appearance depends on the smoothing method and its tuning, so it should be treated as a visual aid rather than a formal test.
What the Calibration Slope Does—and Does Not—Say
The slope \(b\) describes how steeply observed responses change with predictions in the fitted linear trend. A slope below one means that this fitted observed trend changes less steeply than the predictions: for a one-unit increase in \(\hat{Y}\), the fitted observed response increases by \(b\) units. A slope above one means the fitted observed trend changes more steeply. Neither statement, by itself, determines which variable has greater overall dispersion. Scatter around the line and the marginal distributions or ranges of both variables must also be examined.
The algebra makes the limitation clear. With an intercept in ordinary least squares, the fitted slope is the sample covariance of predictions and observations divided by the sample variance of predictions:
Thus, the slope summarizes how the two variables vary together relative to variation in the predictions. It does not use the observed variance alone. A slope between zero and one is therefore not evidence, on its own, that predictions are more dispersed than observations. To assess the distributions themselves, look at the scatter, marginal spreads, and ranges as well as the trend.
Worked Example: A Prediction Trend That Is Too Steep
Suppose three hypothetical PD observations and predictions, in response units, are \((\hat{Y},Y)=(4,2),(5,4),(6,6)\). The points fall below identity except at the largest prediction. Calculate the descriptive calibration line.
The means are \(\overline{\hat{Y}}=(4+5+6)/3=5\) and \(\bar{Y}=(2+4+6)/3=4\). The numerator of the slope is \( (4-5)(2-4)+(5-5)(4-4)+(6-5)(6-4)=2+0+2=4\). The denominator is \( (4-5)^2+(5-5)^2+(6-5)^2=1+0+1=2\). Therefore \(\hat{b}=4/2=2\), and the intercept is \(\hat{a}=\bar{Y}-\hat{b}\overline{\hat{Y}}=4-2(5)=-6\).
The points follow the exact line \(Y=-6+2\hat{Y}\). At \(\hat{Y}=4\), the fitted observation is \(-6+2(4)=2\); at 5 it is 4, and at 6 it is 6. The fitted trend in observed response is twice as steep as the prediction trend. Relative to identity, the observed-minus-predicted differences are \(-2,-1,0\), so predictions exceed observations at the first two values and agree at the third. This small, perfectly aligned example illustrates a trend departure; it does not establish how well a model would predict new observations.
Worked Examples: Compression and Curvature
Worked Example: A Trend Compressed Relative to Predictions
Consider hypothetical pairs \((10,14),(20,22),(30,30),(40,38)\), with prediction first and observed response second, all in the same PD units. The means are \(\overline{\hat{Y}}=25\) and \(\bar{Y}=26\). The slope numerator is \( (-15)(-12)+(-5)(-4)+(5)(4)+(15)(12)=180+20+20+180=400\). The prediction sum of squares is \( (-15)^2+(-5)^2+5^2+15^2=225+25+25+225=500\). Thus \(\hat{b}=400/500=0.800\), and \(\hat{a}=26-0.800(25)=6.00\). The fitted trend is \(Y=6.00+0.800\hat{Y}\).
The fitted observed response changes by 0.800 units for each one-unit increase in prediction, so its trend is less steep than the prediction scale. The observed-minus-predicted differences are \(4,2,0,-2\): the model underpredicts at the lower values and overpredicts at the highest value in this set. Do not infer from the slope alone that predictions have greater overall spread. Here the listed values can be compared directly, but in noisier data the point scatter and marginal spreads also need inspection.
Worked Example: A Straight-Line Summary Hides Curvature
Suppose hypothetical predictions are 1, 2, 3, 4, and 5 response units, and the observed responses are 2, 3, 3, 3, and 4. The identity differences \(Y-\hat{Y}\) are \(1,1,0,-1,-1\). Their sum is zero, so the average observed-minus-predicted difference is zero. That average alone could suggest no overall offset.
However, the differences are positive at the low end, zero in the middle, and negative at the high end. A smooth through the points would bend relative to identity, showing that the model overpredicts at the high end and underpredicts at the low end. A single mean difference misses this pattern; a straight calibration line may also be an incomplete summary. The example motivates checking the point pattern and smooth curve rather than relying on one statistic.
Prediction Intervals and Their Interpretation
A prediction interval describes uncertainty for an individual future observation under a specified predictive model and conditions. It is not the same as a confidence interval for a mean response or a confidence band for a fitted calibration line. For example, under a normal predictive model with known mean \(\hat{Y}_i\) and known standard deviation \(\sigma_i\), a two-sided 90% prediction interval is \(\hat{Y}_i\pm 1.645\sigma_i\). In applications, predictive uncertainty may also involve estimated parameters and other sources of variation; the interval should reflect the model and prediction target actually being assessed.
Plotting such intervals vertically at each \(\hat{Y}_i\) can show whether observed responses fall in plausible predictive ranges. Across repeated independent observations generated under a correctly specified model, intervals with nominal 90% coverage should contain about 90% of the corresponding outcomes in the long run. A finite sample’s coverage need not equal 90%, and apparent coverage alone does not establish calibration: intervals can be too wide and cover often, or too narrow and miss often. Examine interval width and misses as well as the coverage proportion.
Worked Example: Checking Prediction-Interval Coverage
Assume a hypothetical model gives ten independent observations the same predicted response, \(\hat{Y}=20\) units, with known predictive standard deviation \(\sigma=2\) units. Under a normal predictive distribution, the 90% interval for each observation is \(\hat{Y}\pm1.645\sigma=20\pm1.645(2)=20\pm3.29\), or \([16.71,23.29]\) units.
Suppose the ten hypothetical observed responses are 17, 20, 21, 24, 18, 22, 16, 25, 19, and 23 units. Values 17, 20, 21, 18, 22, 19, and 23 fall inside the interval; 24, 16, and 25 fall outside. The empirical coverage is \(7/10=0.700\), or 70%. This is below the nominal 90%, but ten observations provide little precision for assessing coverage. It is a prompt to examine more data and the predictive assumptions, not by itself a decisive rejection of the model. The calculation assumes the stated normal predictive distribution and independence; if observations are correlated, this simple coverage interpretation needs to account for that dependence.
Diagnostics, Comparisons, and Limitations
A practical reading of the plot proceeds from basic checks to pattern interpretation. Confirm that axes use the same response scale and units, and that the plotted prediction corresponds to each observation’s intended target. Add identity as the agreement reference. Then inspect the cloud for vertical scatter, systematic displacement, changing spread, curvature, and unusual points. Compare a smooth with identity and, if useful, report the descriptive intercept and slope with appropriate uncertainty. A smooth or regression line estimated from a small sample can be unstable.
State what response is observed, what quantity is predicted, and the scale, time, and units shared by both axes.
Identify whether points tend to lie above or below \(Y=\hat{Y}\), and whether departures change across the prediction range.
Use a smooth or calibration regression to describe trend, and assess point scatter and marginal ranges separately from the slope.
When prediction intervals are shown, check their construction, widths, and observed coverage; do not treat a confidence interval for a mean as an individual prediction interval.
Observed-versus-predicted plots and residual plots answer related but different visual questions. A residual plot displays the size and direction of prediction errors against a chosen quantity; an observed-versus-predicted plot places those errors in the response scale and makes identity agreement explicit. A calibration regression compresses the trend into two parameters, while a smooth can expose curvature but is more dependent on smoothing choices. Prediction-interval coverage adds an uncertainty check, but does not replace inspection of bias and spread. These approaches are complementary rather than competing substitutes.
Several limitations matter in PD data. Repeated measurements from one participant are not automatically independent points; treating them as independent can make visual patterns seem more persuasive than warranted. Predictions evaluated on the data used to fit the model may look more favorable than predictions on new data, so interpret apparent agreement in light of how predictions were obtained. A plot can also hide important structure when many observations overlap or when different groups are combined. Use relevant context and other diagnostics, including residuals against time, concentration, dose, baseline, and covariates, to investigate patterns rather than attributing every departure to one cause.
Likewise, the identity line is a reference, not a requirement that all observations lie exactly on it. Random response variation produces scatter even when predictions are well calibrated on average. Conversely, a cloud centered near identity can still conceal curvature, group-specific bias, or poor interval coverage. Use the plot to formulate and refine diagnostic questions, not to declare model validity from one visual feature.
Check Your Understanding
Answer each question, then compare with the solutions.
- On an observed-versus-predicted plot, what does a point below the identity line mean?
- A calibration regression has slope 0.6. State what this says about the fitted trend and one conclusion it does not support by itself.
- Why might a smooth curve reveal a problem that a mean observed-minus-predicted difference misses?
- In a sample of 40 independent outcomes, 34 fall inside their nominal 90% prediction intervals. What is the empirical coverage, and does that alone prove calibration?
- Explain why a confidence interval for a mean response is not interchangeable with an individual prediction interval.
Solutions. 1. The observed response is less than its prediction, \(Y_i<\hat{Y}_i\). 2. The fitted observed response changes by 0.6 units per one-unit increase in prediction, so its trend is less steep. The slope alone does not establish which variable has greater overall dispersion. 3. Positive differences at one end and negative differences at another can cancel in the average while a smooth displays the changing pattern. 4. Coverage is \(34/40=0.850\), or 85%. It is an estimate based on a finite sample and does not alone prove calibration; interval construction, uncertainty, and dependence also matter. 5. A confidence interval for a mean quantifies uncertainty about the mean, whereas a prediction interval must also represent variability of an individual outcome around that mean.