Checking NPDE Marginal Calibration

Why Check Marginal Calibration?

NPDE are designed so that, when the predictive model is correctly specified and the NPDE construction is valid, their marginal distribution follows a standard normal reference distribution. Checking that distribution is a direct way to look for systematic location, scale, or shape discrepancies between observed outcomes and model predictions. A histogram or summary close to the reference is reassuring, but it is not proof that the model is adequate.

Marginal calibration asks whether the values, taken one at a time, behave like draws from \(N(0,1)\). It does not ask whether NPDE from different observations are independent or whether their ordering over time has structure. Those are distinct questions. In particular, a normal-looking histogram can coexist with dependence or with errors in a clinically important subgroup.

Definition: NPDE are marginally calibrated when, under the model and the assumptions of their construction, each NPDE has distribution \(N(0,1)\): mean zero, variance one, and the standard normal distributional shape. A finite sample will not match these properties exactly; calibration is assessed by looking for departures that are substantial relative to sampling and simulation variability.

Assumptions Behind the Standard Normal Reference

The earlier material on NPDE construction defines predictive deviations and uses their predictive distribution to form normal-reference deviations, with a covariance transformation to account for relationships among observations. For the standard normal reference to apply, the predictive distribution used in that construction must represent the model correctly for the observed design and the observations being assessed. The simulation procedure must also approximate that predictive distribution adequately.

A common mathematical justification applies when the predictive deviation vector \(D\) has a multivariate normal distribution with mean zero and positive-definite covariance matrix \(\Sigma\). Positive definiteness matters: it ensures a nonsingular Cholesky factor and permits ordinary whitening. If \(LL^{T}=\Sigma\), define \(W=L^{-1}D\). Then:

E(W)=L^{-1}E(D)=0,\qquad \operatorname{Cov}(W)=L^{-1}\Sigma(L^{-1})^{T} =L^{-1}LL^{T}(L^{-1})^{T}=I.

Because a linear transformation of a multivariate normal vector is multivariate normal, \(W\sim N(0,I)\). Each component therefore has the standard normal marginal distribution. This argument depends on \(\Sigma\) being positive definite and \(L\) being nonsingular; the equation \(LL^{T}=\Sigma\) alone does not guarantee that an inverse exists. A singular covariance matrix requires separate treatment and cannot be passed through this ordinary inverse-based argument.

In practice, NPDE may be estimated from simulated replicates, so the reference is approximate. Materially too few or poorly representative simulations can distort estimated predictive distributions and NPDE. Marginal plots should therefore be interpreted as model checks subject to the quality of the NPDE calculation, not as exact tests with a guaranteed reference in every finite simulation.

Histogram: Compare Counts With Normal Probabilities

A histogram is useful for noticing broad location and shape features. Centering near zero is expected; roughly symmetric values and tails compatible with a standard normal distribution are also expected. Choose bins that make these features visible, and compare observed counts with counts expected under \(N(0,1)\). For an interval \((a,b]\), the expected count among \(n\) NPDE is \(n\{ \Phi(b)-\Phi(a)\}\), where \(\Phi\) is the standard normal cumulative distribution function.

The expected counts are reference values, not exact targets. Random samples fluctuate, and a histogram can conceal structure depending on its bin boundaries and widths. In particular, a small sample may show uneven bars even when there is no meaningful departure, while a large sample can make minor departures visible.

Worked Example: Reading a Binned NPDE Histogram

Suppose a hypothetical diagnostic contains 100 NPDE. The observed counts in six intervals are: below \(-1.96\), 5; from \(-1.96\) to \(-1\), 12; from \(-1\) to 0, 31; from 0 to 1, 35; from 1 to 1.96, 13; and above 1.96, 4. The counts sum to \(5+12+31+35+13+4=100\).

For \(N(0,1)\), the approximate probabilities in those intervals are \(0.0250, 0.1337, 0.3413, 0.3413, 0.1337,\) and \(0.0250\). Multiplying by 100 gives expected counts of \(2.50, 13.37, 34.13, 34.13, 13.37,\) and \(2.50\), respectively.

The observed central counts are somewhat lower than their references, while the two outermost counts total 9 rather than the 5 expected from the two \(2.5\%\) tails. This suggests checking the tails and overall spread more closely; it is not, by itself, a formal rejection or proof of miscalibration. The sample is hypothetical, and modest count differences can occur by chance. Other bins or a quantile plot may reveal whether the apparent tail excess is part of a consistent pattern.

Normal Quantile Plots and Summary Statistics

A normal quantile plot compares ordered NPDE with the corresponding theoretical quantiles of \(N(0,1)\). If the marginal distribution is close to the reference, points should fall approximately along a straight line with intercept zero and slope one. A displaced line suggests a location difference; a line that is systematically steeper or flatter suggests a scale difference. Curvature or asymmetric tail departures may indicate skewness, heavy or light tails, or other shape differences.

The sample mean \(\bar{x}\) and sample standard deviation \(s\) give compact summaries of location and spread. Values near zero and one, respectively, are consistent with the reference, but they do not establish normal shape. A distribution can have mean zero and standard deviation one while being skewed or having unusually heavy tails. Always compare summaries with a histogram or quantile plot, and state the sample size.

Worked Example: Interpreting a Normal Quantile Plot

Consider a hypothetical set of nine ordered NPDE: \(-1.8,-1.1,-0.7,-0.3,0.0,0.4,0.8,1.2,2.4\). For a simple visual comparison, approximate standard normal quantiles at plotting positions \(i/(n+1)\), for \(i=1,\ldots,9\), are about \(-1.28,-0.84,-0.52,-0.25,0,0.25,0.52,0.84,1.28\). A normal quantile plot pairs each ordered observation with its corresponding theoretical quantile.

The middle observations are reasonably close to the theoretical sequence, but the largest observed value, 2.4, is far above the largest listed theoretical quantile, about 1.28. This pattern points to an unusually high upper-tail value relative to the normal reference. It does not establish that the model is wrong: with only nine observations, the plot is highly variable, and one point can dominate the impression. The next step is to check the full data, simulation quality, and relevant model context rather than infer a specific cause from this plot alone.

Worked Example: Calculating NPDE Summary Statistics

For eight hypothetical NPDE, suppose the values are \(-1.4,-0.8,-0.3,0.1,0.4,0.7,1.1,2.2\). Their sum is \(2.0\), so the sample mean is

\bar{x}=\frac{2.0}{8}=0.250.

The sum of squared deviations from this mean is \(8.90\). Therefore the sample variance and standard deviation are

s^2=\frac{8.90}{8-1}\approx1.271,\qquad s=\sqrt{\frac{8.90}{7}}\approx1.128.

The mean is above zero and the spread is above one, so these summaries suggest a positive location shift and a somewhat wider distribution than the reference. With only eight values, they are descriptive signals, not decisive evidence. The value 2.2 also warrants checking the upper tail in a plot. Neither the mean nor the standard deviation alone indicates whether the remaining distribution is symmetric or otherwise normal-shaped.

How to Use the Checks Responsibly

A practical assessment combines several views rather than relying on a single numerical cutoff. The following sequence keeps the interpretation tied to what each diagnostic can show.

Verify the reference and data.
Confirm that the NPDE correspond to the intended observations and that the predictive simulations and NPDE calculation are adequate for the design.

Inspect the histogram.
Look for a shift, excess spread, asymmetry, or tail differences, while remembering that bin choice and sample size affect its appearance.

Inspect the normal quantile plot.
Check whether departures from the reference line are consistent across the center or concentrated in one tail.

Report the summaries and scope.
Give \(n\), \(\bar{x}\), and \(s\), then describe the visible marginal pattern without claiming that these checks establish overall model adequacy.

NPDE differ from raw residuals because their construction uses the predictive distribution and accounts for its covariance structure; raw residuals can have different expected scales across observations. A visual predictive check (VPC) instead compares observed outcome summaries with predictive intervals across time or other design features. These approaches complement one another: NPDE marginal plots compress deviations into a common reference scale, while a VPC retains outcome-scale and design-specific patterns. A satisfactory marginal NPDE distribution cannot rule out a time trend, subgroup problem, or other structured discrepancy.

Diagnostics, Pitfalls, and Limitations

A striking mean shift, spread difference, curved quantile plot, or tail excess is a reason to investigate, not an automatic diagnosis. Possible sources include model misspecification, an unsuitable predictive distribution, data or record-handling errors, and inadequate simulation. Consider whether a departure is stable across reasonable plotting choices and whether it aligns with the observed design or model predictions. Avoid searching many subgroups and reporting only the most extreme pattern without acknowledging that selection.

Do not treat expected histogram counts as exact quotas. They are expectations under the reference distribution; observed counts fluctuate.
Do not equate mean zero and standard deviation one with normality. Those two summaries do not determine distributional shape.
Do not infer independence from a normal-looking histogram. A histogram discards observation order and dependence information; marginal calibration is a different diagnostic question.
Do not overinterpret a few extreme values. Tail behavior is important, but small samples have unstable tails and simulation error can affect NPDE.
Do not treat marginal calibration as proof of model adequacy. It cannot show that predictions are correct for every time, dose, covariate group, or clinically relevant feature.

The appropriate conclusion is therefore bounded: say whether the marginal NPDE appear broadly compatible with \(N(0,1)\), name the departures that motivate concern, and state what the check cannot resolve. If the pattern is questionable, examine the predictive simulation and complementary diagnostics before deciding what model changes, if any, are warranted.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What does marginal calibration of NPDE mean?
Under the whitening argument, why must \(\Sigma\) be positive definite?
For 200 NPDE, how many values are expected below \(-1.96\) under the standard normal reference?
A quantile plot is nearly straight but has slope greater than one. What departure does that suggest?
Why does a normal-looking NPDE histogram not establish independence or overall model adequacy?

Solutions. (1) Each NPDE has the standard normal marginal distribution under the model and valid construction. (2) Positive definiteness permits a nonsingular Cholesky factor \(L\), so \(L^{-1}\) exists; singular covariance cannot use this ordinary whitening argument. (3) The lower-tail probability is approximately \(0.025\), so \(200(0.025)=5.00\) values are expected. (4) A slope greater than one suggests NPDE spread larger than the standard normal reference, though the plot's variability must be considered. (5) A histogram ignores ordering and dependence, and a good marginal distribution can coexist with structured errors across time or subgroups.