Why Test NPDE Distributions Formally?
A plot can reveal whether normalized prediction distribution errors (NPDEs) look compatible with their target distribution, but visual judgments vary across readers. A formal distribution test gives a rule-based assessment of a specified departure. It can be useful as a supplement to graphical evaluation, but it does not replace it: a test result depends on the chosen test, sample size, and assumptions about the observations.
The NPDE target is the standard normal distribution, \(N(0,1)\): it has mean 0 and variance 1. A formal test can therefore ask whether NPDEs are compatible with that particular distribution. This is more specific than asking whether they resemble some normal distribution. For example, values that are approximately normal with mean 0.4 and standard deviation 1.2 are not calibrated to the standard normal target, even though their shape may appear normal.
For the usual textbook calibration of these tests, the tested observations must be independent and identically distributed under the null. They must also be drawn from the distribution specified by \(H_0\), and the test’s reference distribution must match how its parameters were handled. Those conditions matter for NPDEs: repeated observations from the same subject may remain dependent, and some tests assess an unspecified normal distribution rather than the required \(N(0,1)\).
Tests of a Fixed Target and Tests of Normal Shape
Several familiar tests answer related but distinct questions. A one-sample Kolmogorov–Smirnov test can compare an empirical distribution function with a fully specified continuous distribution such as \(N(0,1)\). Its standard null calibration assumes independent observations and parameters specified independently of the tested sample. If the normal mean and variance are estimated from the same data, the ordinary reference distribution no longer applies; an adjustment such as a Lilliefors procedure or a suitable simulation calibration is needed.
The Shapiro–Wilk test assesses whether data are compatible with a normal distribution whose mean and variance are not fixed in advance. Thus, a non-significant Shapiro–Wilk result alone does not establish that NPDEs follow \(N(0,1)\). It can be useful for detecting non-normal shape, but it should be accompanied by checks of location and spread relative to the target. The Anderson–Darling test also assesses distributional compatibility and places relatively greater emphasis on the tails than the Kolmogorov–Smirnov test. A chi-square goodness-of-fit test compares counts in bins with their expected counts under a stated distribution.
No test is best in every setting. Tail departures, asymmetry, and a shift in the mean can produce different amounts of evidence for different procedures. Before selecting a test, specify whether the scientific target is the fixed standard normal distribution or merely the broader family of normal distributions.
A Binned Goodness-of-Fit Test
A chi-square test is straightforward to calculate and makes the comparison between observed and expected frequencies explicit. Choose bins in advance, calculate the probability of each bin under \(N(0,1)\), and multiply those probabilities by the sample size to obtain expected counts. The test statistic is the sum, over bins, of squared observed-minus-expected differences divided by expected counts.
Here \(O_j\) and \(E_j\) are the observed and expected counts in bin \(j\), and \(k\) is the number of bins. If bin probabilities are fixed under \(H_0\), the observations are independent, and expected counts are sufficiently large for the approximation, the reference distribution is approximately chi-square with \(k-1\) degrees of freedom. If parameters are estimated from the tested observations, degrees of freedom generally need to be reduced for the estimated parameters, subject to the specific test construction.
Worked Example: A Standard-Normal Test With 40 NPDEs
Suppose a hypothetical model assessment produces 40 NPDEs. For a simple fixed-target test, divide the standard normal distribution into four bins: below \(-1\), from \(-1\) to 0, from 0 to 1, and above 1. The standard normal probabilities of these bins are approximately 0.1587, 0.3413, 0.3413, and 0.1587. The observed counts are 5, 15, 14, and 6, respectively.
The expected counts are \(40(0.1587)=6.348\), \(40(0.3413)=13.652\), \(40(0.3413)=13.652\), and \(40(0.1587)=6.348\). Each expected count exceeds 5, so the usual expected-count guideline is met. The statistic is:
The bins have fixed probabilities and no distribution parameters were estimated, so \(df=4-1=3\). For a chi-square distribution with 3 degrees of freedom, the upper-tail p-value is approximately 0.930. At a prespecified significance level of \(\alpha=0.05\), this result does not provide evidence against the standard normal target. It does not prove that the NPDE distribution is standard normal; with only 40 observations, meaningful departures may be difficult to detect. The independence assumption also needs to be credible for this p-value to have its stated calibration.
Why Sample Size Changes the Interpretation
A test’s p-value reflects both the size of the departure and the information in the data. With a small sample, even a noticeable mismatch can yield a large p-value. With a very large sample, small departures that may have little practical impact can produce small p-values. Therefore, a p-value should not be read as a direct measure of how far the NPDEs depart from \(N(0,1)\).
Worked Example: A Small Departure Detected With Many Observations
Consider a second hypothetical data set with 1,000 NPDEs, divided using the same four bins. The observed counts are 180, 320, 320, and 180. Under the standard normal target, the expected counts are approximately 158.7, 341.3, 341.3, and 158.7. All expected counts are large, and the bins and target were fixed in advance.
The chi-square statistic is:
With 3 degrees of freedom, the upper-tail p-value is approximately 0.0389. At \(\alpha=0.05\), this is evidence against the standard normal target, assuming independent NPDEs and the other test conditions hold. The observed excess is in the two outer bins: 360 observations were outside \([-1,1]\), compared with an expected total of 317.4. Whether this departure matters for a particular predictive use requires examining its magnitude and pattern, not just noting that the p-value is below 0.05.
Dependence Can Miscalibrate a Test
A large number of NPDE values is not equivalent to the same number of independent observations. Repeated measurements from one subject can share information about that subject, and resulting errors may be correlated. In that case, applying a test calibrated for independent observations to every row can produce a p-value that is too small or too large. The direction and amount of miscalibration depend on the dependence structure and test.
Worked Example: Paired NPDEs and the Effective Information
Suppose a hypothetical data set has 50 subjects with two NPDEs per subject, giving 100 values. For illustration, assume the pairs have correlation \(\rho=0.6\), while different subjects are independent. If each value has variance 1 under the target, the variance of the overall mean is not \(1/100\). The 100 observations contribute 100 individual variances and 100 covariance terms, because each of the 50 pairs contributes twice its covariance in the variance of a sum:
Thus the standard error of the mean is \(\sqrt{0.016}\approx0.1265\), rather than the independence-based value \(1/\sqrt{100}=0.1000\). If the observed mean were 0.25, the independence-based standardized value would be \(0.25/0.1000=2.50\), whereas this illustrative covariance-aware value would be \(0.25/0.1265\approx1.976\). This calculation concerns the mean component, not an omnibus normality-test p-value; it shows why dependence can change the uncertainty underlying a test. A formal test’s null calibration must account for the actual dependence, for example through a simulation or resampling procedure that preserves the subject-level structure.
The intended NPDE construction includes decorrelation, but that does not justify assuming independence automatically. Check how NPDEs were generated and whether the analyzed observations can reasonably be treated as independent. Where dependence remains, a standard independent-observation p-value should not be presented as though its nominal error rate were guaranteed.
Use Graphs Alongside Tests
A Q–Q plot compares ordered NPDEs with the corresponding standard normal quantiles. Agreement with the target appears approximately as points following a straight reference line. Curvature, asymmetry, unusually distant tail points, or a shifted or steeper pattern can reveal features that a single test statistic does not identify. A histogram can provide a complementary view of shape, although its appearance depends on the bin width and its assessment is less precise in the tails.
A formal test compresses the data into a statistic and p-value; it does not explain the form of a discrepancy. A Q–Q plot can suggest whether a departure is primarily a location shift, a scale mismatch, skewness, or a tail difference, but it does not supply a calibrated p-value. Read them together: a statistically significant result with a nearly straight Q–Q plot may indicate a small departure made detectable by a large sample, while a non-significant test does not rule out an important pattern in a small sample.
| Approach | What it contributes | Important limitation |
|---|---|---|
| Formal goodness-of-fit test | A statistic and a p-value for a specified test and null distribution | Depends on test assumptions, calibration, and sample size; may not identify the departure’s form |
| Q–Q plot | A visual comparison with standard normal quantiles that can expose shape and tail patterns | Interpretation is subjective and does not by itself provide a calibrated test |
| Histogram | A broad view of the distribution’s shape and location | Appearance depends on binning and can conceal tail discrepancies |
Diagnostics, Limitations, and Common Mistakes
Before reporting a formal result, state the null distribution, identify which test was used, and check whether its assumptions are plausible. For a chi-square test, inspect expected counts and confirm that bin probabilities were set under the intended target. For tests with estimated parameters, use the corresponding calibrated procedure rather than assuming the fixed-parameter reference distribution applies. For NPDEs with repeated subject contributions, consider whether the test preserves the dependence structure.
Common mistake: “The Shapiro–Wilk p-value is greater than 0.05, so the NPDEs are standard normal.” This reasoning is incorrect. Shapiro–Wilk tests compatibility with a normal distribution whose mean and variance are not fixed at 0 and 1, and a non-significant result is not proof of the null. Check the NPDEs against the specific standard normal target, inspect location and spread, and use plots to assess the form and practical size of any departure.
A formal distribution test is one diagnostic, not a stand-alone verdict on predictive adequacy. It can detect departures from its null distribution under its assumptions, but it cannot establish why a departure occurred or certify that all relevant aspects of a population PK model are adequate. Report the test result with graphical findings and an interpretation in the context of the sample size and dependence.
State whether the question concerns the fixed \(N(0,1)\) distribution or normal shape with parameters estimated from the data.
Assess independence, parameter handling, and, for binned tests, expected counts and bin selection.
Compare the p-value with the prespecified \(\alpha\), then consider sample size and the estimated size and form of the departure.
Use a Q–Q plot and, where helpful, a histogram to see what kind of mismatch the test may be detecting.
Check Your Understanding
Answer each question, then compare with the solutions below.
- What is the null hypothesis for a test of the standard-normal NPDE target?
- Why does a non-significant Shapiro–Wilk test not establish that NPDEs follow \(N(0,1)\)?
- A test gives \(p=0.004\) in a very large sample, while the Q–Q plot shows only a slight, consistent departure. What can and cannot be concluded?
- Why might a test applied to 100 repeated NPDE observations give an invalid p-value if those observations come from 50 correlated pairs?
Solutions. (1) \(H_0\) states that the NPDEs follow \(N(0,1)\); \(H_a\) states that their distribution differs from that target. (2) Shapiro–Wilk assesses compatibility with a normal distribution whose mean and variance are not fixed at 0 and 1, and failure to reject is not proof of the null. (3) Under the test’s assumptions, the result is evidence against its null at conventional significance levels, but it does not show that the departure is practically important; the plot and magnitude matter, and large samples can detect small departures. (4) The ordinary test’s reference distribution may assume 100 independent observations, while correlation reduces or otherwise changes the information and can alter the null distribution of the statistic. A method that preserves the dependence structure is needed for calibrated inference.