Why Compare More Than Means and Variances?
A simulated sample can have a mean and variance close to their target values while still having the wrong shape. For example, it might contain too few large losses, too many observations near the center, or an incorrect frequency of zero outcomes. Comparing cumulative probabilities and quantiles provides a broader check of how the sample is distributed.
This tutorial considers simulated observations \(X_1,\ldots,X_n\) drawn independently from a specified target distribution with cumulative distribution function \(F(x)=P(X\leq x)\). The empirical CDF, quantiles, and Q-Q plot are descriptive diagnostics: they show how the realized sample compares with the target. A difference by itself does not establish that the simulation is incorrect; random samples naturally vary.
The empirical CDF is a nondecreasing step function. It increases by \(1/n\) at each observation, or by a multiple of \(1/n\) when observations are tied. In contrast, the target CDF \(F(x)\) describes probabilities in the model, not proportions in one particular simulated sample. Comparing \(F_n(x)\) with \(F(x)\) at useful thresholds can reveal where the sample has more or fewer observations than the model would suggest.
Empirical and Theoretical Quantiles
A quantile gives a location in a distribution corresponding to a specified cumulative probability. For a target CDF, use the generalized-inverse definition, which also works for distributions with jumps. For a sample, a convention is needed because an empirical distribution is discrete. This tutorial uses the nearest-rank empirical quantile.
For a continuous target distribution, \(F^{-1}(p)\) is the value below which probability \(p\) lies. For a discrete distribution, the quantile is the first value at which the CDF reaches or exceeds \(p\). The empirical quantile also selects an observed value, so it need not equal the target quantile even when the simulation is correctly implemented.
Worked Example: A Bernoulli Sample With a Tie at the Quantile
A hypothetical simulation targets a Bernoulli distribution with success probability \(0.2\), so each observation is 1 with probability \(0.2\) and 0 otherwise. The sample of 10 simulated values is \(0,0,0,1,0,1,0,0,0,1\). Find \(F_n(0)\), compare it with \(F(0)\), and calculate both empirical and theoretical 75th percentiles.
Method and assumptions. The observations are intended to be independent draws from the stated Bernoulli target. Count observations at or below 0 for the CDF comparison; use the nearest-rank convention for the empirical percentile and the generalized inverse for the theoretical percentile.
Empirical CDF at zero. There are seven zeros among the ten observations, so \(F_n(0)=7/10=0.7000\). Under the target model, \(F(0)=P(X\leq0)=P(X=0)=1-0.2=0.8000\).
Empirical 75th percentile. Sort the sample: \(0,0,0,0,0,0,0,1,1,1\). The nearest-rank position is \(\lceil 10(0.75)\rceil=\lceil7.5\rceil=8\). The eighth ordered value is 1, so the empirical percentile is 1.
Theoretical 75th percentile. The target CDF is \(F(0)=0.8\), which is already at least \(0.75\). Hence \(F^{-1}(0.75)=0\). The empirical and theoretical answers differ: 1 and 0, respectively.
Check and common mistake avoided. The sample has exactly seven zeros, not eight. A tempting but incorrect calculation treats the 75th percentile as the seventh ordered value, which would give 0. Under the stated nearest-rank rule, the position is the ceiling of \(np\), namely 8, and the correct empirical percentile is 1. The discrepancy is possible in a finite sample and does not, on its own, prove an error in the simulation.
Why the Empirical CDF Tracks the Target
At a fixed threshold \(x\), each indicator \(I(X_j\leq x)\) is a Bernoulli random variable with success probability \(F(x)\). If the observations are independent and identically distributed from the target model, then averaging these indicators gives an unbiased estimate of \(F(x)\). The following calculation explains both the expected agreement and the natural sampling variation.
Independence also gives the variance. Each indicator has variance \(F(x)(1-F(x))\); the variance of their average is therefore \(\operatorname{Var}(F_n(x))=F(x)(1-F(x))/n\). The standard deviation is \(\sqrt{F(x)(1-F(x))/n}\). This is a measure of typical variation at one fixed threshold, not a guarantee that every observed difference will be small. Checking many thresholds also makes it more likely that at least one difference looks noticeable by chance.
The assumptions matter. If simulated observations are dependent, the expectation at a threshold may still equal \(F(x)\) when each observation has the target marginal distribution, but the variance formula above need not hold because indicator covariances can be nonzero. If the observations are generated from a different distribution, the target CDF is not the correct benchmark.
Worked Example: Comparing a Uniform Sample With Its Target
A hypothetical simulation is intended to generate independent observations from a uniform distribution on \([0,1]\). Its six ordered observations are \(0.12,0.28,0.44,0.61,0.79,0.93\). Compare empirical and target cumulative probabilities at \(x=0.50\) and \(x=0.70\), then compare their 75th percentiles.
Method and assumptions. For \(0\leq x\leq1\), the target CDF is \(F(x)=x\). The observations are already sorted, so count values at or below each threshold. Use the nearest-rank rule for the empirical quantile.
At \(x=0.50\), the observations \(0.12,0.28,0.44\) are at or below the threshold. Thus \(F_n(0.50)=3/6=0.5000\), while \(F(0.50)=0.5000\).
At \(x=0.70\), four observations are at or below the threshold: \(0.12,0.28,0.44,0.61\). Therefore \(F_n(0.70)=4/6=0.6667\), whereas \(F(0.70)=0.7000\). The difference, empirical minus target, is \(0.6667-0.7000=-0.0333\) to four decimal places.
For the empirical 75th percentile, the rank is \(\lceil6(0.75)\rceil=\lceil4.5\rceil=5\), giving \(X_{(5)}=0.79\). The theoretical 75th percentile solves \(F(x)=0.75\), so it is \(0.75\).
Check and common mistake avoided. At \(0.70\), exactly four—not five—sample values are at or below the threshold. Also, the empirical quantile is the fifth ordered observation under nearest rank; it is not found by interpolating between the fourth and fifth values. The sample percentile \(0.79\) is above the target percentile \(0.75\), a modest finite-sample difference.
Constructing and Reading a Q-Q Plot
A quantile-quantile (Q-Q) plot compares sample quantiles with quantiles from a target distribution. First sort the simulated observations. Then assign a probability to each order statistic and calculate the target quantile at that probability. One common plotting-position choice is \(p_i=(i-0.5)/n\), for \(i=1,\ldots,n\). Plot the target quantile \(F^{-1}(p_i)\) on the horizontal axis and the simulated order statistic \(X_{(i)}\) on the vertical axis.
If the simulation follows the target distribution reasonably well, the plotted points should generally lie near the line \(y=x\). This is an approximate visual reference, not an exact requirement: even a correct finite sample will usually not fall exactly on the line. A consistent shift can indicate a location difference; a pattern that fans outward or inward can indicate a scale difference; and departures concentrated at one end can signal a tail mismatch. Such patterns are clues to investigate, not diagnoses by themselves.
Worked Example: A Normal Q-Q Comparison
A hypothetical simulator targets \(X\sim N(10,4)\), where the second parameter is the variance, so the target standard deviation is \(2\). Five simulated values, in increasing order, are \(8.0,9.2,10.1,11.3,12.8\). Construct a Q-Q comparison using \(p_i=(i-0.5)/5\), calculate observed-minus-target differences, and describe the pattern.
Method and assumptions. Treat the observations as independent draws from the stated normal target. For each plotting position \(p_i\), calculate the standard-normal quantile \(z_{p_i}\), then use \(F^{-1}(p_i)=10+2z_{p_i}\). The normal quantiles below are rounded to four decimal places; monetary values are not involved.
The plotting positions are \(0.1,0.3,0.5,0.7,0.9\). Their standard-normal quantiles are approximately \(-1.2816,-0.5244,0,0.5244,1.2816\). The corresponding target quantiles are:
The plotted pairs \((\text{target quantile},\text{simulated order statistic})\) and their vertical differences are shown below. Differences are calculated from unrounded normal quantiles and rounded to four decimal places.
| \(i\) | \(p_i\) | Target quantile | Simulated \(X_{(i)}\) | Observed minus target |
|---|---|---|---|---|
| 1 | 0.1 | 7.4369 | 8.0 | 0.5631 |
| 2 | 0.3 | 8.9512 | 9.2 | 0.2488 |
| 3 | 0.5 | 10.0000 | 10.1 | 0.1000 |
| 4 | 0.7 | 11.0488 | 11.3 | 0.2512 |
| 5 | 0.9 | 12.5631 | 12.8 | 0.2369 |
For example, at \(p=0.1\), the unrounded target quantile is \(10+2(-1.2815516)=7.4368968\). The difference is \(8.0-7.4368968=0.5631032\), which rounds to \(0.5631\). The same subtraction gives the other listed differences. All five simulated quantiles are above their target counterparts, but five points are too few to conclude that the whole distribution is shifted. The differences are also not constant, so they do not describe a simple exact location shift.
Check and common mistake avoided. The variance is 4, so the standard deviation used to scale normal quantiles is \(\sqrt{4}=2\), not 4. The first target quantile is \(7.4369\), and its difference is \(8.0-7.4369=0.5631\). Subtracting a target quantile from the observation is essential for the stated observed-minus-target convention; reversing the subtraction reverses every sign.
Practical Interpretation and Common Traps
Use these diagnostics to inspect different aspects of a simulated distribution. A few CDF comparisons answer questions at chosen loss thresholds. Quantiles focus on particular percentiles, including the upper tail when high losses matter. A Q-Q plot displays a range of percentiles at once and can make systematic departures easier to see. None of these views should be replaced by a single summary such as the mean.
- Do not treat a small sample discrepancy as proof of a bug. Even under the correct model, \(F_n(x)\) varies from sample to sample. Consider sample size, the threshold, and the overall pattern.
- Do not mix quantile conventions. Nearest rank selects an order statistic. Other sample-quantile methods may interpolate, so their answers can differ, especially for small samples.
- Handle ties and jumps correctly. For a discrete target, \(F^{-1}(p)\) is the smallest value whose CDF is at least \(p\). Do not interpolate across a jump as if the target were continuous.
- State the Q-Q plotting positions. Different plotting-position conventions can produce slightly different target quantiles. Comparisons are clearest when the convention is consistent and explicit.
- Use the correct target parameters and units. A Q-Q plot against the wrong scale, parameterization, or unit can show apparent disagreement even when the generator is functioning as intended.
The empirical CDF and Q-Q plot are complementary. The CDF directly compares proportions below thresholds; the Q-Q plot compares locations at probability levels. Both are most informative when interpreted alongside the known target model and the limitations of a finite sample.
Check Your Understanding
Use the stated conventions and show the counting or quantile calculation before checking each answer.
- A Bernoulli target has success probability \(0.3\), and the sample is \(0,1,0,0,1,0,1,0\). Find \(F_n(0)\), \(F(0)\), and the empirical and theoretical medians. (Answer: \(F_n(0)=0.6250\), \(F(0)=0.7000\); empirical median \(=0\), theoretical median \(=0\).)
- Five ordered observations from a uniform target on \([0,20]\) are \(2,6,9,15,18\). Compare \(F_n(10)\) with \(F(10)\), and find the empirical and theoretical 60th percentiles using nearest rank for the sample. (Answer: \(F_n(10)=0.6000\), \(F(10)=0.5000\); empirical 60th percentile \(=9\), theoretical 60th percentile \(=12\).)
- A sample has sorted values \(1,2,2,5,9\), and its target is uniform on \([0,10]\). At the five observed values, calculate the largest absolute difference between the right-continuous empirical CDF and target CDF. (Answer: \(0.4000\), attained at \(x=2\), where \(F_n(2)=3/5=0.6\) and \(F(2)=0.2\).)
- A simulator targets \(N(50,100)\), with variance 100. For the Q-Q plotting position \(p=0.875\), use \(z_{0.875}\approx1.1503\) to find the target quantile. If the corresponding simulated order statistic is 63, find the observed-minus-target difference. (Answer: target quantile \(=50+10(1.1503)=61.503\); difference \(=63-61.503=1.497\).)
- For a correctly specified continuous target, explain why an empirical CDF at a fixed threshold need not equal the target CDF in a finite independent sample, and state its variance. (Answer: it is an average of \(n\) Bernoulli indicators with success probability \(F(x)\), so \(E[F_n(x)]=F(x)\), but \(\operatorname{Var}(F_n(x))=F(x)(1-F(x))/n\), which is positive when \(0<F(x)<1\).)