Graphical Diagnostics Alongside Goodness-Of-Fit Tests

Why Look at a Goodness-of-Fit Plot?

A goodness-of-fit statistic reduces a sample to a number. That number can indicate whether a fitted distribution is in tension with the data, but it does not show where the mismatch occurs. An empirical-versus-fitted distribution plot or a probability plot preserves more of the pattern: it can suggest that a discrepancy is concentrated in the lower tail, the center, or the upper tail. These plots complement, rather than replace, formal tests such as Kolmogorov-Smirnov, Anderson-Darling, and Pearson chi-square.

The comparison must be made against the correct fitted distribution and with the data features in view. In particular, ties create larger jumps in the empirical distribution function, and a probability plot requires an explicit choice of plotting positions. A plot is diagnostic evidence, not a calibrated significance test.

Definition: For observations $x_1,\ldots,x_n$, the empirical distribution function (EDF) is $$ F_n(x)=\frac{1}{n}\sum_{j=1}^{n} I(x_j\le x), $$ where $I(\cdot)$ equals 1 when its condition is true and 0 otherwise. It is a nondecreasing, right-continuous step function. At a distinct observed value occurring $k$ times, it jumps by $k/n$; when there are no ties, each jump is $1/n$. A fitted CDF $F_0(x)$ is the cumulative distribution function specified by the candidate model, using its fixed or estimated parameter values.

Overlaying the Empirical and Fitted CDFs

An empirical-versus-fitted CDF plot places loss value $x$ on the horizontal axis and cumulative probability on the vertical axis. Draw $F_n(x)$ as a step function and $F_0(x)$ as a smooth curve when the fitted model is continuous. A departure is visible wherever the steps lie persistently above or below the model curve. A difference at one point may be less informative than a pattern across a range of losses.

The vertical difference $F_n(x)-F_0(x)$ has a direct interpretation: it compares the observed proportion of losses no greater than $x$ with the proportion predicted by the model. A positive difference means the sample has accumulated more probability by $x$ than the model predicts. For a continuous fitted CDF, comparing just to the right of a data value is not the whole story: the empirical function also has a left-hand limit before its jump. The plot displays both sides of that change.

Worked Example: Reading an EDF With a Tie

A hypothetical sample of five claim amounts, in thousands of dollars, is $1,2,2,4,7$. A continuous candidate model has CDF $F_0(x)=1-e^{-x/3}$ for $x\ge0$. Construct the empirical steps at the observed values and compare them with the model.

Method: build the EDF from cumulative counts. At $x=1$, one of five observations is no greater than $x$, so $F_n(1)=1/5=0.2$. At $x=2$, three observations are no greater than $x$, so $F_n(2)=3/5=0.6$. The two observations equal to 2 create one jump of $2/5=0.4$, not two separately visible jumps at different loss values. At $x=4$, $F_n(4)=4/5=0.8$, and at $x=7$, $F_n(7)=5/5=1$.

Evaluate the fitted curve at the same values. At $x=1$, $F_0(1)=1-e^{-1/3}\approx0.2835$. At $x=2$, $F_0(2)=1-e^{-2/3}\approx0.4866$. At $x=4$, $F_0(4)=1-e^{-4/3}\approx0.7364$. At $x=7$, $F_0(7)=1-e^{-7/3}\approx0.9030$.

The right-hand differences $F_n(x)-F_0(x)$ at these observations are approximately $-0.0835,\ 0.1134,\ 0.0636,$ and $0.0970$. There is also an important comparison just before the tied jump at 2: $F_n(2-)=1/5=0.2$, while the continuous model approaches $F_0(2)=0.4866$. The difference there is approximately $-0.2866$. Thus the plot shows a substantial gap immediately before the jump, followed by a smaller gap immediately after it.

Check. The empirical values are nondecreasing and finish at 1, and the tied observations account for a jump of $0.4$. The common mistake avoided is assigning a jump of $1/5$ at each distinct observed value even when two observations are tied.

Derivation: At any $x$, each observation contributes either 0 or $1/n$ to $F_n(x)$. As $x$ passes a distinct value $z$ occurring $k$ times, exactly those $k$ indicator terms change from 0 to 1. The total increase is therefore $k(1/n)=k/n$. This also explains why the EDF is a step function and why ties produce a larger step.

When there are no ties and $x_{(1)}\le \cdots\le x_{(n)}$ are the ordered observations, the vertical gaps at and just before each observation are closely related to the one-sided departures summarized by the Kolmogorov-Smirnov statistic. The plot provides context for those gaps; it does not change how a test statistic or its critical values are calculated. For a fitted model with parameters estimated from the same data, the distribution used to calibrate a formal test must account for that estimation.

Probability Plots

A probability plot compares observed ordered values with quantiles predicted by a model. For a continuous fitted CDF $F_0$, its theoretical quantile at probability $p$, where $0<p<1$, is $F_0^{-1}(p)$. One common convention assigns the $i$th ordered observation the plotting position $$ p_i=\frac{i-\tfrac12}{n},\qquad i=1,\ldots,n. $$ The plot then pairs $x_{(i)}$ with $F_0^{-1}(p_i)$. Plotting positions are conventions rather than observed probabilities; other choices exist. State the convention so the construction is reproducible.

The axes can be arranged either way. If observed quantiles $x_{(i)}$ are plotted vertically against theoretical quantiles $F_0^{-1}(p_i)$ horizontally, agreement with the model is suggested by points near the line $y=x$. If the axes are reversed, the same comparison still uses the equality line, but the interpretation of vertical deviations changes. Always label the axes before interpreting a pattern.

A probability plot is especially helpful for seeing systematic curvature or tail departures. A roughly consistent offset may suggest a location mismatch; a systematic change in the size of deviations across the range may suggest a scale mismatch. Curvature or an isolated tail bend may indicate a shape or tail mismatch. These are clues, not unique diagnoses: different causes can generate similar patterns, and small samples can produce irregular plots even when a model is reasonable.

Worked Example: Constructing an Exponential Probability Plot

For a hypothetical set of five complete, uncensored losses, the ordered amounts are $1,2,2,4,7$, in thousands of dollars. Compare them with a specified exponential candidate distribution having mean 3 thousand dollars, so $F_0(x)=1-e^{-x/3}$. Use midpoint plotting positions and put observed losses on the vertical axis.

Method: calculate a model quantile for each plotting position. The positions are $p_i=(i-\tfrac12)/5$, giving $0.1,0.3,0.5,0.7,0.9$. For this model, $$ F_0^{-1}(p)=-3\ln(1-p). $$

At $p=0.1$, the theoretical quantile is $-3\ln(0.9)\approx0.3161$. At $p=0.3$, it is $-3\ln(0.7)\approx1.0700$. At $p=0.5$, it is $-3\ln(0.5)\approx2.0794$. At $p=0.7$, it is $-3\ln(0.3)\approx3.6119$. At $p=0.9$, it is $-3\ln(0.1)\approx6.9078$.

The plotted pairs $(F_0^{-1}(p_i),x_{(i)})$ are approximately $(0.3161,1)$, $(1.0700,2)$, $(2.0794,2)$, $(3.6119,4)$, and $(6.9078,7)$. Relative to the equality line, the observed-minus-theoretical differences are $0.6839,\ 0.9300,\ -0.0794,\ 0.3881,$ and $0.0922$ thousand dollars. The points do not show a large, steadily increasing upper-tail departure in this small sample; the first two points are notably above the line.

Check. The positions increase from 0.1 to 0.9, and the resulting theoretical quantiles increase from about 0.3161 to 6.9078. The common mistake avoided is using $i/n$ without stating that choice, or reversing the axes and then interpreting a vertical difference as though the axes were unchanged.

Reading Patterns Without Overclaiming

The following interpretations are useful starting points when inspecting an overlay or probability plot. They should be checked against the data collection and fitting procedure before drawing a conclusion.

If $F_n$ lies above $F_0$ over an interval, the empirical sample has accumulated more probability by those loss levels than the model predicts. On an observed-versus-theoretical quantile plot, the direction of the visual departure depends on the chosen axes.
A discrepancy that develops mainly near the largest observations suggests inspecting upper-tail fit. The Anderson-Darling statistic gives particular weight to tail discrepancies, but the plot shows which observed region contributes to the concern.
One unusually distant point may reflect a single observation, a data error, or genuine tail variation. It does not by itself establish that the entire model is unsuitable.
Repeated losses or values rounded to fixed increments create tied EDF jumps and repeated observed quantiles. They should not be treated as if they were distinct, continuously measured values.
With small samples, a jagged or curved pattern can occur by chance. Visual inspection alone does not provide a p-value or a decision rule.

Worked Example: Locating a Departure From CDF Values

A hypothetical sample of ten losses, in thousands of dollars, is $1,2,2,4,5,5,6,7,10,12$. A fitted continuous model has the following CDF values at selected thresholds: $F_0(2)=0.20$, $F_0(5)=0.55$, and $F_0(10)=0.95$. Use the EDF overlay to describe the direction of the departures at those thresholds.

Method: count observations at or below each threshold. At 2, three observations are at or below the threshold, so $F_n(2)=3/10=0.30$. At 5, six are at or below it, so $F_n(5)=6/10=0.60$. At 10, nine are at or below it, so $F_n(10)=9/10=0.90$.

Subtract the model values. At 2, $F_n(2)-F_0(2)=0.30-0.20=0.10$. At 5, the difference is $0.60-0.55=0.05$. At 10, it is $0.90-0.95=-0.05$. Thus, at the first two thresholds, the empirical sample has accumulated more probability than the model; by 10, it has accumulated less. The final observation is 12, so the sample still has one loss above 10, whereas the model assigns only 0.05 probability above 10.

Check. The empirical probabilities are nondecreasing, and the model probabilities supplied are also nondecreasing. The common mistake avoided is saying that a positive difference means the model has more observations below the threshold; the positive difference means the empirical proportion is larger.

Use Plots Alongside Tests

A plot is most informative when it answers a focused question: Is the discrepancy concentrated in a tail? Does the empirical distribution accumulate faster than the model over a range? Are a few observations driving the visual difference? The Kolmogorov-Smirnov statistic summarizes the largest absolute empirical-to-model CDF gap, Anderson-Darling emphasizes tail behavior, and chi-square compares observed and expected counts in chosen categories. A graph can help explain a result from any of these tests, but it does not replace their assumptions, calibration, or decision rules.

Parameter fitting deserves particular care. If the same observations determine the fitted parameters and appear in the diagnostic plot, the empirical and fitted distributions are not independent. Visual comparison remains useful, but formal test calibration must reflect parameter estimation. Also, a plot made from censored or truncated data must use an empirical estimator appropriate to that observation scheme; the ordinary complete-sample EDF defined here is not automatically valid for such data.

Key takeaway: Use an EDF overlay to locate differences in cumulative probability and a probability plot to compare observed and model quantiles. State the plotting convention, handle ties by their full jump size, and treat graphical patterns as diagnostic evidence to be considered alongside calibrated goodness-of-fit tests.

Check Your Understanding

For each problem, state the calculation or comparison that supports your conclusion.

A sample is $3,3,4,8,8,8$. What is the size of the EDF jump at 3 and at 8? (Answer: $2/6=0.3333$ at 3 and $3/6=0.5000$ at 8.)
For observations $1,2,4,6,9$, calculate $F_n(4)$. If a fitted continuous model has $F_0(4)=0.50$, what is the right-hand difference there? (Answer: $F_n(4)=3/5=0.6000$, so $F_n(4)-F_0(4)=0.1000$.)
A probability plot uses midpoint positions for $n=7$. What plotting position is assigned to the fourth ordered observation? (Answer: $p_4=(4-\tfrac12)/7=0.5000$.)
At two thresholds, a sample gives $F_n(3)=0.4$ and $F_n(8)=0.9$; the fitted CDF gives $F_0(3)=0.5$ and $F_0(8)=0.8$. State the signed differences and their interpretation. (Answer: $-0.1$ at 3 and $+0.1$ at 8; the empirical distribution has accumulated less probability by 3 and more by 8.)
On a probability plot with observed quantiles on the vertical axis, several of the largest points lie above the equality line, while central points lie close to it. What region merits investigation, and what does the plot not establish? (Answer: Investigate the upper-tail fit; the pattern alone does not establish statistical significance or prove that the model is unsuitable.)

What You'll Learn