Why the KS Critical Value Depends on How the Model Was Obtained
The one-sample Kolmogorov-Smirnov (KS) test compares an empirical distribution with a proposed cumulative distribution function (CDF). Its familiar critical values apply when the reference CDF is fully specified before the tested observations are examined. A different calibration is needed when parameters of that CDF are estimated from those same observations. The distinction matters because fitting the model changes the distribution of the test statistic, and can make the standard cutoff misleading.
For example, a hypothetical insurer may compare observed claim severities with an exponential model. If the exponential rate is fixed from an independent source and treated as known, a fully specified KS reference distribution may be appropriate under the test assumptions. If the rate is estimated from the very claims being tested, the usual fully specified critical value is not automatically valid. The correction must reflect the estimation step.
The null hypothesis in a fitted-model setting is composite: the observations follow a member \(F_\theta\) of a model family, but the parameter \(\theta\) is unknown. The usual plug-in statistic is \(D_n=\sup_x|F_n(x)-F_{\hat\theta}(x)|\), where \(\hat\theta\) is estimated from the sample. Its critical values must account for how \(\hat\theta\) was obtained.
Why Estimation Changes the Reference Distribution
When \(F_0\) is a fully specified continuous CDF and the null hypothesis is true, the probability integral transform gives \(U_i=F_0(X_i)\sim\mathrm{Uniform}(0,1)\). The transformed observations are independent, and the KS statistic has the same null distribution for every fully specified continuous \(F_0\). This is the distribution-free property behind the standard KS critical values.
That reasoning does not carry over unchanged when parameters are estimated from the tested data. The fitted CDF \(F_{\hat\theta}\) depends on the same observations used to form \(F_n\). Consequently, the difference \(F_n-F_{\hat\theta}\) is not the difference between an empirical CDF and a fixed null CDF. In common fitted-model settings, fitting tends to bring the fitted CDF closer to the observations, so using the fully specified reference distribution can misstate the rejection probability. The exact effect depends on the model and the estimation method; do not assume a universal adjustment.
For a two-sided test, a commonly used large-sample approximation to the 5% critical value for a fully specified continuous CDF is \(1.36/\sqrt{n}\). This is an approximation, not an exact finite-sample value. The issue addressed here is not the approximation itself: it is using that fully specified-model cutoff after fitting parameters to the tested data.
Two Suitable Corrections
A Lilliefors correction uses critical values derived for a particular model family when its parameters are estimated from the tested sample. For example, there are Lilliefors procedures for normality when the mean and variance are estimated. Critical values depend on the specified model and fitting procedure; a correction for one family should not be transferred to another.
A parametric bootstrap provides a flexible alternative when a suitable tabulated correction is unavailable. Fit the null model, simulate samples from that fitted model, refit the model to every simulated sample using the original estimation method, and recalculate the KS statistic against each refitted CDF. The simulated statistics then approximate the null distribution of the statistic with parameter estimation included.
Estimate \(\hat\theta\) from the observed sample and calculate \(D_{\mathrm{obs}}=\sup_x|F_n(x)-F_{\hat\theta}(x)|\).
Generate \(B\) independent samples, each of size \(n\), from \(F_{\hat\theta}\).
For each simulated sample, estimate its parameter using the same method as for the observed data, then calculate its KS statistic against its own fitted CDF.
Compare \(D_{\mathrm{obs}}\) with the bootstrap statistics. An estimated upper-tail p-value is \((r+1)/(B+1)\), where \(r\) is the number of bootstrap statistics at least as large as \(D_{\mathrm{obs}}\).
The \(+1\) convention avoids reporting a simulated p-value of zero when none of the \(B\) bootstrap statistics exceed the observed statistic. The bootstrap is an approximation: its quality depends on the model, sample size, and number of simulations. It must also reproduce relevant features of the data collection process. For example, if the tested data are censored, truncated, or rounded, the simulation and statistic need to reflect that structure; the simple complete, continuous-sample procedure above does not automatically apply.
Worked Example: A Fitted Normal Model and the Standard Cutoff
A hypothetical portfolio has 80 complete claim severities. A normal distribution is fitted by estimating its mean and standard deviation from these 80 observations. The largest positive and negative KS deviations are \(D_n^+=0.105\) and \(D_n^-=0.091\), respectively. At the 5% level, assess the tempting calculation using the fully specified critical value, then give an appropriate fitted-model decision if a parametric bootstrap reports 47 of 999 statistics at least as large as the observed statistic.
Step 1: Calculate the observed statistic. By definition, \(D_n=\max(0.105,0.091)=0.105\).
Step 2: Show the tempting, but invalid, comparison. The large-sample fully specified 5% cutoff is approximately \(1.36/\sqrt{80}\). Since \(\sqrt{80}\approx8.9443\), $$ \frac{1.36}{\sqrt{80}}\approx\frac{1.36}{8.9443}=0.1521. $$ The comparison \(0.105<0.1521\) would appear to say “do not reject.” But the mean and standard deviation were estimated from these same claims, so this fully specified cutoff does not calibrate the test. That apparent decision is not justified.
Step 3: Use the fitted-model bootstrap result. Each bootstrap sample is generated from the fitted normal model, and the mean and standard deviation are re-estimated before calculating that sample’s KS statistic. With \(r=47\) and \(B=999\), $$ \hat p=\frac{r+1}{B+1}=\frac{47+1}{999+1}=\frac{48}{1000}=0.0480. $$ At the 5% significance level, this bootstrap result leads to rejection of the fitted normal model. The appropriate decision differs from what the invalid standard-cutoff comparison suggests.
Check. The reported \(D_n\) is the larger of the two one-sided deviations, and the bootstrap p-value is below 0.05. The method avoids the common mistake of treating fitted parameters as known.
Calculating the Statistic Before Choosing Its Calibration
The KS statistic is calculated from the sample and fitted CDF; the correction is applied afterward. Keeping those tasks separate helps prevent a common mix-up: estimating a model does not change the definition of \(D_n\), but it does change the reference distribution used to assess \(D_n\). The following example calculates the statistic explicitly for an exponential severity model.
Worked Example: An Exponential Rate Estimated From the Claims
A hypothetical sample of five complete loss severities, in thousands of dollars, is \(0.2, 0.5, 0.7, 1.0, 1.6\). Assume an exponential model with rate \(\lambda\), density \(f(x)=\lambda e^{-\lambda x}\) for \(x\geq0\), and CDF \(F_\lambda(x)=1-e^{-\lambda x}\). Estimate \(\lambda\) by maximum likelihood, calculate the KS statistic, and interpret a supplied bootstrap result of 421 out of 999 statistics at least as large as the observed value.
Step 1: Estimate the rate. For a complete exponential sample, the maximum-likelihood estimate is the reciprocal of the sample mean. Here, $$ \bar{x}=\frac{0.2+0.5+0.7+1.0+1.6}{5}=\frac{4.0}{5}=0.8, \qquad \hat\lambda=\frac{1}{0.8}=1.25. $$ The units of \(\hat\lambda\) are per thousand dollars.
Step 2: Evaluate the fitted CDF at the ordered observations. Using \(F_{\hat\lambda}(x)=1-e^{-1.25x}\), the values are approximately \(0.2212, 0.4647, 0.5831, 0.7135,\) and \(0.8647\), in observation order.
Step 3: Calculate both one-sided deviations. The empirical-CDF steps are \(i/5=0.2,0.4,0.6,0.8,1.0\), while the preceding steps are \((i-1)/5=0,0.2,0.4,0.6,0.8\). Thus the positive deviations \(i/5-F_{\hat\lambda}(x_{(i)})\) are approximately \(-0.0212,-0.0647,0.0169,0.0865,0.1353\), giving \(D_n^+=0.1353\). The negative deviations \(F_{\hat\lambda}(x_{(i)})-(i-1)/5\) are approximately \(0.2212,0.2647,0.1831,0.1135,0.0647\), giving \(D_n^-=0.2647\). Therefore, $$ D_n=\max(0.1353,0.2647)=0.2647. $$
Step 4: Use the stated bootstrap calibration. Because the rate was estimated from these five observations, do not compare \(0.2647\) with a fully specified KS critical value. The supplied bootstrap result gives \(r=421\) and \(B=999\), so $$ \hat p=\frac{421+1}{999+1}=\frac{422}{1000}=0.4220. $$ At the 5% level, this result does not lead to rejection. It is a bootstrap-based conclusion, not a conclusion from standard fully specified critical values.
Check. The sample mean is \(0.8\), the fitted rate is its reciprocal, and the larger one-sided deviation is \(D_n^-\). The calculation keeps the fitted rate and the calibration issue distinct.
Known Parameters Versus Parameters Estimated From the Test Sample
The status of a parameter is determined by the analysis, not by whether its numerical value appears in the question. A value specified as fixed under the null is different from the same value obtained by fitting the tested data. If the parameter is truly fixed for the test, the fully specified KS calibration may apply. If it was estimated from the tested sample, use a fitted-model correction. If the value came from separate data, the dependence issue is different from estimating it on the test sample, but uncertainty in the external estimate may still matter for the overall inference. Do not silently treat an uncertain external estimate as known.
Worked Example: The Same Statistic Under Two Different Null Specifications
A hypothetical study tests 100 complete loss severities against an exponential CDF. The observed KS statistic is \(D_n=0.120\), and the sample mean happens to be \(1{,}000\) dollars. Compare the decision when the null rate is fixed in advance at \(1/1{,}000\) per dollar with the decision when that rate is estimated from these 100 losses. For the fitted-rate case, a valid procedure-specific bootstrap has supplied a 5% critical value of \(0.091\).
Fixed-rate case. The rate is specified before examining the sample, so the null CDF is fully specified. The large-sample 5% cutoff is approximately $$ \frac{1.36}{\sqrt{100}}=\frac{1.36}{10}=0.1360. $$ Since \(0.120<0.1360\), this approximation does not lead to rejection at 5%.
Estimated-rate case. The sample mean is \(1{,}000\) dollars, so estimating the exponential rate by maximum likelihood also gives \(\hat\lambda=1/1{,}000\) per dollar. The numerical fitted CDF is therefore the same, but now the rate was estimated from the tested observations. Use the supplied fitted-model bootstrap cutoff instead: \(0.120>0.091\), so reject at 5% under that calibration.
Check. The numerical rate and \(D_n\) are the same in both cases, but the null procedures differ. A fitted model needs a calibration that accounts for fitting; the standard cutoff is not made valid merely because its numerical CDF matches the fitted CDF.
Common Exam Trap and Practical Checks
The central trap is to calculate the standard KS cutoff correctly and then use it in the wrong setting. In the first worked example, the arithmetic \(1.36/\sqrt{80}\approx0.1521\) is correct, and \(0.105<0.1521\) is also correct. What fails is the inference that follows: the cutoff is for a fully specified continuous null, while the normal mean and standard deviation were estimated from the tested data. The bootstrap correction can yield a different decision.
Before making a KS decision, check the following:
- Was the null CDF fully specified? If any model parameter was estimated from the tested sample, standard fully specified critical values are not automatically appropriate.
- Does the correction match the model? A Lilliefors correction is specific to its model family and estimation procedure. Otherwise, use an appropriate fitted-model calibration.
- Does the bootstrap repeat the fit? Simulating from the fitted null is not enough. The parameter must be re-estimated within every simulated sample before that sample’s statistic is calculated.
- Does the simulated sample match the data process? The basic procedure assumes independent, complete observations from a continuous distribution. Ties, censoring, truncation, or other sampling features may require a test and calibration that account for them.
- Is the reported cutoff an approximation? State whether a critical value is a large-sample approximation, a finite-sample table value, or a bootstrap result.
A KS test can be a useful model diagnostic, but its conclusion depends on both the observed statistic and a correctly chosen reference distribution. This distinction is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus: identify whether parameters were estimated from the tested data, then select a calibration that reflects that estimation.
Check Your Understanding
For each problem, focus on whether the null CDF is fixed or fitted and use the stated calibration only in the setting for which it is appropriate.
- A sample of 100 independent complete observations is tested against a continuous CDF fully specified before sampling. The observed statistic is \(D_n=0.120\). Using the large-sample 5% fully specified cutoff, calculate the cutoff and state the decision. (Answer: \(1.36/\sqrt{100}=0.1360\); do not reject at 5%.)
- A normal model’s mean and standard deviation are estimated from the same 64 observations used in the KS test. The observed statistic is \(0.140\). A candidate compares it with \(1.36/\sqrt{64}=0.1700\) and concludes that the model is not rejected. What is the error, and what correction is suitable? (Answer: The fully specified cutoff is not justified after fitting both parameters; use an appropriate fitted-normal Lilliefors correction or a parametric bootstrap that re-estimates both parameters.)
- For a fitted severity model, a parametric bootstrap uses \(B=999\) simulated samples and finds \(r=29\) bootstrap statistics at least as large as the observed statistic. Calculate the estimated p-value and state the 5% decision. (Answer: \((29+1)/(999+1)=0.0300\); reject at 5%.)
- A fully specified continuous null is tested with \(n=50\) and observed \(D_n=0.160\). Using the large-sample 5% cutoff, determine the decision. (Answer: \(1.36/\sqrt{50}\approx0.1923\); do not reject at 5%.)
- A candidate simulates samples from a fitted exponential model but keeps the original estimated rate fixed when calculating every simulated KS statistic. What essential step is missing? (Answer: Re-estimate the rate in every bootstrap sample, then compare that sample’s empirical CDF with its own refitted exponential CDF.)