Bandwidth Selection For Heavy-Tailed Losses

Why Heavy-Tailed Losses Need Special Bandwidth Checks

A kernel density estimate smooths observed losses over a scale set by its bandwidth $h$. A single global bandwidth is convenient, but it is selected to balance error across the distribution as a whole. If extreme losses are sparse, that overall balance may not produce a useful estimate where actuarial decisions are most sensitive: the upper tail.

A bandwidth that is too large can blur a tail feature or spread mass from more common losses into a region with little data. A bandwidth that is too small can make a few extreme observations look like separate spikes. Neither visual smoothness nor the fact that a rule returns a numerical answer establishes that the tail estimate is reliable. The appropriate response is to compare bandwidths using diagnostics that explicitly examine tail behavior.

Model and scope: Let $X_1,\ldots,X_n$ be independent, identically distributed, exact observations from a continuous loss density $f_X$. The ordinary kernel density estimator is $$ \hat f_h(x)=\frac{1}{nh}\sum_{i=1}^{n}K\!\left(\frac{x-X_i}{h}\right), $$ where $K$ is a nonnegative density kernel and $h>0$ is measured in the same units as $X$. This tutorial concerns bandwidth choice for such exact observations; it does not adjust for truncation or censoring.

Global Rules Are Starting Points, Not Tail Guarantees

For a smooth density with suitable curvature and an appropriate finite-variance kernel, an asymptotic mean integrated squared error calculation gives a bandwidth proportional to $n^{-1/5}$. That criterion averages error over the full loss axis. It does not give extra importance to a specified tail region merely because losses there are financially consequential.

Reference rule: For a Gaussian kernel and data that are reasonably close to a normal reference shape, a commonly used rule is $h=1.06s n^{-1/5}$, where $s$ is the sample standard deviation. A robust alternative often used as a scale-based starting point is $h=0.9\min(s,\mathrm{IQR}/1.34)n^{-1/5}$. These are reference rules, not universal optima. Their normal-reference calibration can be misleading for a strongly heavy-tailed or multimodal distribution.

The standard-deviation rule can react sharply to a few very large observations. That reaction can increase $h$ throughout the distribution, including around the body where most data lie. The robust alternative limits the influence of such observations on its scale estimate, but it does not prove that a smaller bandwidth is correct in the tail. A robust scale describes the central spread more stably; it does not create tail information.

Worked Example: Comparing Two Reference Bandwidths

A hypothetical insurer has $n=1000$ exact positive loss observations. Suppose their sample standard deviation is $s=\$20$, and their interquartile range is $\mathrm{IQR}=\$8.04$. Calculate both reference bandwidths and interpret the difference.

Method. Calculate $n^{-1/5}$, then apply each rule using the same loss units.

First, $1000^{1/5}\approx3.98107$, so $n^{-1/5}\approx0.251189$. The standard-deviation rule gives $$ h=1.06(20)(0.251189)=5.3252\text{ dollars}\approx \$5.33. $$

For the robust rule, $\mathrm{IQR}/1.34=8.04/1.34=6.00$. Thus $\min(s,\mathrm{IQR}/1.34)=\min(20,6)=6$, and $$ h=0.9(6)(0.251189)=1.35642\text{ dollars}\approx \$1.36. $$

Rounded to the nearest cent, the reference values are $\$5.33$ and $\$1.36$. The large gap indicates that a few losses may be affecting the standard deviation. It does not establish that $\$1.36$ is the right tail bandwidth: the robust rule may be too narrow for the tail. Tail-focused checks are still needed. Common mistake avoided: treating the robust rule as a guaranteed correction for heavy tails rather than as a more resistant starting point.

Measure How Much Local Data Support the Estimate

At a target loss $x$, observations close to $x$ contribute more to the estimate than distant observations. With a Gaussian kernel, the contribution is proportional to $\exp[-(x-X_i)^2/(2h^2)]$. A useful descriptive diagnostic is the effective local count: it summarizes whether the kernel contributions are spread across several observations or concentrated in one or two.

Definition: For nonnegative kernel weights $w_i=K((x-X_i)/h)$, define the effective local count at $x$ by $$ n_{\mathrm{eff}}(x;h)=\frac{\left(\sum_{i=1}^{n}w_i\right)^2}{\sum_{i=1}^{n}w_i^2}, $$ provided at least one weight is positive. It is a diagnostic of weight concentration, not a replacement for the sample size and not, by itself, a formal standard error.

To see why this diagnostic has the stated interpretation, normalize the contributions as $a_i=w_i/\sum_j w_j$. Then $\sum_i a_i=1$ and $n_{\mathrm{eff}}=1/\sum_i a_i^2$. If $m$ observations contribute equally, each $a_i=1/m$, so $n_{\mathrm{eff}}=1/(m(1/m)^2)=m$. If one observation dominates, its normalized weight is near one and the remaining weights are near zero, giving $n_{\mathrm{eff}}$ near one. Thus the value responds to concentration, not merely to how many observations are present in the full data set.

Worked Example: Checking Local Support Near a Large Loss

At target loss $x=\$100$, suppose the only three observations with appreciable Gaussian-kernel weights are $\$100,\$110,\$120$, and the candidate bandwidth is $h=\$10$. Calculate their effective local count and the resulting kernel density estimate using only these three observations.

Method. The standardized distances are $0,1,2$. Ignoring the common Gaussian normalizing constant, the weights are $1,e^{-1/2},e^{-2}$, or approximately $1,0.606531,0.135335$.

Their sum is $1+0.606531+0.135335=1.741866$. The sum of squared weights is $$ 1^2+0.606531^2+0.135335^2 \approx 1+0.367879+0.018316=1.386195. $$ Therefore, $$ n_{\mathrm{eff}}(100;10)\approx\frac{1.741866^2}{1.386195} \approx 2.189. $$

For the standard Gaussian kernel, the common constant is $1/\sqrt{2\pi}\approx0.398942$. Hence the estimate from these three observations is $$ \hat f_{10}(100)=\frac{1.741866}{3(10)\sqrt{2\pi}} \approx 0.023163\text{ per dollar}. $$

The estimate is based on three observations, but their unequal contributions correspond to an effective local count of about $2.189$. This is limited local support, so the estimate should be treated as uncertain. Common mistake avoided: claiming that a target near the tail has support from all $n$ observations just because the full data set has $n$ records. The diagnostic uses the observations’ contributions at the target, not the total sample size.

Use Validation That Looks at the Tail

A direct way to compare candidate bandwidths is to reserve validation observations and score the density estimates at those losses. For a held-out observation $X_i$, estimate its density using the training observations only. A log score penalizes a candidate that assigns very low density to a validation loss. To focus the comparison on important large losses, average the scores over validation observations above a preselected threshold $u$.

Tail validation score: Given a training set of size $m$, define its estimate $\hat f_{h,\mathrm{train}}$ by the usual kernel formula with $m$ in place of $n$. For validation losses $X_i\ge u$, a tail-focused negative log score is $$ -\frac{1}{r}\sum_{i:X_i\ge u}\log\{\hat f_{h,\mathrm{train}}(X_i)\}, $$ where $r$ is the number of qualifying validation observations. Lower scores are preferred among the bandwidths compared, provided all scores use the same data split, loss units, and threshold. The threshold should be specified independently of the validation outcomes, or estimated using training data only.

A tail score is noisy when $r$ is small. A single split can favor a bandwidth by chance, so in practice the comparison should use several validation splits or folds and report the variability of the scores as well as their averages. With a compact-support kernel, an estimate can be zero at a held-out loss, producing an infinite negative log score. That is a warning that the candidate bandwidth and kernel provide no local support there, not a reason to silently discard the observation.

Worked Example: Comparing Bandwidths With a Tail Validation Score

A hypothetical validation exercise uses the uniform kernel $K(t)=1/2$ for $|t|\le1$ and $K(t)=0$ otherwise. The training losses, in dollars, are $100,110,130,150$; the two held-out tail losses are $120$ and $140$. Compare $h=\$10$ and $h=\$20$ using the mean negative log score. The uniform kernel includes observations exactly at the bandwidth boundary.

Method. For each held-out value, count training observations within distance $h$. If the count is $c$, the density estimate is $c/(2mh)$, with $m=4$.

For $h=10$, the training losses within distance $10$ of $120$ are $110$ and $130$, so $c=2$ and the estimate is $2/[2(4)(10)]=0.025$. At $140$, the qualifying losses are $130$ and $150$, again giving $0.025$. The mean negative log score is therefore $$ -\frac{\log(0.025)+\log(0.025)}{2} =-\log(0.025)\approx3.688879. $$

For $h=20$, the losses within distance $20$ of $120$ are $100,110,130$, giving $3/[2(4)(20)]=0.01875$. At $140$, the qualifying losses are $130,150$, giving $2/[2(4)(20)]=0.0125$. Thus $$ -\frac{\log(0.01875)+\log(0.0125)}{2} \approx\frac{3.976562+4.382027}{2} =4.179295. $$

The lower score in this split favors $h=10$. The comparison is illustrative, not conclusive: there are only two validation losses, and the uniform kernel makes estimates change abruptly at its support boundary. Common mistake avoided: choosing the bandwidth with the smoother-looking graph without checking how it scores on held-out large losses.

Interpret Tail Diagnostics Carefully

Tail-focused diagnostics answer different questions. A local effective count indicates how concentrated the kernel contributions are at a target. A tail validation score compares predictive density assignments for held-out tail observations. A plot of several bandwidth estimates reveals whether a feature is stable or created by one particular smoothing choice. None is a substitute for the others, and none can manufacture information absent from the data.

It is also important to distinguish estimating a density within the observed range from extrapolating beyond it. A kernel estimate is constructed from observed losses and the chosen kernel. If the largest observed loss is $x_{\max}$, an estimate at $x>x_{\max}$ may reflect only the kernel’s mathematical tails, not observed evidence about how losses continue beyond $x_{\max}$. A smooth curve extending to the right is not, by itself, a fitted heavy-tail model or a justified extrapolation.

For positive losses, a transformation can sometimes make the shape easier to estimate, but the density on the original scale must include the appropriate change-of-variable factor. The choice of scale should be consistent when comparing bandwidths, and density values on different scales should not be compared as if they had the same units.

Define the tail region.
Specify the loss threshold or range relevant to the analysis before comparing candidate bandwidths.

Generate plausible candidates.
Use a reference rule as a starting point, then examine a range around it rather than treating one formula as decisive.

Check local support and validation.
Inspect effective local counts and compare tail-focused validation scores across candidates using the same splits.

Report sensitivity and limits.
Show whether tail conclusions change with $h$, and state when sparse observations do not support a stable estimate or extrapolation.

Common Exam Trap

A frequent error is to confuse a large overall sample with strong support at every loss level. For example, if a data set contains $1000$ losses but only three receive appreciable kernel weight at a particular extreme target, writing “the effective sample size is $1000$” is wrong. It confuses the total sample size with local weight concentration. In the Gaussian-weight calculation above, the effective local count is approximately $2.189$, not $1000$.

Another error is to assume that the robust reference rule necessarily solves the tail problem. It may reduce sensitivity to extreme values in the scale estimate, but it can also yield a bandwidth that is unsuitable for estimating a sparse tail. The correction is to treat reference rules as candidates, evaluate the tail region directly, and report the remaining uncertainty. This careful distinction is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Key takeaway: For heavy-tailed losses, no global bandwidth formula guarantees a reliable tail density. Use reference rules to generate candidates, then assess local support, tail-focused validation, and sensitivity to $h$. Do not interpret a smooth curve beyond the observed data as evidence of a known tail shape.

Check Your Understanding

Work each problem before checking its answer. Treat the stated observations as exact and the specified kernels and thresholds as fixed.

A hypothetical sample has $n=32$, $s=\$12$, and $\mathrm{IQR}/1.34=\$8$. Using the robust reference rule, calculate $h$. (Answer: $\$3.60$)
At a target loss, three kernel weights are $1,0.5,0.25$. Calculate the effective local count to three decimals. (Answer: $2.333$)
Using a uniform kernel and a training sample of size $m=5$, two training losses lie within bandwidth $h=\$10$ of a held-out loss. What density estimate is assigned to that held-out loss? (Answer: $0.0200$ per dollar)
A validation split has tail scores $2.4, 3.0$ for one bandwidth and $2.7, 2.8$ for another. Calculate each mean score and identify which is preferred by this split. (Answer: Means $2.70$ and $2.75$; the first bandwidth is preferred.)
For a positive loss $X$, let $Z=\log X$. If the estimated density of $Z$ at $\log(200)$ is $0.12$, what is the corresponding density estimate of $X$ at $200$? (Answer: $0.0006$ per dollar)

What You'll Learn