Why Estimation Errors Begin With the Observation Mechanism
A density estimate can be calculated correctly from the records in hand and still estimate the wrong distribution. The central diagnostic question is how did each loss become an observed record? If a record appears only when a loss exceeds a threshold, the observed values come from a conditional distribution. If a loss is known only to exceed a reporting time or threshold, it is censored rather than absent. Kernel estimates bring a separate concern: smoothing near a boundary and in a sparse tail can produce misleading shapes.
This tutorial focuses on identifying and correcting these errors. The methods rely on the distinction between truncation and censoring, the conditional-distribution adjustment for truncated samples, and kernel density estimation. Those distinctions matter before any estimate is computed.
Truncation: Do Not Omit the Conditioning Probability
Suppose losses follow a continuous distribution with density \(f_X\), and only losses exceeding a fixed threshold \(t\) are included. For an observed value \(x>t\), the relevant density is conditional:
The denominator is required because the observed sample is drawn from the part of the population that passes the inclusion rule. For \(n\) independently observed losses \(x_1,\ldots,x_n\), the likelihood under this fixed threshold is the product of these conditional densities. Omitting the denominator treats the selected observations as though they were an unselected sample.
The normalization follows directly from conditional probability. For any interval of possible values within the observed range, the conditional probability is the probability of falling in that interval and passing the threshold, divided by the probability of passing the threshold. In density form this gives \(f_X(x)/S_X(t)\). It is normalized because
Worked Example: Correcting an Exponential Estimate for Left Truncation
A hypothetical portfolio records only losses exceeding 2 units. Three observed losses are 3, 4, and 5 units. Assume the underlying losses are independent exponential variables with rate \(\lambda>0\), so \(f_X(x)=\lambda e^{-\lambda x}\) and \(S_X(t)=e^{-\lambda t}\). Estimate \(\lambda\) from the truncated observations.
Method. Use the conditional density for each observed loss. For \(x>2\), \(f_{X\mid X>2}(x)=\lambda e^{-\lambda x}/e^{-2\lambda}=\lambda e^{-\lambda(x-2)}\).
The three excesses above the threshold are \(3-2=1\), \(4-2=2\), and \(5-2=3\). Thus the likelihood is \(L(\lambda)=\lambda^3 e^{-\lambda(1+2+3)}=\lambda^3e^{-6\lambda}\). The log-likelihood is \(\ell(\lambda)=3\log\lambda-6\lambda\), and its derivative is \(3/\lambda-6\). Setting the derivative to zero gives \(\hat\lambda=3/6=0.5\) per unit.
Check. The second derivative is \(-3/\lambda^2<0\), so this stationary point is a maximum. The estimate is the number of records, 3, divided by their total excess above the threshold, 6. The common error is to use the untruncated likelihood, which would give \(3/(3+4+5)=0.25\) per unit. That calculation incorrectly treats the losses as an ordinary sample from the full exponential distribution.
Censoring: Keep the Information in the Incomplete Record
With right censoring, an exact loss \(x\) contributes its density \(f_X(x)\), while a record censored at \(c\) contributes the probability that the loss exceeds \(c\), namely \(S_X(c)\). Under independent, noninformative censoring, the censoring mechanism does not carry additional information about the loss model parameters. A censored record is not discarded: its observed duration contributes to the likelihood through the survival probability.
Worked Example: Including a Right-Censored Loss in an Exponential Estimate
A hypothetical study has two exact event times, 1.5 and 4 years, and one loss record right-censored at 6 years. Assume independent exponential loss times with rate \(\lambda\) per year and independent, noninformative censoring. Find the maximum likelihood estimate.
Method. Each exact event contributes \(f_X(x)=\lambda e^{-\lambda x}\); the censored record contributes \(S_X(6)=e^{-6\lambda}\). Therefore \(L(\lambda)=(\lambda e^{-1.5\lambda})(\lambda e^{-4\lambda})(e^{-6\lambda}) =\lambda^2e^{-11.5\lambda}\).
The log-likelihood is \(\ell(\lambda)=2\log\lambda-11.5\lambda\). Its derivative is \(2/\lambda-11.5\), so the estimate is \(\hat\lambda=2/11.5=0.173913\ldots\), or 0.1739 per year to four decimal places. The second derivative, \(-2/\lambda^2\), is negative, confirming a maximum.
Check and trap. Total observed exposure is \(1.5+4+6=11.5\) years and there are two exact events, so the estimate is two events divided by total exposure. If the censored record were dropped, the incorrect estimate would be \(2/(1.5+4)=0.3636\) per year. Treating the censored duration as though an event occurred at year 6 would also be wrong: it contributes survival through year 6, not an event density at 6.
Kernel Estimates Near a Boundary
For complete observations \(x_1,\ldots,x_n\), a kernel density estimate at \(x\) with bandwidth \(h>0\) is
where \(K\) is a density function, often the standard normal density, and \(h\) controls the degree of smoothing. For losses constrained to be nonnegative, an ordinary symmetric kernel places some estimated probability below zero when observations are near zero. This boundary leakage can depress the estimate near zero and distort its shape.
One simple correction at a zero boundary is reflection. For a symmetric kernel, add a reflected observation \(-x_i\) for each observed positive loss \(x_i\), or equivalently include both kernel contributions at \(x_i\) and \(-x_i\) when estimating for \(x\geq0\). The reflected estimate is
This correction is not a universal solution: it imposes a locally reflected shape at the boundary. It is a diagnostic and estimation choice that should be stated, rather than silently doubling or renormalizing contributions without explanation.
Worked Example: Quantifying Boundary Bias and Reflection
Suppose two hypothetical losses are 0.2 and 0.4 thousand dollars, and a Gaussian kernel is used with bandwidth \(h=0.2\) thousand dollars. Estimate the density at zero both with the ordinary estimator and with reflection. Report densities per thousand dollars to four decimal places.
For the standard normal kernel, \(\phi(z)=e^{-z^2/2}/\sqrt{2\pi}\), so \(\phi(1)=0.2419707\) and \(\phi(2)=0.0539910\). The standardized differences at zero are \(-1\) and \(-2\). By symmetry, their kernel values are \(\phi(1)\) and \(\phi(2)\).
The ordinary estimate is \(\{\phi(1)+\phi(2)\}/(2h) =(0.2419707+0.0539910)/0.4 =0.739904\ldots\), or 0.7399 per thousand dollars.
At zero, the reflected observations have the same kernel contributions as the original observations. The reflected estimate is therefore twice the ordinary estimate: \(2(0.739904\ldots)=1.479808\ldots\), or 1.4798 per thousand dollars.
Check and trap. Reflection raises the boundary estimate, as expected when the ordinary symmetric kernels leak mass below zero. Using the unrounded Gaussian values is important: rounding the kernel values first can change the final digit. The common error is to report a doubled estimate using prematurely rounded intermediate values.
Bandwidth and Tail Diagnostics
Bandwidth choice is a bias-variance trade-off. A small \(h\) preserves local detail but can produce a highly variable estimate; a large \(h\) smooths noise but can erase gaps, peaks, and tail structure. In a sparse tail, an estimate can look substantial merely because a wide kernel spreads the influence of observations far from the evaluation point. A small estimated density at one point is not evidence that the tail is absent, and a smoothed curve should not be extrapolated beyond the data without a model-based justification.
A practical diagnostic compares plausible bandwidths, checks the distance from the evaluation point to each observation in bandwidth units, and separately inspects boundary behavior. In incomplete-data settings, first establish what distribution the observations represent. Kernel smoothing does not repair selection bias from truncation or recover exact values hidden by censoring.
Worked Example: Checking Whether Bandwidth Hides a Sparse Region
Consider hypothetical losses, in thousands of dollars, equal to 1, 2, 3, 4, and 20. Use a Gaussian kernel to compare the estimate at \(x=10\) for \(h=1\) and \(h=2\). The purpose is to diagnose bandwidth sensitivity, not to select a universally optimal bandwidth.
For \(h=1\), the standardized differences \((10-x_i)/h\) are 9, 8, 7, 6, and \(-10\). The exponential factors in the Gaussian kernel are, respectively, \(e^{-40.5}=2.5768\times10^{-18}\), \(e^{-32}=1.2664\times10^{-14}\), \(e^{-24.5}=2.2897\times10^{-11}\), \(e^{-18}=1.5230\times10^{-8}\), and \(e^{-50}=1.9287\times10^{-22}\). Their sum is approximately \(1.52529\times10^{-8}\). Including the Gaussian normalizing constant and the estimator denominator gives \(\hat f_1(10)=1.52529\times10^{-8}/(5\sqrt{2\pi})\approx1.2169\times10^{-9}\) per thousand dollars.
For \(h=2\), the standardized differences are 4.5, 4, 3.5, 3, and \(-5\). The exponential factors sum to approximately \(e^{-10.125}+e^{-8}+e^{-6.125}+e^{-4.5}+e^{-12.5} =0.01367574\). Thus \(\hat f_2(10)=0.01367574/(5\cdot2\sqrt{2\pi}) \approx0.0005457\) per thousand dollars.
Check and trap. The estimate at 10 is far larger with \(h=2\), because the wider kernels spread more mass into the gap between 4 and 20. This does not establish that the underlying distribution has appreciable density there. The common error is to interpret a bandwidth-dependent smoothed value as direct evidence about the tail without checking the observations, the scale, and sensitivity to \(h\).
A Systematic Error-Checking Sequence
A short review before accepting an estimate can catch errors that arithmetic checks alone will not. The sequence below separates the data mechanism, the mathematical adjustment, and the diagnostic review.
Decide whether a record is exact, censored, or absent because it failed an inclusion rule. State the threshold and direction precisely.
Use a density for an exact value, a tail or interval probability for a censored record, and a conditional density for an observed truncated value.
Confirm that the conditional density integrates to one over the observable region and that bandwidth and density units match the loss scale.
For kernel methods, inspect boundaries, compare reasonable bandwidths, and examine how distant observations contribute in standardized bandwidth units.
Common Exam Traps
- Dropping a truncation denominator. The observed sample is conditional on passing the selection rule; use the corresponding survival or distribution probability to normalize.
- Calling censoring truncation. A censored record is present and contributes a probability statement. A truncated record is not observed at all unless it passes the inclusion condition.
- Counting a censored value as an event. A right-censored observation at \(c\) contributes \(S_X(c)\), not \(f_X(c)\).
- Ignoring support boundaries. For nonnegative losses, a symmetric kernel near zero leaks mass to impossible negative values; consider and state a boundary correction.
- Choosing bandwidth by appearance alone. A smooth curve can conceal sparse tails or gaps. Compare estimates across bandwidths and examine the actual data support.
- Rounding too early. Keep kernel values and likelihood calculations at adequate precision, then round the final requested result.
Check Your Understanding
For each problem, identify the relevant correction before calculating. Answers are rounded as stated.
- A hypothetical exponential loss sample is observed only above 3 units. The observed losses are 4, 6, and 7. Under an exponential model with rate \(\lambda\), find the maximum likelihood estimate using the truncated likelihood. (Answer: \(0.3750\) per unit.)
- Under an exponential model, exact event times are 2 and 5 years, and a third record is right-censored at 4 years. Assuming independent, noninformative censoring, find the rate estimate. (Answer: \(2/11=0.1818\) per year.)
- A Gaussian-kernel estimate uses losses 2, 3, and 4, with \(h=1\), all in thousands of dollars. Calculate the estimate at \(x=3\). (Answer: \(\{2\phi(1)+\phi(0)\}/3=0.2943\) per thousand dollars.)
- A Gaussian kernel with \(h=0.1\) is used for losses 0.1 and 0.3, in thousands of dollars. At \(x=0\), find the reflected estimate. (Answer: \(\{\phi(1)+\phi(3)\}/0.1=2.4640\) per thousand dollars.)
- For losses 1, 2, 3, 4, and 20, all in thousands, compare the Gaussian-kernel estimate at \(x=10\) using \(h=1\) and \(h=2\). Which bandwidth gives the larger estimate, and what is the diagnostic interpretation? (Answer: \(h=2\) gives the larger estimate, approximately \(0.0005457\) versus \(1.2169\times10^{-9}\) per thousand dollars; the difference shows strong bandwidth sensitivity in the gap, not proof of substantial underlying density there.)