Why a Truncated Sample Needs Risk Sets
Suppose a hypothetical study records a loss \(X\) only when it does not exceed an associated observation threshold \(T\). The observed pairs therefore satisfy \(X\leq T\). Losses above their thresholds are absent, so the distribution of observed losses need not match the distribution in the population. In particular, a loss value is observed only for records whose threshold was high enough to include it.
The Lynden-Bell estimator is a nonparametric product-limit estimator for this kind of randomly right-truncated data. Its key adjustment is a risk set: for each observed loss value, count how many observed records had an opportunity to show a loss at least that large, within their observable interval. The risk set, rather than the total sample size, accounts for the unequal observation opportunities.
Write the distinct observed losses in increasing order as \(z_1<\cdots<z_m\). The standard Lynden-Bell product-limit estimate is formed from the factors \(1-d_j/C_j\). The product runs from larger loss values down toward smaller ones. This direction matters: the risk at a given loss concerns intervals that reach that value, not simply the number of observations remaining in an ordinary, untruncated sample.
The Product-Limit Formula
These expressions use the observed support values \(z_1,\ldots,z_m\). The convention \(\widehat F(z_0)=0\) sets the estimated mass below the smallest observed loss to zero. In the usual setting, \(C_j\geq d_j\), since every record with \(X_i=z_j\) also has \(T_i\geq z_j\). Consequently, each factor is between zero and one, and the estimated masses are nonnegative.
The equations telescope: summing the estimated masses from \(z_1\) through \(z_j\) gives \(\widehat F(z_j)\). At the largest observed value, \(\widehat F(z_m)=1\). Thus, when the masses are calculated from these product-limit cumulative probabilities, they sum to one. This does not mean that every observed value must receive positive mass; a factor can be zero.
Why the Risk-Set Adjustment Works
For ordinary complete data, every observation has an opportunity to appear at every loss value. Under truncation, a record with a low threshold could not have revealed a loss above that threshold. At \(z_j\), \(C_j\) counts the records whose intervals actually include \(z_j\); \(d_j\) counts the observed losses at \(z_j\) among those records. The product-limit step uses \(d_j/C_j\), rather than \(d_j/n\), as the risk-set fraction at that value.
The product follows by relating the cumulative estimate immediately below a value to the estimate at that value. In the risk-set product-limit equations, the fraction of the cumulative probability assigned at \(z_j\) is \(d_j/C_j\). Therefore, the probability remaining immediately below \(z_j\) is the fraction \(1-d_j/C_j\) of the cumulative probability through \(z_j\):
Starting with \(\widehat F(z_m)=1\) and applying this relation successively down the ordered loss values gives \(\widehat F(z_j)=\prod_{k>j}(1-d_k/C_k)\). Subtracting the probability just below \(z_j\) from the probability through \(z_j\) gives \(\widehat p_j=(d_j/C_j)\widehat F(z_j)\), the stated mass formula. The adjustment is needed because records with small thresholds were not at risk of revealing large losses.
Worked Risk Sets and Masses
Worked Example: A Boundary-Limit Estimate
A hypothetical sample contains the observed pairs \((X,T)=(1,1),(2,3),(3,3)\). Calculate the risk sets, the Lynden-Bell cumulative probabilities and masses. Then assess the conditional likelihood at the resulting estimate.
Method. Order the observed losses \(z_1=1,z_2=2,z_3=3\), and count intervals containing each value. The intervals are \([1,1]\), \([2,3]\), and \([3,3]\).
At \(z_1=1\), only \([1,1]\) contains 1, so \(C_1=1\). At \(z_2=2\), only \([2,3]\) contains 2, so \(C_2=1\). At \(z_3=3\), \([2,3]\) and \([3,3]\) contain 3, so \(C_3=2\). Each value occurs once, giving \(d_1=d_2=d_3=1\). The factors are \(1-d_1/C_1=0\), \(1-d_2/C_2=0\), and \(1-d_3/C_3=1/2\).
Using the reverse product, \(\widehat F(1)=(0)(1/2)=0\), \(\widehat F(2)=1/2\), and \(\widehat F(3)=1\). The masses are therefore \(\widehat p_1=0-0=0\), \(\widehat p_2=1/2-0=1/2\), and \(\widehat p_3=1-1/2=1/2\). They sum to \(0+1/2+1/2=1\), as required.
For the conditional likelihood, the record \((1,1)\) contributes \(p_1/F(1)=1\) when \(p_1>0\). The other records have threshold 3, so together contribute \(p_2p_3\), since \(F(3)=1\). With \(p_1+p_2+p_3=1\), the supremum of \(p_2p_3\) is \(1/4\): it is approached as \(p_1\downarrow0\) and \(p_2,p_3\to1/2\). At \(p_1=0\), the first conditional probability is undefined because \(F(1)=0\). Thus the product-limit masses are the boundary limit, not an attained interior maximum-likelihood estimate. The common mistake is to count all three records in every risk set; the interval check shows why that would be wrong.
Worked Example: Unequal Opportunities Across Four Loss Values
A hypothetical portfolio study records \((X,T)=(2,5),(3,3),(4,6),(5,5)\). Calculate each risk set and the estimated mass at every observed loss. Also calculate the estimated mean of the observed-support distribution.
Method. The ordered values are \(2,3,4,5\), each with multiplicity one. At 2, only \([2,5]\) contains the value, so \(C_1=1\). At 3, \([2,5]\) and \([3,3]\) contain it, so \(C_2=2\). At 4, \([2,5]\) and \([4,6]\) contain it, so \(C_3=2\). At 5, \([2,5]\), \([4,6]\), and \([5,5]\) contain it, so \(C_4=3\).
The factors in increasing order are \(0,1/2,1/2,2/3\). Applying the reverse products gives \(\widehat F(2)=(1/2)(1/2)(2/3)=1/6\), \(\widehat F(3)=(1/2)(2/3)=1/3\), \(\widehat F(4)=2/3\), and \(\widehat F(5)=1\).
Taking successive differences gives \(\widehat p_2=1/6\), \(\widehat p_3=1/3-1/6=1/6\), \(\widehat p_4=2/3-1/3=1/3\), and \(\widehat p_5=1-2/3=1/3\). The check is \(1/6+1/6+1/3+1/3=1\). The estimated mean is \(2(1/6)+3(1/6)+4(1/3)+5(1/3)=1/3+1/2+4/3+5/3=23/6\), or \(3.8333\) to four decimal places. The risk-set adjustment avoids treating the four records as equally informative at every loss value.
Worked Example: Handling Tied Losses
A hypothetical sample has pairs \((1,4),(2,4),(2,5),(4,4)\). Calculate the Lynden-Bell estimate, grouping the two losses equal to 2.
Method. The distinct values are \(z_1=1,z_2=2,z_3=4\), with multiplicities \(d_1=1,d_2=2,d_3=1\). At 1, only \([1,4]\) contains the value, so \(C_1=1\). At 2, the first three intervals contain it, so \(C_2=3\). At 4, all four intervals contain it, so \(C_3=4\).
The factors are \(1-1/1=0\), \(1-2/3=1/3\), and \(1-1/4=3/4\). Thus \(\widehat F(1)=(1/3)(3/4)=1/4\), \(\widehat F(2)=3/4\), and \(\widehat F(4)=1\). The estimated masses are \(1/4\) at 1, \(3/4-1/4=1/2\) at 2, and \(1-3/4=1/4\) at 4. They total \(1/4+1/2+1/4=1\). Grouping ties is essential: use \(d_2=2\) in the risk-set fraction, rather than processing the two observations as separate distinct loss values.
Common Traps and Practical Checks
- Using the sample size as every risk set. A record belongs to \(C_j\) only if \(X_i\leq z_j\leq T_i\). Check the interval condition at each distinct value.
- Reversing the product. For the convention \(X\leq T\), \(\widehat F(z_j)\) uses factors for values larger than \(z_j\). Reversing the order can produce an incorrect cumulative distribution.
- Ignoring tied losses. Use the multiplicity \(d_j\) in \(d_j/C_j\), and compute one mass for each distinct loss value.
- Assuming every estimated mass is positive. A product-limit mass can be zero. In the first worked example, \(\widehat p_1=0\), and the associated conditional likelihood has a supremum at a boundary rather than an attained interior maximum.
- Applying the estimator under informative truncation without adjustment. If the threshold mechanism depends on the loss, independent random truncation may fail and the risk-set correction alone does not remove that bias.
The estimator is useful because it reconstructs a population distribution from observations that had different chances to appear. Its risk sets make that adjustment explicit, but the result is only as defensible as the truncation assumptions and the overlap in the data. These distinctions are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
Check Your Understanding
For each problem, calculate risk sets at distinct losses and use the reverse product-limit formula. Assume independent random truncation unless the question says otherwise.
- A hypothetical sample consists of \((1,2)\) and \((2,2)\). Find the risk sets and estimated masses. (Answer: \(C=(1,2)\); masses at 1 and 2 are \(1/2,1/2\).)
- A hypothetical sample consists of \((2,4),(3,5),(5,5)\). Find the risk sets and estimated masses. (Answer: \(C=(1,2,2)\); masses at 2, 3, and 5 are \(1/4,1/4,1/2\).)
- A hypothetical sample consists of \((1,3),(2,3),(2,4),(3,3)\). Group tied losses, then find the estimated masses. (Answer: \(d=(1,2,1)\), \(C=(1,3,4)\); masses at 1, 2, and 3 are \(1/4,1/2,1/4\).)
- For the sample in Question 2, calculate the estimated mean loss. (Answer: \(2(1/4)+3(1/4)+5(1/2)=15/4=3.75\).)
- A study observes losses only when \(X\leq T\), but its threshold \(T\) is deliberately set higher for risks already known to have large losses. Is the independent-truncation Lynden-Bell estimate justified without further modeling? (Answer: No; the threshold depends on the loss, so the independent random-truncation assumption fails.)