Inverse-Probability Weighting For Truncated Losses

Why Weight a Truncated Loss Sample?

A truncated loss sample contains only losses that satisfy a selection rule. If larger losses are more likely to be observed, the observed distribution over-represents large losses; if smaller losses are more likely to be observed, it over-represents small losses. Inverse-probability weighting corrects this imbalance when the selection mechanism is known and every part of the target population has a positive chance of appearing.

The key is to weight each observed loss in proportion to how unlikely it was to be included. A loss with a small inclusion probability receives a large weight because it stands in for more source-population losses. This adjustment does not create information about losses that could never have been observed: it requires a positive inclusion probability across the support of the target loss distribution.

Definition: Let $X$ be a loss in the target population, and let $R=1$ indicate that the loss is observed under the truncation mechanism. The inclusion probability at loss value $x$ is $\pi(x)=P(R=1\mid X=x)$. For an observed loss $X_i$, its inverse-probability weight is $w_i=1/\pi(X_i)$. The basic positivity condition is $\pi(x)>0$ wherever the target distribution $F_X$ has positive probability.

The target is the source-population distribution $F_X$, not the distribution conditional on being observed. A useful estimate of its cumulative distribution function is the normalized inverse-probability-weighted empirical distribution:

\widehat F_{\mathrm{IPW}}(x) = \frac{\sum_{i:R_i=1} \dfrac{1}{\pi(X_i)}\,\mathbf{1}(X_i\le x)} {\sum_{i:R_i=1} \dfrac{1}{\pi(X_i)}}.

Here $\mathbf{1}(X_i\le x)$ equals 1 when $X_i\le x$ and 0 otherwise. The weights can also estimate a target-population expectation by replacing the indicator in the numerator with a function $g(X_i)$. For example, set $g(X_i)=X_i$ to estimate the mean loss. The normalized estimate divides the weighted sum of $g(X_i)$ by the sum of weights.

Deriving the Weight From the Sampling Mechanism

The inclusion probability must come from the actual selection rule. For example, suppose an independent random threshold $L$ is applied and a loss is recorded only when $X>L$. Conditional on a particular loss value $x$, the loss is included exactly when $L<x$. Therefore,

\pi(x)=P(L<x).

This is not necessarily the same at every loss value. A loss well above most thresholds is likely to be observed and receives a modest weight; a loss just above the lower end of the loss range may be observed rarely and receives a larger weight. For right truncation, where the loss is observed only when $X<T$ for an independent random threshold $T$, the corresponding probability is $\pi(x)=P(T>x)$.

Derivation: In a sample of losses conditional on inclusion, the density or probability mass at $x$ is proportional to $\pi(x)$ times its source-population value. Thus, for any function $g$,

E\!\left[\frac{g(X)}{\pi(X)}\middle|R=1\right] = \frac{E[g(X)]}{P(R=1)}, \qquad E\!\left[\frac{1}{\pi(X)}\middle|R=1\right] = \frac{1}{P(R=1)}.

Dividing the first expression by the second gives $E[g(X)]$, the source-population expectation. This explains why the normalized weighted estimate targets the source distribution: inverse weighting cancels the factor $\pi(x)$ that made some losses more likely to enter the observed sample. In finite samples, the ratio is generally an estimate rather than an exactly unbiased quantity.

If a finite source population of known size $N$ is sampled with individual inclusion probabilities $\pi_i$, the Horvitz–Thompson estimate of its total loss is $\sum_{i:R_i=1}X_i/\pi_i$. Under the specified sampling design, this total is unbiased. Dividing by $N$ estimates the finite-population mean. The normalized weighted mean instead divides by the sum of observed weights; it is often useful when the population size is not known, but it is not generally exactly unbiased in a finite sample.

Worked Example: Reweighting a Left-Truncated Loss Sample

A hypothetical source population has four equally sized loss groups, with losses of $100, $200, $400, and $800. Losses are included with probabilities $0.50, 0.75, 0.50,$ and $0.25$, respectively. The observed sample has 10, 15, 10, and 5 losses in those groups. Estimate the source-population distribution and mean. This is a pure severity calculation: no interest or mortality assumption applies. The stated finite-support loss distribution and inclusion probabilities fully specify the example.

Method and weights. Use the normalized inverse-probability-weighted empirical distribution. The weights for the four groups are $1/0.50=2$, $1/0.75=4/3$, $1/0.50=2$, and $1/0.25=4$.

Weighted group masses. Multiplying each observed count by its weight gives $10(2)=20$, $15(4/3)=20$, $10(2)=20$, and $5(4)=20$. The total weighted mass is $20+20+20+20=80$, so the estimated probability of each loss value is $20/80=0.25$.

Estimated mean. The weighted loss total is $20(100)+20(200)+20(400)+20(800)=2{,}000+4{,}000+8{,}000+16{,}000=30{,}000$. The weighted mean is $30{,}000/80=\$375.00$.

Check and trap avoided. The unweighted observed mean is $(10(100)+15(200)+10(400)+5(800))/40=12{,}000/40=\$300.00$, which gives too much influence to the more frequently included $200 losses. The weighted probabilities sum to 1, and the corrected estimate is $375.00. The error to avoid is treating the observed sample proportions as source-population proportions.

Weights Must Match the Selection Rule

A probability such as $\pi(x)$ is not an arbitrary adjustment factor. It is determined by the probability that a loss of size $x$ would have entered the data under the actual sampling process. If truncation thresholds vary independently from claim to claim, the probability can be calculated from their distribution. If the threshold is fixed, losses on the excluded side have inclusion probability zero; no inverse weight can recover their contribution without additional assumptions or data.

Worked Example: Deriving Weights From a Random Threshold

A hypothetical portfolio has loss values $100, $200, and $400. An independent threshold $L$ equals $0 with probability 0.20, $150 with probability 0.50, and $350 with probability 0.30; a loss is recorded only if $X>L$. The observed records contain 2 losses of $100, 7 of $200, and 10 of $400. Estimate the source-population mean and the probability that a loss is at most $200. No interest or mortality assumption applies; the loss values and threshold distribution specify the model.

Step 1: Calculate inclusion probabilities. A $100 loss is included only when $L=0$, so $\pi(100)=0.20$. A $200 loss is included when $L=0$ or $L=150$, so $\pi(200)=0.20+0.50=0.70$. A $400 loss is included at all three thresholds, so $\pi(400)=1.00$.

Step 2: Calculate weights and weighted masses. The weights are $5$, $1/0.70=10/7$, and $1$. The weighted masses are $2(5)=10$, $7(10/7)=10$, and $10(1)=10$. Their sum is 30.

Step 3: Estimate the requested quantities. The weighted probability of a loss at most $200 is $(10+10)/30=2/3=0.6667$. The weighted loss total is $10(100)+10(200)+10(400)=1{,}000+2{,}000+4{,}000=7{,}000$; therefore, the weighted mean is $7{,}000/30=\$233.33$, rounded to the nearest cent.

Check and trap avoided. The estimated masses are equal and sum to 30, and the probability for losses at most $200 is $20/30$. A common error is to use the threshold probabilities themselves as the weights. The correct weights are their reciprocals after calculating the chance of inclusion at each loss value.

Stability, Positivity, and Interpretation

Inverse-probability weighting can be unstable when some inclusion probabilities are small. A small $\pi(X_i)$ creates a large $1/\pi(X_i)$, so one or a few observations can dominate a weighted mean or distribution estimate. This is a variance problem as well as a practical warning: the weights can magnify the influence of observations that are sparse in the selected sample.

A useful descriptive diagnostic is the effective sample size,

n_{\mathrm{eff}} = \frac{\left(\sum_{i:R_i=1}w_i\right)^2} {\sum_{i:R_i=1}w_i^2}.

This quantity is at most the number of observed records and is smaller when the weights are uneven. It is a diagnostic, not a substitute for a design-based variance calculation or a guarantee that an estimate is reliable. In particular, a large observed sample can have a small effective sample size if a few records carry most of the total weight.

The positivity condition is substantive. If $\pi(x)=0$ for losses in a region with positive source-population probability, the observed data contain no information about that region through weighting alone. If $\pi(x)$ is merely very small, an estimate is theoretically defined but may be highly variable. Possible responses include collecting more data in underrepresented regions, using a defensible model for the selection mechanism, or reporting a sensitivity analysis. Truncating or capping weights can reduce variability, but it changes the estimator and can introduce bias; it should not be presented as a cost-free correction.

Worked Example: Diagnosing an Unstable Weighted Mean

In a hypothetical sample, the observed losses are $100, $200, and $1,000, with counts 8, 4, and 1. Their inclusion probabilities are 0.80, 0.40, and 0.05. Estimate the weighted mean and effective sample size, and compare the result with the raw mean. No interest or mortality assumption applies; the three loss values, counts, and inclusion probabilities specify the empirical model.

Step 1: Calculate weights and weighted counts. The weights are $1.25$, $2.5$, and $20$. The weighted counts are $8(1.25)=10$, $4(2.5)=10$, and $1(20)=20$, for a total weight of 40.

Step 2: Calculate the weighted mean. The weighted loss total is $10(100)+10(200)+20(1{,}000)=1{,}000+2{,}000+20{,}000=23{,}000$. Thus, the weighted mean is $23{,}000/40=\$575.00$. The raw mean is $(8(100)+4(200)+1(1{,}000))/13=(800+800+1{,}000)/13=\$200.00$, rounded to the nearest cent.

Step 3: Calculate effective sample size. The sum of squared weights is $8(1.25^2)+4(2.5^2)+1(20^2)=8(1.5625)+4(6.25)+400=12.5+25+400=437.5$. Therefore, $n_{\mathrm{eff}}=40^2/437.5=1{,}600/437.5=3.6571$, to four decimal places.

Check and trap avoided. There are 13 observed records, but the effective sample size is only about 3.66 because the single $1,000 loss carries half the total weight. The large difference between the raw and weighted means is consistent with the stated selection mechanism, but it also signals sensitivity. Do not interpret the weighted estimate as equally precise as an ordinary mean based on 13 equally weighted observations.

Common Exam Traps

Weighting in the wrong direction. The weight is $1/\pi(x)$, not $\pi(x)$. More easily included losses must receive less weight, not more.
Using the wrong threshold event. For observation when $X>L$, calculate $P(L<x)$. For observation when $X<T$, calculate $P(T>x)$. The inequality comes from the stated selection rule.
Ignoring zero inclusion probabilities. A weight cannot be assigned when $\pi(x)=0$. The target distribution is not identifiable nonparametrically on that part of the support from these selected observations alone.
Confusing a weighted total with a weighted mean. A Horvitz–Thompson total and a normalized weighted mean have different denominators and interpretations.
Overlooking estimated weights. If the selection probabilities are estimated rather than known, their estimation uncertainty also affects the final result.

Key takeaway: Derive each inclusion probability from the sampling mechanism, then weight observed losses by its reciprocal. The adjustment targets the source population only under appropriate selection assumptions and positivity; very small probabilities can make estimates unstable.

Check Your Understanding

For each problem, use the stated selection mechanism and show the weighted masses before calculating the requested estimate.

A hypothetical source has losses of $50, $150, and $300. Their inclusion probabilities are 0.50, 0.25, and 1.00, and observed counts are 4, 6, and 4. Find the weighted mean and the estimated probability of a loss at most $150. (Answer: weighted mean $144.44; probability 0.8889)
An independent threshold $L$ equals $0, $100, or $300 with probabilities 0.20, 0.50, and 0.30. A loss is observed only when $X>L$. Observed counts are 2 at $80, 7 at $200, and 5 at $500. Find the inclusion probabilities and weighted mean. (Answer: inclusion probabilities 0.20, 0.70, and 1.00; weighted mean $212.00)
Losses of $100, $250, and $500 have inclusion probabilities 0.75, 0.50, and 0.20, with observed counts 6, 4, and 2. Find the estimated probability of a loss at most $250 and the weighted mean. (Answer: probability 0.6154; weighted mean $300.00)
A weighted sample has weights 1, 2, 10, and 10. Calculate its effective sample size and explain whether it equals the number of records. (Answer: effective sample size $529/205=2.5805$, not 4)
A fixed threshold records losses only when $X>500$. The target distribution assigns positive probability to losses below $500. Can inverse-probability weighting alone estimate the full source-population distribution? (Answer: No; losses at or below $500 have zero inclusion probability, so positivity fails.)

What You'll Learn