Conditional Densities Under Right Truncation

Why the Observation Rule Changes the Likelihood

Suppose a hypothetical insurer receives detailed records only for losses below a reporting threshold. The losses above that threshold are absent from the data; they are not recorded as equal to the threshold. This observation rule changes the distribution of the losses that appear in the sample. In particular, the observed losses are not an unselected sample from the original severity distribution.

The adjustment is essential in likelihood estimation. A loss that is observed has not only taken its recorded value; it has also met the condition for being observed. The likelihood must account for that condition. This tutorial treats a fixed, known upper threshold and an independent sample consisting of losses selected by falling below it.

Definition: Let $X$ be a loss with density $f_\theta$ and distribution function $F_\theta$, where $\theta$ is an unknown parameter. Right truncation at a known threshold $c$ means a loss is observed only when $X<c$. For $F_\theta(c)>0$, the density of an observed loss is the conditional density $$ f_{\theta,c}(x)=f_\theta(x\mid X<c) =\frac{f_\theta(x)}{F_\theta(c)},\qquad x<c. $$ It is zero outside the values that can be observed. For a continuous loss, using $X\le c$ instead makes no difference to the conditioning probability.

Deriving the Conditional Density

For a small interval of width $dx$ around a value $x<c$, the probability that a loss is in that interval and is eligible for observation is approximately $f_\theta(x)\,dx$. The probability that a loss is eligible at all is $P(X<c)=F_\theta(c)$. Dividing the first probability by the second gives the probability of the interval among eligible losses:

P(x\le X<x+dx\mid X<c) \approx \frac{f_\theta(x)\,dx}{F_\theta(c)}.

Dividing by $dx$ and taking the limit gives the conditional density. The denominator is also a useful check: integrating over all observable values gives

\int_{-\infty}^{c}\frac{f_\theta(x)}{F_\theta(c)}\,dx =\frac{F_\theta(c)}{F_\theta(c)}=1.

The corresponding conditional distribution function, for $x<c$, is

P(X\le x\mid X<c)=\frac{F_\theta(x)}{F_\theta(c)}.

These formulas assume that losses are observed exactly when they meet the stated threshold rule, that $c$ is known, and that the probability of inclusion is positive. They also assume the parameterized severity model describes the losses before selection. If records at or above $c$ are instead reported as values of $c$, that is a different observation mechanism: those records contain an inequality about the loss, rather than an exact loss below the threshold.

Worked Example: A Conditional Exponential Density

A hypothetical portfolio has loss $X$, measured in thousands of dollars, with an exponential distribution of rate $\lambda=0.5$ per thousand dollars. Only losses below $c=2$ thousand dollars are observed. Find the density of an observed loss at $x=0.8$, and the probability that an observed loss is at most 1 thousand dollars.

Method. Condition the original density and distribution function on inclusion, $X<2$. For an exponential loss, $f_\lambda(x)=\lambda e^{-\lambda x}$ and $F_\lambda(x)=1-e^{-\lambda x}$ for $x\ge0$.

First calculate the inclusion probability:

F_\lambda(2)=1-e^{-0.5(2)}=1-e^{-1}=1-0.367879=0.632121.

At $x=0.8$, the original density is $0.5e^{-0.5(0.8)}=0.5e^{-0.4}=0.335160$. Therefore the conditional density is

f_{\lambda,2}(0.8) =\frac{0.335160}{0.632121} =0.5302

per thousand dollars. For the conditional probability of a loss at most 1, calculate $F_\lambda(1)=1-e^{-0.5}=0.393469$, then divide by the inclusion probability:

P(X\le1\mid X<2) =\frac{0.393469}{0.632121} =0.6225.

Check. The requested event $X\le1$ is contained in the inclusion event $X<2$, and its conditional probability is between 0 and 1. The density at a point is not itself a probability; its value may exceed 1 for other parameter choices without violating probability rules.

Likelihood for a Right-Truncated Sample

Suppose $x_1,\ldots,x_n$ are independent exact losses observed under the same fixed threshold $c$, with every recorded loss below $c$. The sample is modeled as independent draws from the original severity distribution conditional on inclusion. Its likelihood is the product of the conditional densities:

Theorem: For a fixed common threshold $c$, the conditional likelihood for an independent right-truncated sample is $$ L_c(\theta) =\prod_{i=1}^{n}\frac{f_\theta(x_i)}{F_\theta(c)} =\frac{\prod_{i=1}^{n}f_\theta(x_i)}{[F_\theta(c)]^n}, \qquad x_i<c. $$ The log-likelihood is $$ \ell_c(\theta) =\sum_{i=1}^{n}\log f_\theta(x_i)-n\log F_\theta(c). $$

The term $-n\log F_\theta(c)$ is not generally a constant with respect to $\theta$. If $F_\theta(c)$ changes as $\theta$ changes, this term changes the likelihood and can change the maximum likelihood estimate. Only when the inclusion probability does not depend on the unknown parameter can the denominator be omitted without changing the maximizer.

This likelihood conditions on the number $n$ of observed losses. It does not include a separate count of losses that failed the reporting rule. If the number of screened losses or the number excluded is also part of the data, the sampling information is different and must be modeled as such.

Worked Example: Constructing the Conditional Likelihood

A hypothetical sample consists of exact losses $0.4$, $0.9$, and $1.6$, in thousands of dollars. They are observed only when below $c=2$. Assume the original losses are exponential with known rate $\lambda=0.5$ per thousand dollars. Write and evaluate the conditional likelihood for these observed losses.

Method. Use the product of the three exponential densities divided by the probability that each loss is included. The inclusion probability is the same for all three records.

The sum of the recorded losses is $0.4+0.9+1.6=2.9$. The numerator is

\prod_{i=1}^{3} f_\lambda(x_i) =(0.5)^3e^{-0.5(2.9)} =0.125e^{-1.45} =0.029321.

From the preceding calculation, $F_\lambda(2)=0.632121$, so the denominator is $(0.632121)^3=0.25258$. Thus

L_c(0.5) =\frac{0.029321}{0.25258} =0.1161.

Equivalently, the log-likelihood is $3\log(0.5)-1.45-3\log(0.632121)=-2.1534$, whose exponential is approximately $0.1161$. The likelihood is a product of density values and is not itself a probability.

Check. Every recorded value is below 2, as required. The conditional likelihood is larger than the unadjusted product of densities because conditioning divides by an inclusion probability below 1. Omitting that adjustment would use the wrong sampling model.

Estimating an Exponential Rate

For an exponential model with rate $\lambda>0$, threshold $c>0$, and observed values $x_i<c$, substitute $f_\lambda(x)=\lambda e^{-\lambda x}$ and $F_\lambda(c)=1-e^{-\lambda c}$ into the conditional log-likelihood:

\ell_c(\lambda) =n\log\lambda-\lambda\sum_{i=1}^{n}x_i -n\log(1-e^{-\lambda c}).

Differentiating with respect to $\lambda$ gives the score equation

\frac{d\ell_c}{d\lambda} =\frac{n}{\lambda}-\sum_{i=1}^{n}x_i -\frac{nc}{e^{\lambda c}-1}=0.

This equation usually requires numerical solution. It shows directly why using the complete-data exponential estimate $n/\sum x_i$ is generally wrong: the truncated likelihood contains an additional term involving the threshold. A numerical root must also be checked against the likelihood or the curvature, rather than accepted solely because a solver returned a value.

Worked Example: Exponential Rate from Truncated Losses

A hypothetical portfolio has exact observed losses $0.4$, $0.7$, and $1.0$, measured in thousands of dollars. Only losses below $c=2$ thousand dollars are recorded. Assume the original losses are independent exponential random variables with unknown rate $\lambda$ per thousand dollars. Estimate $\lambda$ by maximum likelihood.

Method. Maximize the conditional exponential likelihood. Here $n=3$ and $\sum x_i=0.4+0.7+1.0=2.1$, so the score equation becomes

\frac{3}{\lambda}-2.1-\frac{6}{e^{2\lambda}-1}=0,

or, after dividing by 3,

\frac{1}{\lambda}-\frac{2}{e^{2\lambda}-1}=0.7.

Solve this equation numerically for $\lambda>0$. At $\lambda=0.953$, the left side is approximately $1/0.953-2/(e^{1.906}-1)=1.0493-0.3493=0.7000$. The root is approximately $0.9531$, so the estimate to three significant figures is $0.953$ per thousand dollars.

For a likelihood check, at $\lambda=0.953$ the conditional log-likelihood is

3\log(0.953)-0.953(2.1) -3\log(1-e^{-1.906}) \approx -1.663.

The score decreases through zero at this root; equivalently, its derivative there is negative, so the root is a local maximum. For this truncated exponential model the score is strictly decreasing because its derivative is $-n\operatorname{Var}(X\mid X<c)$, which is negative for a non-degenerate distribution. Thus this root is the unique maximum. The estimate is larger than the unadjusted estimate $3/2.1=1.429$ only if the calculation says so; here it does not: the unadjusted estimate is $1.429$, and the truncation-adjusted estimate is $0.953$. The difference is a reminder to rely on the score equation, not an intuition about the direction of adjustment.

Check. Substitution gives an average observed loss of about $0.700$, matching the sample mean $2.1/3=0.700$. All three values satisfy the threshold condition. The estimate is expressed in the reciprocal of the loss unit, as required for an exponential rate.

Common Exam Traps and Practical Checks

Right truncation changes which losses enter the sample. It does not change the definition of the original severity distribution. For each record, the observed value contributes its original density in the numerator and the probability of satisfying the observation rule in the denominator. With a common threshold and independent observations, that denominator appears once per record.

Omitting the normalizer. The incorrect likelihood is $\prod_i f_\theta(x_i)$. The correction is $\prod_i f_\theta(x_i)/[F_\theta(c)]^n$, because the data are sampled conditional on falling below $c$.
Using a survival probability. For observations below $c$, inclusion has probability $F_\theta(c)$, not $S_\theta(c)=1-F_\theta(c)$. The latter concerns losses above the threshold.
Treating missing large losses as threshold-valued records. Under truncation, such losses do not appear as observations. Recording a value at $c$ to indicate that the loss exceeded $c$ describes censoring, not this conditional sample.
Dropping a parameter-dependent term. The term $-n\log F_\theta(c)$ may be omitted from an optimization only if it is constant in $\theta$. Check this before simplifying the likelihood.
Ignoring units. The threshold and all loss values must use the same units. An exponential rate has units reciprocal to those used for loss.

Key takeaway: With a fixed upper reporting threshold, an observed exact loss has conditional density $f_\theta(x)/F_\theta(c)$. For $n$ independent observed losses, the likelihood includes the factor $[F_\theta(c)]^{-n}$; retaining it is central to valid estimation under right truncation.

Check Your Understanding

For each problem, treat the recorded losses as independent exact observations selected by the stated threshold. Loss values and thresholds use the same units.

Write the conditional density for a loss with original density $f_\theta$ and distribution function $F_\theta$, observed only when $X<c$. (Answer: $f_\theta(x)/F_\theta(c)$ for $x<c$.)
An exponential loss has rate $0.5$, and only losses below 2 are observed. Find $P(X\le1\mid X<2)$, to four decimal places. (Answer: $0.6225$.)
Two exponential losses with known rate 1 are observed as $0.5$ and $1.5$, with threshold $c=3$. Give their conditional likelihood and its approximate value. (Answer: $e^{-2}/(1-e^{-3})^2\approx0.1499$.)
Five losses are observed below a fixed threshold $c$, where the model gives $F_\theta(c)=0.8$. What factor multiplies the product of the five original density values in the conditional likelihood? (Answer: $0.8^{-5}=3.0518$, approximately.)
Two exponential losses are observed as $0.5$ and $1.0$ below $c=2$. Write the conditional score equation and give the maximum likelihood estimate of the rate to three decimals. (Answer: $2/\lambda-1.5-4/(e^{2\lambda}-1)=0$; $\hat\lambda\approx0.780$.)

What You'll Learn