The EM Algorithm for Censored Severity Likelihoods

Why Use EM for Censored Severity Data?

A right-censored loss is known only to exceed a threshold. For example, if a hypothetical policy pays at most a stated limit, the insurer may observe that the underlying loss exceeded the limit without observing its full amount. The likelihood can be maximized directly for simple models, but the expectation-maximization (EM) algorithm provides a systematic way to handle such incomplete observations: it alternates between estimating the missing losses conditionally and maximizing as if those losses had been observed.

This tutorial develops the method for independent losses from an exponential severity distribution with unknown rate $\lambda>0$. The censoring thresholds are treated as known, and the censoring mechanism is assumed independent of the loss process in the sense required for the censoring contributions to be ignorable. The examples use hypothetical losses measured in thousands of dollars; the rate is therefore per thousand dollars.

Definition: Let $X_1,\ldots,X_n$ be independent exponential losses with density $f_X(x)=\lambda e^{-\lambda x}$ for $x\geq 0$, where $\lambda>0$. For an exact observation, record its value $x_i$. For a right-censored observation, record its threshold $c_i$, which tells us only that $X_i>c_i$. Write $m$ for the number of exact observations, $r=n-m$ for the number of censored observations, and $T=\sum_{\text{exact }i}x_i+\sum_{\text{censored }j}c_j$.

The Observed-Data Likelihood

An exact exponential observation contributes its density, $\lambda e^{-\lambda x_i}$. A right-censored observation contributes the probability that the loss exceeds its threshold, $S_X(c_j)=e^{-\lambda c_j}$. Thus, apart from factors that do not depend on $\lambda$, the likelihood and log-likelihood are

L_{\mathrm{obs}}(\lambda)\propto \lambda^m e^{-\lambda T}, \qquad \ell_{\mathrm{obs}}(\lambda)=m\log\lambda-\lambda T+\text{constant}.

If $m>0$ and $T>0$, differentiating the log-likelihood gives $m/\lambda-T=0$; its second derivative is $-m/\lambda^2<0$. Therefore the unique maximum is

\widehat{\lambda}_{\mathrm{direct}}=\frac{m}{T}.

This direct result is useful both as a check on the EM iteration and as a reminder that a censored observation contributes its threshold to $T$, not an unobserved full loss. If the censoring mechanism depends on the loss in a way that is not ignorable, this likelihood need not be valid; the mechanism may need to be modeled as part of the likelihood.

Deriving the EM Update

For the EM construction, temporarily regard every loss as fully observed. If the complete values were $X_1,\ldots,X_n$, the complete-data log-likelihood, omitting constants, would be

\ell_{\mathrm{complete}}(\lambda) =n\log\lambda-\lambda\sum_{i=1}^n X_i.

The exact losses are already known. For a censored loss, the E-step replaces its unknown value by its conditional expectation given that it exceeded the observed threshold, using the current rate $\lambda^{(k)}$. For the exponential model, the memoryless property gives

E[X\mid X>c]=c+\frac{1}{\lambda}.

One way to justify this identity is to write $X=c+(X-c)$ conditional on $X>c$. For $y\geq 0$,

P(X-c>y\mid X>c) =\frac{P(X>c+y)}{P(X>c)} =\frac{e^{-\lambda(c+y)}}{e^{-\lambda c}} =e^{-\lambda y}.

Thus $X-c$, given $X>c$, has the same exponential distribution with mean $1/\lambda$, proving the conditional-mean formula. In the E-step, the rate in that formula is the current value $\lambda^{(k)}$, not the unknown rate being optimized in the M-step.

Summing the exact values and the conditional means for the censored values gives an expected complete-data total of $T+r/\lambda^{(k)}$. The expected complete-data log-likelihood, as a function of a candidate rate $\lambda$, is therefore

Q(\lambda\mid\lambda^{(k)}) =n\log\lambda-\lambda\left(T+\frac{r}{\lambda^{(k)}}\right)+\text{constant}.

Differentiating with respect to $\lambda$ and setting the derivative to zero yields the EM update:

Theorem: For independent exponential losses with ignorable right censoring, $n$ total observations, $m$ exact observations, $r=n-m$ censored observations, and $T$ equal to the sum of exact values and censoring thresholds, the EM rate update is $$ \lambda^{(k+1)}=\frac{n}{T+r/\lambda^{(k)}}. $$ This update applies for a positive starting value $\lambda^{(0)}$. When $m>0$ and $T>0$, its fixed point is the direct maximum likelihood estimate $m/T$.

The fixed-point claim follows by setting $\lambda^{(k+1)}=\lambda^{(k)}=\lambda^*$. Then $\lambda^*T+r=n$, so $\lambda^*T=n-r=m$, and hence $\lambda^*=m/T$. This is a check on the algebra, not a claim that EM reaches the maximum in one iteration. The conditional expectation is recalculated at each iteration, and convergence is generally approached over successive updates.

Worked Iterations and Likelihood Comparison

Worked Example: One Censored Loss and a Corrected First Update

A hypothetical portfolio has three exact losses of 2, 3, and 5 thousand dollars and one loss right-censored at 4 thousand dollars. Starting from $\lambda^{(0)}=0.5$ per thousand dollars, find the direct estimate and the first three EM updates. Compare their direction with the direct estimate.

Method and sufficient total. There are $n=4$ observations, $m=3$ exact observations, and $r=1$ censored observation. The observed exact-loss sum is $2+3+5=10$, so $T=10+4=14$ thousand dollars.

Direct maximization. The observed-data maximum likelihood estimate is $m/T=3/14=0.2142857$ per thousand dollars. The censored loss contributes its threshold of 4 to $T$; it is not treated as an exact loss of 4.

First EM update. At the initial rate, the censored loss has conditional expected value $4+1/0.5=6$. The expected complete-data total is $2+3+5+6=16$, so $\lambda^{(1)}=4/16=0.25$ per thousand dollars. Equivalently, the denominator is $T+r/\lambda^{(0)}=14+1/0.5=16$.

Second update. At $\lambda^{(1)}=0.25$, the conditional expected censored loss is $4+1/0.25=8$. The expected total is $2+3+5+8=18$, giving $\lambda^{(2)}=4/18=2/9=0.2222222$.

Third update. At $\lambda^{(2)}=2/9$, the conditional expected censored loss is $4+9/2=8.5$. The expected total is $10+8.5=18.5$, giving $\lambda^{(3)}=4/18.5=8/37=0.2162162$.

The iterates $0.25,\ 0.2222222,\ 0.2162162$ approach $3/14=0.2142857$. The common mistake this example avoids is using the corrected first update $4/(14+1/0.5)=4/16=0.25$; omitting the arithmetic in the censored-loss expectation can produce an incorrect update.

Worked Example: Varying Censoring Thresholds and Direct Maximization

Five hypothetical losses are observed in thousands of dollars. Three are exact, with values 1, 2.5, and 4; the other two are right-censored at thresholds 1 and 3. Starting at $\lambda^{(0)}=0.3$, calculate two EM updates, the direct estimate, and the observed log-likelihood at the start, after one update, and at the direct estimate. Omit constants that do not depend on $\lambda$.

Set up the data totals. Here $n=5$, $m=3$, and $r=2$. The exact sum is $1+2.5+4=7.5$; the sum of censoring thresholds is $1+3=4$. Thus $T=7.5+4=11.5$, and the direct estimate is $3/11.5=6/23=0.2608696$ per thousand dollars.

First update. At $\lambda^{(0)}=0.3$, the conditional mean losses for the censored observations are $1+1/0.3=4.3333333$ and $3+1/0.3=6.3333333$. The expected complete-data total is $7.5+4.3333333+6.3333333=18.1666667$. Therefore $\lambda^{(1)}=5/18.1666667=30/109=0.2752294$ per thousand dollars.

Second update. Since $1/\lambda^{(1)}=109/30$, the expected complete-data total is $11.5+2(109/30)=11.5+7.2666667=18.7666667$. Thus $\lambda^{(2)}=5/18.7666667=150/563=0.2664298$ per thousand dollars.

Likelihood comparison. The observed log-likelihood, up to an additive constant, is $3\log\lambda-11.5\lambda$. At the initial value it is $3\log(0.3)-11.5(0.3)=-7.0619$. After one update it is $3\log(30/109)-11.5(30/109)=-7.0356$. At the direct estimate it is $3\log(6/23)-11.5(6/23)=-7.0312$. The values increase toward the direct maximum, as expected. Values are rounded to four decimal places here.

The method handles each threshold separately in the conditional expected loss, but the rate update uses their aggregate through $T$ and $r$. Treating both censored losses as if they were censored at the same threshold would change $T$ and would not represent these data.

Worked Example: All Observations Censored and a Boundary Supremum

Three hypothetical losses are right-censored at 1, 2, and 4 thousand dollars, with no exact observations. Determine whether a finite positive maximum likelihood estimate exists, and calculate two EM updates from $\lambda^{(0)}=0.5$ per thousand dollars.

Observed likelihood. Here $n=r=3$, $m=0$, and $T=1+2+4=7$. The likelihood is proportional to $e^{-7\lambda}$, which decreases as $\lambda$ increases over $\lambda>0$. Its supremum occurs as $\lambda$ approaches zero, but zero is not in the parameter space $\lambda>0$. There is no finite positive MLE.

First EM update. The denominator is $T+r/\lambda^{(0)}=7+3/0.5=13$, so $\lambda^{(1)}=3/13=0.2307692$ per thousand dollars.

Second EM update. Since $3/\lambda^{(1)}=13$, the next denominator is $7+13=20$, and $\lambda^{(2)}=3/20=0.15$ per thousand dollars. The sequence decreases toward the boundary value zero; it does not converge to a positive finite rate estimate.

This example illustrates why a numerical iteration should not be interpreted without checking the likelihood’s shape and the parameter space. The observed data establish lower bounds on the losses but contain no exact event to support a positive rate estimate.

Convergence, Checks, and Exam Traps

For $m>0$ and $T>0$, the direct estimate provides a benchmark for the EM sequence. Its update can also be written as

\lambda^{(k+1)} =\frac{n\lambda^{(k)}}{T\lambda^{(k)}+r}.

This expression is useful for checking the direction of an update. If the current rate is above $m/T$, then the next update is smaller; if it is below $m/T$, the next update is larger. For example, subtracting $\lambda^{(k)}$ from the update gives a numerator proportional to $m-T\lambda^{(k)}$, which is positive below $m/T$ and negative above it. For $r>0$, the update is increasing as a function of the current positive rate, and the iterates move toward the fixed point without crossing it. If $r=0$, there is no missing data and the update gives $n/T=m/T$ immediately.

EM is valuable as a general incomplete-data procedure, but it is not necessarily the most efficient way to compute this particular estimate: direct maximization is available in closed form. In more complex models, an EM iteration can simplify a difficult optimization, but it may converge slowly near the maximum. A practical stopping rule can monitor the change in the rate or in the observed log-likelihood, while still checking that the result is consistent with the likelihood and parameter constraints.

Do not treat a censoring threshold as an exact loss. The observed-data likelihood uses the survival probability at that threshold.
Do not add the current conditional mean twice. The censored loss contributes $c+1/\lambda^{(k)}$; equivalently, $T$ already includes $c$, so the update adds only $1/\lambda^{(k)}$ per censored observation to $T$.
Use the current rate in the E-step. The expectation is computed with $\lambda^{(k)}$, while $\lambda^{(k+1)}$ is the value that maximizes the resulting expected complete-data log-likelihood.
Check for boundary behavior. With no exact observations, $m=0$, and the likelihood has no positive finite maximizer for this model.
Keep units consistent. If losses are measured in thousands of dollars, both $T$ and $1/\lambda$ are in thousands of dollars, and $\lambda$ is per thousand dollars.

These calculations are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus. The key habits are to write each observation’s likelihood contribution correctly, distinguish a threshold from an unobserved loss, and verify any iterative result against the observed-data likelihood when direct maximization is available.

Check Your Understanding

For each problem, identify the exact and censored contributions before applying an update. Rates are per thousand dollars unless stated otherwise.

A hypothetical sample has exact losses 2 and 6, and one loss censored at 3. Find $T$, the direct rate estimate, and the first EM update from $\lambda^{(0)}=0.5$. (Answer: $T=11$, $\widehat{\lambda}_{\mathrm{direct}}=2/11=0.1818$, and $\lambda^{(1)}=3/(11+2)=3/13=0.2308$.)
There are two exact losses, 1 and 3, and two censored losses at 2 and 5. Find the direct rate estimate. (Answer: $T=1+3+2+5=11$, so $\widehat{\lambda}_{\mathrm{direct}}=2/11=0.1818$.)
A sample contains four observations: one exact loss of 3 and three censoring thresholds of 1, 2, and 4. Find the first EM update from $\lambda^{(0)}=0.25$. (Answer: $T=10$, $r=3$, and $\lambda^{(1)}=4/(10+3/0.25)=4/22=0.1818$.)
For $n=5$, $m=2$, $r=3$, and $T=12$, calculate the EM update from $\lambda^{(k)}=0.25$, and compare it with the direct estimate. (Answer: $\lambda^{(k+1)}=5/(12+3/0.25)=5/24=0.2083$, while $\widehat{\lambda}_{\mathrm{direct}}=2/12=0.1667$; the update is above the direct estimate and moves downward.)
Four losses are all right-censored, with thresholds 1, 2, 3, and 4. State whether a positive finite MLE exists and find the first EM update from $\lambda^{(0)}=0.5$. (Answer: No positive finite MLE exists; the likelihood supremum is at the boundary $\lambda\to0$. Here $T=10$, $r=4$, and $\lambda^{(1)}=4/(10+4/0.5)=4/18=0.2222$.)

What You'll Learn