Why Conditioning Errors Change the Likelihood
Incomplete data do not all require the same adjustment. With truncation, some observations are absent because they did not meet a selection rule; the likelihood for an included observation is therefore conditional on meeting that rule. With censoring, an observation is present but its exact value is not known; its likelihood contribution is the probability of the event consistent with what was observed. Confusing these mechanisms can introduce an incorrect denominator, an extra density factor, or a conditioning event that the study never imposed.
A reliable diagnostic is to describe the observation process before writing a likelihood: Was an observation selected because its loss crossed a threshold? Was everyone enrolled, with some losses only known to exceed a censoring time? Were losses below the threshold recorded as well? The answers determine which probabilities belong in the likelihood.
These statements presume the specified loss model is appropriate and, for the usual censoring likelihood, that censoring is independent of the loss (or is otherwise ignorable under the observation design). If the censoring mechanism depends on the loss and that dependence is not modeled, the ordinary contribution \(S_X(c)\) need not describe the observed data correctly. The examples below use continuous exponential losses, so \(S_X(t)=e^{-\lambda t}\) and \(f_X(t)=\lambda e^{-\lambda t}\), for \(\lambda>0\).
A Short Derivation: The Truncation Denominator
Suppose the sampling rule retains a loss only when \(X>a\). Once an observation is in the sample, its exact value is known to satisfy \(X>a\). The conditional probability of finding it in a small interval around \(x>a\), divided by the interval width, is the unconditional probability in that interval divided by the probability of being included. Taking the interval width toward zero gives the conditional density.
The denominator is the probability of the event that caused selection. It is not automatically the probability of the observed value, nor the probability of a different event such as exceeding a censoring time. For a sample of independent, identically distributed observations selected by the same rule, this conditional density applies to each included exact observation.
Match Each Likelihood Factor to the Observation
For a continuous loss \(X\), an exact value contributes a density. A right-censored record at \(c\) contributes the probability that the loss is at least \(c\), because every loss in that tail is consistent with the record. A truncated exact value contributes a conditional density, because its appearance in the data already tells us that it passed the selection rule.
| What the design says | What is observed | Contribution for a continuous \(X\) |
|---|---|---|
| Everyone in the cohort is observed | Exact loss \(x\) | \(f_X(x)\) |
| Everyone in the cohort is observed | Only known that \(X\ge c\) | \(S_X(c)\) |
| Only losses with \(X>a\) enter the sample | Exact included loss \(x>a\) | \(f_X(x)/S_X(a)\) |
| Only losses with \(X>a\) enter the sample | Included loss is censored at \(c>a\) | \(S_X(c)/S_X(a)\) |
In the last row, inclusion establishes \(X>a\), while censoring establishes \(X\ge c\). Since \(c>a\), the observation means \(X\ge c\) conditional on \(X>a\), so its contribution is \(P(X\ge c\mid X>a)\). This is not the same as conditioning on \(X\ge c\) and then using a density at \(c\).
Worked Examples
Worked Example: Using the Selection Threshold, Not the Loss Itself
A hypothetical study includes only claims exceeding \(a=\$2{,}000\). Treat claim amounts as continuous and suppose \(X\sim\operatorname{Exponential}(\lambda)\), with rate \(\lambda\) per thousand dollars. Three included claims are \(\$3{,}000\), \(\$4{,}000\), and \(\$6{,}000\). Find the maximum likelihood estimate of \(\lambda\), conditional on the sample size being fixed.
Method and model. The selection event is \(X>2\), with amounts measured in thousands of dollars. For an exponential model, \(f_X(x)=\lambda e^{-\lambda x}\) and \(S_X(2)=e^{-2\lambda}\). Each exact included claim contributes \(f_X(x)/S_X(2)\).
The three observed amounts in thousands are \(3,4,6\). Their conditional density contributions are \( \lambda e^{-3\lambda}/e^{-2\lambda}=\lambda e^{-\lambda}\), \( \lambda e^{-4\lambda}/e^{-2\lambda}=\lambda e^{-2\lambda}\), and \( \lambda e^{-6\lambda}/e^{-2\lambda}=\lambda e^{-4\lambda}\).
Setting the score equal to zero gives \(3/\lambda-7=0\), so \(\widehat{\lambda}=3/7=0.428571\ldots\) per thousand dollars. The second derivative is \(\ell''(\lambda)=-3/\lambda^2<0\), confirming a maximum. The estimate is based on the three excess amounts \(3-2=1\), \(4-2=2\), and \(6-2=4\): their sum is \(7\), and the estimate is \(3/7\).
Common mistake avoided. Using the unconditioned densities would give \(L(\lambda)=\lambda^3e^{-13\lambda}\) and the incorrect estimate \(3/13\). That calculation treats the claims as though they could have entered the sample regardless of size. The stated selection rule requires conditioning on \(X>2\).
Worked Example: A Censored Record Contributes a Tail Probability
In a hypothetical cohort, claim amounts follow \(X\sim\operatorname{Exponential}(\lambda)\), with \(\lambda\) per thousand dollars. All claims are followed from the start, and independent administrative censoring gives two exact amounts, \(\$1{,}000\) and \(\$2{,}000\), and two amounts known only to exceed \(\$3{,}000\) and \(\$4{,}000\). Estimate \(\lambda\).
Method and model. There is no loss-based selection into this cohort, so exact amounts contribute \(f_X(x)\). The censored claims contribute \(S_X(3)\) and \(S_X(4)\). In thousands of dollars, the two exact contributions are \(\lambda e^{-\lambda}\) and \(\lambda e^{-2\lambda}\); the censored contributions are \(e^{-3\lambda}\) and \(e^{-4\lambda}\).
The log likelihood is \(\ell(\lambda)=2\log\lambda-10\lambda\), with score \(2/\lambda-10\). Thus \(\widehat{\lambda}=2/10=0.2\) per thousand dollars. Since \(\ell''(\lambda)=-2/\lambda^2<0\), this is the maximum. The denominator \(10\) is the total observed amount of exposure in this exponential likelihood: \(1+2+3+4=10\), counting each censored record through its censoring threshold.
As a check, at \(\widehat{\lambda}=0.2\), the likelihood is \(0.2^2e^{-10(0.2)}=0.04e^{-2}\). The two censored contributions are probabilities of exceeding their respective thresholds, not probabilities of taking exactly those values.
Common mistake avoided. Replacing both censored tail probabilities with densities at 3 and 4 adds two factors of \(\lambda\). The resulting incorrect likelihood is \(\lambda^4e^{-10\lambda}\), whose score gives the incorrect estimate \(4/10=0.4\). A censored value is not an exact observation at its censoring threshold.
Worked Example: Do Not Condition a Full Cohort on Selection
A hypothetical study observes all 10 claims in a cohort and records only whether each amount exceeds \(\$5{,}000\). Three claims exceed that threshold and seven do not. Suppose claim amounts are independent with \(X\sim\operatorname{Exponential}(\lambda)\), where \(\lambda\) is per thousand dollars. Estimate \(\lambda\) from these indicators.
Method and model. The full cohort is observed; the study did not sample only claims exceeding 5. Each indicator is a Bernoulli observation with exceedance probability \(p=P(X>5)=e^{-5\lambda}\). A claim that does not exceed 5 contributes \(1-p=1-e^{-5\lambda}\). Therefore the likelihood is \(L(\lambda)=p^3(1-p)^7\).
To maximize this likelihood, first maximize it over \(p\). Its log likelihood is \(3\log p+7\log(1-p)\), with derivative \(3/p-7/(1-p)\). Setting the derivative to zero gives \(3(1-p)=7p\), so \(\widehat p=3/10=0.3\). Since \(p=e^{-5\lambda}\),
Thus the estimate is \(0.2408\) per thousand dollars, rounded to four decimal places. The result is consistent with the fitted exceedance probability: \(e^{-5(0.240794\ldots)}=0.3\).
Common mistake avoided. Dividing the three exceedance contributions by \(S_X(5)\) treats those records as though the study had selected only claims above 5. That is not this design: all 10 claims are observed, including the seven that do not exceed 5. The correct likelihood includes both the three exceedance indicators and the seven non-exceedance indicators. If a study instead sampled only claims above 5 and then recorded their exact amounts, a conditional density \(f_X(x)/S_X(5)\) would be appropriate for those selected amounts; it would answer a different sampling problem.
A Practical Conditioning Check
Before differentiating an incomplete-data likelihood, label each factor by the event it represents. For an exact, left-truncated observation, the denominator is the probability of inclusion. For a right-censored observation, the numerator is the probability of all values still compatible with the record. When selection and censoring both occur, keep their roles separate: under left truncation at \(a\) and right censoring at \(c>a\), the censored contribution is \(S_X(c)/S_X(a)\). The numerator describes censoring; the denominator describes selection.
Identify who could enter the data and whether inclusion required a loss event such as \(X>a\).
An exact value, a threshold exceedance, and an interval each specify different observed information.
Use a conditional density for selected exact losses and an event probability for a censored record.
Do not divide by an inclusion probability unless the observation was selected by that event.
A frequent exam trap is to condition on a convenient threshold simply because it appears in the data. A threshold may mark censoring, truncation, or only a recorded category. Only a sampling rule that excludes observations on one side of the threshold justifies conditioning the likelihood on inclusion. Conversely, when the cohort is fully observed, the presence of non-exceedances is information and should not be discarded by pretending that the sample was truncated.
A second useful check is dimensional and structural. In an exponential likelihood, an exact observation contributes one factor of \(\lambda\), while a right-censored observation contributes none. If the number of factors of \(\lambda\) equals the number of records rather than the number of exact observations, inspect whether a density has mistakenly been assigned to a censored record.
Check Your Understanding
For each problem, identify the observation mechanism before calculating. Amounts are measured in thousands of dollars where stated.
- A hypothetical study selects only exponential losses above 3. The three observed losses are 4, 7, and 9. With rate \(\lambda\) per thousand dollars and sample size fixed, find the conditional maximum likelihood estimate of \(\lambda\). (Answer: \(3/11=0.2727\) per thousand dollars.)
- In a fully observed cohort with exponential losses, rate \(\lambda\) per year, two event times are exact at 0.5 and 1 year, and one independent right-censoring time is 2 years. Find the maximum likelihood estimate. (Answer: \(2/(0.5+1+2)=4/7\) per year.)
- A loss is known to exceed 5, having entered a sample truncated at 2. For an exponential model with rate \(\lambda=0.25\) per thousand dollars, find its conditional probability contribution. (Answer: \(S_X(5)/S_X(2)=e^{-0.75}=0.4724\).)
- All 20 claims in a hypothetical cohort are observed as indicators of whether they exceed 4; eight exceed. Assume exponential amounts and estimate the rate \(\lambda\) per thousand dollars. (Answer: \(-\log(8/20)/4=0.2291\) per thousand dollars.)
- With no truncation, a hypothetical exponential cohort has exact losses at 1 and 2 and an independent right-censored loss at 4. Write the likelihood and find the maximum likelihood estimate of \(\lambda\). (Answer: \(L(\lambda)=\lambda^2e^{-7\lambda}\), so \(\widehat{\lambda}=2/7\) per unit of loss.)