Regression with Top-Coded Outcomes

Why a Top-Coded Value Is Different

Suppose a continuous outcome, such as annual earnings, is reported only up to a known maximum. A reported value at that maximum does not necessarily mean the underlying outcome equals it: the actual value may be higher. The observation rule has replaced all such higher values with the cap. This changes the likelihood, because a capped report tells us that the outcome reached or exceeded a threshold, not its exact value.

The distinction matters in regression. A likelihood that uses the information in the observation rule can account for values above the cap without pretending to know them. A likelihood that instead treats each capped report as an exact measurement at the cap discards that distinction and specifies a different data model.

Latent Outcome and Observation Rule

Let \(Y_i^*\) be the continuous outcome that would be observed without the reporting cap, and let \(x_i\) be a vector of observed covariates for unit \(i\). Consider the normal regression model

Y_i^* = x_i'\beta + \epsilon_i, \qquad \epsilon_i \mid x_i \sim N(0,\sigma^2), \qquad \sigma>0.

Here, \(\beta\) is a vector of regression coefficients and \(\sigma\) is the error standard deviation. Assume that, conditional on the covariates, the errors are independent across units and have this common normal distribution. The cap \(C\) is known and applies to every unit. The observation rule is

Y_i = \min(Y_i^*,C).

Thus a report below \(C\) is the exact latent outcome, while a report equal to \(C\) indicates that the latent outcome is at least \(C\). Under the continuous model, the event \(Y_i^*=C\) has probability zero, so whether the threshold event is written \(Y_i^*\ge C\) or \(Y_i^*>C\) makes no difference to its probability.

Definition: A top-coded outcome is reported at a known upper threshold \(C\) whenever its underlying value reaches or exceeds that threshold. A report below \(C\) gives an exact value; a report at \(C\) gives the event that the underlying value is at least \(C\).

This is a censoring observation rule: units with outcomes above the threshold remain in the data, but their exact outcomes are not recorded. It is not a rule that removes those units from the sample.

Likelihood Contributions

Write \(\mu_i=x_i'\beta\), and let \(f(y\mid x_i)\) denote the conditional density of \(Y_i^*\). For an uncapped observation \(y_i<C\), the observed value is the latent value itself, so its likelihood contribution is the normal density evaluated at \(y_i\):

f(y_i\mid x_i) = \frac{1}{\sigma} \phi\!\left(\frac{y_i-\mu_i}{\sigma}\right), \qquad y_i<C,

where \(\phi\) is the standard normal density. For a top-coded observation, the data report only that \(Y_i^*\ge C\). Its contribution is therefore the conditional probability of that event:

P(Y_i^*\ge C\mid x_i) = 1-\Phi\!\left(\frac{C-\mu_i}{\sigma}\right) = \Phi\!\left(\frac{\mu_i-C}{\sigma}\right),

where \(\Phi\) is the standard normal cumulative distribution function. The second equality uses the symmetry of the standard normal distribution.

Key result: An uncapped observation contributes a density; a top-coded observation contributes a tail probability. The latter is the probability of all latent values at or above \(C\), not the density at \(C\).

Let \(D_i=1\) when unit \(i\) is top-coded and \(D_i=0\) otherwise. If \(y_i\) denotes the observed report, then the individual contribution can be written compactly as

L_i(\beta,\sigma) = \left[ \frac{1}{\sigma} \phi\!\left(\frac{y_i-x_i'\beta}{\sigma}\right) \right]^{1-D_i} \left[ 1-\Phi\!\left(\frac{C-x_i'\beta}{\sigma}\right) \right]^{D_i}.

The likelihood for an independent sample is the product of these contributions. Equivalently, its log is the sum of the log-density terms for uncapped observations and the log-tail-probability terms for top-coded observations:

\ell(\beta,\sigma) = \sum_{i:D_i=0} \left[ -\log(\sigma) + \log\phi\!\left(\frac{y_i-x_i'\beta}{\sigma}\right) \right] + \sum_{i:D_i=1} \log\!\left[ 1-\Phi\!\left(\frac{C-x_i'\beta}{\sigma}\right) \right].

Worked Contributions

Worked Example: One Uncapped and One Top-Coded Observation

Suppose \(C=10\), \(\sigma=2\), and the conditional mean for each of two hypothetical units is \(\mu_i=8\). One unit is reported at \(y=7\); the other is reported at \(y=10\). Find their likelihood contributions.

For the uncapped report \(y=7\), the standardized value is \((7-8)/2=-0.5\). Therefore,

L_{\text{uncapped}} = \frac{1}{2}\phi(-0.5) = \frac{1}{2}(0.3521) \approx 0.1760.

This is a density, with units inverse to the units in which the outcome is measured. For the report at the cap, the information is \(Y_i^*\ge10\), so

L_{\text{top-coded}} = 1-\Phi\!\left(\frac{10-8}{2}\right) = 1-\Phi(1) = 1-0.8413 \approx 0.1587.

This is a probability and is dimensionless. The second report does not establish that the latent outcome was exactly 10; it records the event that it was at least 10.

Why the Tail Probability Appears

The contribution for a top-coded observation follows directly from the observation rule. Conditional on \(x_i\), the latent outcome has density \(f(y\mid x_i)\). Every latent value at or above \(C\) produces the same recorded value \(Y_i=C\). The probability of that recorded event is the integral of the latent density over the values that produce it:

\begin{aligned} P(Y_i=C\mid x_i) &=P(Y_i^*\ge C\mid x_i)\\ &=\int_C^\infty \frac{1}{\sigma} \phi\!\left(\frac{y-\mu_i}{\sigma}\right)\,dy. \end{aligned}

Set \(z=(y-\mu_i)/\sigma\), so \(dy=\sigma\,dz\). The lower limit becomes \((C-\mu_i)/\sigma\), and the integral becomes

\int_{(C-\mu_i)/\sigma}^{\infty}\phi(z)\,dz = 1-\Phi\!\left(\frac{C-\mu_i}{\sigma}\right).

This calculation also shows why a density evaluated at \(C\) is not the right contribution. The top-coded report corresponds to an entire range of latent values, so its probability is obtained by integrating over that range.

Worked Examples: Comparing the Correct and Incorrect Contributions

Worked Example: A Capped Report Is Not an Exact Measurement

Use \(C=10\), \(\mu_i=8\), and \(\sigma=2\). For a report at 10, compare the correct contribution with the contribution from incorrectly treating 10 as an exact latent outcome.

The correct contribution is the upper-tail probability:

P(Y_i^*\ge10\mid x_i) = 1-\Phi(1) \approx 0.1587.

Under the incorrect exact-value treatment, the contribution would instead be the density at 10:

f(10\mid x_i) = \frac{1}{2}\phi\!\left(\frac{10-8}{2}\right) = \frac{1}{2}\phi(1) = \frac{1}{2}(0.2420) \approx 0.1210.

These numbers are not two estimates of the same kind of quantity: \(0.1587\) is a probability, while \(0.1210\) is a density per outcome unit. The key error is not simply that the numbers differ. It is that the density describes an exact latent value, whereas the report at 10 describes the event that the latent value reaches or exceeds 10.

The difference also affects how the likelihood responds to the conditional mean. For the correct top-coded term, the derivative of its log contribution with respect to \(\mu_i\) is

\frac{\partial}{\partial\mu_i} \log\!\left[ 1-\Phi\!\left(\frac{C-\mu_i}{\sigma}\right) \right] = \frac{\phi((C-\mu_i)/\sigma)} {\sigma[1-\Phi((C-\mu_i)/\sigma)]}.

At the stated values this is \(\phi(1)/(2[1-\Phi(1)])=0.7626\) per outcome unit. The incorrect exact-value density has log-derivative \((C-\mu_i)/\sigma^2=0.5000\) per outcome unit. Thus even the way a capped observation pulls on the fitted mean differs between the two specifications.

Worked Example: Building a Sample Likelihood

Consider four hypothetical independent observations with cap \(C=10\) and \(\sigma=2\). Their covariates give conditional means \(7,9,8,\) and \(11\). The first two reports are uncapped at \(7\) and \(9\); the last two reports are both top-coded at 10. Evaluate the likelihood at these parameter values.

For the first observation, \(y_1=\mu_1=7\), so its contribution is \[ \frac{1}{2}\phi(0)=0.1995. \] For the second, \(y_2=\mu_2=9\), giving the same contribution, \(0.1995\).

For the third observation, \(\mu_3=8\) and the report is top-coded. Its contribution is \[ 1-\Phi\!\left(\frac{10-8}{2}\right) = 1-\Phi(1) \approx 0.1587. \] For the fourth, \(\mu_4=11\), so the chance of reaching the cap is \[ 1-\Phi\!\left(\frac{10-11}{2}\right) = 1-\Phi(-0.5) = \Phi(0.5) \approx 0.6915. \]

By independence, multiply the four contributions:

L(\beta,\sigma) = (0.1995)(0.1995)(0.1587)(0.6915) \approx 0.00437.

Equivalently, the log likelihood is approximately \[ 2\log(0.1995)+\log(0.1587)+\log(0.6915) \approx -5.434. \] The two uncapped observations contribute densities, and the two top-coded observations contribute probabilities. This mixed form reflects exactly what was recorded for each unit.

Estimation and Interpretation

To estimate the model by maximum likelihood, choose \(\beta\) and \(\sigma>0\) to maximize the log likelihood. The covariates affect both the density terms for uncapped observations and the probabilities of exceeding \(C\) for top-coded observations. In particular, a top-coded report still provides information: a parameter value that makes exceeding the cap very unlikely receives a small contribution for that report.

The model describes the latent conditional outcome \(Y_i^*\) and the process that converts it into the reported outcome \(Y_i\). A top-coded observation does not reveal how far above the cap the latent outcome lies. Its likelihood contribution therefore uses only the event information supplied by the observation rule; it does not assign an invented exact value above the threshold.

The normal regression assumptions are substantive. The conditional distribution of the latent outcome must be adequately represented by the specified normal model with common variance, and the cap and reporting rule must be correctly known. Conditional independence is also used to multiply individual contributions. If the cap varies by unit, the likelihood must use the relevant known cap for each unit rather than a single common \(C\).

Common Mistake

Wrong reasoning: “The data file records 10, so the outcome was 10; use the normal density at 10 just as for any other observation.”

Correction: The recorded value is determined by the reporting rule. A value of 10 means the latent outcome was at least 10, not that it equalled 10. Use \(P(Y_i^*\ge C\mid x_i)\) for a top-coded report and the density only for an uncapped exact observation.

A related pitfall is to compare a tail probability and a density as though both were probabilities of the same event. A density is not the probability of observing one exact continuous value; probabilities for continuous outcomes are obtained by integrating a density over a range. For top-coding, the relevant range is the full upper tail.

Key takeaway: With a known upper cap, an uncapped value contributes its conditional density and a report at the cap contributes the probability of exceeding the cap. Treating the cap as an exact observation uses the wrong likelihood contribution.

Check Your Understanding

Use the observation rule and likelihood contributions to answer the following.

If \(C=12\) and a unit is reported at 12, what event about its latent outcome is observed?
For a normal latent regression with conditional mean \(\mu_i\) and standard deviation \(\sigma\), write the likelihood contribution of a report \(y_i<C\).
Write the likelihood contribution of a top-coded report and explain why it is not the density evaluated at \(C\).
With \(C=10\), \(\mu_i=10\), and \(\sigma=2\), express the top-coded contribution using \(\Phi\). What is its approximate value?
Why is a top-coded observation still informative about the regression parameters, even though its exact latent value is unknown?