Worked Analysis of a Limited-Outcome Dataset

From an Observation Rule to an Analysis

A dataset with many zeros does not, by itself, tell us why the zeros occurred. They may be recorded values created by censoring, or genuine outcomes generated by a process with a separate zero state. The observation rule matters: it determines the likelihood and the interpretation of the fitted model. This worked analysis follows a hypothetical dataset from its rule to estimation, fit checks, and a comparison with a two-part model.

Suppose an outcome \(Y\), measured in hundreds of dollars of monthly spending, cannot be negative. For every person in the sample, the recorded value is \(Y=\max(0,Y^*)\), where \(Y^*\) is a latent, potentially negative outcome. Everyone is observed, including people whose recorded spending is zero. This is left censoring at zero, not left truncation: with truncation, units whose latent outcomes fall below the threshold would be missing from the sample. The distinction changes the likelihood.

Model and observation rule: The left-censored Tobit model assumes \(Y_i^*=x_i'\beta+\varepsilon_i\), with \(\varepsilon_i\sim N(0,\sigma^2)\), \(\sigma>0\), and \(Y_i=\max(0,Y_i^*)\). The covariates \(x_i\) and the threshold are observed, the stated censoring rule applies to every unit, and the errors are independent across observations conditional on the covariates. A recorded zero means \(Y_i^*\leq 0\); a recorded positive value equals \(Y_i^*\).

The Likelihood for Censored Observations

Write \(\mu_i=x_i'\beta\). A zero does not reveal the value of \(Y_i^*\); it reveals only the event \(Y_i^*\leq 0\). Its likelihood contribution is therefore a probability. A positive observation reveals the latent outcome, so its contribution is a density. Multiplying these contributions gives the likelihood for the observed sample.

L(\beta,\sigma)= \prod_{i:Y_i=0}\Phi\left(-\frac{x_i'\beta}{\sigma}\right) \prod_{i:Y_i>0}\frac{1}{\sigma} \phi\left(\frac{Y_i-x_i'\beta}{\sigma}\right).

Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. The positive-observation term is a density, not a probability at one exact value. For computation, it is usual to maximize the log-likelihood, the sum of the logarithms of these contributions. The model parameters are estimated jointly: changing \(\beta\) or \(\sigma\) changes both the probability of a zero and the distribution of positive outcomes.

Likelihood principle for this rule: include a cumulative probability contribution for each observed zero and a normal density contribution for each positive observation. Do not drop the zeros as if they were unobserved, and do not treat them as exact observations of the latent \(Y_i^*\).

Worked Example: Two Individual Likelihood Contributions

Consider a hypothetical fitted parameter vector with \(\beta_0=1.9\), \(\beta_1=3.2\), and \(\sigma=5.2\), where \(\mu_i=\beta_0+\beta_1x_i\) and \(x_i\) is zero or one. Calculate the likelihood contributions for a zero at \(x=0\) and a positive observation \(Y=4\) at \(x=1\).

For the zero, \(\mu=1.9\), so its contribution is

P(Y=0\mid x=0)=P(Y^*\leq 0\mid x=0) =\Phi\left(-\frac{1.9}{5.2}\right) =\Phi(-0.3654)\approx 0.3574.

For the positive observation, \(\mu=1.9+3.2=5.1\). Its density contribution is

\frac{1}{5.2}\phi\left(\frac{4-5.1}{5.2}\right) =\frac{1}{5.2}\phi(-0.2115) \approx \frac{0.3901}{5.2} \approx 0.07502.

These contributions are different kinds of quantities: the zero contribution is a probability, while the positive contribution is a density per hundred dollars of monthly spending. If these were the only two observations, their joint likelihood contribution would be approximately \(0.3574\times0.07502=0.02681\), using the displayed rounded values.

A Hypothetical Dataset and Its Fitted Model

Now consider a constructed sample of 40 people, with \(x=0\) for 20 people and \(x=1\) for 20 people. The covariate indicates participation in a hypothetical program; the outcome is monthly spending in hundreds of dollars. The observation rule is known: every person is in the dataset, and negative latent spending is recorded as zero. Thus the zeros are censored observations, not people excluded by a sampling rule.

There are 7 zeros among the 20 people with \(x=0\), and 3 among the 20 people with \(x=1\). The 13 positive outcomes in the first group sum to 62, giving a positive-outcome mean of \(62/13=4.769\). The 17 positive outcomes in the second group sum to 109, giving a mean of \(109/17=6.412\). For this hypothetical dataset, suppose maximizing the censored-normal likelihood gives the rounded estimates \(\hat{\beta}_0=1.9\), \(\hat{\beta}_1=3.2\), and \(\hat{\sigma}=5.2\). They are illustrative rounded fit results; actual maximization uses all individual observations, not just the group summaries shown here.

The fitted latent mean is 1.9 hundreds of dollars per month when \(x=0\), and 5.1 when \(x=1\). The coefficient \(\hat{\beta}_1=3.2\) is the fitted difference in the conditional mean of the latent outcome \(Y^*\) for a one-unit increase in \(x\). It is not automatically the difference in the mean of the recorded, censored outcome \(Y\), because censoring changes the observed mean.

Worked Example: Boundary Probabilities and Expected Zero Counts

Use the fitted estimates to predict the probability of a recorded zero in each group and compare the predictions with the observed counts.

For \(x=0\), the fitted latent mean is 1.9, so

\widehat{P}(Y=0\mid x=0) =\Phi\left(-\frac{1.9}{5.2}\right) \approx 0.3574.

The predicted number of zeros among 20 people is \(20\Phi(-1.9/5.2)\approx7.148\), compared with 7 observed. For \(x=1\), the fitted latent mean is \(1.9+3.2=5.1\), giving

\widehat{P}(Y=0\mid x=1) =\Phi\left(-\frac{5.1}{5.2}\right) =\Phi(-0.9808) \approx 0.1634.

The predicted number of zeros in this group is \(20(0.1634)=3.268\), compared with 3 observed. The model therefore predicts fewer zeros in the \(x=1\) group, consistent with the observed group proportions of \(7/20=0.35\) and \(3/20=0.15\). This comparison is a useful diagnostic, but two close counts alone cannot establish that the entire model fits well.

Predicting the Recorded Outcome

For this model, a positive observation has the distribution of \(Y^*\) conditional on \(Y^*>0\), which is a normal distribution truncated below at zero. Its conditional mean is

E(Y\mid Y>0,x) =\mu+\sigma\frac{\phi(\mu/\sigma)}{\Phi(\mu/\sigma)}, \qquad \mu=x'\beta.

The mean of the recorded outcome, including zeros, instead follows by taking the expectation of \(\max(0,Y^*)\):

E(Y\mid x)=\mu\Phi(\mu/\sigma)+\sigma\phi(\mu/\sigma).

These formulas answer different questions. The first describes the average among people with positive recorded outcomes; the second averages over everyone, including those at zero. In particular, the latent index coefficient is not itself the effect on either of these observed-outcome means.

Justification of the recorded mean: Since \(Y=Y^*\) when \(Y^*>0\) and \(Y=0\) otherwise, \(E(Y\mid x)=E[Y^*1(Y^*>0)\mid x]\). Write \(Y^*=\mu+\sigma Z\), where \(Z\sim N(0,1)\). Then \(E(Y\mid x)=\mu P(Z>-\mu/\sigma)+\sigma E[Z1(Z>-\mu/\sigma)]\). The first probability is \(\Phi(\mu/\sigma)\); integrating \(z\phi(z)\) from \(-\mu/\sigma\) to infinity gives \(\phi(\mu/\sigma)\). Substitution yields the stated formula.

Worked Example: Comparing Predicted Outcome Means

Calculate the fitted mean among positive outcomes and the fitted mean including zeros for both covariate values. Use the estimates \(\hat{\beta}_0=1.9\), \(\hat{\beta}_1=3.2\), and \(\hat{\sigma}=5.2\).

For \(x=0\), \(\mu=1.9\), \(\mu/\sigma=0.3654\), \(\phi(0.3654)\approx0.3732\), and \(\Phi(0.3654)\approx0.6426\). The fitted mean among positive outcomes is

1.9+5.2\frac{0.3732}{0.6426} \approx 4.920

hundreds of dollars per month. The fitted mean across everyone is

1.9\Phi(1.9/5.2)+5.2\phi(1.9/5.2) \approx 3.161

hundreds of dollars per month.

For \(x=1\), \(\mu=5.1\), \(\mu/\sigma=0.9808\), \(\phi(0.9808)\approx0.2466\), and \(\Phi(0.9808)\approx0.8366\). Therefore the positive-outcome mean is

5.1+5.2\frac{0.2466}{0.8366} \approx 6.633

hundreds of dollars per month, while the mean including zeros is

5.1(0.8366)+5.2(0.2466) \approx 5.549

hundreds of dollars per month. The fitted difference in the recorded-outcome means is about \(5.549-3.161=2.388\) hundreds of dollars per month. It differs from the latent-index coefficient of 3.2 because the observed means also reflect the probability of being at the boundary.

Fit Checks and a Two-Part Comparison

A useful check compares the model’s predictions with distinct features of the data, rather than looking only at one summary. Here, the fitted zero probabilities imply expected zero counts of 7.148 and 3.268, close to the observed 7 and 3. The fitted positive-outcome means, about 4.920 and 6.633 hundreds of dollars per month, are also near the observed positive means of 4.769 and 6.412. These checks are reassuring for these summaries, but they do not test every implication of the normal latent-error assumption. A fuller diagnostic examines the distribution of positive outcomes and whether the fitted model systematically misses patterns across covariates.

A two-part alternative models two features separately: the probability that the recorded outcome is positive, and the distribution or mean of the outcome conditional on being positive. It does not require the same latent normal process to generate both features. In the simple group-specific empirical version for this dataset, the estimated positive probabilities are \(13/20=0.65\) and \(17/20=0.85\), and the estimated positive-outcome means are \(62/13=4.769\) and \(109/17=6.412\) hundreds of dollars per month.

Worked Example: Comparing Two-Part and Tobit Predictions

For each group, multiply the two-part estimate of the probability of a positive outcome by the estimated mean among positive outcomes. This gives the two-part estimate of the mean recorded outcome.

For \(x=0\), the two-part mean prediction is

\frac{13}{20}\frac{62}{13}=\frac{62}{20}=3.100

hundreds of dollars per month. For \(x=1\), it is

\frac{17}{20}\frac{109}{17}=\frac{109}{20}=5.450

hundreds of dollars per month. The difference is \(5.450-3.100=2.350\) hundreds of dollars per month, compared with the Tobit predicted difference of about 2.388. The predictions are similar in this example, but similarity does not prove that the Tobit restrictions hold. The models also differ in what they assume about the relation between the boundary probability and the positive-outcome distribution.

The comparison is informative because it separates model choice from the arithmetic of prediction. A Tobit model imposes a common latent index and error distribution that link the probability of a zero to the positive outcomes. A two-part model allows the two parts to vary more independently. Whether that flexibility is warranted depends on the observation mechanism, substantive assumptions, fit evidence, and the inferential question. A comparison of predicted means alone does not establish causation or settle the choice of model.

Common Mistakes

Calling every zero a censored value. A pile of zeros is not proof of censoring. Establish the observation rule. If zeros are actual outcomes generated by a distinct process, a two-part model may be more appropriate.
Using a density contribution for a censored zero. A zero records the event \(Y^*\leq0\), so its contribution is \(\Phi(-x_i'\beta/\sigma)\), not a normal density evaluated at zero.
Confusing censoring with truncation. Under censoring, the unit remains in the sample and the boundary value is recorded. Under truncation, units on one side of a threshold are absent from the observed sample, requiring a likelihood conditional on inclusion.
Interpreting \(\beta_1\) as the change in the observed mean. It is the latent-index change in this specification. Calculate the desired observed-outcome prediction from the model before interpreting its change.
Treating a close fit on zero counts as proof of adequacy. Check positive outcomes and other model implications as well; a few matching summaries do not verify the full distributional assumptions.

Key takeaway: Start with the observation rule. For left censoring, zeros contribute probabilities and positive values contribute densities to the likelihood. Estimate the model jointly, distinguish latent from observed-outcome effects, and compare its predictions with both the boundary mass and positive outcomes. A two-part model provides a useful alternative when the shared Tobit mechanism is not credible.

Check Your Understanding

Try these questions using the observation rule and calculations in the tutorial.

Why does an observed zero contribute a probability to the left-censored Tobit likelihood rather than a density?
For a fitted latent mean \(\mu=2\) and standard deviation \(\sigma=4\), write the model probability of a recorded zero.
In the worked dataset, what does the Tobit coefficient \(\hat{\beta}_1=3.2\) describe, and why is it not the fitted difference in recorded-outcome means?
What is the observed rule that distinguishes left censoring from left truncation?
Give one reason why similar Tobit and two-part mean predictions do not, by themselves, show that the Tobit restrictions are appropriate.