Posterior Classification Uncertainty

Why Class Assignment Is Uncertain

A latent class joint model represents each individual as belonging to one unobserved class, while allowing the observed longitudinal and event-time records to be compatible with more than one class. Even after fitting the model, an individual’s class is not directly observed. The model instead provides probabilities of class membership conditional on that individual’s data and the fitted parameters. These probabilities make uncertainty visible; replacing them immediately with one assigned label can hide it.

The probabilities are conditional on the model and the observed information used to calculate them. They are not statements that the latent classes are objectively known categories, and they do not capture all uncertainty about estimated model parameters. In the plug-in calculations below, the fitted parameter estimate is held fixed.

Definition: Let $C_i$ be the latent class for individual $i$, with $K$ possible classes. Let $O_i$ denote that individual’s observed record used for classification, and let $\hat{\theta}$ denote the fitted model parameters. The posterior class probability is $$ \tau_{ig}=P(C_i=g\mid O_i;\hat{\theta}),\qquad g=1,\ldots,K. $$ For each individual, $0\leq \tau_{ig}\leq 1$ and $\sum_{g=1}^{K}\tau_{ig}=1$. Under the fitted mixture model, it has the form $$ \tau_{ig}= \frac{\hat{\pi}_{ig} f_g(O_i;\hat{\theta})} {\sum_{h=1}^{K}\hat{\pi}_{ih} f_h(O_i;\hat{\theta})}, $$ where $\hat{\pi}_{ig}$ is the fitted class-membership probability before using the class-specific contribution of $O_i$, and $f_g$ is that contribution under class $g$.

The record $O_i$ must match the model’s intended conditioning information. In a joint model, it may include the observed longitudinal measurements and the event or censoring information, along with relevant covariates as specified by the model. A posterior calculated from only part of the record answers a different conditional question. The formula presumes that the fitted model supplies the relevant membership probabilities and class-specific contributions and that these have been calculated for the observed records using the model as specified.

Individual Uncertainty: Maximum Probability and Entropy

Two related summaries answer different questions. The largest posterior probability indicates how strongly the most probable class is favored over the other classes. Entropy summarizes the spread of the entire probability vector: it is low when probability is concentrated in one class and high when it is spread across several classes.

Definition: The maximum posterior probability for individual $i$ is $m_i=\max_g\tau_{ig}$. A maximum-posterior (or modal) assignment is $\hat{C}_i=\arg\max_g\tau_{ig}$; ties require an explicit tie rule. The posterior entropy, using natural logarithms, is $$ H_i=-\sum_{g=1}^{K}\tau_{ig}\log(\tau_{ig}), $$ with $0\log(0)=0$. The normalized entropy is $H_i/\log(K)$ when $K>1$. Entropy is measured in nats; normalized entropy ranges from 0 to 1.

Maximum probability and entropy are complementary, not interchangeable. The maximum uses just the largest entry, whereas entropy uses every entry. For example, with more than two classes, the same maximum probability can accompany different distributions of the remaining probability and hence different entropy. Entropy is also not the probability that a modal assignment is wrong: under the fitted model, the posterior probability of error for a modal assignment is $1-m_i$.

Worked Example: Comparing Two Individuals’ Uncertainty

Suppose a fitted three-class model gives one hypothetical individual posterior probabilities $(0.80,0.15,0.05)$, and another $(0.45,0.40,0.15)$. Calculate entropy, normalized entropy, and the posterior probability of error for each modal assignment.

For the first individual, class 1 is modal and $m_i=0.80$. The posterior probability of error under that assignment is $1-0.80=0.20$. Entropy is $$ H_i=-[0.80\log(0.80)+0.15\log(0.15)+0.05\log(0.05)] =0.1785+0.2846+0.1498=0.6129\text{ nats}. $$ Since $\log(3)=1.0986$, normalized entropy is $0.6129/1.0986=0.5579$.

For the second individual, class 1 is again modal, but $m_i=0.45$, so the posterior probability of error for the modal assignment is $1-0.45=0.55$. Its entropy is $$ H_i=-[0.45\log(0.45)+0.40\log(0.40)+0.15\log(0.15)] =0.3593+0.3665+0.2846=1.0104\text{ nats}. $$ Normalized entropy is $1.0104/1.0986=0.9197$. Thus the second individual’s probabilities are much more diffuse. These are hypothetical, model-conditional calculations, not externally verified error rates.

Entropy’s Range and Its Interpretation

Entropy is nonnegative because each term $-\tau_{ig}\log(\tau_{ig})$ is nonnegative for a probability between 0 and 1. It is zero when one class has probability 1 and all others have probability 0. Its largest value is $\log(K)$, attained when every class has probability $1/K$. The following argument establishes the upper bound.

Compare the probabilities with a uniform distribution.
Let $q_g=\tau_{ig}$ and $u_g=1/K$. For any positive $q_g$, the inequality $\log(x)\leq x-1$, applied to $x=u_g/q_g$, gives $-q_g\log(q_g/u_g)\leq u_g-q_g$. Summing yields $-\sum_g q_g\log(q_g/u_g)\leq 0$, or $\sum_g q_g\log(q_g/u_g)\geq 0$. Zero entries follow by continuity.

Rearrange the inequality.
Because $u_g=1/K$, $\sum_g q_g\log(q_g/u_g)=\sum_g q_g\log(q_g)+\log(K)=\log(K)-H_i$. Therefore $H_i\leq\log(K)$. Equality holds for the uniform vector; entropy is zero at a vector concentrated entirely on one class.

The proof gives useful endpoints, not a universal threshold for calling an individual “well classified.” Normalized entropy helps compare the scale across models with different numbers of classes, but values still depend on the fitted model and the information used for classification. A high entropy indicates ambiguity among classes under that model; it does not by itself show why the ambiguity arose or whether the model is adequate.

Summarizing Uncertainty Across Individuals

Uncertainty can be summarized at the sample level, but an aggregate should not conceal individuals whose assignments remain ambiguous. Two common summaries are the average individual entropy, $n^{-1}\sum_i H_i$, and the average posterior probability of modal-assignment error, $n^{-1}\sum_i(1-m_i)$. The latter is the model-based expected fraction that would be misassigned by modal classification if the fitted posterior distributions correctly represented the latent classes. It is not an observed error fraction because the classes are unobserved.

The posterior expected number of individuals in class $g$ is $\sum_i\tau_{ig}$, because the conditional expectation of the indicator $I(C_i=g)$ is $\tau_{ig}$. It need not be an integer. In contrast, hard classification counts each person entirely in one modal class. The two summaries answer different questions: one preserves uncertainty, while the other reports a deterministic partition.

Worked Example: Expected Counts and Average Uncertainty

For three hypothetical individuals in a two-class model, suppose the posterior vectors are $(0.90,0.10)$, $(0.60,0.40)$, and $(0.50,0.50)$. Calculate the posterior expected class counts, average entropy, and average posterior modal-assignment error.

The expected count for class 1 is $0.90+0.60+0.50=2.00$; for class 2 it is $0.10+0.40+0.50=1.00$. Modal classification assigns all three to class 1 if ties are resolved in favor of class 1, producing hard counts of 3 and 0. That tie decision is a convention, not evidence favoring class 1 for the third individual.

The entropies are $-[0.90\log(0.90)+0.10\log(0.10)]=0.3251$, $-[0.60\log(0.60)+0.40\log(0.40)]=0.6730$, and $-[0.50\log(0.50)+0.50\log(0.50)]=0.6931$ nats. The average is $(0.3251+0.6730+0.6931)/3=0.5637$ nats. Since the maximum for two classes is $\log(2)=0.6931$, average normalized entropy is $0.5637/0.6931=0.8133$.

The modal probabilities are $0.90, 0.60,$ and $0.50$. Thus the average posterior modal-assignment error is $[(1-0.90)+(1-0.60)+(1-0.50)]/3=1.00/3=0.3333$. Under the fitted probabilities, the modal assignments have an expected error fraction of about 33.3%; this is not an observed validation error.

Why Hard Labels Can Distort Later Analyses

A hard label replaces a probability vector with a single zero-or-one membership indicator. This discards uncertainty, makes arbitrary distinctions around ties, and treats the resulting assignments as if they had been observed. A downstream analysis that ignores this generated-label uncertainty can misstate class-specific summaries, covariate relationships, standard errors, or tests. The direction and size of distortion are not universal; they depend on the classification uncertainty and the downstream question.

The next example illustrates the change in a descriptive class-specific mean for an external variable not used to calculate the posterior probabilities. It does not claim that posterior weighting alone is a generally valid inferential correction. Its purpose is to show how the same fitted membership probabilities lead to different summaries when uncertainty is retained rather than discarded.

Worked Example: A Downstream Mean Under Soft and Hard Membership

Suppose six hypothetical individuals have class-1 posterior probabilities $0.90,0.80,0.55,0.45,0.20,0.10$, respectively. Their external measurements $Z_i$, not used in fitting the class model, are $10,12,14,16,18,20$ units. Compare posterior-weighted class means with means after modal assignment.

A descriptive posterior-weighted mean for class 1 is $\sum_i\tau_{i1}Z_i/\sum_i\tau_{i1}$. The weights sum to $0.90+0.80+0.55+0.45+0.20+0.10=3.00$, and the weighted total is $9.0+9.6+7.7+7.2+3.6+2.0=39.1$. Correction: using the listed values, $9.0+9.6=18.6$, $18.6+7.7=26.3$, $26.3+7.2=33.5$, $33.5+3.6=37.1$, and $37.1+2.0=39.1$. Therefore the class-1 mean is $39.1/3.00=13.03$ units.

The class-2 weights are the complements, also summing to 3.00. Since the total of all six measurements is 90, the class-2 weighted total is $90-39.1=50.9$, and its mean is $50.9/3.00=16.97$ units. The weighted class-2 minus class-1 contrast is $(50.9-39.1)/3.00=3.9333\ldots$, or about 3.93 units.

Modal classification assigns the first three individuals to class 1 and the last three to class 2. The corresponding means are $(10+12+14)/3=12$ units and $(16+18+20)/3=18$ units, a contrast of $18-12=6$ units. Thus the hard-label contrast is larger in this hypothetical example. Neither contrast alone establishes an inferential result: posterior-weighted summaries condition on the fitted probabilities, whereas valid uncertainty quantification and adjustment for classification error require methods appropriate to the downstream estimand.

When the quantities of interest are part of the joint model, the full likelihood-based analysis retains the mixture structure rather than treating estimated labels as observed. When a separate downstream analysis is needed, appropriate classification-error-adjusted or three-step methods may be preferable to naïve classify-then-analyze procedures. A posterior-weighted descriptive summary can be informative, but it is not automatically a corrected estimator, and treating posterior weights as ordinary fixed sampling weights does not by itself account for uncertainty in the fitted model.

Diagnostics, Labeling, and Limitations

Useful reporting shows more than a single average. Inspect the distribution of maximum posterior probabilities or entropies, identify how many individuals have diffuse probabilities, and summarize uncertainty by assigned class. A class with many low maximum probabilities deserves particular attention when interpreting class-specific summaries. Also check that the posterior probabilities sum to one for every individual and that the records used to compute them match the model’s conditioning information.

Posterior probability coordinates depend on the names assigned to classes. If class labels are permuted, the posterior probability vector is permuted in the same way; its unordered set of probabilities and the individual’s entropy remain unchanged. The modal class name changes accordingly, while the maximum probability and entropy do not. This distinction matters when comparing fitted models whose class labels have been ordered differently.

A frequent mistake is to say, “This person has a 70% probability of belonging to the true class, so I can treat the assigned class as certain.” The correction is that 0.70 is a model-conditional posterior probability for the modal class, leaving posterior probability 0.30 for the other classes combined. A modal label is a useful summary when a single label is required, but it does not erase the uncertainty or justify analyses that assume the label was observed.

Finally, entropy measures uncertainty in class assignment conditional on the fitted model; it is not a measure of uncertainty in the fitted parameters, a test of model fit, or proof that latent classes are substantively real. Plug-in posterior probabilities hold estimated parameters fixed. If parameter uncertainty is important, it requires additional modeling or resampling beyond these individual assignment summaries.

Key takeaway: Preserve and report the posterior probability vector when classification uncertainty matters. Use entropy and maximum probability to describe ambiguity, and avoid treating a modal class label as observed truth in downstream inference.

Check Your Understanding

Answer each question, then compare with the solutions below.

For a two-class posterior vector $(0.75,0.25)$, what is the modal class, its posterior probability of error, and the entropy in nats?
For four classes, what are the minimum and maximum possible entropy, and which probability vectors attain them?
Three people have class-1 probabilities $0.8,0.3,0.2$. What is the posterior expected class-1 count, and how many are assigned to class 1 by modal classification?
What happens to an individual’s entropy when the class labels are permuted?
Why is a posterior-weighted downstream mean not automatically a valid inferential correction for hard classification?

Solutions. (1) Class 1 is modal. Its posterior probability of error is $1-0.75=0.25$. Entropy is $-[0.75\log(0.75)+0.25\log(0.25)]=0.5623$ nats. (2) The minimum is 0, attained when one class has probability 1; the maximum is $\log(4)=1.3863$ nats, attained by $(0.25,0.25,0.25,0.25)$. (3) The expected count is $0.8+0.3+0.2=1.3$; modal classification assigns one person to class 1. (4) The coordinates of the probability vector are permuted, but its unordered probabilities and entropy are unchanged. (5) The weighted mean is a descriptive summary conditional on fitted probabilities. By itself, it does not account for classification error, parameter uncertainty, or the estimand and sampling structure of a downstream analysis; an appropriate inferential method must address those features.