Categorical Models For Unordered Outcomes

One Response, Several Unordered Outcomes

A response need not be numerical or binary. A patient’s discharge destination might be home, another care facility, or a rehabilitation facility. These outcomes are different categories, but none is naturally “larger” or “smaller” than another. A probability model for such a response assigns a probability to each possible category.

The central requirement is simple but essential: each category probability must be nonnegative, and the probabilities across all possible categories must add to one. These conditions make the assignments a valid probability distribution for a single response.

Definition: Let \(X\) be a response whose possible values are the distinct categories \(c_1,\ldots,c_K\), where \(K\) is a positive integer. A categorical model for \(X\) specifies probabilities \(p_1,\ldots,p_K\) such that

p_k=P(X=c_k),\qquad p_k\geq 0,\qquad \sum_{k=1}^{K}p_k=1.

Equivalently, its probability mass function is \(f(c_k)=p_k\) for \(k=1,\ldots,K\), and \(f(x)=0\) for values outside the stated support.

The categories must be distinct, mutually exclusive, and exhaustive: exactly one of them occurs for the response being modeled. If a response could belong to more than one listed category at once, or if a possible outcome has been omitted, the categories do not form the stated support without further refinement. The order in which the categories are listed is only a way to index them. It does not imply a ranking.

For example, assigning the numbers 1, 2, and 3 to “home,” “rehabilitation,” and “other facility” can be convenient for data storage. It does not make the outcomes quantitative: subtracting the code for home from the code for rehabilitation has no inherent interpretation. The model’s probabilities attach to the categories, not to arithmetic on their labels.

Checking the Probability Constraints

A proposed list of category probabilities defines a valid categorical model exactly when it satisfies both constraints in the definition. Nonnegativity prevents an impossible negative probability. The sum-to-one condition ensures that the probability of getting one of the possible outcomes is one.

Theorem: Suppose \(X\) has support exactly \(\{c_1,\ldots,c_K\}\), with these values distinct. Assign \(f(c_k)=p_k\) and \(f(x)=0\) outside the support. Then \(f\) is a probability mass function if and only if \(p_k\geq 0\) for every \(k\) and \(\sum_{k=1}^{K}p_k=1\).

Proof. By definition, a probability mass function has nonnegative values, so \(f(c_k)=p_k\geq 0\) for every category. Its total mass on the support must be one. Since the support consists of the distinct values \(c_1,\ldots,c_K\),

\sum_{x} f(x)=\sum_{k=1}^{K}f(c_k)=\sum_{k=1}^{K}p_k=1.

This proves necessity. Conversely, if every \(p_k\) is nonnegative and their sum is one, then \(f\) is nonnegative everywhere, and its total mass is

\sum_x f(x)=\sum_{k=1}^{K}p_k=1.

Thus \(f\) satisfies the defining conditions for a probability mass function. \(\square\)

Worked Example: A Valid Three-Category Model

Suppose a single patient’s discharge destination \(X\) has three possible categories: home, rehabilitation, and another care facility. A proposed model assigns probabilities \(0.62\), \(0.28\), and \(0.10\), respectively. Check that the model is valid, then find the probability of discharge somewhere other than home.

Check nonnegativity. Each probability is at least zero: \(0.62\geq 0\), \(0.28\geq 0\), and \(0.10\geq 0\).

Check the total.

$$ 0.62+0.28+0.10=1.00. $$

Both constraints hold, so these assignments define a valid categorical model. “Somewhere other than home” is the union of the rehabilitation and other-facility categories. Its probability is \(0.28+0.10=0.38\).

The probability of an event that combines categories is obtained by adding the probabilities of those categories. This follows from the fact that distinct categories cannot occur simultaneously.

Result: For a categorical response with probabilities \(p_1,\ldots,p_K\), and any subset \(A\) of the category indices,

P(X\in\{c_k:k\in A\})=\sum_{k\in A}p_k.

Proof. The event \(X\in\{c_k:k\in A\}\) is the union of the events \(\{X=c_k\}\) for \(k\in A\). These events are pairwise disjoint because the category values are distinct. By finite additivity of probability, the probability of their union is the sum of their probabilities, namely \(\sum_{k\in A}P(X=c_k)=\sum_{k\in A}p_k\). \(\square\)

Worked Checks: Validity and Interpretation

Worked Example: Finding an Invalid Probability Assignment

A survey classifies one respondent’s preferred mode of transportation as walking, cycling, public transit, or driving. A proposed model assigns probabilities \(0.20\), \(0.25\), \(0.40\), and \(0.25\), in that order. Is this a valid model?

Every individual value is nonnegative. However, their sum is

$$ 0.20+0.25+0.40+0.25=1.10. $$

Because the categories are mutually exclusive and exhaustive, the probability that the response is one of them must be one. A total of \(1.10\) violates that requirement, so the proposed assignment is not a valid categorical model. One could divide each value by \(1.10\) to create a different probability assignment that sums to one, but that would change the proposed probabilities. Renormalizing is not evidence that the original assignment was valid.

Worked Example: Combining Several Categories

A laboratory records one specimen’s result as negative, inconclusive, or positive. The model assigns probabilities \(0.84\), \(0.06\), and \(0.10\), respectively. Find the probability that the result is not inconclusive, and verify the result in two ways.

First, the probabilities are nonnegative and \(0.84+0.06+0.10=1\), so they define a valid categorical model. “Not inconclusive” consists of the negative and positive categories. Adding their probabilities gives

P(\text{not inconclusive})=0.84+0.10=0.94.

Alternatively, “not inconclusive” is the complement of the inconclusive category. Therefore,

P(\text{not inconclusive})=1-P(\text{inconclusive})=1-0.06=0.94.

The two calculations agree, as they must.

Indicator Representation of a Categorical Response

A categorical response can also be represented by indicators that record which category occurred. Define \(Z_k=1\) if \(X=c_k\), and \(Z_k=0\) otherwise. For each single response, exactly one of \(Z_1,\ldots,Z_K\) equals one, and all the others equal zero. This is sometimes called a one-hot representation.

The indicators are useful because they express category probabilities as expectations. They also make clear why the categories cannot be treated as independent binary responses: if one indicator is one, all the others must be zero.

Result: For \(Z_k=\mathbf{1}\{X=c_k\}\) in a categorical model, \(E[Z_k]=p_k\). For any indices \(j,k\), the covariance is

\operatorname{Cov}(Z_j,Z_k)= \begin{cases} p_k(1-p_k),&j=k,\\ -p_jp_k,&j\ne k. \end{cases}

Derivation. Since \(Z_k\) is one for category \(c_k\) and zero otherwise, its expectation is \(1\cdot p_k+0\cdot(1-p_k)=p_k\). Also \(Z_k^2=Z_k\), so \(\operatorname{Var}(Z_k)=E[Z_k^2]-E[Z_k]^2=p_k-p_k^2=p_k(1-p_k)\). If \(j\ne k\), the indicators cannot both equal one, so \(Z_jZ_k=0\) for every possible outcome. Hence \(E[Z_jZ_k]=0\) and \(\operatorname{Cov}(Z_j,Z_k)=E[Z_jZ_k]-E[Z_j]E[Z_k]=0-p_jp_k=-p_jp_k\). \(\square\)

Worked Example: Indicator Probabilities and Covariances

Let \(X\) have three unordered categories \(c_1,c_2,c_3\), with probabilities \(0.50\), \(0.30\), and \(0.20\). Verify the constraints and find the expectation vector and covariance matrix of the one-hot indicators.

All probabilities are nonnegative, and \(0.50+0.30+0.20=1.00\), so this is a valid model. From \(E[Z_k]=p_k\), the expectation vector is

E[(Z_1,Z_2,Z_3)^\mathsf{T}]=(0.50,0.30,0.20)^\mathsf{T}.

The diagonal variances are \(0.50(1-0.50)=0.25\), \(0.30(1-0.30)=0.21\), and \(0.20(1-0.20)=0.16\). The off-diagonal covariances are \(-0.50(0.30)=-0.15\), \(-0.50(0.20)=-0.10\), and \(-0.30(0.20)=-0.06\). Thus

\operatorname{Cov}((Z_1,Z_2,Z_3)^\mathsf{T})= \begin{pmatrix} 0.25&-0.15&-0.10\\ -0.15&0.21&-0.06\\ -0.10&-0.06&0.16 \end{pmatrix}.

The negative off-diagonal entries reflect mutual exclusivity: observing one category rules out the others for this response.

Common Pitfalls and Why the Constraints Matter

A categorical model is a probability distribution for one response, not a ranking of outcomes and not a collection of unrelated yes-or-no probabilities. Its categories must describe all possible outcomes exactly once. Once that is true, the probability assigned to each category determines the probability of any event formed by grouping categories.

Using category codes as measurements. Codes such as 1, 2, and 3 are labels unless the problem gives them a meaningful numerical interpretation. The ordering of codes alone does not create an ordering of outcomes.
Checking only that each value is between zero and one. Even if every proposed value is in that range, the assignments are invalid if they do not sum to one. For example, three probabilities of \(0.4\), \(0.4\), and \(0.4\) are individually within the range but total \(1.2\).
Forgetting that categories are exclusive. If categories overlap, adding their probabilities can double-count outcomes. Define distinct outcomes first, or use the appropriate probability rule for overlapping events.
Treating one-hot indicators as independent. Their values are constrained: exactly one is one. For distinct categories their covariance is \(-p_jp_k\), not generally zero.

For two categories, the categorical model reduces to a binary response model: if \(P(X=c_1)=p\), then the other category has probability \(1-p\). With more than two categories, the same logic extends by assigning a probability to every possible category while preserving nonnegativity and a total of one.

Key takeaway: To specify a model for one unordered categorical response, list its distinct exhaustive categories, assign each a nonnegative probability, and check that the probabilities sum to one. Probabilities of grouped categories are sums of their component probabilities.

Check Your Understanding

Use the probability constraints and the event interpretation to answer the following.

A response has four categories with probabilities \(0.15\), \(0.25\), \(0.35\), and \(0.25\). Is the assignment valid? Show both checks.
For probabilities \(0.15\), \(0.25\), \(0.35\), and \(0.25\), what is the probability of the event consisting of the second and fourth categories?
Why does assigning numerical codes to unordered categories not by itself make the response quantitative?
For two distinct one-hot indicators \(Z_j\) and \(Z_k\), explain why their covariance is \(-p_jp_k\).
A list of purported categories leaves out one possible response. Which requirement for a categorical model has failed, and why does that matter for the sum of the listed probabilities?