Posterior Responsibilities

Why Component Membership Is Probabilistic

A finite Gaussian mixture represents a distribution as a weighted combination of component distributions. Each observation is associated with an unobserved component label, but observing a point does not generally reveal that label with certainty. Instead, the model assigns a conditional probability to each possible component. These probabilities are called posterior responsibilities.

A responsibility is not a newly estimated parameter, nor is it automatically a final classification. It describes how the fitted mixture model allocates probability to the possible sources of a particular observation. Its value depends on both the component weights and the component densities at that observation.

Definition: Consider a \(G\)-component Gaussian mixture in \(d\) dimensions, with weights \(\pi_g>0\), \(\sum_{g=1}^G\pi_g=1\), means \(\mu_g\in\mathbb{R}^d\), and positive-definite covariance matrices \(\Sigma_g\), for \(g=1,\ldots,G\). Let \(Z\in\{1,\ldots,G\}\) be the latent component label and \(X\in\mathbb{R}^d\) the observation. The model specifies \(P(Z=g)=\pi_g\) and \(X\mid Z=g\sim N_d(\mu_g,\Sigma_g)\). For an observed value \(x\), the posterior responsibility of component \(g\) is \(\gamma_g(x)=P(Z=g\mid X=x)\), using the density-based conditional probability defined below.

Deriving the Responsibility Formula

Write the \(d\)-variate Gaussian density for component \(g\) as

\phi_d(x;\mu_g,\Sigma_g) = \frac{1}{(2\pi)^{d/2}|\Sigma_g|^{1/2}} \exp\!\left\{-\frac{1}{2}(x-\mu_g)^\top\Sigma_g^{-1}(x-\mu_g)\right\}.

Because \(X\) has a continuous distribution, \(P(X=x)=0\) for an individual point \(x\). The conditional probability at \(x\) is therefore obtained from the joint and marginal densities; it is a version of the regular conditional distribution of \(Z\) given \(X\). In this mixture, the joint density for \(X=x\) and \(Z=g\) is \(\pi_g\phi_d(x;\mu_g,\Sigma_g)\), while the marginal density of \(X\) is the sum of those joint densities.

Theorem: For every \(x\in\mathbb{R}^d\), the posterior responsibility of component \(g\) is

\gamma_g(x) = \frac{\pi_g\phi_d(x;\mu_g,\Sigma_g)} {\displaystyle\sum_{h=1}^G\pi_h\phi_d(x;\mu_h,\Sigma_h)}.

The denominator is the mixture density at \(x\), which is positive under the stated assumptions.

To justify the formula as a conditional probability, consider any measurable set \(A\subseteq\mathbb{R}^d\). The joint probability that \(X\in A\) and \(Z=g\) is

P(X\in A,Z=g) = \int_A\pi_g\phi_d(x;\mu_g,\Sigma_g)\,dx.

The marginal density of \(X\) is \(f_X(x)=\sum_{h=1}^G\pi_h\phi_d(x;\mu_h,\Sigma_h)\). Multiplying this marginal density by the proposed conditional probability gives

\int_A \gamma_g(x)f_X(x)\,dx = \int_A \frac{\pi_g\phi_d(x;\mu_g,\Sigma_g)} {\sum_{h=1}^G\pi_h\phi_d(x;\mu_h,\Sigma_h)} \left(\sum_{h=1}^G\pi_h\phi_d(x;\mu_h,\Sigma_h)\right)dx = P(X\in A,Z=g).

This is the defining integral identity for the conditional distribution of \(Z\) given \(X\), so the formula is valid. \(\square\)

For \(n\) observations \(x_1,\ldots,x_n\), write \(\gamma_{ig}=\gamma_g(x_i)\). Each observation has its own vector of responsibilities. Under the mixture model, the labels for different observations are generated independently conditional on the component probabilities; the formula for one observation uses its own value \(x_i\), not the other observations.

What the Formula Says

Each numerator is the component’s prior weight multiplied by its density at the observed point. Thus a component can receive substantial responsibility because it is common in the mixture, because it fits the observation well, or both. The denominator normalizes these weighted densities into probabilities.

Result: For every observation \(x\), the responsibilities are nonnegative and sum to one:

\gamma_g(x)\geq 0 \qquad\text{and}\qquad \sum_{g=1}^G\gamma_g(x)=1.

The nonnegativity follows because the weights and Gaussian densities are positive and the denominator is positive. For the sum, use the common denominator:

\sum_{g=1}^G\gamma_g(x) = \frac{\sum_{g=1}^G\pi_g\phi_d(x;\mu_g,\Sigma_g)} {\sum_{h=1}^G\pi_h\phi_d(x;\mu_h,\Sigma_h)} =1.

So the responsibilities form a probability vector. This property is also a useful arithmetic check: if the computed probabilities do not sum to one, there is an error in the calculation or normalization. \(\square\)

For any two components \(g\) and \(h\), their posterior odds have a particularly clear form:

Result: If \(g\) and \(h\) are components with positive weights, then

\frac{\gamma_g(x)}{\gamma_h(x)} = \frac{\pi_g}{\pi_h} \frac{\phi_d(x;\mu_g,\Sigma_g)} {\phi_d(x;\mu_h,\Sigma_h)}.

To prove this, substitute the responsibility formula for both components. The common mixture-density denominator cancels, leaving the prior-weight ratio times the density ratio. \(\square\)

This identity emphasizes that the component with the greatest density at \(x\) need not have the greatest responsibility: a sufficiently larger mixture weight can compensate for a lower component density.

Worked Calculations

Worked Example: Two Components With Equal Covariance

Suppose \(X\in\mathbb{R}^2\), with weights \(\pi_1=0.4\) and \(\pi_2=0.6\). Both components have covariance \(I_2\), and their means are \(\mu_1=(0,0)^\top\) and \(\mu_2=(2,0)^\top\). Calculate the responsibilities for \(x=(1.5,0)^\top\).

For component 1, the squared Mahalanobis distance is \((1.5-0)^2+(0-0)^2=2.25\). For component 2, it is \((1.5-2)^2+(0-0)^2=0.25\). Since both covariance matrices are \(I_2\), the Gaussian density factor \(1/(2\pi)\) is the same for both components. The weighted densities, after omitting this common factor, are

0.4e^{-2.25/2}=0.4e^{-1.125}\approx 0.129861, \qquad 0.6e^{-0.25/2}=0.6e^{-0.125}\approx 0.529498.

Their sum is approximately \(0.659359\). Therefore,

\gamma_1(x)\approx\frac{0.129861}{0.659359}=0.1970, \qquad \gamma_2(x)\approx\frac{0.529498}{0.659359}=0.8030.

Although \(x\) is closer to the second mean, the calculation also accounts for the larger prior weight of component 2. The responsibilities sum to \(0.1970+0.8030=1.0000\), up to rounding.

Worked Example: Unequal Covariance Matrices

Consider two bivariate components with equal weights, zero means, \(\Sigma_1=\operatorname{diag}(1,4)\), and \(\Sigma_2=I_2\). For \(x=(1,0)^\top\), find the responsibilities.

For both components, the quadratic form is \(x^\top\Sigma_g^{-1}x=1\). But the determinants differ: \(|\Sigma_1|=4\), while \(|\Sigma_2|=1\). Thus

\phi_2(x;0,\Sigma_1) = \frac{1}{2\pi\sqrt{4}}e^{-1/2} = \frac{1}{4\pi}e^{-1/2}, \qquad \phi_2(x;0,\Sigma_2) = \frac{1}{2\pi}e^{-1/2}.

The second density is twice the first. Equal mixture weights preserve that ratio in the weighted densities, so

\gamma_1(x) = \frac{1}{1+2} = \frac{1}{3}, \qquad \gamma_2(x) = \frac{2}{1+2} = \frac{2}{3}.

The quadratic forms alone would not distinguish the two components here. The determinant term in the Gaussian density matters: it reflects the difference in overall spread and changes the relative density at \(x\).

Worked Example: Three Components and Unequal Weights

Let three bivariate components have common covariance \(I_2\), means \(\mu_1=(0,0)^\top\), \(\mu_2=(1,1)^\top\), and \(\mu_3=(2,2)^\top\), and weights \(0.2,0.5,0.3\), respectively. For \(x=(0,0)^\top\), calculate all three responsibilities.

The squared distances from \(x\) to the three means are \(0,2,8\). The common density factor \(1/(2\pi)\) cancels from every responsibility. The weighted densities after removing that factor are

a_1=0.2e^0=0.2,\qquad a_2=0.5e^{-2/2}=0.5e^{-1}\approx0.183940,\qquad a_3=0.3e^{-8/2}=0.3e^{-4}\approx0.005495.

The normalizing sum is \(a_1+a_2+a_3\approx0.389434\). Dividing each term by this sum gives

\gamma_1(x)\approx0.5136,\qquad \gamma_2(x)\approx0.4723,\qquad \gamma_3(x)\approx0.0141.

These values sum to approximately \(1.0000\). Component 2 has the largest prior weight, but component 1 receives the largest responsibility because \(x\) is centered at its mean. Component 3 has both a smaller weight and a much smaller density at \(x\).

Responsibilities and Component Assignments

A responsibility retains the full distribution of probability over the latent labels. If a single label is required, a common rule is to assign \(x\) to a component with the largest responsibility; this is a maximum a posteriori assignment. It is a separate decision made from the probabilities, not the definition of responsibility itself. For instance, probabilities \(0.51\) and \(0.49\) produce a hard assignment to one component, but they indicate a much closer competition than probabilities \(0.99\) and \(0.01\).

The component labels are names for mixture terms, not intrinsically ordered categories. If the component labels, weights, means, and covariance matrices are all permuted together, the same mixture distribution results and the responsibility entries are permuted in the same way. The probability attached to a particular numerical label therefore depends on the chosen labeling convention.

Common Mistake

Using only the nearest mean. This ignores component weights and covariance matrices. The responsibility uses the entire weighted density, including both the Mahalanobis-distance term and the determinant factor.
Using only the component density. The density is not itself a posterior probability. Multiply it by the prior mixture weight, then normalize across all components.
Treating the largest responsibility as certainty. Selecting the largest entry gives a hard assignment, but the responsibility vector may still spread substantial probability over several components.

Posterior responsibilities matter whenever an analysis needs to represent latent membership without pretending that unobserved labels are known. They provide the model-based allocation probabilities for each observation. In applications, they should be calculated using the full component densities and checked to sum to one; replacing them prematurely with hard labels discards information in those probabilities.

Check Your Understanding

Use the definitions and calculations above to test your understanding.

In the responsibility formula, what does the denominator represent?
Why can a component with a lower density at \(x\) still have a greater responsibility than another component?
For two components, what factors determine the posterior odds?
What role does the determinant of a covariance matrix play in the multivariate Gaussian density?
How does a maximum a posteriori assignment differ from reporting the full responsibility vector?

What You'll Learn