Gaussian Mixture Models for Profile Clustering

Why Model Profiles With a Mixture?

A profile may sit between several plausible groups rather than being an obvious member of just one. Assigning it to the nearest cluster hides that ambiguity. A Gaussian mixture model instead represents a population of profiles as a weighted combination of component distributions. It can describe groups with different centers and different shapes, and it gives each profile a probability of membership in every component.

Here, a profile is a vector of measurements, such as expression values for a selected set of genes. The standard Gaussian mixture model assumes those vectors are independent observations from the same finite mixture distribution. Within each latent component, the profile follows a multivariate normal distribution. The mixture model describes a distribution of profiles; by itself, it does not establish that the components are biological cell types or causal mechanisms.

The Mixture Model and Its Covariance Structures

Definition: A $K$-component Gaussian mixture model for a $d$-dimensional profile $\boldsymbol{x}_i$ has density $$ f(\boldsymbol{x}_i)=\sum_{k=1}^{K}\pi_k\phi_d(\boldsymbol{x}_i;\boldsymbol{\mu}_k,\boldsymbol{\Sigma}_k). $$ Here, $\pi_k$ is the mixing proportion for component $k$, with $\pi_k\geq 0$ and $\sum_{k=1}^{K}\pi_k=1$; $\boldsymbol{\mu}_k$ is its mean vector; $\boldsymbol{\Sigma}_k$ is its covariance matrix; and $\phi_d$ is the $d$-dimensional normal density. Each covariance matrix must be positive definite for this density to be defined in the usual nonsingular form.

One way to express the model is to introduce a latent label $Z_i$, which is not observed. It takes value $k$ with probability $\pi_k$; conditional on $Z_i=k$, the profile $\boldsymbol{x}_i$ has distribution $N_d(\boldsymbol{\mu}_k,\boldsymbol{\Sigma}_k)$. The labels are latent because the data contain measured profiles, not known component assignments. This construction implies the mixture density above after averaging over the possible labels.

The covariance structure determines which differences in profile shape the model can represent and how many parameters it must estimate. A spherical covariance is $\boldsymbol{\Sigma}_k=\sigma_k^2\boldsymbol{I}$: within a component, all directions have equal variance and there is no covariance between features. A diagonal covariance allows different feature variances but assumes zero within-component covariance between features. A full covariance allows different variances and correlations. These choices can be made separately by component, or constrained to be shared; for example, a shared full covariance uses one common matrix $\boldsymbol{\Sigma}$ for every component.

These are modeling choices, not facts guaranteed by the data. With $d$ features, a full symmetric covariance matrix has $d(d+1)/2$ distinct entries per component. Estimating many such parameters requires enough information within each component. A diagonal or spherical structure reduces the parameter count, but imposes stronger assumptions about the shape of clusters.

Soft Membership From Bayes’ Rule

For a profile $\boldsymbol{x}_i$, the soft membership probability for component $k$ is the conditional probability that its latent label is $k$, given the observed profile. Bayes’ rule gives this probability by multiplying the component density by its mixing proportion and normalizing across all components.

Definition: The posterior membership probability, also called the responsibility, is $$ \tau_{ik}=P(Z_i=k\mid \boldsymbol{x}_i) =\frac{\pi_k\phi_d(\boldsymbol{x}_i;\boldsymbol{\mu}_k,\boldsymbol{\Sigma}_k)} {\sum_{\ell=1}^{K}\pi_\ell\phi_d(\boldsymbol{x}_i;\boldsymbol{\mu}_\ell,\boldsymbol{\Sigma}_\ell)}. $$ For each profile, $\tau_{ik}\geq 0$ and $\sum_{k=1}^{K}\tau_{ik}=1$.

The numerator is the joint density contribution for profile $i$ arising from component $k$: it combines how prevalent the component is, through $\pi_k$, with how well its distribution fits the profile. The denominator adds those contributions over all components, converting them to probabilities. A largest-probability assignment is possible, but it discards the remaining membership probabilities.

Worked Example: A One-Feature Profile

Suppose a hypothetical standardized gene-expression measurement $x$ is modeled by two components. Their parameters are $\pi_1=\pi_2=0.5$, $X\mid Z=1\sim N(0,0.25)$, and $X\mid Z=2\sim N(2,4)$. For an observed value $x=0.5$, calculate its soft memberships. The model assumes the observations are independent draws from this mixture and that each component is normal with the stated variance.

For a univariate normal distribution with mean $\mu$ and variance $\sigma^2$, the density at $x$ is $\{1/\sqrt{2\pi\sigma^2}\}\exp\{-(x-\mu)^2/(2\sigma^2)\}$. For component 1,

\phi_1(0.5)=\frac{1}{\sqrt{2\pi(0.25)}}\exp\left[-\frac{(0.5-0)^2}{2(0.25)}\right] =0.7979\exp(-0.5)\approx 0.4839.

For component 2,

\phi_2(0.5)=\frac{1}{\sqrt{2\pi(4)}}\exp\left[-\frac{(0.5-2)^2}{2(4)}\right] =0.1995\exp(-0.28125)\approx 0.1506.

The mixing proportions are equal, so they cancel when the memberships are normalized:

\tau_{i1}=\frac{0.5(0.4839)}{0.5(0.4839)+0.5(0.1506)} =\frac{0.4839}{0.4839+0.1506}\approx 0.7627, \qquad \tau_{i2}=1-\tau_{i1}\approx 0.2373.

Under these specified parameters, the profile has about 76.27% posterior membership in component 1 and 23.73% in component 2. This is not a claim that the profile belongs biologically to component 1 with a 76.27% chance under every possible model; it is conditional on the fitted mixture and its assumptions.

Allowing Different Covariance Shapes

In more than one dimension, the covariance matrix affects both the normalizing constant of the density and the distance from a profile to the component center. For a positive-definite covariance matrix, the density is

\phi_d(\boldsymbol{x};\boldsymbol{\mu},\boldsymbol{\Sigma}) =\frac{1}{(2\pi)^{d/2}|\boldsymbol{\Sigma}|^{1/2}} \exp\left[-\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu})^\mathsf{T} \boldsymbol{\Sigma}^{-1}(\boldsymbol{x}-\boldsymbol{\mu})\right].

The quadratic expression in the exponent measures distance relative to the covariance: a deviation along a high-variance direction contributes less than the same-sized deviation along a low-variance direction. The determinant term $|\boldsymbol{\Sigma}|^{1/2}$ also matters; a broad component has a lower density near its center than a narrow one, all else equal. Thus, mixture membership is not generally equivalent to choosing the nearest mean by ordinary Euclidean distance.

Worked Example: Two Features With Diagonal Covariances

Consider two hypothetical standardized expression measurements and two components with equal mixing proportions. Let component 1 have mean $(0,0)$ and covariance $\operatorname{diag}(1,0.25)$; let component 2 have mean $(2,1)$ and covariance $\operatorname{diag}(4,1)$. For the profile $\boldsymbol{x}=(1,0.5)$, calculate the posterior membership probabilities. The diagonal covariance assumption means that the two features have zero covariance within each component; the variances are positive, so the matrices are positive definite.

For component 1, the determinant is $0.25$, and the quadratic term is $1^2/1+0.5^2/0.25=2$. Its density is therefore

\phi_2(\boldsymbol{x};\boldsymbol{\mu}_1,\boldsymbol{\Sigma}_1) =\frac{1}{2\pi\sqrt{0.25}}\exp(-2/2) =\frac{1}{\pi}\exp(-1)\approx 0.1171.

For component 2, the determinant is $4$. The deviations are $(-1,-0.5)$, so the quadratic term is $(-1)^2/4+(-0.5)^2/1=0.5$. Thus,

\phi_2(\boldsymbol{x};\boldsymbol{\mu}_2,\boldsymbol{\Sigma}_2) =\frac{1}{2\pi\sqrt{4}}\exp(-0.5/2) =\frac{1}{4\pi}\exp(-0.25)\approx 0.0620.

Because the mixing proportions are equal, the normalized density contributions give

\tau_{i1}=\frac{0.1171}{0.1171+0.0620}\approx 0.6539, \qquad \tau_{i2}=1-\tau_{i1}\approx 0.3461.

This profile is more compatible with component 1 under the specified model, although it retains substantial membership probability in component 2. Both covariance matrices influence that comparison; using only distances to $(0,0)$ and $(2,1)$ would not reproduce these probabilities.

What the Mixing Proportions Contribute

A component’s mixing proportion is its probability before observing a particular profile. Consequently, when two components have similar densities at a profile, a component with a larger mixing proportion receives a larger posterior membership probability. The following example isolates that effect.

Worked Example: Unequal Mixing Proportions

Suppose one standardized feature follows a two-component mixture with $\pi_1=0.8$, $\pi_2=0.2$, and both component variances equal to 1. Their means are 0 and 2, respectively. For a hypothetical observed value $x=2$, the normality and independence assumptions are as in the univariate mixture above.

The density under component 1 is

\phi_1(2)=\frac{1}{\sqrt{2\pi}}\exp[-(2-0)^2/2] =0.3989\exp(-2)\approx 0.0540.

The observed value equals the mean of component 2, so

\phi_2(2)=\frac{1}{\sqrt{2\pi}}\exp(0)\approx 0.3989.

Weighting by the mixing proportions and normalizing gives

\tau_{i1}=\frac{0.8(0.0540)}{0.8(0.0540)+0.2(0.3989)} \approx \frac{0.0432}{0.1230}\approx 0.3512, \qquad \tau_{i2}\approx 0.6488.

Although component 2 has a much higher density at $x=2$, component 1’s larger mixing proportion matters. Here, component 2 still has the larger posterior membership probability, but neither the density nor the prior proportion alone is the posterior.

Assumptions, Diagnostics, and Limitations

A standard finite Gaussian mixture model makes several substantive assumptions: profiles are independent observations from a common population mixture; the number of components $K$ has been specified; the component probabilities are nonnegative and sum to one; and each component is adequately represented by a multivariate normal distribution with a valid covariance matrix. For expression profiles, choices about transformation, feature selection, and scaling affect the geometry and therefore the fitted components and memberships. Measurements with very different scales can dominate the covariance-based distances unless scaling is appropriate for the scientific question.

Diagnostics should examine whether these assumptions appear reasonable, rather than treating a numerical output as proof of meaningful groups. Useful checks include looking for strong skewness, heavy tails, or outlying profiles within proposed components; inspecting whether estimated covariance matrices are nearly singular; and checking whether a component has very few profiles relative to its number of covariance parameters. A component that is extremely narrow around one observation may indicate a poorly behaved fit rather than a real cluster. With full covariance matrices, the parameter burden grows quickly as the number of features increases.

Membership probabilities are conditional on the chosen model. They do not account automatically for uncertainty about the number of components, the covariance structure, or the estimated parameters. Moreover, component labels are arbitrary: swapping labels 1 and 2 together with their parameters leaves the same mixture distribution. A component number is therefore not an intrinsic biological identity.

Gaussian mixtures differ from the main alternatives in what they represent. K-means assigns profiles to nearest centers under a squared-distance objective and yields hard assignments; it does not estimate a probability distribution for each cluster. A Gaussian mixture can express soft assignments and elliptical component shapes, but depends on its distributional assumptions and covariance estimation. Hierarchical clustering organizes profiles through pairwise distances and linkage choices, producing a nested grouping rather than a probability model with component densities. No approach guarantees that its groups correspond to scientifically meaningful biological classes.

A common mistake is to treat the largest posterior membership probability as certainty: for example, calling a profile with memberships 0.51 and 0.49 an unambiguous member of the first component. The correction is to report or inspect the full probability vector and recognize the assignment as weakly separated. Another mistake is to compare only distances to component means. Covariances and mixing proportions also determine the posterior probabilities, as the worked examples show.

Key takeaway: A Gaussian mixture represents a profile distribution as weighted component densities. Soft membership probabilities combine component prevalence, fit to the observed profile, and covariance-defined shape; they are model-based uncertainty summaries, not unquestionable labels.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For a profile with posterior memberships 0.62, 0.30, and 0.08 across three components, what is its largest-probability assignment, and why is that not a certain label?
What assumption is imposed by a diagonal covariance matrix that a full covariance matrix does not impose?
Two components have equal densities at a profile, but their mixing proportions are 0.7 and 0.3. What are the posterior memberships?
Why might a full covariance model be difficult to estimate when the number of features is large relative to the number of profiles in a component?

Solutions. (1) The largest-probability assignment is component 1, but 0.62 is not 1, so the model retains 0.38 total membership probability in the other components. (2) A diagonal covariance assumes zero covariance between distinct features within each component; a full covariance permits those covariances to be nonzero. (3) Equal densities leave only the mixing proportions to normalize, so the posterior memberships are 0.7 and 0.3. (4) A full covariance in $d$ dimensions has $d(d+1)/2$ distinct entries, so limited component-specific data may provide insufficient information for stable estimates and can lead to nearly singular covariance matrices.