Sensitivity to Covariance-Model Choice

Why Covariance-Model Sensitivity Matters

A Gaussian mixture assigns observations to components using both their locations and their covariance structures. A component with a particular mean may be a close match under spherical covariance but a poor match under a tilted, elongated covariance. Thus, the same observations and the same number of components can produce different memberships when the covariance constraints change.

A useful sensitivity analysis changes the covariance parameterization while holding other choices fixed. This makes it possible to ask a specific question: which conclusions depend on the covariance model, and which remain stable across plausible alternatives? The comparison should include both allocations and estimated group geometry; agreement on one does not imply agreement on the other.

Set Up a Controlled Comparison

Definition: A covariance-model sensitivity comparison fits the same observations under two or more specified covariance parameterizations, with the other analysis choices held fixed as far as possible. These choices include the number of components, variables and preprocessing, fitting and restart rules, convergence criteria, and any selection criterion used. For each fit, record the estimated component means, mixing proportions, covariance matrices, posterior memberships, and hard allocations.

For a fitted \(K\)-component Gaussian mixture, let \(\hat{\pi}_g\), \(\hat{\mu}_g\), and \(\hat{\Sigma}_g\) denote the mixing proportion, mean, and positive-definite covariance estimate for component \(g\). For observation \(x_i\), define its component log score by

s_{ig} = \log \hat{\pi}_g -\frac{1}{2}\left[ p\log(2\pi)+\log|\hat{\Sigma}_g| +(x_i-\hat{\mu}_g)^\mathsf{T}\hat{\Sigma}_g^{-1}(x_i-\hat{\mu}_g) \right].

The posterior membership is \(\hat{\tau}_{ig}=\exp(s_{ig})/\sum_{h=1}^K\exp(s_{ih})\), and the hard allocation assigns \(x_i\) to a component with the largest score. A unique largest score gives a unique allocation. A hard allocation discards uncertainty: two fits can assign the same component while giving quite different posterior memberships.

Before comparing component labels across fits, account for label switching: component numbers have no intrinsic meaning. Match components using a stated rule, such as a permutation that maximizes agreement in memberships or matches fitted means. Keep the rule fixed and report it. If the number of components is selected separately under each covariance family, differences in \(K\) become part of the pipeline comparison. For a focused comparison of covariance effects, first compare fits at a common \(K\); report any separately selected \(K\) as a distinct consequence of the selection procedure.

Worked Example: One Observation, Two Covariance Choices

Consider two equally weighted components in two dimensions, with means \(\mu_1=(0,0)^\mathsf{T}\) and \(\mu_2=(2,0)^\mathsf{T}\). Compare a fit with \(\Sigma_1=\Sigma_2=I\) to one with the same shared covariance \(\Sigma_1=\Sigma_2=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). For \(x=(1.2,1.5)^\mathsf{T}\), calculate the hard allocation and posterior memberships under each fit.

With equal weights and a covariance shared by both components, the log determinant and normalizing constant cancel in the score difference. For the identity covariance, the squared distances are

d_1^2=1.2^2+1.5^2=3.69, \qquad d_2^2=(1.2-2)^2+1.5^2=0.64+2.25=2.89.

The log-score difference \(s_2-s_1=(d_1^2-d_2^2)/2=0.40\), so the hard allocation is component 2. Its posterior membership is \(1/(1+\exp(-0.40))\approx0.599\); component 1 has membership about \(0.401\).

For the correlated covariance, the determinant is \(1-0.8^2=0.36\) and the inverse is \(\frac{1}{0.36}\begin{pmatrix}1&-0.8\\-0.8&1\end{pmatrix}\). The squared Mahalanobis distances are

d_1^2=\frac{1.2^2-1.6(1.2)(1.5)+1.5^2}{0.36} =\frac{0.81}{0.36}=2.25,

d_2^2=\frac{(-0.8)^2-1.6(-0.8)(1.5)+1.5^2}{0.36} =\frac{4.81}{0.36}\approx13.361.

Now \(s_1-s_2=(d_2^2-d_1^2)/2\approx5.556\). The allocation switches to component 1, whose posterior membership is \(1/(1+\exp(-5.556))\approx0.996\). The covariance orientation changes how the observation's joint displacement is assessed, even though the data, means, weights, and component count are unchanged.

A Score-Margin Guarantee for Stable Allocations

A direct way to quantify allocation sensitivity is to compare each observation's top score with its strongest competitor. This gap is called its score margin. The following bound says exactly when bounded score changes cannot alter a hard allocation.

Theorem: For one observation, suppose model A has a unique top-scoring component \(g^\star\), with margin \(\Delta=s_{g^\star}-\max_{h\ne g^\star}s_h>0\). Suppose model B's scores satisfy \(|s'_g-s_g|\le\varepsilon\) for every component \(g\). If \(\varepsilon<\Delta/2\), the top-scoring component under model B is also \(g^\star\).

Proof. For any competitor \(h\ne g^\star\), the perturbation bound gives \(s'_{g^\star}\ge s_{g^\star}-\varepsilon\) and \(s'_h\le s_h+\varepsilon\). Therefore

s'_{g^\star}-s'_h \ge s_{g^\star}-s_h-2\varepsilon \ge \Delta-2\varepsilon >0.

So \(g^\star\) still scores strictly higher than every competitor, and the allocation is unchanged. \(\square\)

The condition is sufficient, not necessary: an allocation may remain unchanged even if the bound does not apply. It is also strict. At \(\varepsilon=\Delta/2\), the inequalities only ensure that the former winner's score is at least tied with a competitor, not strictly higher.

Worked Example: Applying the Strict Margin Condition

Suppose an observation's top score exceeds every competing score by \(\Delta=0.7\). What score-change bound guarantees that its allocation is unchanged?

The theorem requires \(\varepsilon<\Delta/2\), so here it guarantees stability when \(\varepsilon<0.7/2=0.35\). The supremum of the guaranteed values is \(0.35\), but there is no largest guaranteed value: at \(\varepsilon=0.35\), the bound permits a tie. For example, if \(\varepsilon=0.30\), the new top-score advantage is at least \(0.7-2(0.30)=0.10>0\), so the allocation is guaranteed to remain unchanged.

Posterior Memberships Can Change Without an Allocation Switch

The same score comparison yields a bound for posterior odds. This is useful because a hard allocation can conceal a substantial change in uncertainty.

Theorem: Suppose, for every component \(g\), two fits' scores for an observation satisfy \(|s'_g-s_g|\le\varepsilon\). For any pair of components \(g,h\), their posterior membership odds satisfy

e^{-2\varepsilon} \le \frac{\hat{\tau}'_g/\hat{\tau}'_h}{\hat{\tau}_g/\hat{\tau}_h} \le e^{2\varepsilon}.

Proof. In either fit, the posterior odds for \(g\) against \(h\) equal the exponential of the score difference: \(\hat{\tau}_g/\hat{\tau}_h=\exp(s_g-s_h)\). Hence the ratio of the two fits' odds is

\frac{\hat{\tau}'_g/\hat{\tau}'_h}{\hat{\tau}_g/\hat{\tau}_h} =\exp\{(s'_g-s_g)-(s'_h-s_h)\}.

Each score change lies between \(-\varepsilon\) and \(\varepsilon\), so their difference lies between \(-2\varepsilon\) and \(2\varepsilon\). Exponentiating gives the stated bounds. \(\square\)

This result controls pairwise odds, not the absolute difference between posterior probabilities. The denominator that normalizes all component scores matters for absolute probabilities, while it cancels exactly in pairwise odds.

Worked Example: Stable Winner, Less Stable Membership

Suppose a two-component fit gives posterior memberships \(0.80\) and \(0.20\), so the odds for component 1 are \(0.80/0.20=4\). If each component score changes by at most \(\varepsilon=0.10\), the odds under the second fit lie between \(4e^{-0.20}\) and \(4e^{0.20}\), or approximately \(3.275\) and \(4.886\).

For two components, odds \(r\) correspond to membership \(r/(1+r)\). The possible component 1 memberships therefore range from approximately \(3.275/4.275=0.766\) to \(4.886/5.886=0.830\). These bounds preserve component 1 as the winner, but they show that its reported membership need not remain exactly \(0.80\).

Compare the Estimated Geometry, Not Just the Labels

A covariance matrix describes a component's scale, orientation, and shape. For a positive-definite covariance estimate, its eigenvectors give the principal directions and its eigenvalues give the variances along those directions. The determinant measures the squared volume scale of the covariance ellipsoid; eigenvalue ratios describe relative elongation. These summaries answer different questions, so comparing only determinants or only allocations can miss important sensitivity.

Worked Example: Same Eigenvalues, Different Orientation

Compare \(\Sigma_A=\begin{pmatrix}4&0\\0&1\end{pmatrix}\) and \(\Sigma_B=\begin{pmatrix}2.5&1.5\\1.5&2.5\end{pmatrix}\). For \(\Sigma_A\), the eigenvalues are \(4\) and \(1\), the principal axes are aligned with the coordinate axes, and \(|\Sigma_A|=4\). For \(\Sigma_B\), the vectors \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\) are eigenvectors with eigenvalues \(4\) and \(1\), respectively; its determinant is \(2.5(2.5)-1.5^2=4\). The two covariances have the same eigenvalues and determinant, but their principal directions differ by \(45\) degrees.

For \(x=(1,-1)^\mathsf{T}\) relative to a component mean of zero, the squared Mahalanobis distance under \(\Sigma_A\) is \(1^2/4+(-1)^2/1=1.25\). Under \(\Sigma_B\), \(x\) lies along the eigenvector with eigenvalue \(1\), so its squared Mahalanobis distance is \(1^2+(-1)^2=2\). Thus identical eigenvalues and determinant do not imply identical fit to a particular observation: orientation matters.

How to Report a Sensitivity Analysis

A clear report separates the choices that were deliberately held fixed from those that changed. For each covariance parameterization, summarize the fitted geometry and membership changes using a consistent set of diagnostics. When components are aligned, useful summaries include the proportion of observations whose hard allocation changes, changes in posterior memberships, and changes in component covariance eigenvalues and orientations. A score-margin summary helps distinguish robust allocations from those near a decision boundary.

Freeze the comparison conditions.
Use the same observations, variables, preprocessing, component count for the focused comparison, and fitting and restart rules.

Fit the covariance alternatives.
Record convergence status and the estimates needed to compute scores, memberships, and component geometry.

Align labels and compare outcomes.
State the label-matching rule, then compare hard allocations, posterior memberships, score margins, and covariance summaries.

Interpret the scope of agreement.
Report which conclusions are stable and which depend on the covariance choice; do not treat a single selected fit as proof that its geometry is uniquely supported.

The phrase “holding the selection procedure fixed” means using the same stated procedure, not necessarily forcing all outcomes of that procedure to be identical. If a criterion is rerun and selects different component counts across covariance families, report that as a pipeline-level sensitivity result. To isolate covariance effects on memberships and geometry, also compare fits at a common component count.

Common Pitfalls

Using the same hard allocation as evidence of full stability. The winning component can remain the same while posterior memberships change substantially. Compare the memberships as well as the labels.
Calling covariance matrices “close” based only on their entries or determinants. The determinant ignores orientation, and small matrix changes can have a large effect when a covariance is nearly singular. Check eigenvalues, orientations, and the resulting scores.
Comparing component numbers without aligning labels. A relabeling can make identical fits appear different. Apply and report a consistent matching rule before comparing labels or component-specific estimates.
Changing several analysis choices at once. If preprocessing, component count, initialization, and covariance family all change, the source of a different result is unclear. Separate controlled comparisons from full-pipeline sensitivity checks.

Key takeaway: Covariance-model sensitivity is about both assignment and geometry. Compare plausible covariance fits under controlled conditions, use score margins to assess allocation guarantees, and examine posterior memberships and covariance shape even when the hard labels agree.

Check Your Understanding

Answer these questions using the definitions and results above.

Why can two covariance parameterizations assign the same observation to different components even when their means and mixing proportions are identical?
A top score has margin \(0.7\). State the condition on \(\varepsilon\) that guarantees its allocation is unchanged, and give the supremum of those guaranteed values.
If each of two component scores changes by at most \(0.2\), what multiplicative bounds apply to the change in their posterior odds?
Why can covariance matrices with the same determinant and eigenvalues still give different Mahalanobis distances for an observation?
Name two analysis choices, other than covariance parameterization, that should be held fixed in a focused sensitivity comparison.