Collapsed Sampling of Cluster Labels

Why Integrate Cluster Profiles Out?

In a shared-cluster model, each sample has one label that links its measurements across omics blocks. A Gibbs update for a label can condition on the cluster-specific parameters, as in the parameter-conditional approach. Collapsed sampling instead integrates those parameters out and uses the evidence the candidate cluster’s existing members provide for the sample’s measurements. This can reduce dependence between labels and nuisance parameters in a Markov chain, although it does not guarantee rapid or reliable exploration.

The key calculation is an integrated predictive probability: for each candidate cluster, ask how probable the sample’s measurements are after averaging over uncertainty in that cluster’s profile. The label update combines this evidence across modalities with the cluster’s allocation probability. The result is not a product of plug-in likelihoods evaluated at estimated profiles.

Definition: Let $z_i\in\{1,\ldots,K\}$ be the shared cluster label for sample $i$, and let $x_i^{(m)}$ denote its data in omics block $m$, for $m=1,\ldots,M$. Write $D_k^{(m)}$ for the observations in block $m$ currently assigned to cluster $k$. The integrated cluster marginal likelihood is $$ m_k(D_k)=\int p(D_k\mid\theta_k)\,p(\theta_k)\,d\theta_k. $$ It averages the likelihood over the prior for the cluster-specific parameters $\theta_k$. The posterior predictive probability for sample $i$ in block $m$, given the other assigned observations in cluster $k$, is the ratio of integrated marginals $$ p(x_i^{(m)}\mid D_{k,-i}^{(m)}) =\frac{m_k^{(m)}(D_{k,-i}^{(m)}\cup\{x_i^{(m)}\})} {m_k^{(m)}(D_{k,-i}^{(m)})}. $$ Here $D_{k,-i}^{(m)}$ excludes sample $i$, if it was assigned to $k$ before its update.

Assumptions and the Collapsed Allocation Rule

For a finite mixture, suppose the allocation probabilities are $\boldsymbol{\pi}=(\pi_1,\ldots,\pi_K)$, with prior $\boldsymbol{\pi}\sim\operatorname{Dirichlet}(\alpha_1,\ldots,\alpha_K)$, where every $\alpha_k>0$. Conditional on $\boldsymbol{\pi}$, labels are independent categorical draws with probabilities $\boldsymbol{\pi}$. Let $n_{k,-i}$ be the number of samples other than $i$ currently assigned to $k$, and let $\alpha_0=\sum_{k=1}^K\alpha_k$.

Assume the likelihood factors across samples conditional on their labels and cluster profiles. For the simple product form below, also assume that, conditional on a cluster label, the modality-specific likelihoods and their parameter priors factor across blocks (and across features when features are modeled separately). These assumptions permit multiplying modality-specific integrated predictive probabilities. If a model includes dependencies across modalities, their joint likelihood and prior must be integrated together instead.

Result: After integrating out the mixture weights and cluster-specific parameters, the conditional probability of assigning sample $i$ to candidate cluster $k$ is proportional to $$ P(z_i=k\mid z_{-i},X) \ \propto\ (n_{k,-i}+\alpha_k) \prod_{m=1}^{M} p(x_i^{(m)}\mid D_{k,-i}^{(m)}). $$ Normalize these nonnegative weights over $k=1,\ldots,K$ to obtain probabilities that sum to one. If a block is genuinely unobserved for sample $i$, its likelihood contribution is omitted, equivalently set to one, provided the missingness mechanism does not itself carry modeled information about the label.

The count term rewards clusters with greater allocation probability under the posterior for $\boldsymbol{\pi}$. Each predictive term measures how well the candidate cluster predicts the sample’s data in that block, with parameter uncertainty included. A low count can be outweighed by strong evidence across the modalities; neither component alone determines the update.

Why the Formula Holds

The justification follows by taking a ratio of collapsed joint probabilities. Integrating a Dirichlet prior against categorical labels gives an allocation probability proportional to $\prod_k \Gamma(\alpha_k+n_k)/\Gamma(\alpha_k)$, with a common denominator $\Gamma(\alpha_0+n)$. When comparing the assignment $z_i=k$ with the label removed, only cluster $k$’s factor changes. The ratio is $\Gamma(\alpha_k+n_{k,-i}+1)/\Gamma(\alpha_k+n_{k,-i})=\alpha_k+n_{k,-i}$; the common normalizing terms do not depend on the candidate $k$.

For the data, conditional independence across samples means that moving sample $i$ into $k$ changes only cluster $k$’s integrated marginal likelihood. Its ratio is the integrated marginal with $x_i$ included divided by the marginal without it. Under the stated factorization across modalities, this ratio is the product of the modality-specific predictive ratios. Multiplying the allocation ratio and data ratio gives the displayed rule. This derivation also explains why the old assignment of $i$ must be removed first: otherwise its own measurements and count would be used to support that same assignment.

Worked Example: A Beta-Bernoulli Predictive Probability

Suppose one hypothetical omics feature is binary, with value 1 indicating detection. In a candidate cluster, four other samples have values $1,1,1,0$. The cluster’s detection probability $\theta_k$ has prior $\theta_k\sim\operatorname{Beta}(1,1)$. Find the integrated predictive probability that the sample being updated has value 1.

With $s=3$ detections and $f=1$ non-detection, conjugacy gives the posterior $\theta_k\mid D_k\sim\operatorname{Beta}(1+s,1+f)=\operatorname{Beta}(4,2)$. The predictive probability is the posterior mean:

p(x_i=1\mid D_k) =\int_0^1 \theta_k\,p(\theta_k\mid D_k)\,d\theta_k =\frac{4}{4+2} =\frac{2}{3} \approx 0.667.

Thus this candidate cluster contributes a predictive factor of 0.667 for this feature. The probability is not obtained by substituting the observed proportion $3/4$ as if it were known: the prior contributes one pseudo-count to each outcome, and the predictive value averages over posterior uncertainty in $\theta_k$.

Worked Example: A Normal-Normal Predictive Probability Density

Consider one continuous feature in a hypothetical omics block. Measurements in a cluster are conditionally independent with $x\mid\mu_k\sim N(\mu_k,1)$, where the variance is known to be $1$. Let the prior be $\mu_k\sim N(0,4)$. Two other cluster members have measurements $1$ and $3$. Calculate the predictive density at a proposed measurement $x_i=2$.

The posterior variance of $\mu_k$ is $(1/4+2)^{-1}=4/9$, and its posterior mean is $(1+3)/(1/4+2)=16/9$. Therefore the predictive distribution for a new measurement is normal with mean $16/9$ and variance $1+4/9=13/9$. The extra $1$ in the variance is the sampling variance of a new measurement, not uncertainty about the cluster mean.

p(x_i=2\mid D_k) =\frac{1}{\sqrt{2\pi(13/9)}} \exp\left\{-\frac{(2-16/9)^2}{2(13/9)}\right\} \approx 0.326.

This is a density, not a probability of observing exactly 2; its units are inverse units of the measured feature. If the feature is measured in log-expression units, the density is per log-expression unit. For comparison, a candidate cluster with measurements $-1$ and $0$ has posterior mean $-4/9$ and the same predictive variance $13/9$; its density at 2 is approximately $0.0419$. The first cluster predicts this value more strongly, but an allocation update must also include cluster counts and evidence from the other blocks.

Worked Example: Combining Two Omics Blocks

Suppose two candidate clusters have counts $n_{A,-i}=3$ and $n_{B,-i}=2$, with $\alpha_A=\alpha_B=1$. For each block, a binary feature has a $\operatorname{Beta}(1,1)$ prior for its cluster-specific detection probability. In a hypothetical RNA block, cluster A has two detections and one non-detection, while cluster B has zero detections and two non-detections. In a hypothetical protein block, cluster A has one detection and two non-detections, while cluster B has two detections and zero non-detections. The sample being updated has a detection in RNA and a non-detection in protein. Assume the two block-specific parameters are independent within each cluster, as well as the observations conditional on those parameters.

For cluster A, the RNA predictive probability of detection is $(2+1)/(3+2)=3/5$. Its protein predictive probability of non-detection is $(2+1)/(3+2)=3/5$. The product of the two block contributions is $9/25=0.36$. For cluster B, the corresponding probabilities are $(0+1)/(2+2)=1/4$ and $(0+1)/(2+2)=1/4$, so their product is $1/16=0.0625$.

The unnormalized allocation weights include the count terms:

w_A=(3+1)(0.36)=1.44,\qquad w_B=(2+1)(0.0625)=0.1875.

The normalizing sum is $1.44+0.1875=1.6275$. Hence $P(z_i=A\mid\cdots)=1.44/1.6275\approx0.885$, and $P(z_i=B\mid\cdots)=0.1875/1.6275\approx0.115$. These are conditional probabilities for this Gibbs update under the specified model and current labels of the other samples. They are not fixed cluster classifications: a sampler draws a label from this distribution.

Diagnostics, Alternatives, and Limitations

Before trusting a collapsed update, check both the model factorization and the computation. In particular, verify that sample $i$ has been removed from its old cluster’s counts and sufficient statistics, that the same observations are used in every modality, and that each candidate’s predictive factor uses the correct posterior distribution and measurement scale. A useful numerical check is that every weight is nonnegative, at least one is positive, and normalization gives probabilities summing to one. For a block with no observed data for sample $i$, confirm that the model justifies omitting that block rather than treating missingness as an ordinary non-detection.

A common failure is to multiply modality-specific predictive probabilities when the model has not made the corresponding conditional-independence assumption. For example, correlated blocks with shared cluster parameters generally require a joint predictive distribution; multiplying separate marginal predictions can count the same evidence more than once. Another warning sign is a predictive calculation that ignores uncertainty in cluster parameters, or that uses sample $i$’s own measurement while evaluating its proposed cluster. These errors change the conditional distribution the sampler is meant to use.

Collapsed and parameter-conditional Gibbs updates target the same posterior when each is implemented under the same model. The former averages over cluster parameters at each label update; the latter samples or conditions on those parameters. Collapsing often reduces dependence between labels and parameters, but predictive evaluations can become more expensive, especially for large feature sets. A plug-in allocation rule that substitutes estimated profiles is not the same posterior update: it omits parameter uncertainty and generally does not define a draw from the model’s full conditional distribution.

The result also relies on a correctly specified likelihood and prior. Conditional independence does not make the modalities scientifically independent, nor does it guarantee that a shared partition is appropriate. Strongly conflicting signals, unmodeled dependence, or poorly chosen profile distributions can produce misleading allocation probabilities. Collapsing is a computational choice, not a remedy for model misspecification, and it does not remove label symmetry or ensure that a chain explores separated regions of the posterior.

For a Dirichlet-process mixture rather than a fixed finite mixture, the allocation prior has a different form: an occupied cluster receives a weight proportional to its count, and a new cluster receives a weight proportional to the concentration parameter, multiplied by its prior predictive data probability. That is a distinct allocation rule; the finite-$K$ expression here uses the Dirichlet prior specified above.

Key takeaway: A collapsed label update balances the candidate cluster’s allocation weight with its integrated predictive evidence. Under conditional independence across blocks, multiply the block-specific predictions; otherwise derive a joint predictive term from the actual model.

Check Your Understanding

Answer each question, then compare with the solutions.

A cluster has five other samples, three with a binary feature equal to 1 and two equal to 0. With a $\operatorname{Beta}(1,1)$ prior, what is the predictive probability that the updated sample has value 0?
Two candidate clusters have counts 4 and 1, with equal Dirichlet parameters of 1. Their combined integrated predictive factors across all observed blocks are 0.10 and 0.50, respectively. Calculate and normalize their allocation weights.
Why must sample $i$ be removed from its old cluster before computing its label probabilities?
Two blocks share a cluster-specific latent parameter and are conditionally dependent under the model. Is multiplying their separately integrated predictive probabilities justified? Explain.

Solutions. (1) The posterior is $\operatorname{Beta}(4,3)$, so the predictive probability of 0 is $3/(4+3)=3/7\approx0.429$. (2) The unnormalized weights are $(4+1)(0.10)=0.50$ and $(1+1)(0.50)=1.00$. Their sum is $1.50$, giving probabilities $0.50/1.50=0.333$ and $1.00/1.50=0.667$. (3) Otherwise its own measurements and its existing membership would contribute evidence for retaining the label, rather than calculating the conditional distribution given the other samples. (4) No. The factorization assumption fails; use the joint likelihood and integrate the shared parameter to obtain a joint predictive probability.

Collapsed Sampling of Cluster Labels

What You'll Learn