Comparing Cluster Counts With Bayesian Predictive Criteria

Why Compare Cluster Counts?

A clustering model can describe the same multi-omics data with different numbers of clusters. A larger count may capture additional structure, but it also changes the model’s flexibility and the way uncertainty is represented. The goal of comparing candidate counts is therefore not simply to reward a better in-sample fit. It is to assess how well each count is supported after accounting for model complexity, or how well it predicts data not used to fit it.

Let $Y=(Y_1,\ldots,Y_n)$ denote the observed multi-omics data for $n$ subjects, with each $Y_i$ containing the measured modalities for subject $i$. Let $K$ be a candidate number of clusters, and let $\theta_K$ collect the model parameters under that candidate. A model may also include latent cluster labels $z=(z_1,\ldots,z_n)$. In a finite mixture with exactly $K$ components, $K$ is the number of components in the model; it need not equal the number of occupied clusters in a particular posterior draw. Make this distinction explicit when comparing an overfitted finite mixture or a nonparametric mixture, where the number of occupied clusters is itself variable.

Definition: The marginal likelihood (also called the model evidence) for candidate $K$ is the probability density of the observed data after averaging over the model’s parameter and latent-variable uncertainty under its prior: $$ p(Y\mid K)=\sum_z\int p(Y\mid z,\theta_K,K)\,p(z,\theta_K\mid K)\,d\theta_K. $$ For continuous data this is a density, not generally a probability of the exact observed data. The Bayes factor comparing $K_a$ with $K_b$ is $BF_{a,b}=p(Y\mid K_a)/p(Y\mid K_b)$.

This expression assumes that each candidate specifies a proper prior distribution for all parameters and latent variables being integrated out, as well as a well-defined sampling model for the observed data. The likelihood must represent the chosen modalities and their dependence structure. For example, if the model factors the modalities conditionally on a subject’s cluster and modality-specific parameters, that factorization is a modeling assumption; it is not guaranteed merely because several data types were measured.

Marginal Likelihoods and Posterior Support

The evidence averages the likelihood over the prior rather than evaluating it only at a best-fitting parameter value. Thus a model that fits well only in a small portion of its prior parameter space may receive less evidence than one that fits nearly as well across a broader portion. This averaging is one way the marginal likelihood accounts for model flexibility.

To update prior probabilities on candidate counts, Bayes’ rule gives

P(K\mid Y)=\frac{p(Y\mid K)P(K)}{\sum_{k\in\mathcal{K}}p(Y\mid k)P(k)},

where $\mathcal{K}$ is the set of candidate counts. For two candidates, posterior odds equal prior odds multiplied by the Bayes factor:

\frac{P(K_a\mid Y)}{P(K_b\mid Y)} = BF_{a,b}\frac{P(K_a)}{P(K_b)}.

A Bayes factor is therefore evidence conditional on the specified models and priors, not a prior-free measure of which cluster count is true. If the count prior is uniform over the candidates, the posterior odds happen to equal the Bayes factor; with unequal count priors, they do not.

Worked Example: Updating Support With a Bayes Factor

Suppose two candidate models have estimated log marginal likelihoods $-103.2$ for $K=2$ and $-101.7$ for $K=3$. Assume equal prior probabilities for these two counts and that the evidence calculations are sufficiently accurate for this comparison.

The log Bayes factor for $K=3$ against $K=2$ is

\log BF_{3,2}=-101.7-(-103.2)=1.5, \qquad BF_{3,2}=\exp(1.5)\approx 4.482.

With equal prior odds, the posterior odds are also about $4.482:1$. The posterior probability of $K=3$, considering only these two candidates, is

P(K=3\mid Y)=\frac{4.482}{1+4.482}\approx 0.8176.

Under the stated model and prior, the data therefore provide about 4.5 times as much marginal-likelihood support for $K=3$ as for $K=2$, and the posterior probability of $K=3$ among these two candidates is about 0.818. This does not show that three clusters are biologically real, and the conclusion could change if other counts or different priors were included.

Held-Out Predictive Comparisons

A predictive comparison asks how well a candidate model predicts observations not used to fit it. A holdout split separates training data $Y_{\mathrm{train}}$ from test data $Y_{\mathrm{test}}$. For each $K$, the relevant predictive distribution averages over uncertainty remaining after training:

p(Y_{\mathrm{test}}\mid Y_{\mathrm{train}},K) = \int p(Y_{\mathrm{test}}\mid\theta_K,K)\, p(\theta_K\mid Y_{\mathrm{train}},K)\,d\theta_K.

For a mixture, this prediction must also account for the unknown cluster membership of each held-out subject. That usually means averaging over possible labels under the model’s predictive distribution, rather than assigning each test subject to a cluster using the full data. The training procedure, transformations, feature selection, and any tuning must be confined to the training portion of each split; otherwise, information from the test data leaks into fitting.

One commonly reported summary is the summed pointwise log predictive score,

S_{\mathrm{point}}(K)=\sum_{i\in\mathrm{test}}\log p(Y_i\mid Y_{\mathrm{train}},K).

Here each $Y_i$ should be defined consistently—for example, as all modalities measured on one held-out subject. This is a valid sum of log predictive densities, and larger values indicate better pointwise predictive performance on the chosen data representation. However, it is not automatically the log of a joint predictive density for all test subjects. In general, posterior uncertainty in shared parameters induces dependence among future observations after those parameters are integrated out.

Important distinction: Conditional on fixed parameters, observations may be independent, so their conditional joint density can factor. After integrating over shared uncertain parameters, the joint posterior predictive density generally does not factor into the product of its individual posterior predictive densities. Use $\log p(Y_{\mathrm{test}}\mid Y_{\mathrm{train}},K)$ for a joint log predictive score. Call the sum of individual log predictive densities a joint score only when the posterior predictive observations actually factor, or when the score is explicitly defined as a sum of pointwise scores.

Worked Example: Comparing Pointwise Held-Out Scores

Suppose the same six subjects are held out for each of three candidate counts. After fitting each model on the training subjects, the summed pointwise log predictive scores are $-18.6$ for $K=2$, $-16.9$ for $K=3$, and $-17.5$ for $K=4$. Assume the same response variables, transformations, held-out subjects, and density units are used for all three candidates.

The scores can be compared directly because they evaluate the same held-out observations on the same scale. The largest score is $-16.9$, so $K=3$ performs best by this pointwise criterion. The differences are

S_{\mathrm{point}}(3)-S_{\mathrm{point}}(2)=1.7, \qquad S_{\mathrm{point}}(3)-S_{\mathrm{point}}(4)=0.6.

These are differences in summed log predictive density, not probabilities that a count is correct. The relatively small difference of $0.6$ from $K=4$ should not be treated as decisive without considering uncertainty across splits or folds. Also, unless the posterior predictive test observations factor, $-16.9$ is a summed pointwise score, not necessarily the log joint density of all six test subjects.

Why Posterior Predictive Scores Need Care

The distinction between conditional and posterior predictive independence is important whenever subjects share unknown parameters. For example, suppose a shared probability $\theta$ has a uniform prior, $\theta\sim\mathrm{Beta}(1,1)$, and two future binary observations are conditionally independent Bernoulli$(\theta)$. For either observation, the posterior predictive probability of a one is $E(\theta)=1/2$. Multiplying the two marginal predictive probabilities gives $1/4$, but this product is not the joint predictive probability that both are one.

Worked Example: Shared Parameter Uncertainty Changes a Joint Score

With no observations yet, the joint predictive probability of two ones is obtained by averaging the conditional joint probability over the prior:

P(X_1=1,X_2=1)=E(\theta^2) =\int_0^1\theta^2\,d\theta =\frac{1}{3}.

The conditional joint probability is $\theta^2$ because $X_1$ and $X_2$ are conditionally independent given $\theta$. In contrast, multiplying their individual posterior predictive probabilities gives

P(X_1=1)P(X_2=1)=\left(\frac{1}{2}\right)\left(\frac{1}{2}\right)=\frac{1}{4},

which differs from $1/3$. The joint log predictive density for the outcome $(1,1)$ is $\log(1/3)\approx-1.099$, whereas the sum of individual log predictive densities is $2\log(1/2)\approx-1.386$. The conditional independence assumption holds, but integrating over the shared uncertain probability makes the future observations dependent. In mixture models, shared mixture weights and cluster-profile parameters can create the same issue.

Estimation, Diagnostics, and Limitations

Marginal likelihoods for mixtures are often difficult to calculate because they integrate over parameters and sum over many possible label configurations. Methods such as bridge sampling, thermodynamic integration, and carefully designed importance sampling can estimate the evidence, but the estimate itself has Monte Carlo error. Evidence comparisons should therefore include checks that the calculation is stable across repeated runs or reasonable computational settings. A tiny estimated difference between counts is not informative if it is smaller than the numerical uncertainty.

Prior sensitivity is especially consequential for evidence: changing the prior’s scale or shape changes the regions over which likelihood is averaged. Proper priors are required for ordinary marginal likelihoods, and apparently reasonable but very diffuse priors can strongly affect comparisons. A large Bayes factor is conditional on the chosen prior specifications. The earlier material on sensitivity of clusters to prior choices concerns cluster results; here the same general caution applies directly to evidence and posterior count probabilities.

Held-out prediction avoids using the training fit to score the test data, but it answers a different question from evidence. A count can predict well without matching a uniquely meaningful biological partition: multiple counts may make similar predictions, and extra clusters may help prediction even when they split a group that is not scientifically useful. Conversely, the best-supported count by evidence need not maximize predictive accuracy on a particular holdout. Neither criterion by itself establishes causation or biological validity.

Use repeated cross-validation when a single split could be unrepresentative. Keep all measurements from one subject together when the prediction target is a new subject; splitting modalities from the same subject across training and test can make prediction unrealistically easy. Compare candidates on identical folds and identical outcomes. Examine the variation of score differences across folds, not only a single average. Predictive densities also depend on data scale: comparisons are meaningful when candidates score the same measured variables with consistent preprocessing and units.

Criterion	What it evaluates	Main caution
Marginal likelihood and Bayes factor	Prior-averaged support for the specified candidate model given all observed data	Can be highly sensitive to parameter and count priors, and hard to estimate accurately
Held-out pointwise log predictive score	Average predictive performance for the scored observations under the chosen split or folds	A sum of pointwise scores is not generally a joint predictive log density
Joint held-out log predictive density	Predictive probability density assigned to the full test set jointly	Requires computing the joint posterior predictive distribution, including dependence from shared uncertainty

Specify the candidates.
State whether $K$ counts mixture components or occupied clusters, and define the same observed modalities and sampling assumptions for each candidate.

Choose the comparison target.
Use evidence for posterior model support under explicit priors, or held-out scores for predictive performance on a stated prediction target.

Check the calculation.
Assess evidence-estimation stability or cross-validation variability, prevent data leakage, and distinguish pointwise from joint prediction.

Report conditional conclusions.
State which criterion favored which candidate and qualify the result by its priors, split design, uncertainty, and scientific purpose.

Common Mistake

A frequent error is to write “the joint predictive density is the product of the individual posterior predictive densities because the test subjects are conditionally independent.” This reasoning stops too early: conditional independence given parameters implies a product of conditional densities, but integrating over shared parameter uncertainty generally does not preserve that product. The correction is to compute the joint posterior predictive density by integrating the conditional joint density, or to report the sum of pointwise log predictive densities under that name and not call it a joint log score. A second error is to treat the best score as proof of the true number of biological clusters. The score supports a choice relative to specified candidates and assumptions; it does not establish a uniquely real partition.

Key takeaway: Marginal likelihoods compare prior-averaged support for specified models, while held-out scores compare predictive performance under a defined validation design. State the criterion precisely, preserve the dependence structure when scoring joint predictions, and interpret any preferred count conditionally rather than as a definitive biological truth.

Check Your Understanding

Answer each question, then check the brief solutions below.

If two candidate counts have a Bayes factor of 3 in favor of $K=4$, but the prior odds favor the other count by $2:1$, what are the posterior odds for $K=4$ versus the other count?
Why can conditional independence of test subjects given shared parameters fail to make their posterior predictive densities factor?
In cross-validation for new-subject prediction, should modalities from one subject be split across training and test sets? Why or why not?
What does a higher summed pointwise log predictive score indicate, and what does it not establish?
Name one important limitation of marginal likelihood comparisons and one limitation of held-out predictive comparisons.

Solutions. (1) Posterior odds are $3\times(1/2)=1.5:1$ in favor of $K=4$. (2) Averaging a product of conditional densities over uncertain shared parameters is generally not the product of the separately averaged densities. (3) No. Keep all modalities for a subject together so information from that subject cannot leak into training. (4) It indicates better pointwise predictive performance for the scored data under the specified split and representation; it does not prove that the count is biologically true or that the sum is a joint log score. (5) Evidence can be strongly prior-sensitive and difficult to estimate; held-out scores can vary with the split and assess prediction rather than uniquely meaningful cluster structure.