Mixed-Type Likelihoods Within a Shared Cluster Model

Why Mixed-Type Likelihoods Need a Shared Model

A sample in an integrative clustering analysis may have a continuous molecular measurement, sequencing counts, and binary indicators. These values do not share a natural measurement scale: a Gaussian density, a count probability, and a binary probability are different kinds of quantities. A shared-cluster model brings them together by asking whether each measurement is plausible under the same cluster assignment, then combining the resulting likelihood contributions.

The key modeling choice is not to force all measurements into one distribution. Instead, specify a suitable likelihood for each type and connect the modalities through a common latent cluster label for each sample. The product of the modality-specific likelihoods is justified only under stated conditional-independence assumptions.

Definition: Let \(i=1,\ldots,n\) index samples, \(m=1,\ldots,M\) index omics modalities, and \(k=1,\ldots,K\) index clusters. The latent variable \(z_i\in\{1,\ldots,K\}\) is the shared cluster assignment for sample \(i\). Within modality \(m\), let \(j\) index features. A mixed-type likelihood assigns each observed feature a distribution appropriate to its data type and multiplies those contributions conditional on \(z_i=k\) and the cluster-specific parameters.

Model and Assumptions

For clarity, consider three kinds of features. Write \(x_{ij}\) for a continuous measurement, \(y_{ij}\) for a count, and \(b_{ij}\in\{0,1\}\) for a binary measurement. Each feature has a cluster-specific parameter. For continuous features, use a Gaussian model with mean \(\mu_{kj}\) and variance \(\sigma_{kj}^{2}>0\). For counts, use a Poisson model with exposure \(e_i>0\) and rate \(\lambda_{kj}>0\), so the expected count is \(e_i\lambda_{kj}\). For binary features, use a Bernoulli model with success probability \(q_{kj}\), where \(0<q_{kj}<1\).

The core assumptions are conditional: given the cluster assignment and the model parameters, observations are independent across included features and modalities, and their distributions have the stated forms. The shared allocation assumption means that all modalities observed on sample \(i\) use the same \(z_i\). It does not mean that the different modalities have the same cluster-specific profile. The cluster-specific parameters may differ by feature and by modality.

\begin{aligned} x_{ij}\mid z_i=k &\sim N(\mu_{kj},\sigma_{kj}^{2}),\\ y_{ij}\mid z_i=k &\sim \operatorname{Poisson}(e_i\lambda_{kj}),\\ b_{ij}\mid z_i=k &\sim \operatorname{Bernoulli}(q_{kj}). \end{aligned}

Let \(O_i^{(G)}\), \(O_i^{(C)}\), and \(O_i^{(B)}\) be the sets of observed continuous, count, and binary features for sample \(i\). An observation missing from the data is omitted from the corresponding set; it is not silently assigned a value of zero. Under the assumptions above, the likelihood for the observed measurements of sample \(i\), conditional on cluster \(k\), is

\begin{aligned} L_{ik} ={}& \prod_{j\in O_i^{(G)}} \frac{1}{\sqrt{2\pi\sigma_{kj}^{2}}} \exp\!\left[-\frac{(x_{ij}-\mu_{kj})^2}{2\sigma_{kj}^{2}}\right]\\ &\times \prod_{j\in O_i^{(C)}} \frac{(e_i\lambda_{kj})^{y_{ij}} \exp(-e_i\lambda_{kj})}{y_{ij}!}\\ &\times \prod_{j\in O_i^{(B)}} q_{kj}^{b_{ij}}(1-q_{kj})^{1-b_{ij}}. \end{aligned}

The likelihood is a function of the model parameters for the observed data; it is not itself the probability that cluster \(k\) is correct. If the cluster weights \(\pi_k\) are specified, with \(\pi_k\geq0\) and \(\sum_k\pi_k=1\), the conditional probability of an assignment given these parameters is proportional to \(\pi_k L_{ik}\). This is useful for comparing clusters, but it does not account for uncertainty in the parameters.

Why the Product and Log-Likelihood Work

Under conditional independence, the joint probability or density of a collection of observations is the product of its feature-wise terms. This follows by repeated application of the definition of conditional independence: for two observations \(A\) and \(B\), \(P(A,B\mid z_i=k)=P(A\mid z_i=k)P(B\mid z_i=k)\); applying the same factorization to each additional feature gives the full product. The same factorization applies to densities when some observations are continuous.

Products of many small terms can be numerically inconvenient. Taking logarithms preserves comparisons because the logarithm is strictly increasing, and converts products into sums. Thus the log-likelihood for sample \(i\) in cluster \(k\) is the sum of the Gaussian, count, and binary log-likelihood contributions. For comparing clusters \(a\) and \(b\), the log-likelihood ratio is

\log\frac{L_{ia}}{L_{ib}} = \sum_{j\in O_i^{(G)}}\log\frac{f_G(x_{ij}\mid a)}{f_G(x_{ij}\mid b)} + \sum_{j\in O_i^{(C)}}\log\frac{f_C(y_{ij}\mid a)}{f_C(y_{ij}\mid b)} + \sum_{j\in O_i^{(B)}}\log\frac{f_B(b_{ij}\mid a)}{f_B(b_{ij}\mid b)}.

Here \(f_G\), \(f_C\), and \(f_B\) denote the corresponding Gaussian density, Poisson probability mass function, and Bernoulli probability mass function. A positive value favors cluster \(a\) over \(b\) by the likelihood ratio; a negative value favors \(b\). If cluster weights are used, the log ratio for assignment probabilities also includes \(\log(\pi_a/\pi_b)\).

Worked Example: Combining One Feature of Each Type

Suppose a hypothetical sample has one continuous measurement \(x=1.2\), one count \(y=3\) with exposure \(e_i=1\), and one binary value \(b=1\). Compare clusters A and B. Let the continuous feature have standard deviation \(0.5\) in both clusters, with means \(1\) and \(0\); let the count rates be \(2\) and \(0.5\); and let the binary success probabilities be \(0.8\) and \(0.3\), respectively. Assume conditional independence and equal cluster weights.

The Gaussian contribution to the likelihood ratio is \(\exp\{[(1.2-0)^2-(1.2-1)^2]/(2(0.5)^2)\}=\exp(2.8)\approx16.44\). The common Gaussian normalizing constant cancels. For the count, the Poisson likelihoods are \(e^{-2}2^3/3!\approx0.1804\) in A and \(e^{-0.5}(0.5)^3/3!\approx0.01264\) in B. The count likelihood ratio is therefore about \(14.28\). The binary likelihoods are \(0.8\) and \(0.3\), giving a ratio of \(2.667\).

Multiplying the three contributions gives \(L_{iA}/L_{iB}\approx16.44\times14.28\times2.667\approx626\). Equivalently, the log ratio is \(2.8+\log(14.28)+\log(2.667)\approx6.44\). With equal weights and these fixed parameters, the conditional assignment probability for A against these two clusters is \(626/(626+1)\approx0.9984\). This is strong relative evidence under the specified model, not a guarantee that the model is correct.

Worked Example: A Missing Modality Is Not a Zero

Consider a hypothetical sample with a continuous value \(x=0.5\), no observed count data, and a binary value \(b=0\). For the continuous feature, suppose both clusters have standard deviation \(0.5\), with mean \(0.5\) in A and \(0\) in B. For the binary feature, let the probability of \(1\) be \(0.2\) in A and \(0.7\) in B. Assume equal cluster weights, conditional independence, and that the missing count is ignorable for this likelihood comparison.

The Gaussian density is \(1/(\sqrt{2\pi}\,0.5)\approx0.7979\) in A. In B it is \(0.7979\exp(-0.5)\approx0.4839\). Since \(b=0\), the Bernoulli likelihoods are \(1-0.2=0.8\) and \(1-0.7=0.3\). Thus \(L_{iA}\approx0.7979\times0.8=0.6383\), and \(L_{iB}\approx0.4839\times0.3=0.1452\). The ratio is about \(4.40\), giving a conditional probability of A of \(4.40/(4.40+1)\approx0.8148\).

There is no count factor in either likelihood because no count was observed. Inserting a zero count would add a Poisson probability and would assert that a count of zero was measured; that is a different observation and can change the comparison. If the fact that a modality is missing depends on the unobserved value or on cluster membership, the missingness mechanism may need to be modeled rather than ignored.

Feature Counts, Dependence, and Diagnostics

A product likelihood does not automatically give each modality equal influence. A modality with many features can contribute many log-likelihood terms, while a modality with few features contributes fewer. This may be appropriate if the features are conditionally independent and the model is correctly specified. It can be misleading if many features are redundant or if one likelihood is poorly calibrated. The following hypothetical calculation illustrates the arithmetic.

Worked Example: Many Count Features Can Dominate

Suppose two clusters have equal weights. A sample has one Gaussian value \(x=0\) with standard deviation \(0.5\) in each cluster, but means \(0.2\) in A and \(0\) in B; one binary value \(b=1\), with success probabilities \(0.6\) and \(0.5\); and 100 count features, each observed as zero, with Poisson rates \(0.2\) in A and \(0.3\) in B and exposure \(1\). Assume all features are conditionally independent, as the model requires.

The Gaussian log-likelihood ratio is \(\{(0-0)^2-(0-0.2)^2\}/[2(0.5)^2]=-0.08\). The binary contribution is \(\log(0.6/0.5)=\log(1.2)\approx0.1823\). For each zero count, the Poisson probability is \(\exp(-\lambda)\), so its log-likelihood ratio is \(\log[\exp(-0.2)/\exp(-0.3)]=0.1\). The 100 counts contribute \(100(0.1)=10\).

The total log-likelihood ratio is \(-0.08+0.1823+10=10.1023\), so the likelihood ratio is \(\exp(10.1023)\approx2.44\times10^4\). With equal weights, the conditional probability for A is approximately \(0.99996\). Most of this evidence comes from the count block, not from an equal vote by each modality. If the count features are strongly dependent, treating all 100 terms as independent can overstate the evidence.

Useful checks therefore concern both the individual likelihoods and the factorization that combines them:

Check the measurement scale and support. A Gaussian model permits any real value; a count model permits nonnegative integers; a Bernoulli model permits only zero or one. A mismatch signals a data-coding or model problem.
Check count exposure and dispersion. Exposure must reflect the relevant sampling scale, such as library size. Counts with much greater variability than a Poisson model predicts may need a more flexible count likelihood, such as a negative-binomial model.
Check conditional dependence. Correlated features can contribute nearly repeated evidence. Residual association within clusters, or a cluster assignment driven by a redundant feature block, suggests the independence model may be too strong.
Check cluster-specific fit. Compare observed and model-implied feature distributions within clusters. Poor Gaussian residual behavior, miscalibrated binary probabilities, or systematic count discrepancies can make the combined evidence unreliable.
Check missingness and modality balance. Confirm that missing measurements are handled as missing, and inspect how each modality contributes to the total log-likelihood ratio. A dominant block is not automatically wrong, but should be understood.

These checks do not prove the model assumptions. They can reveal clear failures and motivate a revised likelihood or dependence structure. A likelihood that fits each data type marginally may still fail jointly if conditional independence is implausible.

Comparisons and Limitations

Approach	How modalities are combined	Main concern
Shared mixed-type likelihood	Type-specific likelihoods multiply conditional on one cluster label.	Depends on the component distributions and conditional-independence assumptions.
Separate clustering by modality	Each modality produces its own grouping.	Does not directly impose a common sample partition; resulting groups may disagree.
Concatenated measurements	Measurements are assembled into one feature matrix for a common procedure.	Requires a procedure suitable for heterogeneous scales and data types; rescaling alone does not make distributions equivalent.

The shared likelihood has a clear interpretation: each observed feature contributes evidence for or against a cluster under its own measurement model. Its limitations are equally important. Conditional independence can overcount evidence, incorrect distributions can distort likelihood ratios, and a shared assignment can be scientifically inappropriate when modalities genuinely imply different sample groupings. The model also does not, by itself, account for uncertainty in cluster-specific parameters or establish that clusters correspond to causal biological states. Those issues require additional modeling and scientific judgment.

Key takeaway: A mixed-type shared-cluster likelihood is built by multiplying the appropriate feature-wise likelihoods conditional on one common sample assignment. The product is justified by conditional independence, and its log form makes clear how each modality contributes to the comparison.

Check Your Understanding

Try each question, then compare your reasoning with the solutions below.

A sample has a continuous value, two observed binary values, and no recorded counts. Which terms appear in its likelihood under the mixed model?
For one binary feature with observed value \(b=0\), what likelihood contribution does a cluster with success probability \(q\) assign?
A count feature has observed value \(y=0\) and exposure \(e_i=2\). Write its Poisson likelihood contribution for cluster rate \(\lambda\).
Two clusters have equal weights and likelihoods \(L_{iA}=0.12\) and \(L_{iB}=0.03\). What is the conditional assignment probability for A among these two clusters?
Why can 50 correlated features produce overconfident cluster evidence when their likelihoods are multiplied as if conditionally independent?

Solutions. (1) Include the Gaussian density for the observed continuous value and the two Bernoulli terms for the observed binary values. Include no count term: an unrecorded count is not a zero count. (2) The contribution is \(q^0(1-q)^1=1-q\). (3) It is \((2\lambda)^0\exp(-2\lambda)/0!=\exp(-2\lambda)\). (4) The probability is \(0.12/(0.12+0.03)=0.80\). (5) Under dependence, the joint likelihood need not equal the product of the marginal likelihoods. Multiplication can count similar information repeatedly, exaggerating the likelihood ratio and the resulting assignment certainty.