Bernoulli and Categorical Likelihoods for Omics Features

Why the Measurement Type Determines the Likelihood

A cluster model describes samples through cluster-specific profiles. The likelihood for a feature must match what was measured: a mutation indicator records whether an event is present, whereas a discrete molecular measurement records one value from a set of categories. Treating either as a continuous measurement, a count, or a set of unrelated binary variables can change the implied probability model and distort how samples contribute evidence for a cluster.

The earlier material on Bayesian generative models and conditional independence established the shared-cluster setting. Here, the focus is the feature-level likelihood: how to write it, what assumptions it entails, and how to interpret its value. These likelihoods describe data conditional on cluster membership and cluster-specific parameters. They do not, by themselves, specify prior distributions for those parameters or a full clustering algorithm.

Notation: Let $i$ index samples, $k$ index clusters, and $j$ index binary features. Write $z_i=k$ when sample $i$ belongs to cluster $k$. For binary feature $j$, $x_{ij}\in\{0,1\}$, and $\theta_{kj}$ is its probability of being 1 in cluster $k$. For categorical feature $g$, $y_{ig}\in\{1,\ldots,C_g\}$, and $\pi_{kgc}$ is the probability of category $c$ for that feature in cluster $k$.

Bernoulli Likelihoods for Binary Features

A binary mutation indicator is a natural Bernoulli measurement when each sample either has or does not have the specified event. Conditional on cluster $k$, the model assigns probability $\theta_{kj}$ to presence and $1-\theta_{kj}$ to absence. The parameter is feature- and cluster-specific: a mutation can be common in one cluster and rare in another.

Definition: For binary observations $x_{ij}\in\{0,1\}$, assume samples are conditionally independent given their cluster assignments and the cluster-specific probability $\theta_{kj}$. Then $X_{ij}\mid(z_i=k,\theta_{kj})\sim\operatorname{Bernoulli}(\theta_{kj})$, with likelihood contribution $$ P(x_{ij}\mid z_i=k,\theta_{kj})=\theta_{kj}^{x_{ij}}(1-\theta_{kj})^{1-x_{ij}}. $$ For a set $I_k=\{i:z_i=k\}$ of $n_k$ samples, the feature likelihood is the product of these contributions over $i\in I_k$.

The exponent form handles both outcomes without separate equations: if $x_{ij}=1$, the contribution is $\theta_{kj}$; if $x_{ij}=0$, it is $1-\theta_{kj}$. Let $m_{kj}=\sum_{i\in I_k}x_{ij}$ be the number of positive observations. Conditional independence gives

L(\theta_{kj})=\prod_{i\in I_k}\theta_{kj}^{x_{ij}}(1-\theta_{kj})^{1-x_{ij}} =\theta_{kj}^{m_{kj}}(1-\theta_{kj})^{n_k-m_{kj}}.

This likelihood treats samples within the cluster as independent Bernoulli trials with a common probability for that feature. It does not require different features to be independent. To multiply likelihoods across features into a joint profile likelihood, however, conditional independence across those features is an additional modeling assumption.

Worked Example: A Mutation Indicator in One Cluster

Suppose a hypothetical cluster contains 10 samples, and a mutation indicator is 1 for three samples and 0 for seven. Under the Bernoulli model, the likelihood at $\theta=0.20$ for the observed sequence is

$$ L(0.20)=(0.20)^3(0.80)^7 =0.008(0.2097152) =0.00167772. $$

The likelihood at the observed positive fraction, $\hat{\theta}=3/10=0.30$, is

L(0.30)=(0.30)^3(0.70)^7 =0.027(0.0823543) \approx 0.00222357.

The observed sequence is more likely at $\theta=0.30$ than at $\theta=0.20$. These are likelihood values for the specified sequence under two parameter values; neither is the probability that a parameter value is true.

Categorical Likelihoods for Discrete Measurements

A categorical feature has one recorded level from a finite set of mutually exclusive and exhaustive categories. Examples could include a discrete assay class or a molecular state recorded as one of several labels. The labels need not have a numerical order. Even if software stores them as integers, their numeric codes should not be interpreted as measurements unless the values genuinely have a quantitative meaning.

Definition: For categorical feature $g$, assume each sample in cluster $k$ independently has one category $c\in\{1,\ldots,C_g\}$, with probabilities $\pi_{kgc}\geq 0$ satisfying $\sum_{c=1}^{C_g}\pi_{kgc}=1$. Its likelihood contribution is $$ P(y_{ig}=c\mid z_i=k,\boldsymbol{\pi}_{kg})=\pi_{kgc}. $$ For samples $I_k$, the likelihood is $L(\boldsymbol{\pi}_{kg})=\prod_{i\in I_k}\pi_{kg,y_{ig}}$.

If $n_{kgc}$ is the number of samples in cluster $k$ observed in category $c$, the likelihood can be written as

L(\boldsymbol{\pi}_{kg})=\prod_{c=1}^{C_g}\pi_{kgc}^{n_{kgc}}, \qquad \sum_{c=1}^{C_g}n_{kgc}=n_k.

A category with count zero contributes a factor of 1 to this expression, including when its probability is zero. A category with positive count and probability zero makes the likelihood zero. This distinction matters when considering boundary estimates.

Worked Example: Likelihood for a Three-Level Feature

In a hypothetical cluster of 12 samples, a discrete feature has counts 5, 4, and 3 across categories A, B, and C. Consider the profile $(\pi_A,\pi_B,\pi_C)=(0.50,0.30,0.20)$, which sums to 1. The likelihood for the observed category sequence is

$$ L=(0.50)^5(0.30)^4(0.20)^3 =0.03125(0.0081)(0.008) =0.000002025. $$

The small numerical value is not a defect: it is a probability for one particular sequence of 12 category outcomes. Many other sequences have positive probability under the same profile. The likelihood is useful for comparing profiles for the same data, not as a standalone measure of model adequacy.

Maximum-Likelihood Profiles and Their Boundaries

The maximum-likelihood estimate is the parameter value that maximizes the likelihood for the observed data. For a Bernoulli feature with $0<m_{kj}<n_k$, the log likelihood is

\ell(\theta_{kj})=m_{kj}\log(\theta_{kj})+(n_k-m_{kj})\log(1-\theta_{kj}).

Differentiating and setting the derivative to zero gives

\frac{d\ell}{d\theta_{kj}} =\frac{m_{kj}}{\theta_{kj}}-\frac{n_k-m_{kj}}{1-\theta_{kj}}=0 \quad\Longrightarrow\quad \hat{\theta}_{kj}=\frac{m_{kj}}{n_k}.

The log likelihood is strictly concave for $0<\theta_{kj}<1$ when both outcomes occur, so this stationary point is the maximum. If every observation is 0, the maximum is at $\hat{\theta}_{kj}=0$; if every observation is 1, it is at $\hat{\theta}_{kj}=1$. These are legitimate boundary maximum-likelihood estimates, not computational errors.

For a categorical feature, the maximum-likelihood profile is the observed category proportion in each category. The boundary case requires care when some categories were never observed. Let $A$ be the set of categories with positive counts, and let $N=n_k>0$. Any profile that assigns probability zero to an observed category has likelihood zero. A profile assigning positive probability to every category in $A$ has positive likelihood, so a zero-likelihood profile cannot maximize it. Now consider a profile with positive probability on all categories in $A$, and set $s=\sum_{c\in A}\pi_{kgc}$. If $s=0$, then the likelihood is zero, already ruled out as a maximum. If $0<s<1$, replace each probability on $A$ by $\pi_{kgc}/s$ and set probabilities outside $A$ to zero. This remains a valid profile, and its likelihood is multiplied by $s^{-N}>1$. Thus a maximizing profile must put all probability on the observed categories, so $s=1$.

On that constrained set, maximizing the log likelihood $\sum_{c\in A}n_{kgc}\log(\pi_{kgc})$, subject to $\sum_{c\in A}\pi_{kgc}=1$, gives the result directly. The Lagrange equations are $n_{kgc}/\pi_{kgc}=\lambda$ for every $c\in A$. Summing the resulting probabilities shows $\lambda=N$, and hence

\hat{\pi}_{kgc}=\frac{n_{kgc}}{N}\quad(c\in A), \qquad \hat{\pi}_{kgc}=0\quad(c\notin A).

The log likelihood is strictly concave over the positive probabilities for observed categories, so this is the maximizing profile. If $N=0$, there are no observations in that cluster for this feature and its likelihood is an empty product equal to 1; the data provide no maximum-likelihood information about its category probabilities.

Worked Example: Estimating a Categorical Profile

Return to the hypothetical counts $(5,4,3)$ for categories A, B, and C. The likelihood-maximizing probabilities are the observed proportions:

\hat{\pi}_A=\frac{5}{12}=0.4167,\qquad \hat{\pi}_B=\frac{4}{12}=0.3333,\qquad \hat{\pi}_C=\frac{3}{12}=0.2500.

They sum to $1.000$, as required. If a fourth category D is part of the defined set but has count zero, its maximum-likelihood probability is 0 in this sample. That boundary estimate does not establish that category D is impossible in the population; it means only that this likelihood, with these observations, is maximized at zero for D.

Using Likelihood Contributions and Checking Fit

For a known feature value, a cluster-specific likelihood contribution describes how compatible that value is with the cluster profile. For a binary feature, a positive observation contributes $\theta_{kj}$ and a negative one contributes $1-\theta_{kj}$. For a categorical feature, observing category $c$ contributes $\pi_{kgc}$. Comparing these contributions across clusters can show which cluster profile better predicts an observation for that feature. A likelihood contribution alone is not a posterior cluster probability: posterior assignment also depends on cluster prevalence and the likelihood contributions from other modeled features.

Worked Example: A Categorical Observation Across Two Clusters

Suppose a hypothetical discrete feature has three categories. For a new sample, the observed value is B. Cluster 1 has profile $(0.60,0.30,0.10)$, cluster 2 has profile $(0.20,0.70,0.10)$, and their prior assignment probabilities are $0.40$ and $0.60$. The feature likelihood contributions for B are $0.30$ and $0.70$. Weighting by the given assignment probabilities gives

0.40(0.30)=0.12,\qquad 0.60(0.70)=0.42.

After normalizing these two values, the assignment probability for cluster 1 based on this feature is $0.12/(0.12+0.42)=0.2222$; for cluster 2 it is $0.42/0.54=0.7778$. This calculation assumes these are the only clusters and this is the only feature being used. It illustrates why a larger feature likelihood does not automatically imply a larger posterior assignment probability when cluster prevalences differ.

Diagnostics should ask whether the recorded data and the assumed sampling process agree. For binary features, inspect whether the indicator truly represents two outcomes for each sample, and whether a single probability within each cluster is plausible. For categorical features, verify that categories are mutually exclusive, exhaustive, and consistently coded. A missing measurement is not automatically a category: coding missingness as a level changes the model to one that treats it as an observed outcome. Likewise, a category that combines biologically distinct states may conceal meaningful heterogeneity.

The conditional-independence assumption also deserves scrutiny. If samples within a cluster remain dependent for a feature, or if features are strongly dependent but a joint likelihood multiplies them as independent, the model can overstate the evidence contributed by repeated or related measurements. Earlier material on posterior predictive checks provides a framework for comparing observed data features with replicated data from a fitted model. Unexpected category frequencies, excessive clusters of positive indicators, or residual patterns can signal an inadequate profile model.

Choosing Between Bernoulli and Categorical Models

Use a Bernoulli likelihood when a feature records one yes/no outcome, such as presence versus absence of a specified mutation. Use a categorical likelihood when each observation falls into exactly one of several distinct levels. These models are not interchangeable merely because software can represent categories in different ways.

Data structure	Appropriate feature likelihood	Important condition
One binary event per sample	Bernoulli	The two outcomes are defined consistently.
Exactly one of several discrete levels	Categorical	Levels are mutually exclusive and exhaustive.
Several mutation indicators that can co-occur	Separate Bernoulli feature likelihoods may be used	A joint product additionally assumes conditional independence across indicators.
A count or continuous measurement	Not Bernoulli or categorical by default	Choose a likelihood that matches its measurement scale and sampling process.

Encoding a three-level measurement as three one-hot binary indicators and multiplying independent Bernoulli likelihoods is generally not a valid substitute for a categorical likelihood: the independent model assigns positive probability to impossible outcomes such as two levels occurring at once. Conversely, a set of mutation indicators that can co-occur should not be forced into one categorical variable, because that would discard combinations. If levels have a meaningful order and the ordering itself matters, an ordinal model may be a more suitable alternative than a nominal categorical model.

Common Pitfalls and Limitations

Confusing likelihood with probability of a model. A likelihood compares parameter values for fixed observed data; it is not the probability that a parameter or cluster is true.
Ignoring boundary estimates. All-zero or all-one binary observations yield boundary maximum-likelihood estimates. An unobserved categorical level likewise receives zero probability under the unconstrained maximum-likelihood profile.
Using arbitrary numeric codes as measurements. Category labels such as 1, 2, and 3 do not make the variable quantitative or ordered by themselves.
Treating missingness as an ordinary level. This changes the model and can make missingness appear to be a molecular state.
Multiplying dependent feature likelihoods without justification. The resulting joint likelihood can count correlated evidence more than once.

These likelihoods are deliberately simple. A single cluster-specific probability for a binary feature or a single category-probability vector may not capture within-cluster heterogeneity, dependence, or measurement error. The likelihood specification should therefore be treated as a scientific assumption to assess, not as an automatic consequence of the data format.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A cluster has 8 samples, with a binary feature positive in 2. What is its Bernoulli maximum-likelihood estimate?
A categorical feature has counts $(6,2,0)$ across three levels in a cluster of 8. Give the maximum-likelihood profile.
Why is a three-level measurement generally not modeled as three independent Bernoulli indicators constrained only by observed one-hot coding?
For a negative binary observation, what likelihood contribution does cluster $k$'s profile assign, and what parameter is required?

Solutions. (1) The estimate is the positive fraction, $\hat{\theta}=2/8=0.25$. (2) The maximum-likelihood profile is $(6/8,2/8,0/8)=(0.75,0.25,0)$; the probabilities are nonnegative and sum to 1. (3) Independent Bernoulli indicators can assign positive probability to impossible outcomes with multiple levels present, whereas the categorical model assigns probability to exactly one level. (4) The contribution is $1-\theta_{kj}$, because a negative observation has $x_{ij}=0$ in the Bernoulli likelihood.