Dirichlet-Multinomial Likelihood For Category Counts

Why Extend the Multinomial?

When each unit contributes a fixed number of observations to $K$ mutually exclusive categories, the multinomial distribution is a natural model: it fixes the total and describes the category counts through a probability vector. It assumes that, within a unit, the category probabilities are constant across observations. If units differ in their underlying category composition, however, counts can vary more from unit to unit than this model predicts. The Dirichlet-multinomial represents that extra variation by allowing each unit’s category probabilities to vary around a common mean composition.

This is a model for nominal category counts, not an assumption that categories have an order. The total count is fixed for each unit, so the model does not address variation in that total. Its extra variability comes from heterogeneity in category probabilities, not from increasing the total.

Definition: For unit $i$, let $Y_{ij}$ be the count in category $j$, $j=1,\ldots,K$, and let $n_i=\sum_{j=1}^K Y_{ij}$ be the fixed total. Let $\boldsymbol{\pi}_i=(\pi_{i1},\ldots,\pi_{iK})$ be a probability vector, with each $\pi_{ij}>0$ and $\sum_j\pi_{ij}=1$, and let $\phi>0$ be a concentration parameter. The Dirichlet-multinomial model is $$ \mathbf{q}_i\sim\operatorname{Dirichlet}(\phi\pi_{i1},\ldots,\phi\pi_{iK}),\qquad \mathbf{Y}_i\mid\mathbf{q}_i\sim\operatorname{Multinomial}(n_i,\mathbf{q}_i). $$ Here $\mathbf{q}_i$ is a latent probability vector for unit $i$; its expected value is $\boldsymbol{\pi}_i$, and the sum of its Dirichlet parameters is $\phi$.

The mixture makes the model’s structure explicit. Conditional on $\mathbf{q}_i$, the category observations are multinomial. Marginally, after $\mathbf{q}_i$ is integrated out, the counts have a Dirichlet-multinomial distribution. One can model $\boldsymbol{\pi}_i$ as a function of covariates, for example by a multinomial logit model, while $\phi$ controls how much the unit-specific compositions vary around those probabilities. The mixture is a probability model for counts; it should not be confused with every possible GLMM specification or link-scale random effect.

Likelihood and Parameters

For a single unit, write observed counts as $y_1,\ldots,y_K$ and their total as $n$. The likelihood is the conditional multinomial probability averaged over the Dirichlet distribution of $\mathbf{q}$. The Dirichlet density has normalizing constant $\Gamma(\phi)/\prod_j\Gamma(\phi\pi_j)$. Integrating the multinomial probability against this density uses the Dirichlet integral with updated parameters $\phi\pi_j+y_j$, giving the following mass function.

Probability mass function: For nonnegative integers $y_j$ summing to $n$, $$ P(\mathbf{Y}=\mathbf{y}) = \frac{n!}{\prod_{j=1}^K y_j!} \frac{\Gamma(\phi)}{\Gamma(n+\phi)} \prod_{j=1}^K \frac{\Gamma(y_j+\phi\pi_j)}{\Gamma(\phi\pi_j)}. $$ The parameters satisfy $\pi_j>0$, $\sum_j\pi_j=1$, and $\phi>0$. The vector $\boldsymbol{\pi}$ determines the expected category proportions; $\phi$ determines the amount of variation around them.

The first factor counts the ways to arrange the observed categories. The remaining factors account for integrating over the latent composition. In likelihood-based modeling, the observed count vector must have the stated total $n$. If there are independent units, their likelihood contributions multiply, conditional on the modeled covariates and parameters. The model typically assumes independent latent compositions across units, unless dependence between units is separately specified.

Worked Example: Evaluating a Three-Category Probability

Suppose a hypothetical unit has a fixed total $n=5$, with observed counts $\mathbf{y}=(3,1,1)$. Set $\boldsymbol{\pi}=(0.50,0.30,0.20)$ and $\phi=4$. The Dirichlet parameters are $(2,1.2,0.8)$; all are positive and sum to 4, and the expected counts are $(2.5,1.5,1.0)$.

Substitute into the mass function: $$ P(\mathbf{Y}=(3,1,1)) = \frac{5!}{3!1!1!} \frac{\Gamma(4)}{\Gamma(9)} \frac{\Gamma(5)}{\Gamma(2)} \frac{\Gamma(2.2)}{\Gamma(1.2)} \frac{\Gamma(1.8)}{\Gamma(0.8)}. $$

The multinomial coefficient is $120/6=20$. The gamma ratios are $\Gamma(4)/\Gamma(9)=6/40320=1/6720$, $\Gamma(5)/\Gamma(2)=24$, $\Gamma(2.2)/\Gamma(1.2)=1.2$, and $\Gamma(1.8)/\Gamma(0.8)=0.8$. Therefore, $$ P(\mathbf{Y}=(3,1,1)) =20\left(\frac{1}{6720}\right)(24)(1.2)(0.8) \approx 0.06857. $$

This is the probability of this particular count vector under the specified model. It is not the probability that the parameters are correct or that the model is true.

Expected Counts and Extra-Multinomial Variation

The Dirichlet distribution has $E(q_j)=\pi_j$, $\operatorname{Var}(q_j)=\pi_j(1-\pi_j)/(\phi+1)$, and, for $j\ne \ell$, $\operatorname{Cov}(q_j,q_\ell)=-\pi_j\pi_\ell/(\phi+1)$. These properties yield the mean and covariance of the observed counts. The derivation makes clear that the total remains fixed even though the marginal count variance is larger than in a multinomial model.

Use conditional moments.
Given $\mathbf{q}$, $E(Y_j\mid\mathbf{q})=nq_j$ and $\operatorname{Var}(Y_j\mid\mathbf{q})=nq_j(1-q_j)$. For distinct categories, $\operatorname{Cov}(Y_j,Y_\ell\mid\mathbf{q})=-nq_jq_\ell$.

Average the conditional means.
The law of total expectation gives $E(Y_j)=E(nq_j)=n\pi_j$.

Apply the law of total variance.
$\operatorname{Var}(Y_j)=E\{\operatorname{Var}(Y_j\mid\mathbf{q})\}+\operatorname{Var}\{E(Y_j\mid\mathbf{q})\}$. Substitution of the Dirichlet moments gives the variance below.

Combine conditional covariance and covariance of means.
For $j\ne\ell$, $\operatorname{Cov}(Y_j,Y_\ell)=E\{\operatorname{Cov}(Y_j,Y_\ell\mid\mathbf{q})\}+\operatorname{Cov}\{E(Y_j\mid\mathbf{q}),E(Y_\ell\mid\mathbf{q})\}$.

\begin{aligned} E(Y_j)&=n\pi_j,\\ \operatorname{Var}(Y_j)&= n\pi_j(1-\pi_j)\frac{n+\phi}{1+\phi},\\ \operatorname{Cov}(Y_j,Y_\ell)&= -n\pi_j\pi_\ell\frac{n+\phi}{1+\phi},\qquad j\ne\ell. \end{aligned}

For the variance, the two terms from the law of total variance are $n\{\pi_j(1-\pi_j)-\operatorname{Var}(q_j)\}$ and $n^2\operatorname{Var}(q_j)$. Substituting $\operatorname{Var}(q_j)=\pi_j(1-\pi_j)/(\phi+1)$ and combining gives $n\pi_j(1-\pi_j)(n+\phi)/(\phi+1)$. Applying the covariance identity with the stated Dirichlet covariance gives the displayed cross-category covariance. The covariance is negative because a fixed total means that a larger count in one category leaves fewer observations available for the others.

Relative to the multinomial variance $n\pi_j(1-\pi_j)$, the Dirichlet-multinomial variance is multiplied by $(n+\phi)/(1+\phi)=1+(n-1)/(1+\phi)$. This factor is at least 1 for $n\ge1$, equals 1 when $n=1$, and approaches 1 as $\phi$ grows. For individual category indicators within one unit, the induced correlation between two observations is $1/(1+\phi)$. Thus a small $\phi$ means greater within-unit similarity and more variable unit-level counts.

Worked Example: Comparing Count Variances

Consider hypothetical fixed-total counts with $n=10$, $\boldsymbol{\pi}=(0.60,0.30,0.10)$, and $\phi=4$. These probabilities are positive and sum to 1. The expected counts are $10\boldsymbol{\pi}=(6,3,1)$. The variance multiplier is $(10+4)/(1+4)=2.8$.

For category 1, the multinomial variance would be $10(0.60)(0.40)=2.4$. Under the Dirichlet-multinomial it is $2.4(2.8)=6.72$ counts squared. The covariance between categories 1 and 2 is $$ -10(0.60)(0.30)(2.8)=-5.04. $$ For comparison, the multinomial covariance is $-10(0.60)(0.30)=-1.80$. The expected counts are unchanged, while the mixture produces greater marginal variances and more negative cross-category covariances. The observed counts still sum to 10 exactly.

Relationship to the Multinomial and Beta-Binomial

The multinomial model is recovered as a limiting case: as $\phi\to\infty$, the Dirichlet distribution concentrates around $\boldsymbol{\pi}$, so the random probability vector becomes effectively constant. Then the variance multiplier tends to 1 and the count distribution tends to the multinomial with probabilities $\boldsymbol{\pi}$. When $n=1$, the two models already give the same categorical probability $\pi_j$, regardless of $\phi$, so a single observation per unit provides no information for estimating the concentration parameter.

With two categories, the Dirichlet distribution is a beta distribution and the Dirichlet-multinomial reduces to the beta-binomial. Specifically, if the probabilities are $(\pi,1-\pi)$, the beta parameters are $\phi\pi$ and $\phi(1-\pi)$. This is the two-category version of the same probability-mixture idea, rather than a separate source of overdispersion.

Worked Example: Two Categories and the Beta-Binomial

Suppose, hypothetically, that $n=4$ observations fall into either category 1 or category 2, with $y_1=3$, $y_2=1$, $\pi_1=0.70$, and $\phi=2$. The Dirichlet parameters are $(1.4,0.6)$, which are positive and sum to 2. The two-category Dirichlet-multinomial is beta-binomial, and its probability for three observations in category 1 is $$ \frac{4!}{3!1!}\frac{\Gamma(2)}{\Gamma(6)} \frac{\Gamma(4.4)}{\Gamma(1.4)} \frac{\Gamma(1.6)}{\Gamma(0.6)} = 4\left(\frac{1}{120}\right)(1.4)(2.4)(3.4)(0.6) =0.22848. $$

A binomial model with fixed probability 0.70 would instead assign probability $\binom{4}{3}(0.70)^3(0.30)=0.4116$. The difference is consistent with the mixture changing the distribution of counts; it does not mean that the expected category-1 count changes. Under both models that mean is $4(0.70)=2.8$.

Assumptions, Diagnostics, and Limitations

The likelihood is appropriate when the response for each unit is a vector of counts with a known fixed total, categories are mutually exclusive and exhaustive, and the probability model is plausible: each unit has a latent composition drawn from a Dirichlet distribution, followed by multinomial sampling conditional on that composition. Conditional on the modeled parameters, units are generally treated as independent. Covariates may explain changes in the mean composition $\boldsymbol{\pi}_i$, while $\phi$ describes residual composition variation in the specified model.

Diagnostics should consider both the mean composition and the dispersion. Compare observed category proportions with fitted mean proportions, and assess whether variability across units is compatible with the fitted Dirichlet-multinomial distribution. Systematic mean discrepancies suggest that the model for $\boldsymbol{\pi}_i$ may be inadequate; residual variation consistently exceeding or falling below the model’s predictions may suggest an unsuitable dispersion structure. Because the totals $n_i$ affect the variance multiplier, comparisons across units should account for their different totals. Sparse categories can also make probability estimates unstable, particularly when the fitted mean probabilities are near zero.

A single $\phi$ imposes a particular, shared form of heterogeneity across the category composition. It does not provide arbitrary category-specific dispersion or an unrestricted covariance structure. The likelihood may also be weakly informative about $\phi$ when totals are small or there are few independent units. At the limit $\phi\to\infty$, the model approaches a multinomial; if estimation runs toward that boundary, the data may provide little evidence for extra-multinomial variation. A fit should not be interpreted as proving that the latent compositions are literally Dirichlet-distributed: it is a distributional assumption that can be useful when it describes the observed count variation adequately.

Model	What varies across observations within a unit?	Key implication
Multinomial	Categories are sampled with one fixed probability vector.	Variance is $n\pi_j(1-\pi_j)$.
Dirichlet-multinomial	A probability vector varies across units according to a Dirichlet distribution.	Counts have extra-multinomial variation but the total stays fixed.
Separate model for a varying total	The total itself is random.	Needed when the number of observations per unit is not fixed by design or conditioning.

Common Mistake

A frequent error is to say, “The Dirichlet-multinomial makes the total count more variable.” That is incorrect: by definition, the counts in each unit sum to the specified $n_i$. The model increases the variation in how that total is allocated among categories. If the total varies and that variation is part of the response process, it requires a model for the total as well as a model for category allocation. Another mistake is to interpret $\phi$ as a category probability: it is a positive concentration parameter, and larger values mean less variation in latent compositions.

Key takeaway: The Dirichlet-multinomial is a multinomial likelihood mixed over Dirichlet-distributed category probabilities. It retains a fixed total and the same expected category proportions as the multinomial, while allowing extra variation in category counts controlled by $\phi$.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For $K$ categories, what conditions must the mean probability vector and concentration parameter satisfy?
If $n=1$, can the Dirichlet-multinomial be distinguished from the multinomial by the response distribution? Explain.
For fixed $n$ and $\boldsymbol{\pi}$, what happens to the variance as $\phi\to\infty$?
Does the Dirichlet-multinomial model allow the total count to vary across units?
For two categories, which familiar distribution is the Dirichlet-multinomial equivalent to?

Solutions. (1) Each $\pi_j$ must be positive, the probabilities must sum to 1, and $\phi>0$. (2) No: with one observation the response is categorical with probabilities $\boldsymbol{\pi}$, independent of $\phi$. (3) The variance multiplier $(n+\phi)/(1+\phi)$ tends to 1, yielding the multinomial variance. (4) No; each unit’s total is fixed at its specified $n$. (5) The beta-binomial distribution.

Model	What varies across observations within a unit?	Key implication
Multinomial	Categories are sampled with one fixed probability vector.	Variance is \(n\pi_j(1-\pi_j)\).
Dirichlet-multinomial	A probability vector varies across units according to a Dirichlet distribution.	Counts have extra-multinomial variation but the total stays fixed.
Separate model for a varying total	The total itself is random.	Needed when the number of observations per unit is not fixed by design or conditioning.