Individual and Grouped Binomial Data

Why the Data Representation Matters

A binomial response can be stored one trial at a time, with each observation recorded as a success or failure, or as a count of successes out of a known number of trials. These representations can encode the same outcomes and use the same fitted success probabilities. Yet their Pearson and deviance diagnostics need not be equal. The reason is not a different mean model: the two representations use different saturated reference models and aggregate the trial-level residual information differently.

This distinction matters when interpreting a dispersion check. A grouped row represents a success count out of \(m_i\) trials; it is not equivalent, for diagnostic bookkeeping, to a single Bernoulli observation. The formulas below assume that all trials within a group have the same fitted probability. If trial-level fitted probabilities differ, the group generally cannot be represented by one ordinary binomial count with a common probability.

Definitions for the Two Representations

For group \(i\), let \(m_i\) be its positive integer number of trials, \(Y_i\) its observed number of successes, and \(\hat p_i\) its fitted success probability, with \(0\leq Y_i\leq m_i\) and \(0<\hat p_i<1\). Write \(\hat\mu_i=m_i\hat p_i\) for its fitted success count. The grouped binomial variance under the fitted model is \(m_i\hat p_i(1-\hat p_i)\).

For individual data, let \(Z_{ij}\in\{0,1\}\) denote trial \(j\) in group \(i\), so \(Y_i=\sum_{j=1}^{m_i} Z_{ij}\). Each trial has fitted probability \(\hat p_i\). Its Pearson contribution is \((Z_{ij}-\hat p_i)^2/\{\hat p_i(1-\hat p_i)\}\). Its Bernoulli deviance contribution is \(2[Z_{ij}\log(1/\hat p_i)+(1-Z_{ij})\log\{1/(1-\hat p_i)\}]\).

The individual deviance compares each trial with a saturated Bernoulli model, whose fitted probability for that observed trial is zero or one. The grouped deviance instead compares the observed group count with a saturated binomial model, whose fitted probability is \(Y_i/m_i\). This difference between reference models is central to the results below.

Worked Examples

Exact Relationships Between the Totals

Consider one group with \(m\) trials, observed success count \(y\), fitted probability \(p\in(0,1)\), and observed proportion \(q=y/m\). The identities in this section hold for any such values, so they also hold when \(p\) is an estimated fitted probability. They are algebraic comparisons of the diagnostics, not claims about their sampling distributions.

Proof for Pearson diagnostics. Summing the individual contributions over \(y\) successes and \(m-y\) failures gives

The grouped contribution is \(P_{\mathrm{grouped}}=m(q-p)^2/\{p(1-p)\}\). Subtracting and putting the terms over a common denominator yields

The last equality follows by expanding the numerator: \(q(1-p)^2+(1-q)p^2-(q-p)^2=q-q^2\). Since \(0\leq q\leq1\) and \(0<p<1\), the difference is nonnegative; it is zero exactly when \(q=0\) or \(q=1\).

Proof for deviance diagnostics. Summing the individual deviance contributions gives \(\displaystyle D_{\mathrm{individual}}=2m\{q\log(1/p)+(1-q)\log(1/(1-p))\}\). The grouped contribution is \(\displaystyle D_{\mathrm{grouped}}=2m\{q\log(q/p)+(1-q)\log((1-q)/(1-p))\}\). Subtracting cancels the terms involving the fitted probability:

For \(0<q<1\), both logarithms inside braces are negative, so the expression in braces is negative and the displayed difference is positive. At \(q=0\) or \(q=1\), the convention \(0\log0=0\) gives a difference of zero. Thus the difference is nonnegative for every allowed \(q\). This completes the proof. \(\square\)

The deviance difference is twice \(m\) times the binary entropy of the observed proportion, using natural logarithms. It is largest for proportions near one half and vanishes at the endpoints. For example, at \(m=4\) and \(q=0.5\), it is \(-8\{0.5\log(0.5)+0.5\log(0.5)\}=8\log2\approx5.5452\), agreeing with the first worked example.

Likelihood, Degrees of Freedom, and Interpretation

When trials in a group have the same covariates and probability, grouping does not change the fitted parameter estimates under the ordinary binomial likelihood. The individual likelihood contribution is \(p^y(1-p)^{m-y}\); the grouped binomial likelihood contribution is \(\binom{m}{y}p^y(1-p)^{m-y}\). The factor \(\binom{m}{y}\) does not depend on the model parameters, so it does not change the maximizer. But the saturated likelihoods used to define deviance do differ, which explains why equal fitted estimates do not imply equal deviances.

Grouping also changes the row count used in the usual residual degrees of freedom. If there are \(G\) grouped rows and the fitted model matrix has rank \(r\), the usual grouped residual degrees of freedom are \(G-r\). If the same trials are entered as \(N=\sum_i m_i\) Bernoulli rows with replicated covariates and the same rank \(r\), the individual-data convention gives \(N-r\). These bookkeeping differences are one reason not to compare the resulting Pearson or deviance ratios as if they were calculated from the same observational units.

The binomial variance benchmark is \(m_i p_i(1-p_i)\) for a grouped count and \(p_i(1-p_i)\) for one Bernoulli trial. The identities proved above describe how the residual totals change under aggregation; they do not establish that either representation is evidence of extra variation. Interpretation still depends on how the data were collected and modeled, as well as on the diagnostic calibration appropriate to that representation.

Common Mistakes

• Treating a group as one Bernoulli trial. A row with \(Y_i=3\) successes out of \(m_i=4\) trials is a binomial count, not a single success indicator. Use its trial total in both its fitted count and its variance.
• Assuming the grouped and individual deviances are equal. The fitted probabilities may match, but the saturated reference models differ. The exact difference is nonnegative, and its sign is \(-2m\{q\log q+(1-q)\log(1-q)\}\), not \(2m\) times the expression in braces.
• Dropping zero- or full-success groups because a logarithm appears to be undefined. Apply \(0\log0=0\), or use the endpoint formulas. With an interior fitted probability, both diagnostics are finite at either endpoint.
• Collapsing trials that have different fitted probabilities. The identities require one common fitted probability within each group. If probabilities vary across trials, a single grouped binomial likelihood generally does not preserve the individual model.