Workshop: Convert Individual Records to Grouped Data

Why Convert Individual Records to Groups?

A study may store one row for every person, trial, or experimental unit, even when many rows have exactly the same covariates. If the response is binary and the model assigns the same success probability to records with the same covariates, those records can be summarized by a success count and a trial total. The resulting grouped data can be easier to inspect and compute with.

The conversion is exact for fitting the specified model, provided that the groups are formed correctly. In particular, every record in a group must have the same covariate pattern, and the individual Bernoulli responses must satisfy the model’s independence assumptions. The grouped likelihood differs from the likelihood of the observed individual sequence by a factor that does not involve the regression coefficients. That factor changes the numerical likelihood value, but not the fitted coefficients or probabilities.

From Individual Records to Binomial Groups

Definition: Suppose \(Y_i\) is a binary response for record \(i\), with \(Y_i=1\) denoting success and \(Y_i=0\) denoting failure. A group \(g\) is a set of records with the same covariate vector \(x_g\). Its trial total is \(n_g\), the number of records in the group, and its success count is \(r_g=\sum_{i\in g}Y_i\). Under independent Bernoulli responses with common probability \(p_g\) within the group, \(r_g\sim\operatorname{Binomial}(n_g,p_g)\).

For logistic regression, the probability shared by group \(g\) is \(p_g(\beta)=\expit(x_g^\mathsf{T}\beta)\), where \(\expit(u)=e^u/(1+e^u)\). The covariate vector must include every term that affects the linear predictor, such as an intercept, transformations, and interactions. Records that share only some of their recorded covariates cannot be combined if another covariate in the model differs.

The individual data retain the order and identities of the observed responses. The grouped data retain the number of successes and trials in each covariate group, but not which particular records succeeded. For estimating the specified regression coefficients, that lost ordering carries no additional likelihood information when the assumptions above hold.

Worked Example: Converting Two Covariate Patterns

Suppose seven independent Bernoulli records have a binary covariate \(x\). For \(x=0\), the responses are \(1,0,0\). For \(x=1\), they are \(1,1,0,1\). Convert these records to grouped data.

For \(x=0\), there are three trials and one success, so \(n_0=3\) and \(r_0=1\). For \(x=1\), there are four trials and three successes, so \(n_1=4\) and \(r_1=3\). The grouped response can therefore be recorded as \((r_0,n_0-r_0)=(1,2)\) and \((r_1,n_1-r_1)=(3,1)\), alongside the two covariate values.

The failure counts are \(3-1=2\) and \(4-3=1\); they are not the trial totals. Each total is recovered by adding successes and failures: \(1+2=3\) and \(3+1=4\).

How the Two Likelihoods Are Related

For a fixed group with \(n_g\) records, the individual likelihood contribution is the product of the Bernoulli probabilities for the particular observed sequence. Since the probability is common within the group, that contribution is \(p_g^{r_g}(1-p_g)^{n_g-r_g}\). The grouped count likelihood also includes the number of different individual sequences that have \(r_g\) successes: \(\binom{n_g}{r_g}\).

L_{\mathrm{individual}}(\beta) =\prod_g p_g(\beta)^{r_g}\{1-p_g(\beta)\}^{n_g-r_g}, \qquad L_{\mathrm{grouped}}(\beta) =\prod_g \binom{n_g}{r_g} p_g(\beta)^{r_g}\{1-p_g(\beta)\}^{n_g-r_g}.

The first likelihood refers to the observed individual response sequence, conditional on the covariates. The second refers to the observed counts. Their difference is the product of binomial coefficients, which depends on the observed counts but not on \(\beta\).

Theorem: If individual responses are independent Bernoulli variables and every record within group \(g\) has the same model probability \(p_g(\beta)\), then \(L_{\mathrm{grouped}}(\beta)=C L_{\mathrm{individual}}(\beta)\), where \(C=\prod_g\binom{n_g}{r_g}\) is positive and independent of \(\beta\). Thus the two likelihoods have exactly the same set of maximizers, score equations, and likelihood-based comparisons between coefficient values.

Proof. For one group, any specified sequence with \(r_g\) successes and \(n_g-r_g\) failures has probability \(p_g(\beta)^{r_g}\{1-p_g(\beta)\}^{n_g-r_g}\). There are \(\binom{n_g}{r_g}\) distinct sequences with those counts, and the sequences are mutually exclusive. Summing their probabilities gives the binomial count probability. Independence across records and groups gives the displayed products, so their ratio is \(C\).

Because \(C\) is positive and does not depend on \(\beta\), multiplying the individual likelihood by \(C\) preserves its ordering over all possible coefficient values. Hence every value maximizing one likelihood maximizes the other. Taking logarithms gives \(\ell_{\mathrm{grouped}}(\beta)=\ell_{\mathrm{individual}}(\beta)+\log C\). Differentiating with respect to \(\beta\) shows that the score vectors are equal wherever the likelihoods are differentiable; differentiating again shows that their Hessian matrices are equal. Therefore their likelihood-based comparisons, which depend on differences in log likelihood across coefficient values, also agree. \(\square\)

This result does not require that a finite maximum exist or be unique. If the model has no finite maximum, both likelihoods share that feature; if several coefficient values maximize one, they all maximize the other. The result establishes equivalence, not existence of a well-behaved fit.

Worked Example: Verifying the Same Fit in Both Representations

In the two-pattern data above, use a logistic model with an intercept and the binary covariate \(x\). There are two parameters and two distinct covariate patterns, so the model can fit the two group probabilities separately. The binomial maximum-likelihood estimate for each probability is its success proportion: \(\hat p_0=1/3\) and \(\hat p_1=3/4\).

The fitted intercept is the log odds at \(x=0\): \(\hat\beta_0=\log\{(1/3)/(2/3)\}=-\log 2\). The fitted probability at \(x=1\) has log odds \(\log\{(3/4)/(1/4)\}=\log 3\). Thus \(\hat\beta_0+\hat\beta_1=\log 3\), so \(\hat\beta_1=\log 3+\log 2=\log 6\). Checking the fitted values, \(\expit(-\log 2)=1/3\) and \(\expit(-\log 2+\log 6)=\expit(\log 3)=3/4\).

The individual-sequence likelihood at this fit is \((1/3)(2/3)^2(3/4)^3(1/4)\). Calculate each group contribution: \((1/3)(2/3)^2=4/27\), and \((3/4)^3(1/4)=27/256\). Their product is \((4/27)(27/256)=4/256=1/64\).

The grouped likelihood includes the coefficients \(\binom{3}{1}=3\) and \(\binom{4}{3}=4\), so it is \(3\cdot4\cdot(1/64)=3/16\). Directly, the two grouped contributions are \(3(1/3)(2/3)^2=4/9\) and \(4(3/4)^3(1/4)=27/64\); their product is \((4/9)(27/64)=3/16\). The likelihood values differ by \(12\), exactly the constant \(3\cdot4\), while the fitted coefficients and probabilities are identical.

Score Equations and Fitted Probabilities

The equality of score equations can also be seen directly in the logistic model. Write \(x_g\) as a column vector. The grouped log likelihood, apart from terms constant in \(\beta\), is \(\sum_g[r_g\log p_g+(n_g-r_g)\log(1-p_g)]\). Since \(\log\{p_g/(1-p_g)\}=x_g^\mathsf{T}\beta\), differentiating gives the grouped score. The individual score reduces to the same expression after summing records within each group.

U(\beta) =\sum_g x_g\{r_g-n_gp_g(\beta)\} =\sum_i x_i\{Y_i-p_i(\beta)\}.

For the first equality, the derivative of \(r_g\log p_g+(n_g-r_g)\log(1-p_g)\) with respect to the linear predictor \(x_g^\mathsf{T}\beta\) is \(r_g-n_gp_g\). Multiplication by \(x_g\) gives its derivative with respect to \(\beta\). For the second expression, all records in group \(g\) have \(x_i=x_g\) and \(p_i=p_g\), so \(\sum_{i\in g}x_i(Y_i-p_i)=x_g(r_g-n_gp_g)\). Summing over groups proves the equality.

The fitted probability for every record in group \(g\) is \(p_g(\hat\beta)\), whether the model is fit from individual records or grouped counts. Multiplying this probability by \(n_g\) gives the fitted success count, and the probability itself is the fitted group proportion. Aggregation changes the data representation, not the fitted probability assigned to a covariate pattern.

Worked Example: Aggregating a Noncontiguous Covariate Pattern

Consider seven records with covariate \(z\): two records at \(z=0\) have one success and one failure; three records at \(z=1\) have two successes and one failure; and two records at \(z=2\) are both successes. The records need not be adjacent in the original data table. Group them by their exact \(z\) values, giving \((n_0,r_0)=(2,1)\), \((n_1,r_1)=(3,2)\), and \((n_2,r_2)=(2,2)\).

To check the likelihood identity at a specified, not necessarily fitted, coefficient value, take a logistic model with intercept \(0\) and slope \(\log 2\). Its probabilities at \(z=0,1,2\) are \(1/2,2/3,4/5\), respectively. The individual likelihood is \((1/2)(1/2)\cdot(2/3)^2(1/3)\cdot(4/5)^2\). The three contributions are \(1/4\), \(4/27\), and \(16/25\), giving \((1/4)(4/27)(16/25)=16/675\).

The grouped coefficients are \(\binom{2}{1}=2\), \(\binom{3}{2}=3\), and \(\binom{2}{2}=1\), whose product is \(6\). Therefore the grouped likelihood is \(6(16/675)=96/675=32/225\). Direct evaluation gives the same result: \([2(1/4)][3(4/27)][1(16/25)]=(1/2)(4/9)(16/25)=32/225\). This verifies the likelihood relationship even away from a maximum.

When Aggregation Is Useful—and What It Does Not Do

Aggregation can reduce a long individual-level file to a compact table, especially when many records share covariates. It can also make trial totals explicit and expose whether a group has zero successes or all successes. These are practical gains, but aggregation is valid only relative to the model being fit.

The exact equivalence requires common probabilities within each group and independent Bernoulli responses. If records differ in a covariate used by the model, combining them generally replaces distinct probabilities by one probability and changes the likelihood. A covariate that is merely rounded to make groups larger is not an exact match. Similarly, if the analysis needs individual-level outcomes for a different purpose, the grouped counts do not reconstruct which records succeeded.

Define the model covariates.
Determine every quantity entering the linear predictor, not just the most prominent predictor.

Group exact matches.
Combine records only when their full model covariate patterns are identical and the response assumptions permit a shared Bernoulli probability.

Count successes and trials.
For each pattern, record \(r_g\) and \(n_g\), with failures given by \(n_g-r_g\).

Fit and verify.
Use the grouped-binomial response with its trial total, then check that the fitted probabilities correspond to the same covariate patterns.

A common mistake is to claim that the two likelihood values must be numerically equal because the fitted coefficients agree. They need not be equal: the grouped likelihood counts all individual sequences compatible with the observed group totals, so it includes the binomial coefficients. The correction is to compare the likelihoods up to their parameter-independent factor. Another mistake is to omit the trial total and treat a success proportion as if it were an individual binary response. The proportion alone does not say whether it came from two trials or two hundred; the denominator is part of the grouped data.

Key takeaway: For independent Bernoulli records sharing an exact covariate pattern, replace the individual responses by their success count and trial total. The grouped likelihood is the individual-sequence likelihood multiplied by a coefficient that is constant in the model parameters, so the fitted coefficients, score equations, and probabilities agree.

Check Your Understanding

Use the likelihood relationship and the grouping conditions to answer these questions.

Four records share the same covariates and have responses \(1,0,1,0\). What are their grouped success count and trial total?
Why does the grouped likelihood include a binomial coefficient that is absent from the likelihood of one specified individual response sequence?
Two records agree on age but have different treatment indicators, and treatment is included in the model. Can they be put in the same group for this fit? Explain.
If the grouped likelihood is twelve times the individual-sequence likelihood for every coefficient value, what can be concluded about their maximum-likelihood estimates?
Why is a success proportion without its trial total insufficient to specify a grouped-binomial observation?