Grouped Binomial Weights in IRLS

Why Grouping Changes the IRLS Weight

A binomial observation records both a number of successes and the number of trials that could have produced them. A group with 40 trials usually contains more information about its success probability than a group with 4 trials, even when their observed proportions are identical. In IRLS, that difference appears as a multiplier in the group’s working weight.

We derive the working response and weight for independent groups with possibly different trial counts. The formulas apply to a smooth, invertible link, and the logit case shows especially clearly how each trial count scales the Bernoulli contribution.

Definition: For group \(i\), let \(Y_i\) be the number of successes in \(m_i\) trials, with \(Y_i\sim\operatorname{Binomial}(m_i,p_i)\). The observed proportion is \(r_i=Y_i/m_i\). Assume the groups are independent, \(m_i\geq 1\), and \(0<p_i<1\). Let \(g\) be a differentiable, invertible link on \((0,1)\), with \(g'(p)\neq 0\), and set \(g(p_i)=\eta_i=x_i^\mathsf{T}\beta\). Here \(x_i\) is the predictor vector for group \(i\).

At a current coefficient value, the fitted probability is \(p_i\), the linear predictor is \(\eta_i=g(p_i)\), and the working response is the first-order adjustment to \(\eta_i\) implied by the observed proportion. The working weight measures the local information contributed by that group. Both depend on the current fitted probability, and the weight also depends on the number of trials.

Deriving the Grouped-Binomial Formulas

The log-likelihood contribution from group \(i\), omitting no parameter-dependent terms, is

\ell_i(\beta)=\log {m_i\choose Y_i}+Y_i\log p_i+(m_i-Y_i)\log(1-p_i).

The binomial coefficient is constant with respect to \(\beta\). Differentiating the remaining terms with respect to \(p_i\) gives \(Y_i/p_i-(m_i-Y_i)/(1-p_i)\). The chain rule then gives the score with respect to the linear predictor:

\frac{\partial\ell_i}{\partial\eta_i} =\frac{Y_i-m_i p_i}{p_i(1-p_i)}\frac{dp_i}{d\eta_i} =\frac{m_i(r_i-p_i)}{p_i(1-p_i)g'(p_i)}.

The last equality uses \(dp_i/d\eta_i=1/g'(p_i)\) and \(Y_i=m_i r_i\). Thus the observed discrepancy in the score is the proportion residual \(r_i-p_i\), while the trial count multiplies its contribution.

Result: For a grouped-binomial response with a differentiable invertible link, the IRLS working response and expected working weight at the current fitted probability \(p_i\) are

z_i=\eta_i+(r_i-p_i)g'(p_i), \qquad w_i=\frac{m_i}{p_i(1-p_i)\{g'(p_i)\}^2}.

Here \(z_i\) is on the linear-predictor scale. The factor \(m_i\) is the group’s trial count.

The working response follows by linearizing the link around the fitted probability: \(g(r_i)\) is approximated by \(g(p_i)+(r_i-p_i)g'(p_i)\). IRLS uses this first-order expression even when \(r_i\) is at a boundary, where \(g(r_i)\) itself may not be finite. The weight is the expected information for \(\eta_i\), as the derivation below verifies.

Worked Example: A Group of Ten Trials with a Logit Link

Suppose \(m=10\), \(Y=7\), and the current fitted probability is \(p=0.5\). Use the logit link \(g(p)=\log\{p/(1-p)\}\).

The observed proportion is \(r=7/10=0.7\). The fitted linear predictor is \(\eta=\log(0.5/0.5)=0\), and the derivative of the link is \(g'(p)=1/\{p(1-p)\}=1/0.25=4\).

The working response is \(z=0+(0.7-0.5)(4)=0.8\). The working weight is \(w=10/\{0.5(0.5)(4^2)\}=10/4=2.5\).

Equivalently, for a logit link the weight is \(mp(1-p)=10(0.25)=2.5\). The group’s weight is ten times the weight \(0.25\) for one Bernoulli trial at the same fitted probability.

Why the Weight Is Expected Information

The expected information is the negative expected second derivative of the log-likelihood. For the grouped-binomial model, it can also be obtained as the variance of the score. The following derivation makes the role of the trial count explicit.

Result: Under the conditions stated above, the score for \(\eta_i\) has mean zero and variance \(m_i/\{p_i(1-p_i)\{g'(p_i)\}^2\}\). This is the expected information for \(\eta_i\).

Write the score.
Let \(s_i=\partial\ell_i/\partial\eta_i\). From the likelihood derivative, \(s_i=(Y_i-m_i p_i)/\{p_i(1-p_i)g'(p_i)\}\).

Take its expectation.
Since \(E[Y_i]=m_i p_i\), it follows directly that \(E[s_i]=0\).

Calculate its variance.
Since \(\operatorname{Var}(Y_i)=m_i p_i(1-p_i)\), division by the squared denominator gives \(\operatorname{Var}(s_i)=m_i/\{p_i(1-p_i)\{g'(p_i)\}^2\}\).

The score variance is the Fisher information because the likelihood is regular at \(0<p_i<1\), with fixed \(m_i\) and a differentiable link. The formula also shows directly that multiplying the number of trials by a factor multiplies the group’s expected information by the same factor, holding \(p_i\) and the link fixed.

Connection to Weighted Least Squares

For a vector of independent groups, let \(X\) be the design matrix whose \(i\)th row is \(x_i^\mathsf{T}\), and let \(W\) be diagonal with entries \(w_i\). The grouped-binomial score for \(\beta\) is \(\sum_i x_i s_i\). The working-response formulas express each scalar score as a weighted residual:

w_i(z_i-\eta_i) = \frac{m_i}{p_i(1-p_i)\{g'(p_i)\}^2} (r_i-p_i)g'(p_i) = \frac{Y_i-m_i p_i}{p_i(1-p_i)g'(p_i)} =s_i.

This identity proves that the Fisher-scoring equation can be written as a weighted least-squares normal equation for the working responses.

Result: At the current iterate, the Fisher-scoring equation for the coefficients is

X^\mathsf{T}W X\,\beta_{\mathrm{new}}=X^\mathsf{T}Wz.

This equation describes the IRLS update when the displayed system is solvable. If it is singular, it does not by itself specify a unique update.

To verify the equation, note that the score for \(\beta\) is \(X^\mathsf{T}s\), where \(s\) has entries \(s_i\). By the identity above, \(s=W(z-X\beta)\). The expected information matrix for \(\beta\) is \(X^\mathsf{T}WX\), since the independent groups’ expected information contributions are \(w_i x_i x_i^\mathsf{T}\). Fisher scoring adds the inverse expected information times the score to the current coefficient vector. Therefore

\beta_{\mathrm{new}} =\beta+(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W(z-X\beta),

whenever \(X^\mathsf{T}WX\) is invertible. Expanding and rearranging gives \(X^\mathsf{T}WX\,\beta_{\mathrm{new}}=X^\mathsf{T}Wz\), as claimed. This is the weighted least-squares representation of the Fisher-scoring step.

Worked Example: Proportion and Count Forms Give the Same Weight

For one group, let \(m=20\), \(Y=12\), and use the current fitted probability \(p=0.5\) with a logit link. The proportion is \(r=12/20=0.6\); the current predictor is \(\eta=0\), and \(g'(0.5)=4\).

Using the proportion response, \(z=0+(0.6-0.5)(4)=0.4\), and \(w=20(0.5)(0.5)=5\). The score contribution for the predictor is therefore \(w(z-\eta)=5(0.4-0)=2\).

Directly from the binomial likelihood, the same score is \((Y-mp)/\{p(1-p)g'(p)\}=(12-10)/(0.25\cdot4)=2\). The two calculations agree. The observed count enters through \(Y-mp\); the equivalent proportion residual is \(r-p=(Y-mp)/m\), while the weight supplies the factor \(m\).

Worked Example: Unequal Trial Counts at the Same Fitted Probability

Consider two groups with current probability \(p=0.5\) and a logit link. Their trial counts are \(m_1=4\) and \(m_2=40\), and their observed proportions are \(r_1=0.75\) and \(r_2=0.55\). Suppose the only coefficient is an intercept, currently \(\eta=0\).

For both groups, \(g'(0.5)=4\). Thus \(z_1=0+(0.75-0.5)(4)=1\) and \(z_2=0+(0.55-0.5)(4)=0.2\). Their weights are \(w_1=4(0.25)=1\) and \(w_2=40(0.25)=10\).

The one-step weighted least-squares update for the intercept is the weighted mean of the two working responses: \(\beta_{\mathrm{new}}=(w_1z_1+w_2z_2)/(w_1+w_2)=(1\cdot1+10\cdot0.2)/11=3/11\), approximately \(0.273\). The second group has ten times the weight because it has ten times as many trials, not because its observed proportion is closer to the current fitted probability.

The Logit Special Case and a Common Pitfall

For \(g(p)=\log\{p/(1-p)\}\), the derivative is \(g'(p)=1/\{p(1-p)\}\). Substituting this derivative into the general formulas yields

z_i=\eta_i+\frac{r_i-p_i}{p_i(1-p_i)}, \qquad w_i=m_i p_i(1-p_i).

Thus the logit working response for a group is the same expression as for a Bernoulli response when written using the group’s observed proportion, but the weight is multiplied by the number of trials. This is the central distinction from treating each group as one Bernoulli observation.

Pitfall: using the Bernoulli weight for a whole group. The incorrect reasoning is that a group has one observed proportion, so it should receive the single-trial weight \(p_i(1-p_i)\). That ignores the likelihood for \(m_i\) trials. The correction is \(w_i=m_i p_i(1-p_i)\) for a grouped binomial logit model.
Pitfall: multiplying by the trial count twice. If the response is already the proportion \(r_i\), use the weight \(m_i p_i(1-p_i)\). Do not also multiply the response residual by \(m_i\). The equivalent count residual is \(Y_i-m_i p_i\), and its score formula already accounts for the count scale.
Pitfall: treating the working response as observed data. The value \(z_i\) is computed from the current fit and used to form a local weighted least-squares update. It is not a newly observed outcome or a replacement for \(Y_i\).

Grouping is valid when trials within a group share the modeled success probability and the same predictor values. If individual trials have different predictors, replacing them with a single group proportion and one predictor generally changes the model. When trials can legitimately be grouped, the trial-count multiplier ensures that the grouped likelihood contributes the same information as its constituent Bernoulli trials.

Check Your Understanding

Use the grouped-binomial formulas to answer these questions.

For a logit model with \(m_i=8\) and \(p_i=0.25\), what is the working weight?
Why does using the proportion \(r_i\) rather than the count \(Y_i\) not remove the trial-count multiplier from the weight?
For a general link \(g\), state the working response and weight for a group with observed proportion \(r_i\).
Explain why the score contribution \(w_i(z_i-\eta_i)\) agrees with the binomial likelihood score for \(\eta_i\).
What assumption about predictors and success probabilities is needed to combine trials into one grouped-binomial observation?