Why Similar-Looking Weights Can Give Different Inference
A model may use a number beside each response to represent trials, replicated records, or relative precision. Those numbers can all multiply terms in a fitting calculation, but they describe different data-generating arrangements. Confusing them can change the likelihood, the amount of information attributed to the data, and the resulting standard errors.
A useful diagnosis starts with three questions: What was observed? What does the weight say about those observations? Which likelihood contribution follows from that description? The numerical value of a weight alone cannot answer these questions. This tutorial uses binomial and Gaussian examples to show how the likelihood and standard errors expose common misinterpretations.
Trial Counts: Information From Bernoulli Trials
For a binomial response, the observed value \(y_i\) is a success count satisfying \(y_i\in\{0,1,\ldots,m_i\}\). If the success probability is \(p_i\), its likelihood contribution is proportional to \(p_i^{y_i}(1-p_i)^{m_i-y_i}\). The binomial coefficient does not depend on the model parameters, but the exponents do: they record how many successes and failures were observed.
For logistic regression, let \(p_i=\exp(\eta_i)/(1+\exp(\eta_i))\), with \(\eta_i=x_i^\mathsf{T}\beta\). The following derivation shows exactly how the trial count enters the score and information.
For observation \(i\), the parameter-dependent term is \(y_i\eta_i-m_i\log(1+\exp(\eta_i))\), because \(\log(p_i)=\eta_i-\log(1+\exp(\eta_i))\) and \(\log(1-p_i)=-\log(1+\exp(\eta_i))\).
Since \(\partial\eta_i/\partial\beta=x_i\) and \(\partial\log(1+\exp(\eta_i))/\partial\eta_i=p_i\), the score contribution is \(x_i(y_i-m_i p_i)\). Summing independent contributions gives the stated score.
Since \(\partial p_i/\partial\eta_i=p_i(1-p_i)\), the Hessian contribution is \(-m_i p_i(1-p_i)x_i x_i^\mathsf{T}\). Its negative is the information contribution. Summing over \(i\) proves the result. \(\square\)
Worked Example: A Binomial Count Mistaken for One Bernoulli Observation
Suppose 20 independent patients each have the same modeled probability \(p\) of an event, and 8 events are observed. The grouped response is \(y=8\) successes out of \(m=20\) trials. Estimate the intercept \(\beta=\operatorname{logit}(p)\) and its model-based standard error.
The likelihood is proportional to \(p^8(1-p)^{12}\). Its maximum occurs at \(\hat p=8/20=0.4\), so \(\hat\beta=\log(0.4/0.6)=\log(2/3)\), approximately \(-0.4055\). The information for the intercept is \(m\hat p(1-\hat p)=20(0.4)(0.6)=4.8\). Therefore the model-based standard error is \(1/\sqrt{4.8}\), approximately \(0.4564\).
If someone instead treats the proportion \(0.4\) as a single Bernoulli response, the resulting fractional-response expression is not the likelihood of an observed Bernoulli variable: a Bernoulli observation can only be 0 or 1. Even if one informally uses the one-observation curvature \(0.4(0.6)=0.24\), the implied standard error is \(1/\sqrt{0.24}\), approximately \(2.041\). The ratio \(2.041/0.4564\) is \(\sqrt{20}\). The error discarded the information from 20 trials. For the estimated probability itself, the correct standard error from the binomial variance is \(\sqrt{0.4(0.6)/20}\), approximately \(0.1095\).
Frequency Weights: Replication of Observed Records
A frequency weight refers to repeated copies of an observed record, not automatically to multiple trials behind a response. If record \(i\) has log-likelihood contribution \(\ell_i(\beta)\) and occurs \(a_i\) times, independence gives the joint likelihood as a product of \(a_i\) identical contributions. Its log-likelihood is therefore \(\sum_i a_i\ell_i(\beta)\).
Worked Example: Replicated Binary Records Versus Trial Counts
Consider an intercept-only Bernoulli model. One observed record has response 1 and frequency 3; another has response 0 and frequency 2. The frequency weights mean three independent copies of the first record and two of the second.
The replicated data contain 3 successes and 2 failures. Their likelihood is proportional to \(p^3(1-p)^2\), giving \(\hat p=3/5=0.6\) and \(\hat\beta=\log(3/2)\), approximately \(0.4055\). The intercept information is \(5(0.6)(0.4)=1.2\), so its standard error is \(1/\sqrt{1.2}\), approximately \(0.9129\). The probability standard error is \(\sqrt{0.6(0.4)/5}\), approximately \(0.2191\).
A mistaken alternative is to call 3 and 2 trial counts while treating the binary responses 1 and 0 as success counts. That would describe 1 success out of 3 trials and 0 successes out of 2 trials: 1 success and 4 failures in total. It gives \(\hat p=1/5=0.2\), not \(0.6\), and intercept information \(5(0.2)(0.8)=0.8\), with standard error \(1/\sqrt{0.8}\), approximately \(1.118\). The two likelihoods differ because the data descriptions differ. A frequency of three on a response of 1 means three observed successes; a trial count of three with one recorded success means one success and two failures.
Precision Weights: Variance and Dispersion Matter
A precision weight describes how variable an observation is relative to others. For example, the Gaussian model \(Y_i\sim N(\mu_i,\sigma^2/w_i)\), with known positive \(w_i\), assigns greater precision to larger \(w_i\). When \(\mu_i=x_i^\mathsf{T}\beta\), its contribution to information for \(\beta\) is \(w_i x_i x_i^\mathsf{T}/\sigma^2\). This can resemble the information scaling from replicated observations, but it does not by itself mean that \(w_i\) new observations were collected.
Worked Example: Precision Weights Do Not Create New Records
Two independent observations follow \(Y_i\sim N(\mu,\sigma^2/w_i)\), with weights \(w_1=1\), \(w_2=4\), and observed values \(y_1=0\), \(y_2=10\). First suppose \(\sigma=2\) is known. Find the estimate of \(\mu\) and its standard error.
The log-likelihood terms involving \(\mu\) are \(-\sum_i w_i(y_i-\mu)^2/(2\sigma^2)\), so the maximum minimizes the weighted sum of squares. Differentiating that sum gives \(2\sum_i w_i(\mu-y_i)=0\), hence \(\hat\mu=(1\cdot0+4\cdot10)/(1+4)=8\). The information for \(\mu\) is \(\sum_i w_i/\sigma^2=5/4\), so \(\operatorname{Var}(\hat\mu)=4/5=0.8\) and its standard error is \(\sqrt{0.8}\), approximately \(0.8944\).
Now suppose \(\sigma^2\) is unknown and estimated by maximum likelihood under this stated precision model. The weighted residual sum of squares is \(1(0-8)^2+4(10-8)^2=64+16=80\). There are two independent observed records, so the Gaussian log-likelihood has a term \(-2\log(\sigma^2)/2\); maximizing it gives \(\hat\sigma^2=80/2=40\). The plug-in standard error for \(\hat\mu\) is \(\sqrt{\hat\sigma^2/5}=\sqrt{8}\), approximately \(2.828\).
If the weights were wrongly treated as frequency counts, the analyst would instead claim five observations: one value of 0 and four values of 10. That is not the observed data. In this example, literal replication would also produce a weighted residual sum of squares of 80, but the estimated Gaussian variance would be \(80/5=16\), because the replicated-data likelihood has five observations. The resulting plug-in standard error would be \(\sqrt{16/5}\), approximately \(1.789\). The mean estimate and known-\(\sigma\) information happen to agree here; the dispersion estimate does not. Counting precision as replication has invented observations and changed uncertainty about the variance.
A Practical Diagnostic and a Common Mistake
When a reported standard error seems implausibly small or large, trace the calculation back to the response and its likelihood contribution. For binomial data, verify that \(y_i\) is a success count and \(m_i-y_i\) is a failure count. For frequency data, verify that the weight really represents repeated copies of the entire record. For precision weights, verify the variance model or other stated precision convention, and do not infer a number of independent records from the weight alone.
A common mistake is to say, “All three kinds of weights multiply the likelihood, so I can substitute one for another.” The conclusion does not follow. A trial count changes the binomial support and the numbers of successes and failures. A frequency weight repeats a complete likelihood contribution. A precision weight changes the variance or weighted estimating contribution under a specified model. The score or coefficient information can coincide in special cases, but that does not establish that the full likelihood, dispersion estimate, or standard error is the same.
A disciplined check is to compare the implied likelihood with the actual sampling story, then inspect how the weight changes the score and information. If only the coefficient estimate is unchanged, that is not enough: uncertainty can still change through information scaling, likelihood normalization, or the number of independent records used to estimate dispersion.
Check Your Understanding
For each situation, identify the weight interpretation and explain what evidence in the model or likelihood supports your conclusion.
- A response records 6 successes among 15 independent trials. What are the binomial response and trial count, and what does the trial count contribute to logistic-regression information?
- One binary record with outcome 0 has frequency weight 4. How many failures does this represent, and why is that not the same as four trials with zero successes attached to a different response record?
- In the Gaussian model \(Y_i\sim N(\mu_i,\sigma^2/w_i)\), what does increasing \(w_i\) say about the observation? Does it say that \(w_i\) independent observations were collected?
- Why can precision and frequency weights produce similar coefficient information in a Gaussian model but different estimates of dispersion?
- An analyst treats a binomial proportion as one Bernoulli observation and obtains a much larger standard error. Which likelihood and information checks would diagnose the error?