Rescaling Weights and Dispersion

Why the Scale of the Weights Matters

Weights can express relative precision, count replicated observations, or define a weighted likelihood criterion. Those interpretations are not interchangeable. A useful first question is: what changes when every weight is multiplied by the same positive constant? The fitted coefficient estimates may stay the same while the reported standard errors change, or a corresponding change in dispersion may cancel that change.

The distinction is important because likelihood curvature measures information on the scale specified by the model. Multiplying a likelihood criterion by a constant increases its curvature, but that does not necessarily mean the data contain more independent information. Whether it does depends on what the weights represent and how dispersion is treated.

Definition: Let $w_i>0$ be the weights for $n$ independent observations, and let $c>0$. A common rescaling replaces every weight by $w_i^*=cw_i$. In a weighted log-likelihood criterion, $\ell_w(\beta)=\sum_{i=1}^n w_i\ell_i(\beta)$, the weights multiply the individual log-likelihood contributions. In a variance-weight GLM, weights instead specify relative precision through $\operatorname{Var}(Y_i)=\phi V(\mu_i)/w_i$. The consequences of rescaling depend on which interpretation is in use.

What Happens to Likelihood Curvature?

First consider a weighted log-likelihood criterion $\ell_w(\beta)$, with dispersion and all other model components held fixed. Assume the criterion is twice differentiable in $\beta$, and that the same parameter space is used before and after rescaling. The rescaled criterion is $\ell_{cw}(\beta)=c\ell_w(\beta)$. The next result describes its maximizer and curvature.

Theorem: If $c>0$, then $\ell_{cw}(\beta)=c\ell_w(\beta)$ has the same maximizers as $\ell_w(\beta)$. Its score and observed information are $c$ times the original score and observed information. If the information at the maximizer is nonsingular, the usual model-based covariance estimate from its inverse is divided by $c$, so the corresponding standard errors are divided by $\sqrt{c}$.

Proof. Since $c>0$, for any two parameter values $\beta_1,\beta_2$,

\ell_{cw}(\beta_1)\geq \ell_{cw}(\beta_2) \quad\Longleftrightarrow\quad c\ell_w(\beta_1)\geq c\ell_w(\beta_2) \quad\Longleftrightarrow\quad \ell_w(\beta_1)\geq \ell_w(\beta_2).

Thus the sets of maximizers agree. Differentiating the rescaled criterion gives $U_{cw}(\beta)=\partial\ell_{cw}(\beta)/\partial\beta=cU_w(\beta)$. Its observed information is $J_{cw}(\beta)=-\partial^2\ell_{cw}(\beta)/\partial\beta\partial\beta^\mathsf{T}=cJ_w(\beta)$. At a common maximizer, if $J_w$ is nonsingular, then $J_{cw}^{-1}=(cJ_w)^{-1}=J_w^{-1}/c$. Taking standard errors as square roots of the diagonal covariance entries gives the factor $1/\sqrt{c}$. $\square$

This is a statement about the curvature of the specified criterion. It is not, by itself, a claim that rescaling arbitrary weights has created new independent observations. If $w_i$ are genuine frequency weights and $c$ is an integer, the rescaled criterion can correspond to replicating each observation $c$ times. If they are relative-precision weights, a common multiplier may simply be a change of scale that should be matched by dispersion.

Worked Example: Fixed Dispersion and a Poisson Intercept

Suppose independent Poisson counts $Y_i$ have means $\mu=\exp(\beta)$, and consider the weighted log-likelihood criterion with observations $y=(2,4)$ and weights $w=(1,2)$. The Poisson dispersion is fixed at 1.

Ignoring terms that do not depend on $\beta$, the criterion is $\ell_w(\beta)=\sum_i w_i\{y_i\beta-\exp(\beta)\}$. Its score is $U_w(\beta)=\sum_i w_i(y_i-\exp(\beta))$. Setting the score to zero gives

\hat{\mu}=\exp(\hat{\beta}) =\frac{\sum_i w_i y_i}{\sum_i w_i} =\frac{1(2)+2(4)}{1+2} =\frac{10}{3}.

The information for $\beta$ at the estimate is $\sum_i w_i\hat{\mu}=3(10/3)=10$, so the model-based standard error is $1/\sqrt{10}$, approximately $0.316$. Now multiply both weights by $c=4$. The estimate remains $10/3$, while the information becomes $40$. The standard error is now $1/\sqrt{40}$, approximately $0.158$, exactly half its original value. Here dispersion is fixed, so there is no dispersion rescaling to offset the curvature change.

When Dispersion Cancels the Rescaling

For a GLM whose variance is written $\operatorname{Var}(Y_i)=\phi V(\mu_i)/w_i$, the weights set relative precision and $\phi$ sets the overall variance scale. Multiplying both by the same positive factor in the sense $w_i^*=cw_i$ and $\phi^*=c\phi$ preserves each observation's variance:

\frac{\phi^* V(\mu_i)}{w_i^*} = \frac{c\phi V(\mu_i)}{cw_i} = \frac{\phi V(\mu_i)}{w_i}.

The same compensation appears in the GLM coefficient estimating equation and its model-based information. The following result states the conditions and conclusion precisely.

Theorem: Consider a GLM with independent responses, fixed design vectors $x_i$, mean $\mu_i$, differentiable link $\eta_i=x_i^\mathsf{T}\beta$, and variance $\operatorname{Var}(Y_i)=\phi V(\mu_i)/w_i$. Assume $V(\mu_i)>0$, $\phi>0$, and that the sums below exist. The quasi-score and its model-based information are $$ U_\beta=\sum_i \frac{w_i}{\phi} \frac{d\mu_i}{d\eta_i}\frac{y_i-\mu_i}{V(\mu_i)}x_i, \qquad I_\beta=\sum_i \frac{w_i}{\phi} \frac{(d\mu_i/d\eta_i)^2}{V(\mu_i)}x_ix_i^\mathsf{T}. $$ If the weights and dispersion are changed to $cw_i$ and $c\phi$, respectively, then the response variances, quasi-score, and model-based information are unchanged. Consequently, whenever the estimating equation has the same solution and the information is nonsingular, the model-based coefficient covariance is unchanged.

Proof. The variance equality follows from $(c\phi)/(cw_i)=\phi/w_i$. In each summand of the score and information, the factor $w_i/\phi$ is also unchanged:

\frac{cw_i}{c\phi}=\frac{w_i}{\phi}.

All other factors in the summands are functions of the same observations, design vectors, and mean parameters. Thus the entire score function and information matrix are unchanged at every $\beta$. Their estimating-equation solutions and, when nonsingular, the inverse-information covariance are therefore unchanged. $\square$

This result applies when weights represent relative precision and dispersion is free to change scale accordingly. It does not say that any weight rescaling can be absorbed into dispersion. In particular, if dispersion is fixed by the model, or if weights represent actual replication, changing dispersion may not preserve the intended statistical model.

Worked Example: Relative-Precision Weights With Estimated Dispersion

Consider a Gaussian identity-link model with an intercept only, two observations $y_1=1$, $y_2=3$, and relative-precision weights $w_1=1$, $w_2=4$. The weighted estimate of the common mean is

\hat{\mu}=\frac{1(1)+4(3)}{1+4}=\frac{13}{5}=2.6.

The weighted residual sum of squares is $1(1-2.6)^2+4(3-2.6)^2=2.56+0.64=3.2$. Using the Pearson estimate with residual degrees of freedom $n-p=2-1=1$, we obtain $\hat{\phi}=3.2$. The model-based variance estimate for the fitted mean is $\hat{\phi}/\sum_iw_i=3.2/5=0.64$, giving standard error $0.8$.

Now multiply both weights by 9. The fitted mean remains $2.6$, and the weighted residual sum of squares becomes $9(3.2)=28.8$. The same residual degrees of freedom give $\hat{\phi}^{*}=28.8$, while the total weight is $45$. The variance estimate is $28.8/45=0.64$, again giving standard error $0.8$. In this relative-precision analysis, dispersion has scaled with the weights, so the model-based uncertainty is unchanged.

Replication Weights and Fixed-Dispersion Models

A frequency weight is different: it represents how many times an observation occurs. Multiplying frequency weights increases the number of represented observations, rather than merely changing a precision scale. For a model with fixed dispersion, the resulting increase in information is meaningful under that replication interpretation.

Worked Example: Replicated Bernoulli Outcomes

Suppose a logistic intercept model has 3 successes and 7 failures, so the fitted probability is $\hat{p}=3/10=0.3$. The information for the intercept $\beta=\log\{\hat{p}/(1-\hat{p})\}$ is $n\hat{p}(1-\hat{p})$. For these ten Bernoulli observations it is $10(0.3)(0.7)=2.1$, giving model-based standard error $1/\sqrt{2.1}$, approximately $0.690$.

If the frequency weights are multiplied by 4, the expanded data contain 12 successes and 28 failures. The fitted probability remains $12/40=0.3$, but information is now $40(0.3)(0.7)=8.4$, and the standard error is $1/\sqrt{8.4}$, approximately $0.345$. The uncertainty is smaller because the frequency-weight interpretation says there are four times as many independent Bernoulli observations. A free dispersion parameter cannot be introduced to erase this change without changing the model.

Common Pitfalls and Practical Checks

A frequent error is to conclude that multiplying all weights by $c$ always leaves fitted standard errors divided by $\sqrt{c}$. That conclusion follows for inverse-information standard errors when the weighted criterion is multiplied by $c$ and dispersion is held fixed. Under variance weights with an estimated dispersion, the dispersion estimate may also scale, cancelling the change. Under frequency weights, the multiplier may instead represent additional observations.

Another distinction concerns robust, or sandwich, standard errors. Under a pure global rescaling of an estimating equation, the model-based inverse information scales as $1/c$. But a sandwich covariance has the form $A^{-1}BA^{-1}$, where $A$ is the derivative of the estimating equation and $B$ is its variability estimate. If both are computed from the same weighted contributions, rescaling gives $A^*=cA$ and $B^*=c^2B$, so

(A^*)^{-1}B^*(A^*)^{-1} =(cA)^{-1}(c^2B)(cA)^{-1} =A^{-1}BA^{-1}.

Thus a sandwich standard error need not follow the $1/\sqrt{c}$ rule for model-based standard errors. The interpretation and construction of the weights, as well as the covariance estimator, must be stated before comparing reported uncertainties.

Key takeaway: With fixed dispersion, scaling a weighted likelihood by $c$ preserves its maximizer and multiplies its information by $c$, reducing model-based standard errors by $1/\sqrt{c}$. With relative-precision weights, scaling both weights and dispersion by $c$ preserves the variance model and model-based covariance. Frequency weights and fixed-dispersion models require their own interpretation; they cannot be treated as interchangeable with a dispersion adjustment.

Check Your Understanding

For each question, distinguish the weight interpretation from the algebraic effect of rescaling.

If a weighted log-likelihood criterion is multiplied by 9 and dispersion is fixed, by what factor do its inverse-information standard errors change?
In a variance-weight GLM, explain why changing both every weight and dispersion by a factor of 5 preserves the response variance.
Why does multiplying frequency weights by 4 in a Bernoulli model represent more than a harmless change of weight units?
In the Gaussian example, what would happen to the model-based variance estimate if weights were multiplied by 9 but the dispersion estimate were incorrectly held fixed?
Why can a sandwich standard error remain unchanged under a common weight multiplier even when an inverse-information standard error changes?