Known Precision Weights and Relative Variance

Why Precision Weights Need an Interpretation

In a Gaussian regression, observations need not have the same variance. One measurement may be more variable than another, and a weighted fit gives relatively more influence to observations that are assumed to be more precise. To interpret this correctly, it is important to distinguish the relative precision specified by the weights from the overall variance scale.

For independent Gaussian responses with means \(\mu_i=x_i^\mathsf{T}\beta\), the standard precision-weight representation is \(\operatorname{Var}(Y_i)=\phi/w_i\). Here each \(w_i\) is known and positive, while \(\phi>0\) is a common dispersion parameter. The weights determine how the variances compare; the common dispersion determines their overall scale.

Definition: In a Gaussian model with known precision weights \(w_i>0\), write \[ Y_i \sim N(\mu_i,\phi/w_i), \qquad \mu_i=x_i^\mathsf{T}\beta, \] with independent observations and a common \(\phi>0\). The weights are relative precision weights: a larger \(w_i\) means a smaller variance relative to observations with smaller weights. The quantity \(\phi\) is the common dispersion scale.

The word “precision” refers to the reciprocal of variance, up to the shared scale: \(1/\operatorname{Var}(Y_i)=w_i/\phi\). Thus \(w_i\) is not necessarily the absolute reciprocal variance. It is the relative part of that reciprocal. Multiplying every weight by the same positive number changes their common scale, not their ratios.

Reading Weight Ratios as Variance Ratios

Suppose observations \(i\) and \(j\) have positive weights \(w_i\) and \(w_j\), and both follow the same dispersion model. Their variance ratio is

\frac{\operatorname{Var}(Y_i)}{\operatorname{Var}(Y_j)} = \frac{\phi/w_i}{\phi/w_j} = \frac{w_j}{w_i}.

The order of the weights reverses in the variance ratio. If \(w_i\) is four times \(w_j\), then observation \(i\) has one quarter of observation \(j\)'s variance. Equivalently, its precision \(1/\operatorname{Var}(Y_i)\) is four times as large. This statement concerns variance, not standard deviation: the standard deviation ratio is the square root of the variance ratio.

Worked Example: Comparing Variability From Weight Ratios

Three independent responses have weights \(w_1=2\), \(w_2=8\), and \(w_3=4\). Their variances are \(\phi/2\), \(\phi/8\), and \(\phi/4\). Compare the variability of the first two observations and then the first and third.

For observations 1 and 2, \[ \frac{\operatorname{Var}(Y_1)}{\operatorname{Var}(Y_2)} =\frac{w_2}{w_1} =\frac{8}{2}=4. \] Thus observation 1 has four times the variance of observation 2; observation 2 is four times as precise.

For observations 1 and 3, \[ \frac{\operatorname{Var}(Y_1)}{\operatorname{Var}(Y_3)} =\frac{w_3}{w_1} =\frac{4}{2}=2. \] So observation 1 has twice the variance of observation 3. These comparisons do not require knowing \(\phi\), because the common scale cancels.

A weight ratio is the reliable way to compare observations. The raw value of a weight by itself does not tell us an absolute variance unless the dispersion scale is also specified. For example, a weight of 8 does not, by itself, imply variance \(1/8\); that conclusion would require \(\phi=1\).

Worked Example: Turning a Variance Pattern Into Weights

Suppose three responses are modeled with variances \(12\), \(3\), and \(6\), respectively. Write these variances in the common-dispersion form and identify the relative weights.

Choose \(\phi=6\). Then the weights required to give the stated variances are \[ w_1=\frac{6}{12}=\frac12,\qquad w_2=\frac{6}{3}=2,\qquad w_3=\frac{6}{6}=1. \] Indeed, \(\phi/w_1=6/(1/2)=12\), \(\phi/w_2=6/2=3\), and \(\phi/w_3=6/1=6\).

The weight for the second response is four times the weight for the first, and its variance is one quarter as large: \(3/12=1/4\). Another choice of common scale would represent the same variance pattern. For instance, doubling all weights and doubling \(\phi\) leaves every variance unchanged.

Why a Common Rescaling Leaves Coefficients Unchanged

For a Gaussian regression with an identity mean, the weighted least-squares criterion is the weighted sum of squared residuals. Assume the design matrix \(X\) has full column rank and all weights are positive. In matrix notation, let \(W\) be the diagonal matrix with entries \(w_1,\ldots,w_n\), and let \(y\) be the response vector. The coefficient estimate minimizes \((y-X\beta)^\mathsf{T}W(y-X\beta)\).

Theorem: Let \(W\) be diagonal with strictly positive entries, and let \(X\) have full column rank. For any \(c>0\), the unique minimizer of \[ Q_W(\beta)=(y-X\beta)^\mathsf{T}W(y-X\beta) \] is also the unique minimizer of \(Q_{cW}(\beta)\). Thus multiplying all precision weights by one common positive constant does not change the weighted least-squares coefficient estimate.

Proof. For every coefficient vector \(\beta\),

Q_{cW}(\beta) =(y-X\beta)^\mathsf{T}(cW)(y-X\beta) =c\,Q_W(\beta).

Since \(c>0\), for any two coefficient vectors \(\beta_1,\beta_2\), \(Q_W(\beta_1)<Q_W(\beta_2)\) if and only if \(cQ_W(\beta_1)<cQ_W(\beta_2)\). Therefore the two criteria have exactly the same minimizers. Full column rank and positive weights make \(X^\mathsf{T}WX\) positive definite, so the minimizer is unique. \(\square\)

The same conclusion follows from the weighted normal equations. A minimizer satisfies \(X^\mathsf{T}WX\hat{\beta}=X^\mathsf{T}Wy\). Replacing \(W\) by \(cW\) multiplies both sides by \(c\), which does not alter the solution. This argument also shows that the fitted values are unchanged. If \(X\) is rank deficient, the set of minimizing coefficient vectors is still unchanged by the positive rescaling, although a unique coefficient vector is not guaranteed.

Worked Example: A Weighted Line Before and After Rescaling

Fit a line \(\mu_i=a+bx_i\) to \((x_i,y_i)=(0,1),(1,2),(2,4)\) with weights \(1,2,1\). Then multiply every weight by 5 and check the coefficient estimates.

For the original weights, the weighted normal equations use \[ \sum_i w_i=4,\quad \sum_i w_ix_i=4,\quad \sum_i w_iy_i=9,\quad \sum_i w_ix_i^2=6,\quad \sum_i w_ix_iy_i=12. \] Thus \[ \begin{pmatrix}4&4\\4&6\end{pmatrix} \begin{pmatrix}a\\b\end{pmatrix} = \begin{pmatrix}9\\12\end{pmatrix}. \] The determinant is \(4(6)-4(4)=8\), and solving gives \[ a=\frac{6(9)-4(12)}{8}=\frac{6}{8}=0.75,\qquad b=\frac{4(12)-4(9)}{8}=\frac{12}{8}=1.5. \]

After multiplying the weights by 5, every entry in the normal equations is multiplied by 5: \[ \begin{pmatrix}20&20\\20&30\end{pmatrix} \begin{pmatrix}a\\b\end{pmatrix} = \begin{pmatrix}45\\60\end{pmatrix}. \] The solution remains \(a=0.75\), \(b=1.5\). For a direct check, the residuals at the three observations are \(0.25,-0.25,0.25\). The original weighted sum of squares is \[ 1(0.25)^2+2(-0.25)^2+1(0.25)^2=0.25. \] With weights multiplied by 5 it is \(5(0.25)=1.25\), but the minimizing line is unchanged.

What the Common Scale Does—and Does Not—Mean

The weight-rescaling result is about coefficient estimates, not about leaving every part of the model untouched. Under the variance model \(\operatorname{Var}(Y_i)=\phi/w_i\), replacing \(w_i\) by \(cw_i\) while keeping \(\phi\) fixed changes each variance to \(\phi/(cw_i)\), which is \(1/c\) of its former value. That is a different variance specification.

To preserve the same variances while using the rescaled weights, replace \(\phi\) by \(c\phi\):

\frac{c\phi}{cw_i}=\frac{\phi}{w_i}.

When the common dispersion is free to vary, a common rescaling of the weights can therefore be understood as a change of scale convention: it changes the numerical representation of the dispersion, but not the relative variance pattern or the fitted coefficients. If the dispersion is fixed at a particular value and is not rescaled, changing all weights changes the assumed variances and is not merely a change of convention.

Worked Example: Keeping Variances Fixed Under Rescaling

Two responses have weights \(w_1=1\), \(w_2=4\), and dispersion \(\phi=3\). Their variances are \(3/1=3\) and \(3/4=0.75\). Now multiply both weights by \(c=2\).

The new weights are 2 and 8. To preserve the same variances, use the new dispersion \(\phi'=c\phi=6\). The resulting variances are \(6/2=3\) and \(6/8=0.75\), exactly as before. The weight ratio remains \(8/2=4\), so the second response remains one quarter as variable as the first.

If instead the dispersion were held at 3, the new variances would be \(3/2=1.5\) and \(3/8=0.375\). The relative pattern would still be the same, but both variances would have been halved. This is a changed variance model, not an unchanged model written with a different scale.

Common Mistakes

Reading a larger weight as a larger variance. The variance is proportional to \(1/w_i\), so larger weight means smaller variance.
Calling \(w_i\) the absolute precision without qualification. Under this convention, the absolute precision is \(w_i/\phi\). The weight alone gives relative precision unless \(\phi\) is specified.
Assuming rescaling weights changes nothing at all. It preserves the weighted least-squares coefficient estimates, but the numerical weighted residual sum of squares scales by the same constant. With fixed dispersion, the implied variances also change.
Confusing a precision weight with a frequency count. A precision weight describes relative measurement variability in this model; it does not, solely by its numerical value, assert that an observation represents that many replicated observations.

Key takeaway: In the model \(\operatorname{Var}(Y_i)=\phi/w_i\), weight ratios determine variance ratios in reverse: \(\operatorname{Var}(Y_i)/\operatorname{Var}(Y_j)=w_j/w_i\). Multiplying every weight by the same positive constant preserves the weighted least-squares coefficient estimates. Interpret the overall scale together with the dispersion parameter.

Check Your Understanding

Use the common-dispersion model to answer each question.

If \(w_1=3\) and \(w_2=12\), what is \(\operatorname{Var}(Y_1)/\operatorname{Var}(Y_2)\), and which observation is more precise?
With \(\phi=5\) and \(w_i=2\), what is \(\operatorname{Var}(Y_i)\)? Why would the weight alone not give this answer?
Explain why multiplying every weight by 7 leaves the weighted least-squares coefficient minimizer unchanged.
If all weights are multiplied by 4 but \(\phi\) is held fixed, by what factor does each variance change?
What change to \(\phi\) preserves the original variances when every weight is multiplied by a positive constant \(c\)?