Gaussian Likelihoods with Precision Weights

Why Precision Weights Change the Likelihood

A Gaussian model with equal response variances assigns the same amount of variability to every observation. In some applications, however, the observations have different, known levels of precision. A precise observation should have a narrower Gaussian distribution around its mean than a less precise one. The likelihood must express both this change in spread and the corresponding change in the density's normalizing constant.

Here the observation-specific variances are specified using positive numbers \(w_i\). The weights are known inputs to the model, not parameters to be estimated. We derive the individual densities and their log-likelihood contributions before combining them into a likelihood for independent observations.

Definition: For observations \(Y_1,\ldots,Y_n\), a Gaussian model with known positive precision weights has, conditionally on the covariates and weights, \(Y_i\sim N(\mu_i,\sigma^2/w_i)\), where \(w_i>0\) is known and \(\sigma^2>0\). Thus the variance is \(\sigma^2/w_i\), and the conditional precision (the reciprocal variance) is \(w_i/\sigma^2\). The responses are assumed conditionally independent.

The mean \(\mu_i\) may be specified by a model for the conditional mean. For example, in an identity-link Gaussian regression model it can be written as \(\mu_i=x_i^\mathsf{T}\beta\). The likelihood construction below only requires the specified values \(\mu_i\); it does not require estimating them or deriving an estimating equation.

One Observation: Density and Log-Likelihood Contribution

For one observation, the normal density with mean \(\mu_i\) and variance \(\sigma^2/w_i\) is

f_i(y_i\mid \mu_i,\sigma^2,w_i) = \frac{1}{\sqrt{2\pi\sigma^2/w_i}} \exp\left\{-\frac{(y_i-\mu_i)^2}{2\sigma^2/w_i}\right\} = \left(\frac{w_i}{2\pi\sigma^2}\right)^{1/2} \exp\left\{-\frac{w_i(y_i-\mu_i)^2}{2\sigma^2}\right\}.

Taking the logarithm gives the contribution of observation \(i\):

\ell_i(\mu_i,\sigma^2) = \frac{1}{2}\log w_i -\frac{1}{2}\log(2\pi) -\frac{1}{2}\log\sigma^2 -\frac{w_i(y_i-\mu_i)^2}{2\sigma^2}.

There are two effects of \(w_i\). It multiplies the squared residual in the exponential, and it also appears in the normalizing term \(\frac{1}{2}\log w_i\). Both are required for this expression to be the log of a probability density. The weights are held fixed when treating this as a likelihood in the model parameters.

Worked Example: A Single Observation with Weight Four

Suppose \(y=2\), \(\mu=1\), \(\sigma^2=4\), and the known weight is \(w=4\). The model variance is \(\sigma^2/w=4/4=1\). Therefore the density at the observed value is

f(2\mid 1,4,4) = \frac{1}{\sqrt{2\pi(1)}}\exp\left\{-\frac{(2-1)^2}{2(1)}\right\} = \frac{1}{\sqrt{2\pi}}e^{-1/2} \approx 0.24197.

The log-likelihood contribution, using the expanded expression, is

\ell = \frac{1}{2}\log 4-\frac{1}{2}\log(2\pi) -\frac{1}{2}\log 4-\frac{4(2-1)^2}{2(4)} = -\frac{1}{2}\log(2\pi)-\frac{1}{2} \approx -1.41894.

The two terms involving \(\log 4\) cancel here because \(\sigma^2/w=1\). This agrees with the direct calculation \(\log f=-\frac{1}{2}\log(2\pi)-\frac{1}{2}\).

Independent Observations: The Full Likelihood

Conditional independence means that the joint density is the product of the individual densities. Consequently, the likelihood for the model parameters is

L(\boldsymbol{\mu},\sigma^2) = \prod_{i=1}^n \left(\frac{w_i}{2\pi\sigma^2}\right)^{1/2} \exp\left\{-\frac{w_i(y_i-\mu_i)^2}{2\sigma^2}\right\}, \qquad \sigma^2>0.

Taking the logarithm converts the product into a sum. The complete log-likelihood is

\ell(\boldsymbol{\mu},\sigma^2) = \frac{1}{2}\sum_{i=1}^n\log w_i -\frac{n}{2}\log(2\pi) -\frac{n}{2}\log\sigma^2 -\frac{1}{2\sigma^2}\sum_{i=1}^n w_i(y_i-\mu_i)^2.

Equivalently, the first two terms can be combined as \(\frac{1}{2}\sum_i\log\{w_i/(2\pi\sigma^2)\}\). The displayed form separates the parts involving the known weights, the common dispersion, and the weighted squared residuals. If \(\mu_i\) is specified through regression parameters, substitute that model for each \(\mu_i\); the likelihood formula itself remains the same.

Result: Under conditional independence and the stated Gaussian marginal distributions, the joint likelihood is the product of the \(n\) Gaussian densities, and its log-likelihood is the sum of their log contributions. In particular, the weight-dependent term \(\frac{1}{2}\sum_i\log w_i\) is part of the full log-likelihood.

Proof. Conditional independence gives the joint density \(f(y_1,\ldots,y_n)=\prod_{i=1}^n f_i(y_i\mid\mu_i,\sigma^2,w_i)\). Substituting the individual density derived above yields the stated product likelihood. Since each factor is positive, its logarithm is defined, and the logarithm of a finite product is the sum of the logarithms. For each \(i\), that logarithm is \(\frac{1}{2}\log w_i-\frac{1}{2}\log(2\pi)-\frac{1}{2}\log\sigma^2-\frac{w_i(y_i-\mu_i)^2}{2\sigma^2}\). Summing over \(i=1,\ldots,n\) gives the displayed expression, including the term \(-\frac{n}{2}\log\sigma^2\). \(\square\)

Worked Example: Computing Three Log-Likelihood Contributions

Consider three independent observations with common dispersion \(\sigma^2=2\), known weights \(w_1=1\), \(w_2=4\), and \(w_3=0.25\), observed values \((1.5,1,4)\), and specified means \((1,2,3)\). The residuals are \(0.5,-1,1\). First calculate the weighted squared residual sum:

\sum_{i=1}^3 w_i(y_i-\mu_i)^2 = 1(0.5)^2+4(-1)^2+0.25(1)^2 = 0.25+4+0.25 = 4.5.

The weights have product \(1\cdot4\cdot0.25=1\), so \(\sum_i\log w_i=0\). The full log-likelihood is therefore

\ell = -\frac{3}{2}\log(4\pi)-\frac{4.5}{2(2)} = -\frac{3}{2}\log(4\pi)-1.125 \approx -4.92154.

The separate contributions provide a check. For observation 1, the variance is \(2/1=2\), so \(\ell_1=-\frac{1}{2}\log(4\pi)-0.25/4\approx-1.32801\). For observation 2, the variance is \(2/4=0.5\), so \(\ell_2=-\frac{1}{2}\log(\pi)-4/4\approx-1.57236\). For observation 3, the variance is \(2/0.25=8\), so \(\ell_3=-\frac{1}{2}\log(16\pi)-0.25/4\approx-2.02116\). Adding them gives \(-1.32801-1.57236-2.02116=-4.92153\), matching the full expression up to rounding.

Why the Normalizing Term Matters

The density must integrate to one for every permitted mean, dispersion, and positive weight. This requirement explains why it is not enough to multiply the residual penalty by \(w_i\): changing the variance also changes the height and spread of the density. The following result makes the role of the normalizing factor explicit.

Result: For \(w_i>0\) and \(\sigma^2>0\), the density \(\left(w_i/(2\pi\sigma^2)\right)^{1/2}\exp\{-w_i(y_i-\mu_i)^2/(2\sigma^2)\}\) integrates to one over \(y_i\in\mathbb{R}\).

Proof. Set \(z=(y_i-\mu_i)\sqrt{w_i}/\sigma\). Because \(w_i>0\) and \(\sigma>0\), this is a one-to-one change of variable from \(\mathbb{R}\) to \(\mathbb{R}\), with \(dy_i=(\sigma/\sqrt{w_i})\,dz\). The integral of the proposed density is

\begin{aligned} \int_{-\infty}^{\infty} \left(\frac{w_i}{2\pi\sigma^2}\right)^{1/2} \exp\left\{-\frac{w_i(y_i-\mu_i)^2}{2\sigma^2}\right\}\,dy_i &= \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}e^{-z^2/2}\,dz\\ &=1, \end{aligned}

where the last equality is the normalization of the standard normal density. Thus the factor \(\sqrt{w_i}/\sigma\) in the density is essential. \(\square\)

Worked Example: Detecting an Omitted Normalizer

Let \(w=4\), \(\sigma^2=1\), and let the mean be any fixed real number \(\mu\). The correct density is

f(y\mid\mu,1,4) = \frac{2}{\sqrt{2\pi}}\exp\{-2(y-\mu)^2\}.

Its integral is one: with \(z=2(y-\mu)\), we have \(dy=dz/2\), and

\int_{-\infty}^{\infty}\frac{2}{\sqrt{2\pi}}e^{-2(y-\mu)^2}\,dy = \int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}}e^{-z^2/2}\,dz = 1.

If the factor \(\sqrt{w}=2\) were omitted while retaining the exponential term, the integral would instead be \(1/2\). That expression would not be a probability density and would give an incorrect likelihood.

Comparing Mean Specifications at Fixed Dispersion

The full expression also gives a simple likelihood comparison when two candidate mean vectors are evaluated using the same observed responses, weights, and dispersion. This comparison is a property of the likelihood; it does not require fitting either candidate.

Result: Let \(\boldsymbol{\mu}\) and \(\boldsymbol{\nu}\) be two specified mean vectors, with the same \(y_i\), \(w_i\), and \(\sigma^2>0\). Then

\ell(\boldsymbol{\mu},\sigma^2)-\ell(\boldsymbol{\nu},\sigma^2) = -\frac{1}{2\sigma^2}\sum_{i=1}^n w_i \left\{(y_i-\mu_i)^2-(y_i-\nu_i)^2\right\}.

Proof. Write the full log-likelihood once for \(\boldsymbol{\mu}\) and once for \(\boldsymbol{\nu}\). In each expression the terms \(\frac{1}{2}\sum_i\log w_i\), \(-\frac{n}{2}\log(2\pi)\), and \(-\frac{n}{2}\log\sigma^2\) are identical because the weights and dispersion are held fixed. Subtracting the expressions cancels those terms, leaving exactly the displayed difference of weighted squared-residual sums multiplied by \(-1/(2\sigma^2)\). \(\square\)

Common Mistakes and Interpretation

Dropping the weight from the normalizer. Keeping \(w_i(y_i-\mu_i)^2\) in the exponent but omitting \(\frac{1}{2}\log w_i\) does not give the stated Gaussian density. The normalizing term follows from the variance \(\sigma^2/w_i\).
Using \(\sigma^2w_i\) as the variance. Under this model, the variance is \(\sigma^2/w_i\), not \(\sigma^2w_i\). A larger \(w_i\) means a smaller variance.
Treating weights as estimated parameters. Here the \(w_i\) are known positive inputs. If their values were unknown, their role and the likelihood's parameterization would need to be specified differently.
Calling every weight a replication count. These weights specify observation-specific variances in a Gaussian model. They are not automatically counts of independent repeated observations; that would be a different sampling description.
Discarding terms without checking the comparison. Terms constant in the mean parameters can be omitted when comparing means at fixed weights and dispersion. They cannot be discarded indiscriminately when the dispersion or weights vary across the likelihoods being compared.

Key takeaway: With independent responses \(Y_i\sim N(\mu_i,\sigma^2/w_i)\) and known \(w_i>0\), each log-likelihood contribution contains both \(\frac{1}{2}\log w_i\) and the weighted residual term \(-w_i(y_i-\mu_i)^2/(2\sigma^2)\). The complete likelihood includes both.

Check Your Understanding

Use the specified variance model throughout your answers.

If \(\sigma^2=6\) and \(w_i=3\), what is the variance and what is the precision of \(Y_i\)?
Write the log-likelihood contribution for one observation \(y_i\), leaving \(\mu_i\), \(w_i\), and \(\sigma^2\) symbolic.
For two independent observations, explain why their Gaussian densities multiply and their log-likelihood contributions add.
For one observation with \(\sigma^2=1\) and \(w_i=4\), what multiplicative factor in the density would be missing if the weight-dependent normalizer were omitted?
When comparing two mean vectors with identical weights and dispersion, which terms in the full log-likelihood cancel?