Tutorials › Generalized Linear Models › Score Equations for Gaussian Regression Coefficients

Home  /  Tutorials  /  Gaussian GLMs and Weights

Gaussian GLMs and Weights · Tutorial 329 of 981

Score Equations for Gaussian Regression Coefficients

Learn to differentiate the Gaussian regression log-likelihood and express its coefficient and variance score equations clearly in residual and matrix notation.

Beginner 9 min read

What You'll Learn

  • Write the Gaussian regression log-likelihood in terms of the design matrix and residual vector
  • Differentiate with respect to the regression coefficients and the variance
  • Express each coefficient score as a residual-weighted design-column sum
  • State the joint zero-score equations and when the variance equation has an interior solution
  • Check score calculations numerically and identify common derivative errors

From the Gaussian Likelihood to Score Equations

In Gaussian regression, the coefficient score describes how the log-likelihood changes when the fitted mean is perturbed through the regression coefficients. Each coefficient score is built from residuals: observations with residuals aligned with a design column push the log-likelihood in that column's direction. The variance score, in turn, compares the residual sum of squares with the current variance scale.

The goal here is to derive these statements precisely. We use a fixed design matrix and independent Gaussian responses with a common positive variance. The scores are derivatives of the log-likelihood; setting them equal to zero gives necessary stationary-point equations for an interior parameter value, not by itself a claim about a global maximum.

Definition: Let \(Y_1,\ldots,Y_n\) be independent, with \(Y_i\sim N(\mu_i,\sigma^2)\), where \(\sigma^2>0\), and let \(\mu_i=\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}\). Here \(\mathbf{x}_i^{\mathsf T}\) is row \(i\) of a fixed \(n\times p\) design matrix \(\mathbf{X}\), \(\boldsymbol{\beta}\in\mathbb{R}^p\), and \(\mathbf{y}=(Y_1,\ldots,Y_n)^{\mathsf T}\). Define the fitted mean vector \(\mathbf{X}\boldsymbol{\beta}\), residual vector \(\mathbf{r}=\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\), and residual sum of squares \(\operatorname{RSS}(\boldsymbol{\beta})=\mathbf{r}^{\mathsf T}\mathbf{r}\). The score for a parameter is the derivative of the log-likelihood with respect to that parameter; for \(\boldsymbol{\beta}\), it is a \(p\)-component gradient.

The assumptions are important: observations are independent conditional on the fixed design, each has the stated Gaussian density, and the common variance parameter is strictly positive. There are no precision weights in this likelihood.

$$ L(\boldsymbol{\beta},\sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left\{ -\frac{(Y_i-\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta})^2}{2\sigma^2} \right\}. $$

Taking logarithms turns the product into a sum. In matrix notation, the sum of squared deviations from the fitted means is \(\mathbf{r}^{\mathsf T}\mathbf{r}\), so the log-likelihood is

$$ \ell(\boldsymbol{\beta},\sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) -\frac{1}{2\sigma^2} (\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^{\mathsf T} (\mathbf{y}-\mathbf{X}\boldsymbol{\beta}). $$

The Coefficient Score

First consider one coefficient, \(\beta_j\). The \(i\)th residual is \(r_i=Y_i-\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}\). Since \(\partial r_i/\partial\beta_j=-x_{ij}\), differentiating its squared contribution gives

$$ \frac{\partial}{\partial\beta_j} \left[-\frac{r_i^2}{2\sigma^2}\right] = -\frac{1}{2\sigma^2}\,2r_i(-x_{ij}) = \frac{x_{ij}r_i}{\sigma^2}. $$

Summing over observations gives the \(j\)th component of the coefficient score. Collecting all \(p\) components yields the compact matrix expression.

Result: Under the Gaussian regression model above, the coefficient score is
$$ \mathbf{U}_{\boldsymbol{\beta}}(\boldsymbol{\beta},\sigma^2) = \frac{\partial\ell}{\partial\boldsymbol{\beta}} = \frac{1}{\sigma^2}\mathbf{X}^{\mathsf T} (\mathbf{y}-\mathbf{X}\boldsymbol{\beta}) = \frac{1}{\sigma^2}\mathbf{X}^{\mathsf T}\mathbf{r}. $$
Its \(j\)th component is \(U_{\beta_j}=\sigma^{-2}\sum_{i=1}^n x_{ij}r_i\).

The matrix formula follows component by component: the \(j\)th entry of \(\mathbf{X}^{\mathsf T}\mathbf{r}\) is \(\sum_i x_{ij}r_i\). Thus the score is large in a coefficient direction when the residuals have a substantial signed association with that design column. The common factor \(1/\sigma^2\) changes the score's scale but, because \(\sigma^2>0\), not the zero-score condition.

Zero-score equation for the coefficients: At any interior point with \(\sigma^2>0\), the coefficient score is zero if and only if
$$ \mathbf{X}^{\mathsf T}\mathbf{r} = \mathbf{X}^{\mathsf T}(\mathbf{y}-\mathbf{X}\boldsymbol{\beta}) = \mathbf{0}. $$
Equivalently, \(\sum_i x_{ij}r_i=0\) for every coefficient \(j\).

Worked Example: Computing Both Scores at a Given Parameter Value

Take three observations with an intercept and one predictor \(x=(0,1,2)^{\mathsf T}\). Thus \(\mathbf{X}=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}\). Let \(\mathbf{y}=(1,2,4)^{\mathsf T}\), \(\boldsymbol{\beta}=(0,1)^{\mathsf T}\), and \(\sigma^2=2\). Calculate the residual vector and both scores.

The fitted vector is \(\mathbf{X}\boldsymbol{\beta}=(0,1,2)^{\mathsf T}\), so \(\mathbf{r}=(1,1,2)^{\mathsf T}\) and \(\operatorname{RSS}=1^2+1^2+2^2=6\). For the coefficient score,

$$ \mathbf{X}^{\mathsf T}\mathbf{r} = \begin{pmatrix}1&1&1\\0&1&2\end{pmatrix} \begin{pmatrix}1\\1\\2\end{pmatrix} = \begin{pmatrix}4\\5\end{pmatrix}, \qquad \mathbf{U}_{\boldsymbol{\beta}} = \frac{1}{2} \begin{pmatrix}4\\5\end{pmatrix} = \begin{pmatrix}2\\2.5\end{pmatrix}. $$

Neither coefficient score is zero at these parameter values. For the variance score derived below, the value is \(-3/(2\cdot2)+6/(2\cdot2^2)=-3/4+3/4=0\). This illustrates that a zero variance score does not imply zero coefficient scores.

The Variance Score and Joint Equations

The parameter in this derivative is the variance \(\sigma^2\), not the standard deviation \(\sigma\). Write the log-likelihood as a function of \(v=\sigma^2>0\), holding \(\boldsymbol{\beta}\) fixed. The residual sum of squares does not depend on \(v\), so differentiating the two terms gives

$$ \frac{\partial\ell}{\partial\sigma^2} = -\frac{n}{2\sigma^2} + \frac{\operatorname{RSS}(\boldsymbol{\beta})}{2(\sigma^2)^2}. $$

In the second term, the derivative of \(-\operatorname{RSS}/(2\sigma^2)\) is positive: differentiating \(1/\sigma^2\) with respect to \(\sigma^2\) gives \(-1/(\sigma^2)^2\). Keeping track of this sign is essential.

Result: The joint zero-score equations for an interior parameter value \((\boldsymbol{\beta},\sigma^2)\), with \(\sigma^2>0\), are
$$ \mathbf{X}^{\mathsf T}(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})=\mathbf{0}, \qquad -\frac{n}{2\sigma^2} + \frac{\operatorname{RSS}(\boldsymbol{\beta})}{2(\sigma^2)^2} =0. $$
If \(\operatorname{RSS}(\boldsymbol{\beta})>0\), the second equation is equivalent to \(\sigma^2=\operatorname{RSS}(\boldsymbol{\beta})/n\). If the residual sum of squares is zero, the variance score is \(-n/(2\sigma^2)\), so it has no zero at any finite positive variance.

To see the algebra in the positive-RSS case, multiply the variance equation by \(2(\sigma^2)^2\), which is positive, to obtain \(-n\sigma^2+\operatorname{RSS}=0\). Rearranging gives the stated equation. This is an equivalence between the zero variance score and a relation involving the residual sum of squares; it does not establish that a joint stationary point exists or is a maximum.

Worked Example: Checking Zero Scores with an Intercept and Slope

Use \(x=(0,1,2)^{\mathsf T}\), an intercept, and \(\mathbf{y}=(3,0,5)^{\mathsf T}\). Evaluate the scores at \(\boldsymbol{\beta}=(2,1)^{\mathsf T}\) and \(\sigma^2=2\).

The fitted values are \((2,3,4)^{\mathsf T}\), giving residuals \(\mathbf{r}=(1,-3,1)^{\mathsf T}\). The residual sum is \(1-3+1=-1\), so this choice does not give a zero intercept score. To check the equations directly,

$$ \mathbf{X}^{\mathsf T}\mathbf{r} = \begin{pmatrix}1&1&1\\0&1&2\end{pmatrix} \begin{pmatrix}1\\-3\\1\end{pmatrix} = \begin{pmatrix}-1\\-1\end{pmatrix}. $$

A zero-score illustration can be obtained by changing the response to \(\mathbf{y}=(3,0,7)^{\mathsf T}\), which changes only the last residual to \(3\). Instead, for a simpler exact construction, take \(\mathbf{y}=(3,0,5)^{\mathsf T}\) and \(\boldsymbol{\beta}=(2,1/2)^{\mathsf T}\): the fitted vector is \((2,2.5,3)^{\mathsf T}\) and residual vector is \((1,-2.5,2)^{\mathsf T}\), whose sum is \(0.5\), again not zero. These checks show why both column sums must be verified rather than presumed.

For a valid exact example, take instead \(\mathbf{y}=(3,0,5)^{\mathsf T}\) and \(\boldsymbol{\beta}=(3,1)^{\mathsf T}\). The fitted vector is \((3,4,5)^{\mathsf T}\) and \(\mathbf{r}=(0,-4,0)^{\mathsf T}\), so the slope score is nonzero. We will use a balanced four-observation design in the next example to construct residuals orthogonal to both columns exactly.

Worked Example: Zero Coefficient Scores in a Two-Predictor Design

Consider four observations and design columns for an intercept, \(x_1=(0,0,1,1)^{\mathsf T}\), and \(x_2=(0,1,0,1)^{\mathsf T}\). Let \(\boldsymbol{\beta}=(1,2,-1)^{\mathsf T}\), and take \(\mathbf{y}=(2,-1,2,3)^{\mathsf T}\).

The fitted vector is \(\mathbf{X}\boldsymbol{\beta}=(1,0,3,2)^{\mathsf T}\), so \(\mathbf{r}=(1,-1,-1,1)^{\mathsf T}\). Its inner products with the three design columns are

$$ \sum_i r_i=1-1-1+1=0,\qquad \sum_i x_{i1}r_i=-1+1=0,\qquad \sum_i x_{i2}r_i=-1+1=0. $$

Therefore \(\mathbf{X}^{\mathsf T}\mathbf{r}=\mathbf{0}\), and the coefficient score is zero for every positive variance. Also, \(\operatorname{RSS}=1+1+1+1=4\). At \(\sigma^2=1\), the variance score is \(-4/2+4/(2\cdot1^2)=0\). Thus both score equations hold at this parameter value.

Interpreting the Equations and Avoiding Pitfalls

The coefficient equations say that the residual vector has zero inner product with every column of the design matrix. This is a statement about the design directions represented in the model, not a requirement that every residual be zero. The variance equation uses the total squared residual, rather than a signed sum, and its denominator involves the variance squared because it is differentiated with respect to \(\sigma^2\).

  • Using the wrong sign. Since \(\partial r_i/\partial\beta_j=-x_{ij}\), differentiating \(-r_i^2/(2\sigma^2)\) gives \(+x_{ij}r_i/\sigma^2\), not its negative.
  • Confusing variance and standard deviation. The formula above differentiates with respect to \(\sigma^2\). Differentiating with respect to \(\sigma\) would require a chain rule and would give a different expression.
  • Demanding zero residuals individually. The coefficient score equations require \(\mathbf{X}^{\mathsf T}\mathbf{r}=\mathbf{0}\); they do not require \(\mathbf{r}=\mathbf{0}\).
  • Calling a score equation a maximum. A zero derivative is a stationary-point condition. Establishing that a point maximizes the likelihood requires additional argument.
Key takeaway: For independent Gaussian responses with identity-link mean \(\mathbf{X}\boldsymbol{\beta}\) and common variance \(\sigma^2\), the coefficient score is \(\sigma^{-2}\mathbf{X}^{\mathsf T}\mathbf{r}\), while the variance score is \(-n/(2\sigma^2)+\operatorname{RSS}/(2(\sigma^2)^2)\). Their zero-score equations follow directly from these derivatives.

Check Your Understanding

Use the score formulas and show the intermediate calculations.

  1. For a model with an intercept and one predictor, explain what the two entries of \(\mathbf{X}^{\mathsf T}\mathbf{r}\) represent.
  2. Why does \(\mathbf{X}^{\mathsf T}\mathbf{r}=\mathbf{0}\) not generally imply that every residual is zero?
  3. If \(n=5\), \(\operatorname{RSS}=10\), and \(\sigma^2=2\), calculate the variance score.
  4. What happens to the variance score equation if the residual sum of squares is zero and \(\sigma^2\) must remain positive?
  5. Explain why differentiating with respect to \(\sigma^2\) is not the same operation as differentiating with respect to \(\sigma\).

Previous

Next

Back to Tutorials
Clinical Biostats