Tutorials › Multivariate Statistics (Graduate) › Variance of a Linear Score

Home  /  Tutorials  /  Population and Sample PCA

Population and Sample PCA · Tutorial 482 of 975

Variance of a Linear Score

Learn to express and calculate the variance of a centered projection using the covariance matrix, while keeping centering, direction length, and cross-covariances straight.

Intermediate 9 min read

What You'll Learn

  • Define a centered linear score from a random vector and a unit direction.
  • Derive its mean and variance from the covariance matrix.
  • Account for cross-covariances when calculating score variance.
  • Compute score variances for explicit covariance matrices.
  • Distinguish centering a score from changing its variance.
  • Identify common errors involving direction length and covariance terms.

Why the Variance of a Score Matters

A multivariate observation can be reduced to one number by projecting it onto a chosen direction. That number is called a linear score. Its variance describes how much the observations vary along that direction. To calculate it correctly, both the component variances and the covariances between components matter.

This calculation is a basic part of population principal component analysis. It gives a precise meaning to the variance associated with a direction, without yet deciding which direction should be chosen. The direction will be taken to have unit length, so that its coefficients specify orientation rather than an arbitrary change of scale.

Definition and Covariance-Matrix Formula

Definition: Let \(X=(X_1,\ldots,X_p)^T\) be a random vector with finite second moments and mean vector \(\mu=E[X]\). Its covariance matrix is \(\Sigma=E[(X-\mu)(X-\mu)^T]\). For a direction \(a\in\mathbb{R}^p\) with \(a^Ta=1\), the centered linear score in direction \(a\) is \(S_a=a^T(X-\mu)\).

The score is a scalar random variable. Centering the vector before projection makes the score have mean zero. The unit-length condition is \(\sum_{i=1}^p a_i^2=1\); it does not require the entries of \(a\) to be positive or to sum to one.

Theorem: If \(X\) has finite second moments and \(\Sigma\) is its covariance matrix, then the centered score \(S_a=a^T(X-\mu)\) has mean zero and variance \( \operatorname{Var}(S_a)=a^T\Sigma a \). The identity holds for any fixed vector \(a\); unit length is a convention for interpreting \(a\) as a direction, not a condition needed for the identity.

Proof. By linearity of expectation,

$$ E[S_a]=E[a^T(X-\mu)]=a^TE[X-\mu]=a^T(\mu-\mu)=0. $$

Since the score has mean zero, its variance is its expected square. Multiplication of a scalar by its transpose gives

$$ \operatorname{Var}(S_a) =E[S_a^2] =E\!\left[a^T(X-\mu)(X-\mu)^Ta\right] =a^TE[(X-\mu)(X-\mu)^T]a =a^T\Sigma a. $$

The expectation can be moved between the matrix entries and the fixed vectors because the entries of \(X\) have finite second moments. Thus every entry of the matrix expectation exists, and the finite sums in the matrix product can be evaluated term by term. \(\square\)

Writing the formula by components makes the covariance contributions explicit:

$$ \operatorname{Var}(S_a) =\sum_{i=1}^p\sum_{j=1}^p a_i a_j\operatorname{Cov}(X_i,X_j) =\sum_{i=1}^p a_i^2\operatorname{Var}(X_i) +2\sum_{1\leq i<j\leq p}a_i a_j\operatorname{Cov}(X_i,X_j). $$

The second equality uses the symmetry \(\operatorname{Cov}(X_i,X_j)=\operatorname{Cov}(X_j,X_i)\). In particular, a calculation that uses only the diagonal entries of \(\Sigma\) is correct only when the off-diagonal covariance terms are zero (or their weighted contributions happen to cancel).

Worked Calculations

Worked Example: A Direction Through Independent Variables

Suppose \(X_1\sim N(0,9)\), \(X_2\sim N(0,4)\), and \(X_3\sim N(0,1)\) are mutually independent. Take \(a=(1/3,\,2/3,\,2/3)^T\). Find the variance of the centered score.

The direction is a unit vector because \(a^Ta=1/9+4/9+4/9=1\). Independence makes each covariance between distinct components zero, so

$$ \Sigma= \begin{pmatrix} 9&0&0\\ 0&4&0\\ 0&0&1 \end{pmatrix}. $$

Therefore the score variance is

$$ \begin{aligned} \operatorname{Var}(S_a) &=a^T\Sigma a\\ &=9(1/3)^2+4(2/3)^2+1(2/3)^2\\ &=1+\frac{16}{9}+\frac{4}{9} =\frac{29}{9}. \end{aligned} $$

The squared coefficients weight the component variances. Here the first component has the largest variance, but the direction also places a substantial coefficient on the second component; both contributions appear in the final value.

Worked Example: Including a Nonzero Covariance

Let \(X\sim N_2(0,\Sigma)\), where \( \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix} \), and take \(a=(3/5,\,4/5)^T\). Calculate the variance of the score.

The matrix is a valid covariance matrix: it is symmetric, its first diagonal entry is positive, and its determinant is \(4(2)-1^2=7>0\), so it is positive definite. The direction has unit length since \(9/25+16/25=1\). Direct multiplication gives

$$ \begin{aligned} \operatorname{Var}(S_a) &=a^T\Sigma a\\ &=4(3/5)^2+2(3/5)(4/5)(1)+2(4/5)^2\\ &=\frac{36}{25}+\frac{24}{25}+\frac{32}{25} =\frac{92}{25}=3.68. \end{aligned} $$

The middle term is the contribution from the covariance between \(X_1\) and \(X_2\). It is positive here because both the covariance and the two direction coefficients are positive. Omitting that term would give \(68/25\), not the variance of this score.

Worked Example: Centering Changes the Mean, Not the Variance

Let \(X\sim N_3(\mu,\Sigma)\), with \( \mu=(1,-2,4)^T \), \( \Sigma=\begin{pmatrix}2&1&0\\1&2&0\\0&0&1\end{pmatrix} \), and \(a=(1/\sqrt{3},\,1/\sqrt{3},\,1/\sqrt{3})^T\). Find the mean and variance of both \(a^T(X-\mu)\) and the uncentered score \(a^TX\).

The direction is unit length because its three squared entries sum to \(3(1/3)=1\). The covariance matrix is positive definite: its upper-left block has eigenvalues \(3\) and \(1\), and its remaining diagonal entry is \(1\). For the centered score, the variance is

$$ \begin{aligned} \operatorname{Var}\!\left(a^T(X-\mu)\right) &=a^T\Sigma a\\ &=\frac{1}{3}(2+2+1+2\cdot 1)\\ &=\frac{7}{3}. \end{aligned} $$

The two diagonal entries \(2,2\) and \(1\) sum to \(5\), while the covariance between the first two components contributes twice, giving \(5+2=7\) before division by \(3\). The centered score has mean zero. For the uncentered score,

$$ E[a^TX]=a^T\mu=\frac{1-2+4}{\sqrt{3}}=\sqrt{3}. $$

Subtracting a constant changes a variable's mean but not its variance, so \(\operatorname{Var}(a^TX)=\operatorname{Var}(a^T(X-\mu))=7/3\). Thus centering is useful for giving the score mean zero, but is not what determines its variance.

Covariance Between Two Scores

Result: For fixed vectors \(a,b\in\mathbb{R}^p\), define centered scores \(S_a=a^T(X-\mu)\) and \(S_b=b^T(X-\mu)\). If \(X\) has finite second moments, then \( \operatorname{Cov}(S_a,S_b)=a^T\Sigma b \).

Derivation. Both centered scores have expectation zero. Therefore their covariance is the expectation of their product. Expanding that product and using the definition of \(\Sigma\) yields

$$ \begin{aligned} \operatorname{Cov}(S_a,S_b) &=E[S_aS_b]\\ &=E\!\left[a^T(X-\mu)(X-\mu)^Tb\right]\\ &=a^TE[(X-\mu)(X-\mu)^T]b\\ &=a^T\Sigma b. \end{aligned} $$

All terms are integrable: each product of two components with finite second moments has a finite absolute expectation by the Cauchy–Schwarz inequality. Taking \(b=a\) recovers the variance formula. For distinct directions, the result describes how their scores co-vary, and it underscores that covariance structure acts on the coefficients from both directions.

How to Read the Formula—and a Common Pitfall

The expression \(a^T\Sigma a\) combines the direction coefficients with the full covariance structure. The diagonal entries describe variation in individual coordinates; off-diagonal entries describe whether pairs of coordinates tend to vary together or in opposite ways. The signs of the coefficients also matter: a negative coefficient can reverse the contribution of a positive covariance.

A common mistake is to argue, “The score is a weighted sum, so its variance is just the sum of the squared weights times the individual variances.” That reasoning silently assumes that all distinct components have zero covariance. In general, the correct expansion includes every pairwise term:

$$ \operatorname{Var}\!\left(\sum_{i=1}^p a_iX_i\right) =\sum_{i=1}^p a_i^2\operatorname{Var}(X_i) +2\sum_{i<j}a_i a_j\operatorname{Cov}(X_i,X_j). $$

Another error is to treat the unit-length condition as \(\sum_i a_i=1\). A direction is normalized by its Euclidean length, \(a^Ta=1\), not by the sum of its entries. Also, replacing \(a\) by \(-a\) reverses every score value but leaves its variance unchanged, because \((-a)^T\Sigma(-a)=a^T\Sigma a\).

The unit convention prevents an arbitrary rescaling of coefficients from changing the numerical scale of the score. It does not itself select a direction or establish that a particular direction is preferable. For any specified direction, the covariance-matrix formula gives its variance directly.

Check Your Understanding

Use the covariance-matrix formula and the role of centering to answer these questions.

  1. What condition on \(X\) is needed for the covariance matrix and the score variance in this tutorial to be finite?
  2. For a unit vector \(a\), why can the variance calculation include off-diagonal entries of \(\Sigma\)?
  3. If \(a^Ta=1\), what is the variance of \(a^T(X-\mu)\) when \(\Sigma=I_p\)?
  4. Does subtracting \(\mu\) change the variance of \(a^TX\)? Explain.
  5. Why is a direction with entries summing to one not necessarily a unit direction?

Previous

Next

Back to Tutorials
Clinical Biostats