The Multiparameter Cramér-Rao Bound

Why a Matrix Bound Is Needed

When a model has several unknown parameters, an estimator is a vector, and its uncertainty is described by a covariance matrix. Estimating one parameter can be linked to estimating another: the estimator components may covary, and the likelihood may contain information shared across parameters. The multiparameter Cramér-Rao bound captures these relationships in one matrix inequality.

The key practical consequence is that the bound applies not only to each coordinate of an estimator, but also to every linear combination of its coordinates. This matters when the quantity of interest is a contrast, a weighted total, or another combination of model parameters.

Definition: Let \(\theta=(\theta_1,\ldots,\theta_p)^T\) be a parameter vector, and let \(T=(T_1,\ldots,T_p)^T\) be an estimator with \(E_\theta[T]=\theta\). Its covariance matrix is \(\operatorname{Cov}_\theta(T)=E_\theta[(T-\theta)(T-\theta)^T]\). A symmetric matrix \(A\) is positive semidefinite if \(c^T A c\geq 0\) for every vector \(c\); write \(A\succeq B\) when \(A-B\) is positive semidefinite.

The Multiparameter Cramér-Rao Inequality

Suppose the observed data have joint density or probability mass function \(f(x;\theta)\), with parameter-independent support. Define the score vector as the gradient of the log likelihood, and define the Fisher information matrix as its covariance matrix. The following regularity conditions are required: the likelihood is differentiable in the parameters, differentiation can be interchanged with integration or summation, the score has finite second moments, and the information matrix exists and is nonsingular. These conditions ensure that the score has mean zero and the usual score identities hold.

U(X;\theta)=\nabla_\theta \log f(X;\theta),\qquad I(\theta)=E_\theta[U U^T]=\operatorname{Cov}_\theta(U).

For an unbiased estimator vector of the full parameter vector, the matrix Cramér-Rao inequality is

Theorem: Under the stated regularity conditions, if \(E_\theta[T]=\theta\) and \(\operatorname{Cov}_\theta(T)\) is finite, then

\operatorname{Cov}_\theta(T)\succeq I(\theta)^{-1}.

In particular, each diagonal entry gives a variance bound, but the theorem is stronger than a list of separate bounds. For any fixed vector \(c\), the variance of the linear combination \(c^T T\) satisfies

\operatorname{Var}_\theta(c^T T) =c^T\operatorname{Cov}_\theta(T)c \geq c^T I(\theta)^{-1}c.

The right-hand side uses the inverse of the whole information matrix. If the estimator is unbiased for a vector \(g(\theta)\) rather than for \(\theta\) itself, the corresponding bound is \(\operatorname{Cov}_\theta(T)\succeq G I(\theta)^{-1}G^T\), where \(G\) is the Jacobian matrix with entries \(G_{ij}=\partial g_i/\partial\theta_j\). The examples below focus on unbiased estimation of the parameter vector and its linear combinations.

Why the Inequality Holds

The argument extends the one-parameter score-covariance reasoning to vectors. For an unbiased estimator vector \(T\), differentiating \(E_\theta[T]=\theta\) component by component gives \(\operatorname{Cov}_\theta(T,U)=I_p\), where \(I_p\) is the \(p\)-dimensional identity matrix. This identity uses the regularity conditions above; \(T\) is a function of the data and has no direct dependence on the unknown parameter.

Use the score identity.
For each pair of components, differentiating the expectation gives \(\operatorname{Cov}_\theta(T,U)=I_p\), while \(\operatorname{Cov}_\theta(U)=I(\theta)\).

Subtract the score-based linear predictor.
Set \(R=T-\theta-I(\theta)^{-1}U\). Its covariance matrix must be positive semidefinite because \(c^T\operatorname{Cov}(R)c=\operatorname{Var}(c^TR)\geq 0\) for every \(c\).

Expand the covariance.
Using \(\operatorname{Cov}(T,U)=I_p\) and \(\operatorname{Cov}(U)=I(\theta)\), the two cross terms each contribute \(I(\theta)^{-1}\), while the final term contributes \(I(\theta)^{-1}\). Thus \(\operatorname{Cov}(R)=\operatorname{Cov}(T)-I(\theta)^{-1}\).

Since \(\operatorname{Cov}(R)\succeq0\), it follows that \(\operatorname{Cov}_\theta(T)-I(\theta)^{-1}\succeq0\), which is the matrix inequality. This also explains why the inequality controls every linear combination: positive semidefiniteness is defined by checking all vectors \(c\), not just coordinate vectors.

Using the Bound in Models

Worked Example: Two Independent Normal Means

Method: Compute the information for each parameter, invert the matrix, then apply the quadratic-form bound. Suppose \(X_1,\ldots,X_4\) are independent with \(X_i\sim N(\mu,4)\), and \(Y_1,\ldots,Y_4\) are independent with \(Y_j\sim N(\nu,9)\). The two samples are independent, both variances are known, and the unknown parameter vector is \(\theta=(\mu,\nu)^T\). Find the Cramér-Rao bound for estimating \(\mu+2\nu\) with \(T=\bar X+2\bar Y\).

For one \(X_i\), the score for \(\mu\) is \((X_i-\mu)/4\), so its information is \(1/4\). Four independent observations contribute \(4(1/4)=1\). Similarly, one \(Y_j\) contributes \(1/9\) for \(\nu\), so four contribute \(4/9\). Independence gives zero cross-information. Therefore,

I(\theta)= \begin{pmatrix} 1&0\\ 0&4/9 \end{pmatrix}, \qquad I(\theta)^{-1}= \begin{pmatrix} 1&0\\ 0&9/4 \end{pmatrix}.

The coefficient vector for \(\mu+2\nu\) is \(c=(1,2)^T\). Hence the variance of any unbiased estimator of that combination is at least

c^T I(\theta)^{-1}c = \begin{pmatrix}1&2\end{pmatrix} \begin{pmatrix}1&0\\0&9/4\end{pmatrix} \begin{pmatrix}1\\2\end{pmatrix} =1+4(9/4)=10.

The proposed estimator is unbiased. Its variance is \(\operatorname{Var}(\bar X)+4\operatorname{Var}(\bar Y)=4/4+4(9/4)=1+9=10\), so it attains the bound. Check: the direct variance and the matrix bound both equal 10. The common mistake avoided is using the bound for \(\nu\) without multiplying it by the squared coefficient \(2^2\).

Worked Example: A Contrast of Two Bernoulli Probabilities

Method: Use the Bernoulli information and the contrast vector. In a hypothetical study, one independent sample contains 20 Bernoulli observations with success probability \(p=0.2\), and another independent sample contains 10 Bernoulli observations with success probability \(q=0.5\). Estimate the vector \((p,q)^T\) by the two sample proportions, and find the bound for estimating \(p-2q\).

For \(n\) independent Bernoulli observations with success probability \(r\), the information for \(r\) is \(n/[r(1-r)]\). Thus the information entries here are \(20/(0.2\cdot0.8)=20/0.16=125\) and \(10/(0.5\cdot0.5)=10/0.25=40\). Independence makes the off-diagonal entries zero:

I(p,q)= \begin{pmatrix}125&0\\0&40\end{pmatrix}, \qquad I(p,q)^{-1}= \begin{pmatrix}1/125&0\\0&1/40\end{pmatrix} = \begin{pmatrix}0.008&0\\0&0.025\end{pmatrix}.

For \(p-2q\), \(c=(1,-2)^T\). The bound is

c^T I(p,q)^{-1}c =0.008+(-2)^2(0.025) =0.008+0.100 =0.108.

The estimator \(\hat p-2\hat q\), formed from the sample proportions, is unbiased. Its variance is \(0.2(0.8)/20+4[0.5(0.5)/10]=0.16/20+1/10=0.008+0.100=0.108\). Check: the computed variance equals the bound. The common mistake avoided is treating the coefficient \(-2\) as \(-2\), rather than squaring it when calculating a variance.

Worked Example: Individual Bounds Need Not Be Attained Together

Method: Compare the covariance matrix of an unbiased estimator vector with the inverse information matrix. Suppose \(X_1,\ldots,X_4\) are independent with \(X_i\sim N(\mu,\sigma^2)\), where both \(\mu\) and \(\sigma^2\) are unknown. Estimate them by \(T=(\bar X,S^2)^T\), where \(S^2=\sum_{i=1}^4(X_i-\bar X)^2/3\). Evaluate the bounds at \(\sigma^2=2\).

For one observation, the score components for \(\mu\) and \(\sigma^2\) are \((X_i-\mu)/\sigma^2\) and \(-1/(2\sigma^2)+(X_i-\mu)^2/(2\sigma^4)\). Their expected squared values are \(1/\sigma^2\) and \(1/(2\sigma^4)\), and their expected product is zero. With four independent observations, the information matrix is diagonal. At \(\sigma^2=2\), its entries are \(4/2=2\) and \(4/[2(2^2)]=1/2\), giving

I(\mu,\sigma^2)= \begin{pmatrix}2&0\\0&1/2\end{pmatrix}, \qquad I(\mu,\sigma^2)^{-1}= \begin{pmatrix}1/2&0\\0&2\end{pmatrix}.

The two diagonal entries say that \(\operatorname{Var}(\bar X)\geq1/2\) and \(\operatorname{Var}(S^2)\geq2\). In this normal model, \(\operatorname{Var}(\bar X)=\sigma^2/4=1/2\). Also, \(3S^2/\sigma^2\) has a chi-square distribution with 3 degrees of freedom, whose variance is 6. Therefore \(\operatorname{Var}(S^2)=(\sigma^2/3)^2(6)=4(6)/9=8/3\). The sample mean and sample variance are independent, so their covariance is zero. Thus

\operatorname{Cov}(T)= \begin{pmatrix}1/2&0\\0&8/3\end{pmatrix} \succeq \begin{pmatrix}1/2&0\\0&2\end{pmatrix}.

Check: the difference has diagonal entries \(0\) and \(8/3-2=2/3\), so it is positive semidefinite. The mean estimator attains its coordinate bound, but the variance estimator does not. The common mistake avoided is assuming that attaining one diagonal bound forces the whole covariance matrix to equal the inverse information matrix.

Common Matrix Trap: Inverting Only the Diagonal

When information has nonzero off-diagonal entries, its inverse is generally not obtained by taking reciprocals of the diagonal entries. For example, suppose a valid information matrix is

I= \begin{pmatrix}4&2\\2&3\end{pmatrix}.

Its determinant is \(4(3)-2(2)=8\), so

I^{-1}=\frac{1}{8} \begin{pmatrix}3&-2\\-2&4\end{pmatrix} = \begin{pmatrix}3/8&-1/4\\-1/4&1/2\end{pmatrix}.

The incorrect calculation would use \(1/4\) as the first coordinate bound by reciprocating \(I_{11}=4\). The correct first-coordinate bound is \((I^{-1})_{11}=3/8\). For the sum of the two estimator components, the correct bound is \((1,1)I^{-1}(1,1)^T=3/8-1/4-1/4+1/2=3/8\). The off-diagonal entries affect the result; discarding them can give an incorrect bound for both coordinates and combinations.

Key takeaway: For an unbiased estimator vector, compare its covariance matrix with the inverse Fisher information matrix. For a linear combination, use the coefficient vector on both sides of that inverse. The matrix inequality gives a valid bound for every combination, not just for each parameter separately.

Check Your Understanding

For each problem, apply the matrix bound and keep the full coefficient squares and matrix inverse where needed.

Two independent normal samples have sizes 5 and 3, with known variances 9 and 4 and unknown means \(\mu_1,\mu_2\). Find the Cramér-Rao variance bound for an unbiased estimator of \(2\mu_1-\mu_2\). (Answer: \(128/15\), approximately \(8.5333\))
Independent Bernoulli samples have sizes 12 and 8, with success probabilities \(p=0.25\) and \(q=0.5\). Find the bound for an unbiased estimator of \(p+q\). (Answer: \(3/64=0.046875\))
A sample of size 6 comes from \(N(\mu,\sigma^2)\), with both parameters unknown and \(\sigma^2=3\). What is the Cramér-Rao bound for the variance of an unbiased estimator of \(\sigma^2\)? (Answer: \(3\))
An information matrix is \(I=\begin{pmatrix}5&1\\1&2\end{pmatrix}\). Find the bound for the variance of an unbiased estimator of \(2\theta_1-\theta_2\). (Answer: \(17/9\))
For an unbiased estimator vector \(T\) with information matrix \(I\), what matrix must be positive semidefinite, and what scalar bound follows for an arbitrary fixed vector \(c\)? (Answer: \(\operatorname{Cov}(T)-I^{-1}\succeq0\), and \(\operatorname{Var}(c^TT)\geq c^TI^{-1}c\))

What You'll Learn