Matrix Bounds And Loewner Efficiency

Why a Vector Bound Is a Matrix Bound

When a parameter has several components, estimation precision cannot generally be summarized by one variance. An estimator may be precise for one linear combination of the parameters and less precise for another. Its asymptotic covariance matrix records both the component variances and their dependence; a matrix information bound therefore describes precision in every direction at once.

The relevant comparison is not entry-by-entry. It is the Loewner ordering: one symmetric matrix is at least as large as another when their difference is positive semidefinite. This gives a precise way to say that an estimator has no smaller asymptotic variance than the information limit, and to compare two estimators when their covariance matrices are not diagonal.

Definition: A symmetric matrix $A$ is positive semidefinite, written $A\succeq 0$, if $a^\mathsf{T}Aa\geq 0$ for every vector $a$. For symmetric matrices $A$ and $B$, write $A\succeq B$ (or $B\preceq A$) when $A-B\succeq 0$. This is the Loewner ordering.

The quadratic form $a^\mathsf{T}Aa$ is the variance associated with direction $a$ when $A$ is a covariance matrix. In particular, if $T$ has covariance matrix $V$, then $\operatorname{Var}(a^\mathsf{T}T)=a^\mathsf{T}Va$. Thus $A\succeq B$ means the variance represented by $A$ is at least the variance represented by $B$, for every linear direction.

The Matrix Information Bound

Let $X_1,\ldots,X_n$ be independent observations from a regular parametric model with parameter $\theta\in\mathbb{R}^p$. Write $I(\theta)$ for the Fisher information matrix in one observation. Assume it is finite and positive definite. For an estimator $\hat\theta$, suppose $\sqrt n(\hat\theta-\theta)\xrightarrow{d}N_p(0,V(\theta))$. To apply the asymptotic information bound, the estimator must also be regular: under each fixed local alternative $\theta+h/\sqrt n$, the limiting distribution of $\sqrt n\{\hat\theta-(\theta+h/\sqrt n)\}$ is the same as under $\theta$, independently of $h$. Assume a differentiable-in-quadratic-mean model and the usual local asymptotic normality conditions, so the regular parametric information bound applies.

Theorem (asymptotic matrix information bound): Under these conditions, the asymptotic covariance matrix satisfies $$ V(\theta)\succeq I(\theta)^{-1}. $$ Equivalently, for every fixed $a\in\mathbb{R}^p$, $$ a^\mathsf{T}V(\theta)a\geq a^\mathsf{T}I(\theta)^{-1}a. $$

The bound is for the covariance of the root-$n$ limit. On the original estimator scale, the corresponding first-order covariance bound is $I(\theta)^{-1}/n$. Equality of the matrices means that the estimator attains the bound in every direction; equality for just one direction says nothing by itself about precision in the others.

The matrix conclusion follows directly from the scalar directional bounds. Fix any $a$. The scalar limit for the linear contrast $a^\mathsf{T}\hat\theta$ has variance $a^\mathsf{T}Va$, while the information bound in that direction is $a^\mathsf{T}I^{-1}a$. The scalar bound therefore gives

a^\mathsf{T}\{V-I^{-1}\}a\geq 0 \quad\text{for every }a.

By the definition of positive semidefiniteness, this is exactly $V-I^{-1}\succeq 0$. The argument also explains why a collection of separate comparisons of diagonal entries is insufficient: the quadratic forms involve all linear combinations, including those affected by covariance terms.

Finite-Sample Version: The Cramér–Rao Matrix Inequality

A related finite-sample result has stronger distributional conditions and requires unbiasedness. Let $X_1,\ldots,X_n$ have a common differentiable density or mass function $f_\theta$. Suppose the support does not depend on $\theta$, differentiation can be interchanged with integration (or summation), the score has mean zero and finite covariance, and the Fisher information $I(\theta)$ is nonsingular. If $T\in\mathbb{R}^p$ is unbiased for $\theta$, with finite covariance, and the derivative identity for $E_\theta[T]$ is valid, then

Theorem (Cramér–Rao matrix inequality): Under the stated conditions, $$ \operatorname{Cov}_\theta(T)\succeq \{nI(\theta)\}^{-1}. $$

To prove it, let $U=\sum_{i=1}^n \nabla_\theta\log f_\theta(X_i)$ be the total score. Independence and the zero-mean score condition give $E[U]=0$ and $\operatorname{Cov}(U)=nI(\theta)$. Differentiating the unbiasedness identity $E_\theta[T]=\theta$ gives $\operatorname{Cov}(T,U)=I_p$: componentwise, the $(j,k)$ entry is $E[(T_j-\theta_j)U_k]=\partial E[T_j]/\partial\theta_k$, which equals $1$ for $j=k$ and $0$ otherwise.

The covariance matrix of the stacked vector $(T^\mathsf{T},U^\mathsf{T})^\mathsf{T}$ must be positive semidefinite. Its block form is

\begin{pmatrix} \operatorname{Cov}(T) & I_p\\ I_p & nI(\theta) \end{pmatrix}\succeq 0.

Since $nI(\theta)$ is positive definite, its Schur complement is positive semidefinite. That complement is $\operatorname{Cov}(T)-I_p\{nI(\theta)\}^{-1}I_p$, which equals $\operatorname{Cov}(T)-\{nI(\theta)\}^{-1}$. This proves the inequality. $\square$

This finite-sample theorem is not a substitute for the asymptotic theorem: it assumes an unbiased estimator and specific differentiation conditions. The asymptotic result instead concerns regular estimators and their limiting distributions. In particular, one should not apply the finite-sample inequality to a biased estimator without an appropriate biased-estimator version.

Worked Examples of Matrix Comparisons

Worked Example: A Loewner Comparison That Diagonal Entries Miss

Consider the symmetric matrices $$ A=\begin{pmatrix}2&1\\1&2\end{pmatrix}, \qquad B=\begin{pmatrix}1&0\\0&1\end{pmatrix}. $$ The diagonal entries of $A$ exceed those of $B$, but the Loewner comparison must also account for covariance terms.

Subtract the matrices: $$ A-B=\begin{pmatrix}1&1\\1&1\end{pmatrix}. $$ For $a=(a_1,a_2)^\mathsf{T}$, its quadratic form is $$ a^\mathsf{T}(A-B)a=(a_1+a_2)^2\geq 0. $$ Therefore $A\succeq B$. The difference has eigenvalues $2$ and $0$, so the comparison is strict in the direction $(1,1)^\mathsf{T}$ but is equality in direction $(1,-1)^\mathsf{T}$. The zero eigenvalue shows that “at least as large” need not mean strictly larger in every direction.

Worked Example: Efficient and Inefficient Estimators in a Two-Parameter Normal Model

Suppose $X_i\sim N_2(\theta,\Sigma)$ independently, where $\theta=(\theta_1,\theta_2)^\mathsf{T}$ and the known covariance matrix is $$ \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}. $$ Its determinant is $8-1=7>0$, so it is positive definite. For a normal observation with known covariance, the information for the mean is $I(\theta)=\Sigma^{-1}$. Direct inversion gives $$ I(\theta)=\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix}, \qquad I(\theta)^{-1}=\Sigma. $$

The sample mean $\bar X$ has covariance $\Sigma/n$. Thus $\sqrt n(\bar X-\theta)\sim N_2(0,\Sigma)$, and its asymptotic covariance equals the bound. It is efficient in every direction.

Now use only the first $m=n/2$ observations, for even $n$, and let $\tilde\theta$ be their mean. This estimator is unbiased and has covariance $\Sigma/m=2\Sigma/n$. Its root-$n$ asymptotic covariance is therefore $V=2\Sigma$, and $$ V-I(\theta)^{-1}=2\Sigma-\Sigma =\begin{pmatrix}4&1\\1&2\end{pmatrix}\succeq 0. $$ The difference is positive definite because its leading principal minor is $4>0$ and its determinant is $7>0$. In every direction, the variance of this estimator is twice the bound, so its directional efficiency—the bound variance divided by its variance—is $1/2$.

Worked Example: Directional Efficiency and Incomparable Estimators

In the same normal location model with $\Sigma=I_2$, the information bound for root-$n$ covariance is $I_2$. For $n\geq 2$, define $$ \tilde\theta=\bar X+\frac{1}{\sqrt n} \begin{pmatrix}1&0\\1&1\end{pmatrix} \frac{X_1-X_2}{\sqrt 2}. $$ The sample mean and $X_1-X_2$ are independent in this Gaussian model: their covariance is zero, and jointly Gaussian vectors with zero covariance are independent. Also, $(X_1-X_2)/\sqrt 2\sim N_2(0,I_2)$. Consequently $\tilde\theta$ is unbiased, and its root-$n$ covariance is $$ V=I_2+ \begin{pmatrix}1&0\\1&1\end{pmatrix} \begin{pmatrix}1&1\\0&1\end{pmatrix} =\begin{pmatrix}2&1\\1&3\end{pmatrix}. $$

For direction $a=(1,1)^\mathsf{T}$, the bound variance is $a^\mathsf{T}I_2a=2$, while $a^\mathsf{T}Va=2+2+3=7$. Its directional efficiency is $2/7$. For $a=(1,-1)^\mathsf{T}$, the bound variance is also $2$, while $a^\mathsf{T}Va=2-2+3=3$, giving efficiency $2/3$. The estimator is therefore relatively better in the second direction, even though the information bound remains the same in both.

For any covariance matrix $V\succeq B=I^{-1}$, the matrix $B^{-1/2}VB^{-1/2}$ has eigenvalues at least $1$. Their reciprocals describe the efficiencies along the corresponding standardized directions. This summarizes the range of directional performance, but it is not a unique scalar efficiency unless a particular direction or summary is specified.

What Loewner Ordering Does—and Does Not—Say

If $V_1\succeq V_2$, then $V_1$ is no more precise than $V_2$ in any linear direction, and is strictly worse in at least one direction if the difference is nonzero. But two covariance matrices may be incomparable: neither difference need be positive semidefinite. In that case, each estimator can be more precise for some contrasts. No single ranking follows without choosing a direction or another explicit criterion.

A useful equivalent check for a symmetric matrix is that all its eigenvalues are nonnegative. For a $2\times2$ symmetric matrix, one can also check that its diagonal entries are nonnegative and its determinant is nonnegative. These tests apply to the difference of covariance matrices, not to a comparison of their entries one at a time.

Pitfall: concluding $V\succeq B$ because each diagonal entry of $V$ is at least the corresponding diagonal entry of $B$. This checks only the coordinate directions.
Correction: check that $V-B$ is positive semidefinite, or equivalently verify $a^\mathsf{T}(V-B)a\geq0$ for every direction $a$. Off-diagonal terms can change the variance of a linear combination.
Pitfall: reporting one efficiency number without identifying the direction or summary used. A vector estimator can have different efficiencies in different directions.

Key takeaway: The asymptotic information bound is a matrix inequality, $V\succeq I^{-1}$. It asserts a variance bound for every linear direction, and equality of the matrices is the condition for attaining that bound in all directions.

Check Your Understanding

Use quadratic forms, not entry-by-entry comparisons, when answering these questions.

What does $A\succeq B$ mean in terms of the variance of every linear contrast?
If a regular estimator has asymptotic covariance $V$, what matrix must be positive semidefinite for it to satisfy the information bound?
Why does the finite-sample Cramér–Rao matrix inequality require conditions beyond those in the asymptotic information bound?
For the two-parameter normal example, what is the asymptotic covariance of the mean based on half the observations, and what is its efficiency in every direction?
Can two estimators be incomparable in Loewner order even if one has a smaller variance for the first component? Explain what must be checked.