Profile-Likelihood Intervals For Covariance Parameters

Why Profile a Covariance Parameter?

A covariance matrix can contain many parameters, but a scientific question may concern just one: a particular variance, covariance, or correlation. A confidence interval for that parameter should account for uncertainty in the other covariance entries. Profile likelihood does this by maximizing the likelihood over the nuisance parameters separately for each proposed value of the parameter of interest.

The resulting interval is based on a likelihood-ratio drop. Values that fit nearly as well as the unrestricted maximum are retained; values that require a substantial drop are excluded. The calculations below use the multivariate normal model with an unknown mean and covariance matrix.

Model, Likelihood, and Profile Interval

Let $X_1,\ldots,X_n$ be independent observations from $N_p(\mu,\Sigma)$, where $\mu\in\mathbb{R}^p$ is unknown and $\Sigma$ is positive definite. Write $\bar X=n^{-1}\sum_{i=1}^n X_i$, $W=\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)^\mathsf{T}$, and $S=W/n$. When $W$ is positive definite, maximizing over $\mu$ gives $\hat\mu=\bar X$, and the unrestricted covariance maximum is $\hat\Sigma=S$.

That condition matters: under the stated nonsingular normal model, $W$ is positive definite almost surely when $n>p$. If $n\leq p$, $W$ is singular almost surely. In that case the likelihood can increase without bound as covariance eigenvalues approach zero, so there is no attained unrestricted maximum over positive-definite covariance matrices. The likelihood-ratio intervals developed here use the positive-definite, attained-maximum case.

Definition: Let $\psi=g(\Sigma)$ be a scalar parameter of interest. Its profile likelihood at a proposed value $\psi_0$ is $$ L_{\mathrm{prof}}(\psi_0)=\sup_{\mu,\Sigma:\,g(\Sigma)=\psi_0}L(\mu,\Sigma). $$ If $\hat L=\sup_{\mu,\Sigma}L(\mu,\Sigma)$, the likelihood-ratio drop is $$ D(\psi_0)=2\{\log\hat L-\log L_{\mathrm{prof}}(\psi_0)\}. $$ A usual asymptotic $100(1-\alpha)\%$ profile-likelihood interval retains values satisfying $D(\psi_0)\leq \chi^2_{1,1-\alpha}$, where $\chi^2_{1,1-\alpha}$ is the $1-\alpha$ quantile of a chi-square distribution with one degree of freedom.

The chi-square cutoff is the usual Wilks calibration for a single regular scalar parameter. The interval is obtained by optimizing the nuisance parameters under each constraint, not by plugging their unrestricted estimates into a one-parameter likelihood. The next tutorial addresses cases in which this calibration may be unreliable.

Profile Interval for a Marginal Variance

A useful exact profile calculation is available for a marginal variance. Partition each observation into its first coordinate and its remaining coordinates. The multivariate normal likelihood factors into the likelihood for the first coordinate and the conditional likelihood for the remaining coordinates given the first. When $\Sigma_{11}=v$ is fixed, the nuisance regression coefficients and conditional covariance can be maximized in that conditional likelihood. Their fitted values do not add a likelihood drop beyond the one from the first-coordinate marginal likelihood.

Theorem: Suppose $W$ is positive definite. For the parameter $\psi=\Sigma_{11}=v$, the profile likelihood-ratio drop is $$ D(v)=n\left\{\log\left(\frac{v}{s_{11}}\right)+\frac{s_{11}}{v}-1\right\}, $$ where $s_{11}=W_{11}/n$. Consequently, the profile-likelihood interval consists of $v>0$ for which this expression is at most $\chi^2_{1,1-\alpha}$.

To verify the calculation, consider the first-coordinate observations after maximizing over their mean. Their variance-dependent log likelihood, omitting constants, is $-\frac n2\{\log v+s_{11}/v\}$. It is maximized at $v=s_{11}$. The conditional likelihood for the remaining coordinates can be maximized over its regression and covariance parameters for every positive $v$; its maximum does not depend on $v$. Thus the difference between the unrestricted and constrained maximized log likelihoods is $$ 2\left[-\frac n2\{\log s_{11}+1\}+\frac n2\left\{\log v+\frac{s_{11}}{v}\right\}\right] =n\left\{\log\left(\frac{v}{s_{11}}\right)+\frac{s_{11}}{v}-1\right\}. $$ This proves the formula. $\square$

Writing $q=v/s_{11}$, the endpoint equation is $n(\log q+1/q-1)=\chi^2_{1,1-\alpha}$. The function $h(q)=\log q+1/q-1$ has derivative $h'(q)=(q-1)/q^2$, decreases on $0<q<1$, increases on $q>1$, and diverges at both extremes. Therefore there are exactly two positive endpoints, one on each side of 1.

Worked Example: A Profile Interval for One Variance

Suppose $n=20$, $p=2$, and the observed sample covariance matrix is $S=\begin{pmatrix}4&1\\1&9\end{pmatrix}$. It is positive definite because its determinant is $36-1=35>0$. Find the 95% profile-likelihood interval for $\Sigma_{11}$.

For 95% confidence use $\chi^2_{1,0.95}\approx3.841$. Here $s_{11}=4$, so the endpoint equation is $$ 20\left(\log q+\frac1q-1\right)=3.841, \qquad\text{or}\qquad \log q+\frac1q-1=0.19205. $$

Solving on either side of 1 gives $q_-\approx0.5702$ and $q_+\approx1.9954$. Substitution checks the rounding: at $q=0.5702$, $\log q+1/q-1\approx0.1920$; at $q=1.9954$, it is also approximately $0.1920$. Multiplying by $s_{11}=4$ gives $$ 4[0.5702,\;1.9954]\approx[2.2808,\;7.9816]. $$ Thus the interval is approximately $[2.28,7.98]$. The nuisance covariance entries were profiled out; they were not held fixed at their unrestricted estimates.

Profile Interval for a Bivariate Correlation

For a bivariate normal sample, let the observed covariance matrix have diagonal entries $s_1,s_2>0$ and sample correlation $r=s_{12}/\sqrt{s_1s_2}$, with $|r|<1$. At a proposed population correlation $\rho$, write the unknown covariance matrix as $$ \Sigma=\begin{pmatrix}a&\rho\sqrt{ab}\\\rho\sqrt{ab}&b\end{pmatrix}. $$ For each fixed $\rho$, the variances $a,b$ are nuisance parameters and must be maximized over, subject to $a,b>0$.

Set $x=\sqrt{s_1/a}$ and $y=\sqrt{s_2/b}$. Apart from constants, minus twice the profiled log likelihood divided by $n$ is minimized over positive $x,y$ by minimizing $$ \log\{ab(1-\rho^2)\}+ \frac{x^2+y^2-2\rho rxy}{1-\rho^2}. $$ The first-order equations have the symmetric solution $x=y=\sqrt{(1-\rho^2)/(1-\rho r)}$. This is the global minimum: the equivalent objective in $x,y$ has a positive-definite quadratic term because $|\rho r|<1$, together with strictly convex negative-log terms. At the unrestricted maximum, $\rho=r$ and $x=y=1$. Substitution gives the profile drop $$ D(\rho)=n\log\left\{\frac{(1-\rho r)^2}{(1-\rho^2)(1-r^2)}\right\}. $$

Worked Example: A Profile Interval for Correlation

Suppose $n=30$ bivariate observations give sample correlation $r=0.50$, with positive-definite sample covariance matrix. Find the 95% profile-likelihood interval for $\rho$.

The profile-drop condition is $$ 30\log\left\{\frac{(1-0.5\rho)^2}{(1-\rho^2)(0.75)}\right\}\leq3.841. $$ At an endpoint, the ratio inside the logarithm equals $K=\exp(3.841/30)\approx1.1366$. Rearranging the equality gives $$ (1-0.5\rho)^2=0.75K(1-\rho^2), $$ or approximately $1.1024\rho^2-\rho+0.1476=0$. The quadratic formula gives roots approximately $0.185$ and $0.722$. The drop is zero at $\rho=r=0.50$, and it increases on either side up to these boundary values, so the interval is approximately $[0.185,0.722]$. Both endpoints lie in the admissible correlation range $(-1,1)$.

Profile Interval for a Covariance

A covariance parameter can also be profiled directly. The following symmetric example makes the nuisance optimization transparent. For a bivariate sample with $S=I_2$, the observed variances are both 1 and the observed covariance is 0. Write the candidate covariance matrix as $\begin{pmatrix}a&c\\c&b\end{pmatrix}$, where $ab>c^2$. Apart from constants, the negative twice log likelihood divided by $n$ is $$ \log(ab-c^2)+\frac{a+b}{ab-c^2}. $$ For a fixed product $ab$, the numerator $a+b$ is minimized at $a=b=\sqrt{ab}$, by $a+b\geq2\sqrt{ab}$. Thus the likelihood is maximized with $a=b=t$, where $t>|c|$.

Worked Example: A Profile Interval for an Off-Diagonal Covariance

Continue with $n=20$ and $S=I_2$, and find the 95% profile interval for $c=\Sigma_{12}$. With $a=b=t$, the quantity to minimize at fixed $c$ is $\log(t^2-c^2)+2t/(t^2-c^2)$. Differentiating with respect to $t$ and setting the derivative to zero yields $$ t^3-t^2-c^2(t+1)=0, \qquad\text{so}\qquad c^2=\frac{t^2(t-1)}{t+1}. $$ The unrestricted maximum is at $t=1,c=0$. At the constrained optimum, the determinant is $t^2-c^2=2t^2/(t+1)$, and the trace contribution is $2t/(t^2-c^2)=1+1/t$. Hence the profile drop is $$ D(c)=20\left\{\log\left(\frac{2t^2}{t+1}\right)+1+\frac1t-2\right\}, $$ where $t$ is linked to $|c|$ by the preceding equation.

Set $D(c)=3.841$ and solve the resulting one-dimensional equation numerically. The positive solution is approximately $t=1.3744$, which gives $c^2\approx1.3744^2(0.3744)/(2.3744)\approx0.2979$, or $|c|\approx0.5458$. The likelihood is unchanged when $c$ is replaced by $-c$, so the interval is approximately $[-0.546,0.546]$. At the boundary, $t>|c|$, as required for a positive-definite covariance matrix.

A Variance of a Linear Combination

The marginal-variance result also applies to the variance of a specified linear combination $a^\mathsf{T}X$, where $a\neq0$ is fixed: its variance is $a^\mathsf{T}\Sigma a$. An invertible linear transformation can place this combination in the first coordinate, after which the same profile calculation applies. Its sample variance is $a^\mathsf{T}Sa$.

Worked Example: A Profile Interval for a Linear-Combination Variance

For the sample covariance matrix $S=\begin{pmatrix}4&1\\1&9\end{pmatrix}$ in the first example, consider $a=(1,1)^\mathsf{T}$. The sample variance of $X_1+X_2$ is $$ a^\mathsf{T}Sa=4+2(1)+9=15. $$ Using $n=20$, the same endpoint multipliers $0.5702$ and $1.9954$ apply. The 95% profile interval for $\operatorname{Var}(X_1+X_2)$ is therefore approximately $$ 15[0.5702,\;1.9954]=[8.553,\;29.931]. $$ This is an interval for the variance of the specified combination, not a simultaneous interval for every entry of $\Sigma$.

Common Pitfall: Fixing Nuisance Parameters

A tempting shortcut is to constrain the parameter of interest and keep every other covariance entry at its unrestricted estimate. That produces a likelihood value, but generally not the profile likelihood: the nuisance parameters can adjust under the constraint and usually improve the fit. Conversely, reporting the unrestricted estimate of a nuisance parameter as though it were fixed ignores that adjustment. The defining operation is to maximize over nuisance parameters at every candidate value of the target parameter.

Also check that the unrestricted covariance maximum exists before forming a likelihood-ratio drop. If $W$ is singular, using $W/n$ as an unrestricted positive-definite maximum is invalid. Finally, interpret the chi-square-calibrated interval as an asymptotic likelihood interval under the model and regularity conditions, rather than as an exact finite-sample interval.

Key takeaway: A profile-likelihood interval compares the unrestricted maximum with the best fit achievable at each proposed parameter value. Its nuisance-parameter optimization is essential, and its likelihood-ratio calibration and existence conditions must be checked.

Check Your Understanding

Use these questions to check the definitions, derivations, and interval calculations.

Why must the nuisance covariance parameters be re-optimized for each proposed value of the variance of interest?
For a marginal variance, what equation determines the two profile-likelihood endpoint multipliers, and why is there one endpoint on each side of 1?
In the bivariate correlation example, which parameters are profiled out after fixing $\rho$?
For the covariance example with $S=I_2$, explain why the constrained optimum has equal diagonal entries.
What condition ensures that $W/n$ is an admissible unrestricted covariance maximum in the normal model considered here?