Confidence Regions for Robust Regression Effects

From Robust Regression Estimates to Joint Uncertainty

A robust multivariate regression analysis may estimate several coefficients at once: for example, the effects of a predictor on several responses, or a collection of contrasts across predictors and responses. Reporting a separate interval for each coefficient does not, by itself, describe uncertainty in the whole vector. A joint confidence region makes that dependence visible and supports statements about selected combinations of effects.

The construction below is asymptotic. It uses the estimated covariance of a robust regression estimator, which may be a sandwich covariance or another consistent covariance estimate appropriate to the estimator and its assumptions. Robustness does not automatically make a procedure distribution-free, nor does it guarantee valid covariance estimation under every form of contamination.

Definition: Let \(\beta\in\mathbb{R}^d\) be a vector of multivariate regression coefficients, and let \(\hat{\beta}\) be a robust estimator. Suppose that, under the model and regularity conditions for this estimator, \(\sqrt{n}(\hat{\beta}-\beta)\xrightarrow{d}N_d(0,V)\), where \(V\) is positive definite, and that \(\hat V\xrightarrow{p}V\). For a fixed \(q\times d\) matrix \(A\) of rank \(q\), the selected effect vector is \(\theta=A\beta\), estimated by \(\hat{\theta}=A\hat{\beta}\). A level-\(1-\alpha\) asymptotic confidence region for \(\theta\) is \(\mathcal{C}_{1-\alpha}=\{t\in\mathbb{R}^q:n(t-\hat{\theta})'(A\hat V A')^{-1}(t-\hat{\theta})\leq\chi^2_{q,1-\alpha}\}\), where \(\chi^2_{q,1-\alpha}\) is the \(1-\alpha\) quantile of a chi-squared distribution with \(q\) degrees of freedom.

The covariance matrix of \(\hat{\theta}\) is estimated by \(A\hat V A'/n\). Thus the factor \(n\) in the definition of the region is essential: \(\hat V\) estimates the covariance of the scaled estimator \(\sqrt n(\hat{\beta}-\beta)\), not the covariance of \(\hat{\beta}\) itself. If a program instead reports an estimated covariance matrix \(\hat\Omega\) for \(\hat{\beta}\), use \(A\hat\Omega A'\) in the quadratic form, rather than dividing by \(n\) again.

For a multivariate response model, write \(E[Y_i\mid x_i]=B'x_i\), where \(x_i\in\mathbb{R}^p\), \(Y_i\in\mathbb{R}^r\), and \(B\) is a \(p\times r\) coefficient matrix. With column-wise vectorization \(\beta=\operatorname{vec}(B)\), the vector of response effects at a predictor contrast \(x_0\) is \(B'x_0=(I_r\otimes x_0')\beta\). Choosing \(A=I_r\otimes x_0'\) gives a joint region for those \(r\) effects. Other choices of \(A\) select, for example, differences between responses or combinations of predictor effects.

Why the Joint Region Has the Stated Coverage

The region is an ellipsoid centered at \(\hat{\theta}\). Its orientation and axis lengths are determined by the estimated covariance, so correlation between estimated effects is represented rather than ignored. The coverage claim follows by standardizing the estimation error.

Theorem: Under the asymptotic normality and covariance consistency conditions in the definition, with \(A\) fixed and of rank \(q\), the region \(\mathcal{C}_{1-\alpha}\) satisfies \(P(\theta\in\mathcal{C}_{1-\alpha})\to 1-\alpha\).

Transform to the selected effects.
Since \(\theta=A\beta\), linearity and the assumed limit give \(\sqrt n(\hat{\theta}-\theta)\xrightarrow{d}N_q(0,AVA')\). The matrix \(AVA'\) is positive definite because \(A\) has full row rank and \(V\) is positive definite.

Standardize the quadratic form.
Covariance consistency implies \(A\hat V A'\xrightarrow{p}AVA'\). Slutsky’s theorem, together with continuity of matrix inversion at a positive definite matrix, yields \(n(\hat{\theta}-\theta)'(A\hat V A')^{-1}(\hat{\theta}-\theta)\xrightarrow{d}\chi^2_q\).

Translate the event into coverage.
The event \(\theta\in\mathcal{C}_{1-\alpha}\) is exactly that the quadratic form in Step 2 is no larger than \(\chi^2_{q,1-\alpha}\). The chi-squared distribution has a continuous distribution function at this quantile, so the probability of that event converges to \(1-\alpha\). \(\square\)

The argument is conditional on the robust estimator having the stated limit and on its covariance estimate being consistent. It does not supply those properties for a particular fitting method; they depend on the estimator, sampling assumptions, and regularity conditions. The result is asymptotic rather than an exact finite-sample guarantee.

Projection Gives Intervals for Individual Effects

A joint ellipsoid can answer a scalar question by projection. For a contrast \(c'\theta\), the projection is the set of all values \(c't\) as \(t\) ranges over the ellipsoid. Its endpoints can be derived by maximizing and minimizing the linear function over the ellipsoid.

Result: Let \(\hat\Gamma=A\hat V A'/n\), assumed positive definite, and let \(k=\chi^2_{q,1-\alpha}\). The projection of \(\mathcal{C}_{1-\alpha}\) onto \(c'\theta\), for a fixed nonzero \(c\in\mathbb{R}^q\), is the interval \([c'\hat{\theta}-\sqrt{k\,c'\hat\Gamma c},\;c'\hat{\theta}+\sqrt{k\,c'\hat\Gamma c}]\).

To derive this, put \(u=t-\hat{\theta}\). Points in the ellipsoid satisfy \(u'\hat\Gamma^{-1}u\leq k\). Write \(u=\hat\Gamma^{1/2}z\), so that \(z'z\leq k\). Then \(c'u=(\hat\Gamma^{1/2}c)'z\). By the Cauchy–Schwarz inequality, its absolute value is at most \(\sqrt{c'\hat\Gamma c}\sqrt{z'z}\), hence at most \(\sqrt{k\,c'\hat\Gamma c}\). Equality is attained by taking \(z\) in the direction of \(\hat\Gamma^{1/2}c\), with length \(\sqrt{k}\), or its negative. Adding \(c'\hat{\theta}\) gives the two endpoints.

For response-specific effects, take \(c\) to be a coordinate vector. Each resulting interval is the projection of the same joint region. Consequently, if the joint region covers the whole effect vector, all these projected intervals cover their respective coordinates together. This makes their simultaneous coverage at least the coverage probability of the joint region. The projection intervals are generally wider than separately constructed marginal intervals: they account for the joint confidence requirement.

Worked Example: Projecting a Two-Effect Ellipsoid

Suppose two selected regression effects have estimate \(\hat{\theta}=(1.20,-0.50)'\), and their estimated covariance matrix is \(\hat\Gamma=\begin{pmatrix}0.04&0.01\\0.01&0.09\end{pmatrix}\). For a 95% joint region with \(q=2\), use \(\chi^2_{2,0.95}=5.991\). The covariance is positive definite because its diagonal entries are positive and its determinant is \(0.04(0.09)-0.01^2=0.0035>0\).

For the first effect, use \(c=(1,0)'\). The projected interval has center \(1.20\) and half-width \(\sqrt{5.991(0.04)}=\sqrt{0.23964}\approx0.4895\). Thus the projection is approximately \([0.7105,1.6895]\).

For the second effect, use \(c=(0,1)'\). Its half-width is \(\sqrt{5.991(0.09)}=\sqrt{0.53919}\approx0.7343\), so the projection is approximately \([-1.2343,0.2343]\).

These intervals are projections of the 95% joint ellipsoid, not two independent 95% marginal intervals. The off-diagonal covariance affects the ellipsoid’s tilt, even though coordinate projections use the diagonal entries of \(\hat\Gamma\).

Worked Example: Joint Region for Two Response Effects

At a fixed predictor contrast, suppose the estimated effects on two responses are \(\hat{\theta}=(0.80,0.30)'\), with estimated covariance \(\hat\Gamma=\begin{pmatrix}0.01&0.006\\0.006&0.0225\end{pmatrix}\). The determinant is \(0.01(0.0225)-0.006^2=0.000189>0\), so this covariance is positive definite. The 95% joint region is the set of \(t=(t_1,t_2)'\) satisfying \((t-\hat{\theta})'\hat\Gamma^{-1}(t-\hat{\theta})\leq5.991\).

For the first response, the projected half-width is \(\sqrt{5.991(0.01)}\approx0.2448\), giving \([0.5552,1.0448]\). For the second, it is \(\sqrt{5.991(0.0225)}\approx0.3671\), giving \([-0.0671,0.6671]\). Both intervals come from one region for the pair of response effects. Their simultaneous interpretation is therefore different from interpreting each as a separately calibrated interval.

Worked Example: An Interval for a Combination of Effects

Suppose \(\hat{\theta}=(0.80,1.10)'\), and the estimated covariance of this pair is \(\hat\Gamma=\begin{pmatrix}0.04&-0.01\\-0.01&0.09\end{pmatrix}\). Consider the contrast \(\psi=\theta_1+2\theta_2\), so \(c=(1,2)'\). The estimated contrast is \(c'\hat{\theta}=0.80+2(1.10)=3.00\).

The variance estimate for the contrast is \(c'\hat\Gamma c=0.04+4(0.09)+4(-0.01)=0.36\). It is important to include the covariance term; here the negative covariance reduces the variance. Projecting the 95% two-dimensional region gives half-width \(\sqrt{5.991(0.36)}=\sqrt{2.15676}\approx1.4686\). The projected interval is therefore approximately \([1.5314,4.4686]\).

Choosing Contrasts and Reading the Result

The matrix \(A\) defines the scientific question, not just the arithmetic of the calculation. Its rows can select individual coefficients, differences between coefficients, or response effects at a predictor contrast. They must be linearly independent for the stated \(q\)-dimensional ellipsoid formula. If selected rows are redundant, reduce them to a full-rank set or formulate the region on the lower-dimensional space of distinct effects.

A scalar quantity \(g'\beta\) can be obtained by projection from a region for \(\theta=A\beta\) when \(g'=c'A\) for some \(c\). If it cannot be expressed in this way, it is not a function of the selected effect vector alone. Construct a region that includes the needed contrast rather than treating an unselected coefficient as though it were represented.

The chi-squared cutoff grows with the number \(q\) of jointly covered effects. This is why a projected interval from a multivariate region is typically broader than a scalar interval built for one contrast alone. Use the joint region when the scientific claim concerns several effects together; use a scalar procedure when the target is one prespecified contrast and only its marginal coverage is required.

Common Mistake

Using a scalar critical value for a joint claim. Applying a one-dimensional normal cutoff separately to every coefficient does not generally produce the requested joint confidence region. The simultaneous probability that all such intervals cover can be below the nominal level.
Dropping covariance terms for a contrast. For \(c'\theta\), the variance is \(c'\hat\Gamma c\), including all cross-products. Replacing it with a sum of diagonal terms is valid only when the relevant covariances are zero.
Scaling the covariance twice. If a covariance estimate already describes \(\hat{\beta}\), divide-by-\(n\) is not appropriate again. Check whether the reported matrix estimates \(V\) or \(V/n\).
Calling an asymptotic region exact. The coverage result is a limit under stated regularity conditions. Robust estimation alone does not establish finite-sample coverage or protect the covariance estimate against every problematic data configuration.

Key takeaway: A robust joint confidence region is built from the estimated covariance of selected regression effects and a chi-squared cutoff. Projecting the region gives intervals for individual effects or specified combinations, while preserving the joint uncertainty structure.

Check Your Understanding

Use the definitions and projection result to answer the following.

What does \(\hat V\) estimate in the stated convention, and how does the formula change if a program reports the covariance of \(\hat{\beta}\) directly?
Why must the rows of \(A\) have full row rank for the displayed \(q\)-dimensional ellipsoid?
For a contrast \(c'\theta\), which covariance terms enter its projected interval’s half-width?
Explain why projection intervals from one joint region support a simultaneous statement about the selected effects.
When can a scalar contrast \(g'\beta\) be obtained by projecting a region for \(A\beta\)?