Testing Linear Restrictions in Robust Multivariate Regression

From Regression Questions to Restrictions

A regression question often concerns a relationship among coefficients rather than a single coefficient. For example, one may ask whether two slopes are equal, whether several effects vanish together, or whether a prespecified weighted combination of effects takes a particular value. These are linear restrictions. Robust inference tests them by combining the estimated departures from the restrictions with the estimated covariance of the coefficient estimator.

Consider observations with a $d$-dimensional response $Y_i$ and a $k$-dimensional predictor vector $x_i$. Write the coefficient matrix as $B$, a $d$-by-$k$ matrix, so that the regression component is $B^T x_i$. Use column stacking to define $\theta=\operatorname{vec}(B)$, a vector of $s=dk$ coefficients. Let $\hat{\theta}$ be a robust estimator of this vector. The particular robust fitting method is not fixed here: the test uses its appropriate estimated covariance, not automatically the covariance formula from ordinary least squares.

Definition: A general linear hypothesis for the coefficient vector $\theta$ is $H_0:R\theta=r$, where $R$ is a specified $q$-by-$s$ matrix of rank $q$, and $r$ is a specified $q$-vector. The alternative is that at least one of these $q$ restrictions fails. Each row of $R$, together with the matching entry of $r$, specifies one restriction.

A row of $R$ gives the coefficients in a linear combination; the corresponding entry of $r$ gives the value that combination is hypothesized to equal. For instance, the row $(1,-1)$ applied to two slopes tests equality only when its right-hand side is $0$. With right-hand side $1$, it tests whether the first slope exceeds the second by $1$. This distinction is essential when translating a scientific question into a test.

The Robust Wald Statistic

Suppose the robust estimator has an asymptotic covariance estimate $\widehat V$ for its scaled error: $\widehat V$ estimates the covariance $V$ in the limit distribution of $\sqrt n(\hat{\theta}-\theta)$. Thus, the estimated covariance of $\hat{\theta}$ itself is $\widehat V/n$. If software instead reports this latter covariance directly, use the equivalent version of the statistic below.

Definition: For a full-row-rank restriction matrix $R$, the robust Wald statistic for $H_0:R\theta=r$ is $$ W_n=n(R\hat{\theta}-r)^T(R\widehat V R^T)^{-1}(R\hat{\theta}-r). $$ Equivalently, if $\widehat C$ estimates the covariance of $\hat{\theta}$, then $$ W_n=(R\hat{\theta}-r)^T(R\widehat C R^T)^{-1}(R\hat{\theta}-r), $$ with $\widehat C=\widehat V/n$. Under the null and the conditions stated below, the reference distribution is asymptotically chi-square with $q$ degrees of freedom.

The quadratic form measures the estimated discrepancy from the null relative to its uncertainty. A discrepancy in a direction with high estimated variance contributes less than the same-sized discrepancy in a direction with low variance; covariance among restrictions is also accounted for. Because the statistic uses the covariance of the robust estimator, the test is not obtained by simply applying a classical regression formula to robust coefficient estimates.

Why the Reference Distribution Is Chi-Square

Here is a precise version of the asymptotic result. Assume that under $H_0$, the true coefficient vector $\theta_0$ satisfies $R\theta_0=r$, and that $$ \sqrt n(\hat{\theta}-\theta_0)\xrightarrow{d}N(0,V),\qquad \widehat V\xrightarrow{p}V, $$ where $V$ is positive definite. Assume also that $R$ is fixed and has rank $q$. Then $S=RVR^T$ is positive definite: for any nonzero $a\in\mathbb R^q$, $R^Ta\ne0$, so $a^TSa=(R^Ta)^TV(R^Ta)>0$. In particular, the inverse required by the test exists asymptotically.

Theorem: Under these conditions, $W_n\xrightarrow{d}\chi^2_q$ under $H_0$.

Proof. Since $R\theta_0=r$, the scaled restriction error is $$ \sqrt n(R\hat{\theta}-r)=R\sqrt n(\hat{\theta}-\theta_0)\xrightarrow{d}N(0,RVR^T)=N(0,S), $$ by the continuous mapping theorem. Also, $R\widehat V R^T\xrightarrow{p}S$. Matrix inversion is continuous at a nonsingular matrix, so $(R\widehat V R^T)^{-1}\xrightarrow{p}S^{-1}$. Slutsky’s theorem therefore gives the limit of the Wald quadratic form as $Z^TS^{-1}Z$, where $Z\sim N(0,S)$. Write $Z=S^{1/2}U$, where $U\sim N(0,I_q)$. Then $Z^TS^{-1}Z=U^TU=\sum_{j=1}^q U_j^2$, which has a chi-square distribution with $q$ degrees of freedom. Hence $W_n\xrightarrow{d}\chi^2_q$. $\square$

This is an asymptotic calibration, not an exact finite-sample claim. At significance level $\alpha$, reject when $W_n$ exceeds the $1-\alpha$ quantile of $\chi^2_q$, or report the corresponding chi-square tail probability. The validity of the conclusion depends on the robust estimator’s asymptotic normality and on a consistent estimate of its covariance.

Worked Examples

Worked Example: Jointly Testing Two Slopes

Suppose a two-response regression has coefficient vector ordered as $\theta=(\beta_{01},\beta_{11},\beta_{02},\beta_{12})^T$, where $\beta_{01},\beta_{02}$ are intercepts and $\beta_{11},\beta_{12}$ are slopes for the two responses. A study with $n=100$ gives slope estimates $(0.4,-0.2)^T$. The relevant submatrix of the estimated covariance $\widehat V$ for the scaled coefficient error is $$ \widehat V_{\text{slopes}}= \begin{pmatrix}4&1\\1&9\end{pmatrix}. $$ Test jointly whether both slopes are zero.

The null is $H_0:R\theta=(0,0)^T$, where $$ R= \begin{pmatrix}0&1&0&0\\0&0&0&1\end{pmatrix}. $$ The restriction error is $d=(0.4,-0.2)^T$. The determinant of the covariance submatrix is $4(9)-1(1)=35$, so its inverse is $$ \widehat V_{\text{slopes}}^{-1} =\frac{1}{35}\begin{pmatrix}9&-1\\-1&4\end{pmatrix}. $$ Multiplying gives $\widehat V_{\text{slopes}}^{-1}d=(3.8,-1.2)^T/35$, and therefore $$ W_n=100d^T\widehat V_{\text{slopes}}^{-1}d =100\frac{0.4(3.8)+(-0.2)(-1.2)}{35} =\frac{176}{35}\approx5.03. $$ There are $q=2$ restrictions, so the reference distribution is $\chi^2_2$. Its upper-tail probability at $5.03$ is approximately $0.081$. At the 5% level, this test does not reject the joint null. This conclusion concerns the two slopes together; it is not the same decision procedure as performing two separate tests.

Worked Example: A Planned Difference Contrast

In a regression with two slopes $\beta_1,\beta_2$, suppose their estimates are $\hat{\beta}_1=0.7$ and $\hat{\beta}_2=0.2$. The estimated covariance matrix of these two coefficient estimates is $$ \widehat C= \begin{pmatrix}0.04&0.01\\0.01&0.09\end{pmatrix}. $$ Test the planned hypothesis that the first slope exceeds the second by $0.3$, so the null is $\beta_1-\beta_2=0.3$.

Here $R=(1,-1)$ and $r=0.3$. The observed restriction error is $R\hat{\theta}-r=0.7-0.2-0.3=0.2$. Its estimated variance is $$ R\widehat C R^T=0.04+0.09-2(0.01)=0.11. $$ Thus $W_n=0.2^2/0.11\approx0.364$, with one degree of freedom. The upper-tail probability is approximately $0.546$, so the data do not provide evidence against this specified difference. Changing $r$ changes the scientific hypothesis: the row $(1,-1)$ with $r=0$ would test equality, whereas the row with $r=0.3$ tests a difference of $0.3$.

Worked Example: Two Restrictions Written as Contrasts

For one response, suppose the two slope estimates are $\hat{\beta}_1=0.3$ and $\hat{\beta}_2=-0.1$, and their estimated covariance matrix is $0.01I_2$. Test jointly whether their sum and their difference are both zero: $$ H_0: \begin{pmatrix}1&1\\1&-1\end{pmatrix} \begin{pmatrix}\beta_1\\\beta_2\end{pmatrix} = \begin{pmatrix}0\\0\end{pmatrix}. $$

The matrix has rank 2. The estimated contrast vector is $d=(0.3-0.1,\,0.3-(-0.1))^T=(0.2,0.4)^T$. The covariance of the contrasts is $$ R\widehat C R^T =0.01\begin{pmatrix}2&0\\0&2\end{pmatrix} =\begin{pmatrix}0.02&0\\0&0.02\end{pmatrix}. $$ Its inverse is the diagonal matrix with entries 50. Therefore $$ W_n=50(0.2^2)+50(0.4^2)=2+8=10. $$ The test has 2 degrees of freedom and an upper-tail probability $e^{-10/2}\approx0.0067$. It rejects the joint null at the 5% level. Because the contrast matrix is invertible, these two restrictions together are equivalent to $\beta_1=\beta_2=0$; the contrast form simply states the joint question in terms of sum and difference.

Equivalent Ways to Write the Same Restrictions

A full-rank change of coordinates among restrictions does not change the Wald statistic. This can be useful when a set of contrasts is easier to interpret than another set of equations. The requirement is that the transformation is invertible and is applied to both sides of the hypothesis.

Result: If $A$ is an invertible $q$-by-$q$ matrix, then $R\theta=r$ and $AR\theta=Ar$ define the same restrictions and give the same Wald statistic.

Proof. Let $d=R\hat{\theta}-r$ and $S=R\widehat V R^T$. Under the transformed restrictions the error and covariance are $Ad$ and $ASA^T$, respectively. Since $A$ and $S$ are invertible, $$ (ASA^T)^{-1}=A^{-T}S^{-1}A^{-1}. $$ Consequently, $$ (Ad)^T(ASA^T)^{-1}(Ad) =d^TA^TA^{-T}S^{-1}A^{-1}Ad =d^TS^{-1}d. $$ The factor $n$, when used, is unchanged as well. Thus the Wald statistic is identical. $\square$

Interpretation and Common Pitfalls

The restriction matrix records which coefficients enter each hypothesis and with what weights. Its rank determines the number of independent restrictions, and hence the chi-square degrees of freedom. Before calculating, check that the rows are independent and that every row has a matching entry in $r$. If rows are redundant, $R\widehat V R^T$ is singular; the displayed inverse does not exist. Remove redundant equations when they express the same compatible restrictions. If the right-hand sides make those equations incompatible, the stated null has no parameter value satisfying it and should be reformulated.

A frequent mistake is to treat a contrast row as the whole hypothesis. For example, the row $(1,-1)$ does not by itself assert equality: with right-hand side $0$, it tests equality; with right-hand side $1$, it tests whether the first coefficient minus the second equals $1$. Another mistake is to ignore covariance among contrasts or to use the wrong covariance scale. If $\widehat V$ estimates the covariance of $\sqrt n(\hat{\theta}-\theta)$, the formula includes $n$; if $\widehat C$ estimates the covariance of $\hat{\theta}$, it does not. The two formulas agree when $\widehat C=\widehat V/n$.

A joint Wald test answers one joint question and uses the full covariance among the restrictions. It cannot generally be replaced by a collection of individual tests: separate decisions do not account for the same joint geometry or control error in the same way. In applied work, report the restrictions, their right-hand side, the degrees of freedom, the statistic, the reference calibration, and the substantive interpretation.

Check Your Understanding

For each question, identify the restrictions and the role of the covariance estimate before deciding how to calculate the test.

What hypothesis is represented by $R=(1,-1)$ and $r=0.5$ for two slopes? What would change if $r=0$?
Under what asymptotic assumptions does the robust Wald statistic have a chi-square reference distribution, and how many degrees of freedom does it use?
If a program reports the estimated covariance of $\hat{\theta}$ directly, should the Wald formula include an additional factor of $n$? Explain.
Why can a joint test of two restrictions differ in interpretation from two separate coefficient tests?
What goes wrong with the inverse in the Wald statistic if the rows of $R$ are linearly dependent?