Wald Inference for Multivariate Regression Coefficients

Why Wald Inference Needs a Sandwich

In multivariate regression, each observation contributes a vector of responses. Those responses may be correlated within an observation, and their errors need not be normally distributed. A coefficient estimate alone does not capture these features: inference also needs a covariance estimate that reflects the estimating procedure and the dependence in the data.

Robust regression estimating equations can limit the effect of unusual residuals on the coefficient estimates. Their sandwich covariance estimates sampling variability without requiring a normal error distribution or a correctly specified likelihood. The resulting Wald tests are generally asymptotic, rather than exact finite-sample tests.

Model, Estimating Equations, and Sandwich Covariance

A common construction uses residuals \(e_i=Y_i-B^\mathsf{T}x_i\), a scatter estimate \(S\), and a bounded multivariate score \(\psi\). A coefficient-equation contribution can be written as \(G_{i,\beta}=\operatorname{vec}\{x_i\psi(S^{-1/2}e_i)^\mathsf{T}\}\). A radial score, for example, can downweight residuals whose standardized lengths are large. The exact score and scatter equations depend on the chosen robust estimator; the sandwich calculation applies to the full system of equations.

Let \(\gamma_0\) denote the true parameter value, with \(E[G_i(\gamma_0)]=0\). Define the sensitivity matrix, sometimes called the bread, and the variability matrix, sometimes called the meat, by

The meat is a covariance matrix because the estimating equation has mean zero. It retains covariance among the components of \(G_i\), including the contribution of correlated responses. In practice, the bread is estimated from derivatives of the fitted estimating equations and the meat from their empirical contributions. If nuisance parameters are estimated, both calculations use the stacked equations, so their effect on coefficient uncertainty is included.

The factor \(1/n\) is present when \(\hat A\) and \(\hat\Omega\) estimate the per-observation matrices \(A\) and \(\Omega\). Equivalently, if bread and meat are formed from the summed equations, the same scaling must be accounted for. Keeping this convention explicit prevents a common factor-of-\(n\) error.

Why the Sandwich Gives Asymptotic Covariance

The following argument applies to independent, identically distributed observations. Suppose the estimating equations are unbiased at \(\gamma_0\), \(\hat\gamma\xrightarrow{p}\gamma_0\), and \(A\) is nonsingular. Assume the contributions have finite second moments, \(E[G_iG_i^\mathsf{T}]=\Omega\), and a central limit theorem applies to their sum. Also assume the equation is differentiable near \(\gamma_0\), with its sample derivative converging uniformly in a neighborhood of \(\gamma_0\) to the continuous population derivative \(A\). These conditions justify the local Taylor expansion and its limiting derivative.

Proof. Expand the sample estimating equation around \(\gamma_0\). The integral form of the mean-value expansion gives

By consistency, the points along the segment from \(\gamma_0\) to \(\hat\gamma\) approach \(\gamma_0\) in probability. The stated uniform convergence and continuity make the matrix in braces converge in probability to \(A\). By the multivariate central limit theorem, \(n^{-1/2}\sum_iG_i(\gamma_0)\xrightarrow{d}N(0,\Omega)\). Since \(A\) is nonsingular, solving the expansion and applying Slutsky’s theorem gives the limit \(-A^{-1}Z\), where \(Z\sim N(0,\Omega)\). Its covariance is \(A^{-1}\Omega A^{-\mathsf{T}}\), as claimed. \(\square\)

Wald Inference for the Coefficient Matrix

Let \(\hat\beta=\operatorname{vec}(\hat B)\), and let \(\hat V_\beta\) be its sandwich covariance estimate. To test a specified complete coefficient matrix \(B_0\), the null hypothesis is \(H_0:B=B_0\). If the coefficient estimator is asymptotically normal and the coefficient covariance is nonsingular, the Wald statistic is

Under \(H_0\), \(W\) converges in distribution to a chi-square random variable with \(pq\) degrees of freedom. The degrees of freedom equal the number of coefficients being tested. A single coefficient can instead be assessed by dividing its estimate minus its null value by its sandwich standard error, yielding an asymptotic standard normal statistic.

Justification of the Wald limit. The coefficient block of the asymptotic result above gives \(\hat V_\beta^{-1/2}(\hat\beta-\beta_0)\xrightarrow{d}N(0,I_{pq})\) under the null, using consistency of the sandwich estimate. The Wald statistic is the squared Euclidean length of this standardized vector. The sum of squares of \(pq\) independent standard normal variables has a chi-square distribution with \(pq\) degrees of freedom. The continuous mapping theorem therefore gives the asserted limiting distribution. \(\square\)

This reference distribution is an asymptotic calibration, not a claim that the statistic has an exact chi-square distribution for a particular finite sample. It also relies on a nonsingular covariance for the coefficient vector; redundant or unidentifiable coefficients require a different parameterization before this full-matrix test is used.

How Correlated Responses Enter the Meat

The response components need not contribute independently to the estimating equation. For example, with two predictors and two responses, a single coefficient-equation contribution has the form \(x_i\psi(u_i)^\mathsf{T}\), a \(2\times2\) matrix. Its vectorization contains contributions for both responses, so its outer product has cross-response entries. Summing these outer products is one way to form the empirical meat.

Common Pitfalls and Interpretation

• Using model-based standard errors automatically. Robust coefficient estimation does not make a normal-error covariance formula valid. Use the estimating-equation sandwich unless its additional model assumptions are justified.
• Ignoring estimated nuisance parameters. If scatter or other nuisance quantities are estimated, use the stacked estimating equations and take the coefficient block from their joint sandwich covariance. Treating the nuisance estimate as fixed can omit its contribution to uncertainty.
• Dropping cross-response covariance. The response components can be correlated even when the coefficient equations are written response by response. The meat must retain the cross-products of their estimating-equation contributions.
• Reading an asymptotic test as an exact finite-sample test. The chi-square and normal reference distributions follow from large-sample results. Small samples may need a separate calibration method.

The sandwich has two distinct jobs: the robust estimating equations determine how the fitted coefficients respond to the data, and the sandwich estimates the variability of those estimates under the actual estimating-equation contributions. Neither job alone justifies ignoring dependence among outcomes or claiming normal errors.