Why Compare Covariate Means Jointly?
A balance table can show the standardized mean difference for each covariate, one at a time. That is useful, but the columns of such a table do not show how the mean differences line up with the covariates’ joint variation. If two covariates are strongly correlated, a difference in one direction may be common in the data, while an equally large difference in another direction may be unusual. Mahalanobis distance summarizes the separation between group mean vectors while accounting for that covariance structure.
This is a descriptive balance diagnostic. It does not establish that treatment assignment is unconfounded, and it does not measure differences in every aspect of the covariate distributions. Its specific target is the joint separation of the selected covariate means, on a scale adjusted for their variances and covariances.
The mean difference vector is \(\boldsymbol{\delta}=\bar{\mathbf X}_1-\bar{\mathbf X}_0\). The covariance matrix must describe the same covariates, in the same order, as this vector. Its diagonal entries are variances; its off-diagonal entries are covariances. The inverse matrix scales differences by variability and accounts for their directions relative to the correlations.
Choosing the Covariance Scale
A common choice is the pooled within-group covariance matrix,
where \(S_1\) and \(S_0\) are the sample covariance matrices in the treated and comparison groups, and \(n_1\) and \(n_0\) are their sample sizes. This is the conventional pooled covariance under a common-covariance model. For a balance diagnostic, it is also possible to use another explicitly defined reference covariance, such as a covariance matrix calculated in the original sample. The choice changes the scale and can change the numerical distance.
When comparing balance before and after matching or weighting, use the same covariate definitions and, when the goal is a direct comparison of distances, the same reference covariance. Re-estimating the scale after each adjustment can make the distances difficult to compare: a change could reflect a changed mean difference, a changed covariance scale, or both. State the reference covariance and how it was obtained.
Mahalanobis distance does not require normally distributed covariates when used as a descriptive summary. Its practical requirements are that the means and reference covariance are meaningful for the covariates being assessed, and that \(S_{\mathrm{ref}}\) is invertible. A singular or nearly singular matrix can arise from exact or near-linear dependence, too many covariates for the available observations, or redundant covariate coding. In that case the inverse is undefined or unstable; do not silently treat the resulting distance as reliable.
Why Covariance Scaling Matters
If the reference covariance is diagonal, the squared distance is the sum of squared mean differences divided by their reference variances. In that special case, each difference is scaled by its standard deviation, much like a standardized mean difference. With correlated covariates, however, the off-diagonal entries affect the answer. The distance depends not only on how large each difference is, but also on whether the difference points along a direction of common or uncommon joint variation.
A short derivation makes this geometric interpretation precise. Suppose \(S_{\mathrm{ref}}\) is positive definite, so it has a symmetric inverse square root \(S_{\mathrm{ref}}^{-1/2}\). Define the covariance-scaled difference as \(\mathbf z=S_{\mathrm{ref}}^{-1/2}\boldsymbol{\delta}\). Then
Thus, Mahalanobis distance is ordinary Euclidean distance after transforming the difference so that the reference covariance is standardized. This also explains why changing measurement units does not change the distance when the covariance is transformed consistently. It is not, however, a universal measure of balance: different covariate sets or reference covariance matrices define different distances.
Worked Example: Equal Marginal Differences, Unequal Joint Significance
Suppose two hypothetical covariates have reference covariance matrix \(S_{\mathrm{ref}}=\begin{pmatrix}1&0.9\\0.9&1\end{pmatrix}\). The treated-minus-comparison mean differences are \(0.5\) standard-deviation units for the first covariate and \(-0.5\) for the second. Calculate the joint distance.
The determinant is \(1-0.9^2=0.19\), so \(S_{\mathrm{ref}}^{-1}=\frac{1}{0.19}\begin{pmatrix}1&-0.9\\-0.9&1\end{pmatrix}\). With \(\boldsymbol{\delta}=(0.5,-0.5)^\mathsf{T}\),
Therefore \(D_M=\sqrt{5}=2.236\). Each marginal difference has absolute size \(0.5\), but their opposing signs point along a low-variance contrast when the covariates are strongly positively correlated. The joint separation is consequently large relative to the reference covariance. The result does not mean either individual covariate has a standardized difference of 2.236; it is a multivariate summary.
Worked Example: Calculating a Distance With Covariates in Different Units
In a hypothetical balance assessment, the covariates are age in years and a laboratory measure in its recorded units. The treated-minus-comparison mean differences are \(4\) years and \(1\) laboratory unit. Use the reference covariance matrix \(S_{\mathrm{ref}}=\begin{pmatrix}100&12\\12&4\end{pmatrix}\).
Its determinant is \(100(4)-12(12)=256\), so \(S_{\mathrm{ref}}^{-1}=\frac{1}{256}\begin{pmatrix}4&-12\\-12&100\end{pmatrix}\). Substituting \(\boldsymbol{\delta}=(4,1)^\mathsf{T}\) gives
Thus \(D_M=\sqrt{0.2656}=0.5154\). The covariance matrix incorporates the squared units of each variable and their covariance, so the distance itself is dimensionless. The arithmetic is valid because the supplied covariance matrix is positive definite: its first diagonal entry is positive and its determinant, \(256\), is positive.
Reading and Comparing Distances
A distance of zero means the two mean vectors are identical. Larger values mean greater separation relative to the chosen reference covariance, but there is no universal cutoff that declares a particular value acceptable. The value depends on the number and choice of covariates, their covariance structure, and the reference matrix. Treat it as a summary to interpret alongside covariate-specific diagnostics, not as a pass/fail test.
The next example shows why the reference scale matters when tracking an adjustment. Suppose a fixed reference covariance is \(S_{\mathrm{ref}}=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). For any difference \((x,y)^\mathsf{T}\), its inverse gives \(D_M^2=(x^2+y^2-1.6xy)/0.36\). Consider hypothetical mean differences before adjustment of \((0.6,-0.4)\), and after adjustment of \((0.1,0.1)\).
Worked Example: Tracking Joint Mean Balance With a Fixed Scale
Before adjustment, substituting \(x=0.6\) and \(y=-0.4\) gives
After adjustment,
On this fixed covariance scale, the joint mean separation decreased substantially. The calculation describes mean balance for these two covariates only; it does not show that every feature of their distributions is balanced, or that treatment groups are exchangeable.
Diagnostics, Alternatives, and Limitations
Before interpreting a computed distance, check the covariate order in the difference vector and covariance matrix, confirm that the matrix corresponds to the intended reference population, and inspect whether it is singular or poorly conditioned. A near-singular matrix can produce a very large distance from small data changes. Highly correlated covariates, duplicate variables, or a categorical variable represented by redundant indicator columns deserve particular attention.
Mahalanobis distance complements, rather than replaces, other balance diagnostics. Standardized mean differences reveal which individual covariates have mean differences and are easier to interpret in context. Variance ratios, quantile comparisons, and distribution plots can reveal dispersion, tail, or shape differences that mean-vector distance misses. A small Mahalanobis distance does not rule out those problems, nor does it assess covariate interactions or nonlinear features not represented in the chosen vector.
| Diagnostic | What it summarizes well | Important limitation |
|---|---|---|
| Covariate-specific standardized mean differences | Mean difference for each covariate separately | Does not combine covariate correlations into a joint summary |
| Mahalanobis distance | Joint separation of selected covariate means on a covariance-scaled metric | Can conceal which covariate contributes; does not assess distribution shape |
| Variance ratios and distribution plots | Differences in spread, tails, or other distribution features | Do not by themselves provide a single covariance-scaled mean-distance summary |
A frequent mistake is to calculate the square root of the sum of squared marginal standardized differences and call it the Mahalanobis distance. That calculation is correct only when the reference covariance is diagonal (or when the covariates have first been transformed to an uncorrelated reference scale). With correlated covariates, omitting the off-diagonal terms ignores the direction of joint variation. Another mistake is to treat a small distance as proof of adequate causal adjustment. Balance diagnostics describe measured covariates; they cannot establish balance on unmeasured factors or guarantee that the analysis design identifies a causal effect.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If the mean vectors are identical, what is the Mahalanobis distance, provided the reference covariance is invertible?
- For uncorrelated covariates with unit variances and mean differences \(0.3\) and \(0.4\), calculate \(D_M\).
- Why can two covariates with marginal standardized differences of \(0.5\) and \(-0.5\) have a large joint distance when their correlation is strongly positive?
- Name one reason the inverse reference covariance may be unstable, and state an appropriate response.
- Does a small Mahalanobis distance establish balance in covariate distributions beyond their means? Explain.
Solutions. (1) The difference vector is zero, so \(D_M^2=\mathbf{0}^\mathsf{T}S_{\mathrm{ref}}^{-1}\mathbf{0}=0\), and \(D_M=0\). (2) With identity covariance, \(D_M=\sqrt{0.3^2+0.4^2}=\sqrt{0.25}=0.5\). (3) Opposing mean differences point along a low-variance contrast under strong positive correlation, so they are large relative to the joint variation. (4) Near-linear dependence among covariates can make the covariance nearly singular; inspect the covariates and matrix, remove redundancy or use a defensible lower-dimensional specification, and report the choice rather than trusting an unstable inverse. (5) No. The distance summarizes mean-vector separation only; distributional shape, spread, tails, and unmeasured covariates require other assessment.