Quantile-Quantile Plots for Covariate Balance

Why Compare Quantiles?

A balance summary such as a standardized mean difference describes a difference in averages. A variance ratio adds information about spread, and earlier material on skewness and tail imbalance highlights why a small mean difference alone can miss important distributional discrepancies. A quantile-quantile (QQ) plot provides a direct visual comparison across the covariate’s distribution: it pairs selected quantiles from the control and treated groups and displays one against the other.

In a propensity-score analysis, the plot is a design diagnostic. It can be drawn before and after matching or weighting to inspect whether the groups’ covariate distributions have become more alike. It does not establish that treatment assignment is unconfounded, and it does not show balance on covariates that were not plotted.

Choosing and Calculating Quantiles

A plot needs a rule for calculating sample quantiles. One common plotting convention, used in the examples here, is linear interpolation between ordered observations. In a group with \(n\) observed values, write the sorted values as \(x_{(1)}\leq \cdots \leq x_{(n)}\). For a probability \(u\), calculate \(h=1+(n-1)u\). If \(h\) is an integer, the sample quantile is \(x_{(h)}\). Otherwise, let \(j=\lfloor h\rfloor\) and \(\gamma=h-j\); the sample quantile is \((1-\gamma)x_{(j)}+\gamma x_{(j+1)}\).

The same probability levels must be used for both groups. Common choices include a regular grid of probabilities, such as \(0.10,0.20,\ldots,0.90\), or selected levels such as \(0.25,0.50,0.75\). A finer grid shows more of the distribution, but does not create additional information beyond the observed data. In small samples, especially at the extremes, sample quantiles can be imprecise.

Quantile definitions vary across software. Another standard definition is the generalized inverse of a cumulative distribution function, \(Q(u)=\inf\{x:F(x)\geq u\}\). For discrete data or ties, the proportion of observations at or below this quantile can exceed \(u\); it need not equal \(u\). The interpolation convention above is a plotting rule, not a claim that exactly a fraction \(u\) of observations falls below each interpolated value. State the convention and use it consistently when comparing groups or reproducing a plot.

Reading Location, Scale, and Shape

The key visual reference is the equality line \(y=x\). Points close to it indicate similar quantiles at the plotted probabilities. For clarity, keep the same units and axis scales on both axes. If the axes have different scales or ranges, apparent departures from the equality line can be misleading.

• Location difference: Points follow a line approximately parallel to \(y=x\), but offset above or below it. A roughly constant vertical difference means the treated and control quantiles differ by a similar amount across much of the distribution.
• Scale difference: Points follow an approximately straight line through or near the origin with a slope different from 1. A slope greater than 1 means treated quantiles are more dispersed relative to the control quantiles; a slope between 0 and 1 means they are less dispersed, when the pattern is otherwise approximately linear.
• Tail difference: The middle quantiles may lie near the equality line while points at one or both ends depart. An upper-tail departure means the high quantiles differ more than the middle quantiles; a lower-tail departure has the analogous interpretation at the low end.
• Shape difference: Systematic curvature, an S-shaped pattern, or changing departures across the plot suggests that one simple location-and-scale description is inadequate. For example, one group may be more skewed, or the groups may differ in more than one part of the distribution.

These are visual descriptions, not automatic diagnoses. A few extreme points can reflect genuine imbalance, sampling variability, or data problems. Examine the underlying covariate values and sample sizes before deciding which explanation is plausible.

Deriving the Straight-Line Pattern

The usual interpretation of a straight-line pattern follows from how quantiles change under a positive affine transformation. Suppose that, in the population, the treated covariate is distributed as \(X_1=a+bX_0\), where \(b>0\), \(a\) is a shift, and \(b\) is a scale factor. For a continuous \(X_0\), its treated cumulative distribution function satisfies \(F_1(y)=F_0((y-a)/b)\). Applying the generalized-inverse quantile definition gives the relationship below. The same relationship holds for the stated interpolation rule when the sorted treated values are obtained by applying this positive transformation to the sorted control values.

Therefore, the QQ points lie on a line with intercept \(a\) and slope \(b\). When \(b=1\), the difference between paired quantiles is the constant \(a\), the location-shift pattern. When the distributions are equal, \(a=0\) and \(b=1\), producing the equality line. This derivation describes an ideal distributional relationship; a real sample can show departures from a straight line even when the underlying relationship is approximately affine.

Making the Diagnostic Useful

A defensible comparison requires that the two sets of quantiles describe the covariate on the same scale and for the analysis population of interest. Use the same measurement definition and units in both groups, and calculate quantiles after applying the matching or weighting procedure whose balance is being assessed. If weights are used, calculate weighted quantiles according to a stated, consistent rule; an unweighted plot of the weighted sample’s rows does not describe the weighted distributions. For a matched design, the QQ plot compares marginal covariate distributions; it does not display which individuals were paired.

A practical diagnostic workflow is to draw the same covariate’s plot before and after matching or weighting, use identical axis limits, show the equality line, and report the quantile convention and probabilities used. If group sizes differ greatly, or weights concentrate heavily on a few observations, the apparent smoothness of a plot should not be mistaken for precision. Extreme quantiles are particularly unstable in small samples.

QQ plots complement, rather than replace, other balance summaries. Standardized mean differences summarize location with a scale-free number; variance ratios compare spread; a QQ plot reveals how differences vary across quantiles. A compact table can be easier to scan across many covariates, while a plot is often more revealing for a selected continuous covariate. No single univariate display establishes joint balance across several covariates.

Common Mistakes and Limitations

• Using mismatched quantile probabilities. Pairing, for example, the control median with a treated third quartile creates a point that has no standard QQ interpretation. Calculate both groups at the same \(u\) values.
• Reversing the axes without saying so. With control on the horizontal axis, a treated value above the equality line means that treated quantile is higher. Reversing axes changes the visual orientation, so label them explicitly.
• Calling any upward departure a location shift. A constant vertical gap suggests a location difference. A gap that grows across quantiles may instead indicate a scale or shape difference.
• Overinterpreting the tails. A tail point is often based on very few observations. Check the values, group sizes, weights, and measurement quality before drawing a substantive conclusion.
• Treating visual agreement as proof of causal validity. The plot assesses one measured covariate’s marginal distribution. It cannot show that all confounding has been controlled, nor can it establish causal identification.

A QQ plot is most useful when it answers a specific diagnostic question: where in this covariate’s distribution do the groups differ, and did the design procedure reduce that difference? It is a descriptive tool, so assess its pattern alongside other diagnostics and substantive knowledge rather than applying a universal pass/fail rule.