Why Look Beyond Means and Variances?
A covariate can have the same mean and variance in treated and control groups while having a different shape. One group may have a long upper tail, more extreme observations, or a concentration of values near its center. Such differences can matter for design: a covariate’s distribution may be poorly aligned even when the familiar summaries of location and spread appear reassuring.
Skewness, kurtosis, and tail summaries add useful detail to balance assessment. They do not replace standardized mean differences or variance ratios. Rather, they diagnose aspects of the distributions those summaries do not capture. As with the earlier material on covariate balance as a design diagnostic, the purpose is to assess the comparability of covariates before using outcomes to estimate effects.
Skewness describes asymmetry: positive values indicate a longer or more influential upper side of the distribution, and negative values indicate a longer or more influential lower side. Excess kurtosis describes the fourth moment relative to the squared variance. A larger value means that observations farther from the mean contribute more heavily to this summary; it is often associated with more prominent tails, but it is not a complete description of tail behavior. For a constant covariate, \(m_{2,r}=0\), so both summaries are undefined.
Comparing Skewness and Kurtosis
Calculate these summaries separately for treated and control groups using the same covariate, sample definition, and weighting scheme. Compare the values between groups, not just with a reference value such as zero. A group’s skewness or kurtosis can be nonzero even when the two groups match on that feature. Conversely, values that look individually modest may still differ enough to merit investigation.
The following calculations use empirical moments with divisor \(n_r\). Small-sample corrections can change the numerical values, particularly when a group is small. For a balance diagnostic, consistency of convention is essential: do not compare a bias-corrected estimate in one group with an uncorrected estimate in the other.
Worked Example: Equal Means and Variances, Different Skewness
Suppose a hypothetical continuous covariate has the following four values in each group, recorded in standardized units. The treated values are \(-1,-1,-1,3\); the control values are \(-3,1,1,1\). These small data sets are constructed for illustration, not taken from a study.
For the treated group, the mean is \((-1-1-1+3)/4=0\), and the second central moment is \((1+1+1+9)/4=3\). Its third central moment is \((-1-1-1+27)/4=6\), so its empirical skewness is:
For the control group, the mean is \((-3+1+1+1)/4=0\), and its second central moment is \((9+1+1+1)/4=3\). Its third central moment is \((-27+1+1+1)/4=-6\), giving:
The group means and variances match exactly: both means are 0 and both empirical variances are 3. Yet the skewness differs substantially and in direction. The treated group has a relatively influential upper observation; the control group has a relatively influential lower observation. Mean and variance diagnostics alone would miss this shape difference.
Kurtosis can differ even when means, variances, and skewness match. Because the fourth power gives greater weight to values far from the mean, a relatively small number of more extreme observations can affect \(m_4\) strongly. But kurtosis is not a direct count of tail observations: distinct distributions can share the same kurtosis, and a difference in kurtosis does not specify which tail is responsible.
Worked Example: Different Kurtosis With Matching Lower Moments
Consider two hypothetical groups of four covariate values. Group A has \(-1,-1,1,1\); Group B has \(-\sqrt{2},0,0,\sqrt{2}\). Both groups have mean 0. For Group A, \(m_2=(1+1+1+1)/4=1\); for Group B, \(m_2=(2+0+0+2)/4=1\). Both groups also have \(m_3=0\), so both have skewness 0.
For Group A, \(m_4=(1+1+1+1)/4=1\), and its excess kurtosis is \(1/1^2-3=-2\). For Group B, \(m_4=(4+0+0+4)/4=2\), and its excess kurtosis is \(2/1^2-3=-1\). Thus the means, variances, and skewness match, but the excess kurtosis differs by 1.
In Group B, two observations are farther from the center than any observation in Group A, while four observations equal the center. This example shows how fourth-moment summaries can flag a difference in how values are distributed away from the center. It does not establish that kurtosis alone captures all relevant features of the tails.
Direct Tail Comparisons
A tail diagnostic can target a threshold with a meaningful interpretation. For example, compare the proportions of treated and control observations above a specified high value, below a specified low value, or outside a common central range. The threshold should be chosen before inspecting outcomes and should be explained in the covariate’s units or relative to a stated reference scale.
Use a common reference for both groups. If each group is centered and scaled by its own mean and standard deviation, a difference in location or spread can be obscured: each group is made to look centered with unit spread by construction. A reference location and scale can come from the pre-match sample or another clearly stated target. Report the original-unit thresholds as well when they are meaningful.
For a common reference location \(\mu_{\mathrm{ref}}\), scale \(s_{\mathrm{ref}}>0\), and threshold \(c>0\), define an upper-tail rate as the proportion with \(X>\mu_{\mathrm{ref}}+c s_{\mathrm{ref}}\), and a lower-tail rate as the proportion with \(X<\mu_{\mathrm{ref}}-c s_{\mathrm{ref}}\). The strict inequalities make the rule explicit; with discrete or rounded data, state how observations on a threshold are handled. The difference in rates is a descriptive comparison, not a test of a causal effect.
Worked Example: Comparing Both Tails at a Common Threshold
Suppose two hypothetical groups each have eight values. Group A has \(-1,-1,-1,-1,1,1,1,1\). Group B has \(-\sqrt{2},-\sqrt{2},0,0,0,0,\sqrt{2},\sqrt{2}\). Both have mean 0 and empirical variance 1. Use the common reference location 0 and scale 1, and set \(c=1.2\), so the lower and upper cutoffs are \(-1.2\) and \(1.2\).
In Group A, no value is below \(-1.2\) and none is above \(1.2\). Each tail rate is therefore \(0/8=0\). In Group B, two values are below \(-1.2\) and two are above \(1.2\). Each tail rate is \(2/8=0.25\), or 25%. The groups have the same mean and variance, but one quarter of Group B lies in each of these outer regions while none of Group A does.
The choice \(c=1.2\) is illustrative, not a universal cutoff for adequate balance. A practical analysis should justify its threshold and examine whether a small number of observations drives the result. If these values were measured in a covariate’s original units, the cutoffs would be reported in those units too.
How to Use These Diagnostics
Apply the summaries to the analysis sample after matching or weighting, using the corresponding matched observations or weights consistently. For weighted data, calculate weighted moments and tail proportions under the same weights used to define the weighted sample; do not combine unweighted moments with weighted tail rates without explaining the distinction. When comparing before and after adjustment, use the same definitions and report the values in a table so that changes are interpretable.
A practical assessment can include group-specific skewness and excess kurtosis, their between-group differences, and selected upper- and lower-tail rates. For a threshold-based comparison, report the threshold, the numerator and denominator for each rate, and the rate difference. If the covariate has meaningful clinical cut points, report those as well as standardized thresholds. This makes the diagnostic understandable even to readers who do not find standardized units intuitive.
These summaries are descriptive. They do not have a universal pass/fail threshold, and a difference does not by itself prove that matching or weighting failed. Small samples can produce unstable third and fourth moments, while large samples can make tiny differences look numerically precise. Focus on the size and practical meaning of discrepancies, the number of observations involved, and whether the covariate is important to the analysis.
Specify the treatment groups, analysis sample, covariate units, and any weights. Use the same choices for every diagnostic being compared.
Compute group-specific skewness and excess kurtosis with a stated convention. Check the underlying observations when a moment difference is pronounced.
Choose justified cutoffs, apply a common reference location and scale, and report the upper- and lower-tail counts and proportions in context.
Read these results together with mean and variance summaries. Treat a mismatch as a prompt for design review, not as an automatic verdict.
Limitations and Common Pitfalls
Moment summaries compress an entire distribution into a few numbers. Two groups can have matching skewness and kurtosis but differ in other features; matching several moments does not establish that the empirical distributions are the same. Tail rates also depend on their chosen thresholds. They are informative when the thresholds address a concrete concern, but selective thresholds chosen after seeing the results can make a balance assessment misleading.
A further limitation is sensitivity to extreme observations. A single very large deviation can substantially affect the third or fourth moment, particularly in a small group. Check the observed values and data quality rather than deleting an observation solely because it changes a diagnostic. If observations are weighted or matched, also check that the summaries reflect the intended analysis sample and its weights.
- Mistake: “The means and variances match, so the covariate distributions match.” Matching those summaries constrains only location and spread; the examples show that skewness, kurtosis, or tail rates can still differ.
- Mistake: “Positive kurtosis means the upper tail is heavier.” Kurtosis does not identify which tail differs. Compare upper and lower tails separately if direction matters.
- Mistake: standardizing each group separately before comparing tail rates. This removes group-specific location and scale differences by construction. Use a common reference and state it.
- Mistake: treating one cutoff or moment threshold as a universal balance rule. Explain why a discrepancy matters in context; no single cutoff settles balance for every covariate or design.
Mean and variance diagnostics remain useful, particularly standardized mean differences and variance ratios, because they summarize location and spread in familiar ways. Skewness, kurtosis, and tail comparisons answer different questions and should be read as complements. Quantile-quantile plots offer another way to compare distributional shape; they provide a visual comparison rather than replacing careful interpretation of these summaries.
Check Your Understanding
Answer each question using the definitions and distinctions in this tutorial.
- A group has \(m_2=4\) and \(m_3=8\). What is its empirical skewness, and what does the sign indicate?
- Two groups have equal means and variances, but one has more observations beyond a prespecified common upper cutoff. What does this establish, and what does it not establish?
- Why is it generally inappropriate to standardize each group using its own mean and standard deviation before comparing tail proportions?
- A covariate is constant within a treatment group. Can its empirical skewness and excess kurtosis be calculated using the definitions here? Explain.
Solutions. (1) \(g_1=8/4^{3/2}=8/8=1\), indicating positive empirical skewness. (2) It establishes a difference in the observed upper-tail rates at that cutoff, despite equal means and variances; it does not establish that the entire distributions differ in a particular way, that the design has failed, or that a causal effect exists. (3) Group-specific standardization forces each group to have its own standardized mean 0 and standard deviation 1, potentially hiding the very location or scale differences relevant to the comparison. (4) No. A constant group has \(m_2=0\), so the skewness and excess kurtosis formulas divide by zero and are undefined.