Tutorials › Introduction to Statistics › Standardized Effect Sizes for Paired Data

Home  /  Tutorials  /  Paired and Matched Designs

Paired and Matched Designs · Tutorial 763 of 996

Standardized Effect Sizes for Paired Data

Learn to calculate a standardized effect for paired data and explain how it complements, rather than replaces, an estimate in practical units.

Intermediate 9 min read

What You'll Learn

  • Define the paired standardized mean difference using the mean and standard deviation of within-pair differences.
  • Calculate the sample effect size from a set of paired changes.
  • Interpret its direction and magnitude relative to the variability of the differences.
  • Explain why the result is unit-free but does not communicate practical importance by itself.
  • Distinguish this paired standardization from standardizations using other standard deviations.

Why Standardize a Paired Change?

A mean change in practical units is often the clearest summary of a paired study. A mean reduction of 2 points, for example, is directly understandable on the score scale. A standardized effect adds a different perspective: it expresses that average change in relation to how much the within-pair changes vary. This can help describe the size of a change on a unit-free scale, but it does not replace the original-unit estimate or determine whether the change matters in practice.

The key is to standardize the differences within pairs. Once each pair has been reduced to one difference, the calculation uses the mean and sample standard deviation of those differences. It does not treat the two measurements as independent groups or use the standard deviation of either measurement alone.

Definition: For \(n\) paired observations, define each difference \(d_i\) using one consistent order for the two measurements. Let \(\bar{d}\) be the sample mean of the differences and \(s_d\) their sample standard deviation. The paired standardized mean difference, commonly denoted \(d_z\), is $$ d_z = \frac{\bar{d}}{s_d}. $$ It reports the mean paired difference in units of the sample standard deviation of the paired differences. The corresponding population quantity is \(\delta_z=\mu_d/\sigma_d\), when the population mean difference and standard deviation are defined and \(\sigma_d>0\).

Calculation and Interpretation

The calculation starts with the differences, not with the two original columns separately. For the chosen direction, calculate their mean and sample standard deviation, then divide the mean by the standard deviation. The sample standard deviation uses denominator \(n-1\):

$$ \bar{d}=\frac{1}{n}\sum_{i=1}^{n} d_i, \qquad s_d=\sqrt{\frac{\sum_{i=1}^{n}(d_i-\bar{d})^2}{n-1}}, \qquad d_z=\frac{\bar{d}}{s_d}. $$

The sign follows the order used to define \(d_i\). If \(d_i\) is before minus after, a positive \(d_z\) means the average before measurement exceeds the average after measurement. If the order is reversed, the sign reverses. The magnitude describes the mean difference relative to the spread of differences: for instance, \(d_z=0.5\) means the sample mean difference is half a sample standard deviation of the differences from zero.

The units cancel. If differences are measured in points, both \(\bar{d}\) and \(s_d\) are in points, so their ratio has no units. This makes the value less tied to the measurement scale, but it does not make different outcomes automatically comparable in meaning. The variability of the differences is part of the denominator, so two studies with the same mean change can have different standardized effects.

Worked Example: Standardizing a Mean Score Reduction

Suppose, hypothetically, five students have scores recorded before and after a short practice program. Define each difference as before minus after, so a positive value is a score reduction.

StudentBefore (points)After (points)Difference \(d_i\) (points)
A10100
B12111
C14122
D16133
E18144

The differences are \(0,1,2,3,4\) points. Their mean is

$$ \bar{d}=\frac{0+1+2+3+4}{5}=2 \text{ points}. $$

The deviations from 2 are \(-2,-1,0,1,2\), whose squared values sum to 10. Therefore,

$$ s_d=\sqrt{\frac{10}{5-1}} =\sqrt{2.5} =1.581 \text{ points}. $$

The standardized mean reduction is

$$ d_z=\frac{2}{\sqrt{10/(5-1)}}=\frac{2}{1.5811388}=1.265. $$

In this hypothetical sample, the mean score reduction is 1.265 sample standard deviations of the paired reductions. The practical-unit estimate remains a reduction of 2 points. Neither number alone establishes whether the practice program caused the change or whether the reduction is educationally important.

What the Standard Deviation Represents

For paired data, \(s_d\) measures how spread out the observed within-pair differences are around their mean. It is not the standard deviation of the before scores, the after scores, or all measurements pooled together. That choice matters because pairing defines the unit of analysis: each pair contributes one difference.

The denominator therefore gives \(d_z\) a specific meaning. It scales the average paired difference by the observed variability in paired differences. The statistic is sensitive both to the mean change and to that variability. A relatively consistent change can yield a larger standardized effect than a more variable set of changes with the same average.

Worked Example: Matched Treatment and Comparison Measurements

Imagine a hypothetical study in which five matched pairs each receive one treatment and one comparison measurement of a laboratory outcome. Define \(d_i\) as treatment minus comparison, measured in units. Suppose the within-pair differences are \(4,6,5,3,7\) units. Positive differences indicate higher treatment measurements.

The mean difference is

$$ \bar{d}=\frac{4+6+5+3+7}{5}=5 \text{ units}. $$

The deviations from 5 are \(-1,1,0,-2,2\), and the sum of their squares is \(1+1+0+4+4=10\). Thus,

$$ s_d=\sqrt{\frac{10}{5-1}}=\sqrt{2.5}=1.581 \text{ units}, \qquad d_z=\frac{5}{1.5811388}=3.162. $$

The sample mean treatment-minus-comparison difference is 3.162 sample standard deviations of the paired differences. The positive sign communicates the direction under the stated coding. It does not mean the treatment measurement is 3.162 units higher; the difference in practical units is 5 units.

The calculation is defined for these five differences without a normality assumption. If the goal is to use the sample to make population-level inferences, the design and inferential conditions for paired differences also matter. In particular, the pairs must be formed and measured meaningfully, and observations from different pairs should be independent for standard one-sample inference on differences. A standardized effect by itself does not check those conditions.

A Short Derivation: Why the Scale Depends on Variability

The standardization follows directly from the units of a difference. Suppose every outcome and difference is converted by a positive factor \(c\), such as converting from one unit to a smaller unit. Then every \(d_i\) is replaced by \(c d_i\). The mean difference becomes \(c\bar{d}\), and the standard deviation becomes \(c s_d\). Consequently,

$$ \frac{c\bar{d}}{c s_d}=\frac{\bar{d}}{s_d}. $$

So changing the measurement unit does not change \(d_z\). If all differences are instead multiplied by a negative factor, their direction reverses and the sign of \(d_z\) reverses as well, while its magnitude remains the same. This establishes that the statistic is unit-free but still retains the direction set by the coding.

The same definition also shows why the standardized effect is not determined by the mean change alone. Holding \(\bar{d}\) fixed while increasing \(s_d\) decreases \(|d_z|\); reducing \(s_d\) increases it. Thus a large standardized value can reflect a large mean difference, a small spread of differences, or both.

Worked Example: Same Mean Change, Different Standardization

Compare the first example's differences, \(0,1,2,3,4\), with a second hypothetical set of five paired differences in points: \(-2,1,2,3,6\). Both sets have mean 2 points, but their spreads differ.

For the second set, the mean is

$$ \bar{d}=\frac{-2+1+2+3+6}{5}=\frac{10}{5}=2 \text{ points}. $$

The deviations from 2 are \(-4,-1,0,1,4\); their squared values sum to \(16+1+0+1+16=34\). Therefore,

$$ s_d=\sqrt{\frac{34}{5-1}}=\sqrt{8.5}=2.915 \text{ points}, \qquad d_z=\frac{2}{2.9154759}=0.6860. $$

The first set had the same mean difference but \(d_z=1.265\), whereas this set has \(d_z=0.6860\). The second set's differences are more spread out relative to their shared average of 2 points. The contrast is about the relationship between average change and variability, not a change in the average practical-unit effect.

How \(d_z\) Relates to the Paired \(t\) Statistic

The paired \(t\) statistic uses the same mean and standard deviation of differences, but it divides by the standard error of the mean difference. For \(n\) paired differences, that standard error is \(s_d/\sqrt{n}\). Therefore the algebraic relationship is

$$ t=\frac{\bar{d}}{s_d/\sqrt{n}} =\left(\frac{\bar{d}}{s_d}\right)\sqrt{n} =d_z\sqrt{n}. $$

This identity helps distinguish an effect-size description from a test statistic. The paired \(t\) statistic depends on both \(d_z\) and the number of pairs; \(d_z\) alone does not. A larger sample can produce a larger absolute \(t\) statistic for the same standardized effect. Conversely, neither a large \(t\) statistic nor a small p-value tells you directly whether a difference is important in practical units.

The relationship is an algebraic identity for the paired \(t\) statistic calculated from the same differences. It does not establish that the assumptions needed for a particular inferential conclusion are satisfied. Those depend on the study design and the distribution and independence of the paired differences.

What the Effect Size Does and Does Not Say

A paired standardized mean difference is useful when the scale-free relation between average change and variability is of interest. It can supplement a report of \(\bar{d}\), its units, and an appropriate interval or test. But standardization removes the original units, and the resulting number alone does not communicate a practical threshold, the range of individual changes, or the design quality.

There are multiple ways to standardize paired data. For example, a calculation using a before-measurement standard deviation or a pooled standard deviation answers a different scaling question. Here \(d_z\) specifically uses \(s_d\), the standard deviation of the differences. A report should state this convention rather than present a paired standardized effect as if every definition used the same denominator.

There is also no universal cutoff that makes a paired effect small, moderate, or large in every subject area. A given value may have different implications depending on the outcome, measurement reliability, decision being made, and consequences of change. Interpret the ratio alongside the mean change in practical units and the context of the question.

Common Mistake

A common error is to divide the mean difference by the standard deviation of the before scores, or by a pooled standard deviation, and call the result \(d_z\). That calculation may be a standardization someone has chosen for another purpose, but it is not the paired effect defined here. The correction is to first form one difference per pair, then calculate both \(\bar{d}\) and \(s_d\) from that same difference column, and use \(\bar{d}/s_d\).

Another mistake is to interpret \(d_z=1\) as a one-unit change. It means the mean paired difference is one standard deviation of the differences from zero. The change in outcome units is \(\bar{d}\), which must be reported separately.

Key takeaway: For paired data, \(d_z=\bar{d}/s_d\) standardizes the mean within-pair difference by the sample standard deviation of those differences. It is unit-free and direction depends on how differences are coded, but it does not replace the estimate in practical units or by itself establish practical importance.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

  1. A study defines each difference as after minus before. What does a negative \(d_z\) indicate about the average direction of change?
  2. For paired differences with \(\bar{d}=3\) minutes and \(s_d=2\) minutes, calculate \(d_z\) and interpret it in context.
  3. Two paired studies have the same mean difference, but one has a larger standard deviation of differences. Which has the smaller absolute standardized effect, and why?
  4. Why is the standard deviation of the before measurements not the denominator in the definition of \(d_z\) used here?
  5. If the outcome is converted from points to tenths of a point, does \(d_z\) change? Explain using the numerator and denominator.

Previous

Next

Back to Tutorials