Why Effect Size Needs More Than a Test Result
A hypothesis test addresses whether data are compatible with a specified null value, such as a zero difference. It does not, by itself, say whether the difference is large enough to matter. A very small estimated difference can be statistically detectable when measurements are precise and the sample is large. Conversely, an estimate that would matter in practice can remain uncertain when the data are limited or variable.
To discuss the magnitude of an effect, begin by stating what is being compared and in what units. A difference in average blood pressure might be measured in millimetres of mercury; a difference in event risks might be measured in percentage points. A standardized effect size expresses a difference relative to a measure of variability. The raw difference and its standardized version answer related but distinct questions.
Raw Differences and Standardized Mean Differences
For two populations, let \(\mu_1\) and \(\mu_0\) be their means and let \(\sigma\) represent an appropriate standard deviation for the outcome. A standardized mean difference is commonly written as \(\delta=(\mu_1-\mu_0)/\sigma\). It describes the mean separation in standard-deviation units. The estimate from sample data is often called Cohen's \(d\). When a common within-group standard deviation is a defensible reference, a conventional estimate is the pooled sample standard deviation.
Here, \(\bar{x}_1-\bar{x}_0\) is the estimated raw difference, and \(s_{\text{pooled}}\) combines the two sample standard deviations. Using this pooled standard deviation as a common reference is most natural when the two populations have similar variances. If variability differs substantially, the analyst should explain and justify the denominator used rather than treating one convention as automatic.
A raw difference retains direct meaning in context: for example, 0.5 points on a symptom scale. A standardized difference makes the magnitude comparable across scales in some settings, but removes the original units and depends on the variability of the population or sample used. Neither one alone decides whether an effect matters.
Worked Example: A Detectable Difference That May Be Too Small to Matter
In a hypothetical randomized study, a symptom score is measured in two independent groups. The treatment group has \(n_1=10{,}000\), mean \(\bar{x}_1=20.5\), and standard deviation \(s_1=10\) points. The control group has \(n_0=10{,}000\), mean \(\bar{x}_0=20.0\), and standard deviation \(s_0=10\) points. Suppose, before examining the results, clinicians define a difference of 2 points as the smallest change important enough to influence care.
For inference about the difference, assume the groups are independent, assignment was randomized, and the observations within each group are independent. With these large sample sizes, the sampling distribution of the difference in means is well approximated by a normal distribution. A Welch two-sample \(t\) procedure does not require equal population variances; the observed standard deviations happen to be equal here.
The estimated treatment-minus-control difference is \(\bar{x}_1-\bar{x}_0=20.5-20.0=0.5\) points. Its estimated standard error is \[ SE=\sqrt{\frac{s_1^2}{n_1}+\frac{s_0^2}{n_0}} =\sqrt{\frac{10^2}{10{,}000}+\frac{10^2}{10{,}000}} =\sqrt{0.02}\approx 0.1414\text{ points}. \] The test statistic for a zero difference is therefore \[ t=\frac{0.5-0}{0.1414}\approx 3.536. \] With approximately 20,000 degrees of freedom, the two-sided p-value is about 0.0004. Under the test's assumptions, this is strong evidence against a zero population mean difference.
The estimated difference is nevertheless only 0.5 points, less than the prespecified 2-point threshold for practical importance. For a standardized estimate, the pooled standard deviation is 10 points, so \[ d=\frac{20.5-20.0}{10}=0.0500. \] Thus the estimated difference is 0.05 standard deviations. The small p-value is evidence about compatibility with a zero difference, not proof that a 0.5-point change is important. Whether the true effect is below the 2-point threshold also requires considering uncertainty around the estimate; the point estimate alone does not settle that question.
Why Standardization Depends on Variability
The same raw difference can represent a large separation in one setting and a small separation in another. If two groups differ by 4 units on an outcome with a standard deviation of 2 units, the standardized difference is 2. If they differ by the same 4 units on an outcome with a standard deviation of 20 units, the standardized difference is 0.2. This does not mean that the raw change has altered: it means the change is large or small relative to the chosen spread of values.
Standardization also helps explain why a unit conversion does not, by itself, change a standardized mean difference. Suppose every measurement is converted using \(x^*=a+bx\), where \(b>0\). The additive constant shifts both means equally, while the multiplier scales both the mean difference and the standard deviation. In symbols,
Therefore, \[ \frac{\mu_1^*-\mu_0^*}{\sigma^*} =\frac{b(\mu_1-\mu_0)}{b\sigma} =\frac{\mu_1-\mu_0}{\sigma}. \] The standardized difference is unchanged by a positive change of measurement units, while the raw difference changes units. If \(b<0\), the standardized effect keeps its magnitude but reverses sign because the scale's direction has been reversed.
Worked Example: Same Raw Difference, Different Standardized Magnitudes
Consider two hypothetical settings in which an outcome is measured in the same units. In Setting A, the group means are 54 and 50, and a defensible common reference standard deviation is 2. In Setting B, the group means are 54 and 50 again, but the reference standard deviation is 20. Calculate the raw and standardized differences, taking the first mean minus the second.
In both settings, the raw difference is \(\Delta=54-50=4\) outcome units. In Setting A, \[ \delta_A=\frac{54-50}{2}=\frac{4}{2}=2. \] The means are separated by 2 standard deviations. In Setting B, \[ \delta_B=\frac{54-50}{20}=\frac{4}{20}=0.200. \] The raw difference is still 4 units, but it is only 0.2 of the reference standard deviation.
These calculations describe relative separation, not practical importance. If 4 units is a meaningful change according to the outcome's real-world consequences, that remains relevant in both settings. Conversely, a large standardized difference does not guarantee that the outcome matters to decision-makers. The meaning of 4 units must come from the context, not from the standardized number alone.
Effect Sizes for Risks and Other Outcomes
Standardized mean differences are designed for quantitative outcomes, but effect size is a broader idea: it is a measure of the magnitude of a contrast. For a binary outcome, such as whether an event occurred, the absolute risk difference keeps the comparison in probability units. Let \(p_1\) be the event probability in group 1 and \(p_0\) the probability in group 0. The risk difference is \(p_1-p_0\). Its sign depends on the order of subtraction, so the groups and direction must be stated.
A risk difference of 0.01 is a difference of one percentage point, not one percent. It can be accompanied by a relative measure, such as the risk ratio \(p_1/p_0\), but these quantities answer different questions: the risk difference describes an absolute change in probability, while the risk ratio describes a multiplicative comparison. Neither one automatically establishes whether the effect is important.
Worked Example: Interpreting an Absolute Difference in Event Risk
In a hypothetical randomized comparison, an event occurs in 20 of 1,000 people assigned to a treatment and in 10 of 1,000 assigned to control during the same follow-up period. Calculate the two risks, the treatment-minus-control risk difference, and the risk ratio.
The treatment risk is \(p_1=20/1{,}000=0.0200\), or 2.00%. The control risk is \(p_0=10/1{,}000=0.0100\), or 1.00%. Therefore, \[ p_1-p_0=0.0200-0.0100=0.0100, \] which is an absolute difference of 1.00 percentage point. The risk ratio is \[ \frac{p_1}{p_0}=\frac{0.0200}{0.0100}=2.00. \] In this sample, the treatment group's observed event risk is twice the control group's, and is higher by one percentage point.
If assignment and follow-up support a causal interpretation, these data estimate an increase in risk associated with treatment, not a reduction. The observed difference corresponds to 10 more events per 1,000 people in the treatment group over the stated follow-up. Whether an increase of this size is practically important depends on the seriousness of the event, potential benefits, uncertainty, and the decisions people face. The counts alone do not establish statistical significance or guarantee that the population risks differ.
Judging Practical Importance
Practical importance is a contextual judgment informed by evidence. For a clinical outcome, it can depend on how much change patients can notice, whether a change affects treatment decisions, possible harms and benefits, cost, and the time period over which outcomes are measured. In other fields, the relevant considerations may include safety, resources, policy consequences, or the scale of the outcome.
A useful approach is to state a smallest effect that would matter for the decision, when such a threshold can be justified, and to do so before interpreting results. Compare the estimate and its uncertainty with that threshold. If a mean difference is estimated as 0.5 points and 2 points is the agreed meaningful threshold, the estimate is below the threshold; but uncertainty may still leave open the possibility of a larger effect. If the estimate is 3 points, that exceeds the threshold, but uncertainty may still make the size imprecise. A test of zero and a judgment against a practical threshold are not identical questions.
Specify the groups, outcome, direction of subtraction, and target quantity, such as a mean difference or risk difference.
Give the estimated difference in its original units so readers can connect it to the outcome and decision.
State the reference standard deviation or other denominator and explain what the standardized effect represents.
Interpret statistical evidence under the model assumptions, then assess magnitude and uncertainty relative to the practical context.
Common Mistake: Treating a Small P-Value as a Large Effect
Wrong reasoning: “The p-value is very small, so the treatment effect must be important.” A p-value measures how unusual the observed test statistic, or a more extreme one, would be under the null model. It is not a measure of effect magnitude or practical value. Precision and sample size affect how readily a difference can be detected.
Correction: Report the estimated effect in meaningful units, add a standardized effect size if it helps answer the question, and judge the result against relevant practical criteria. Also consider uncertainty and study design. Statistical significance does not prove causation, and practical importance cannot be read from a p-value alone.
Check Your Understanding
Answer each question using the distinction between effect magnitude, uncertainty, and practical meaning.
- A study estimates a mean difference of 1.2 points, with a very small p-value. What additional information would you need before calling the difference practically important?
- Two settings have the same 5-unit mean difference, but their standard deviations are 5 and 25. Calculate each standardized difference using the stated standard deviation.
- In a risk comparison, \(p_1=0.03\) and \(p_0=0.02\). State the risk difference in percentage points and identify which group has the higher observed risk.
- Why does multiplying every measurement by a positive conversion factor change the raw mean difference but not the standardized mean difference?
- Explain why a small p-value does not, by itself, show that an effect is large or practically important.