Introduction
A clinical study rarely answers only the question “Is there evidence of a difference?” Investigators also need to know how large the difference is and whether it is clinically meaningful.
That distinction is the purpose of an effect size. An effect size summarizes the magnitude of a treatment difference, association, or change using a quantity that can be interpreted independently of the sample size.
For example, a very large randomized trial may produce a tiny treatment difference with a very small P-value, while a small study may show a potentially important difference without achieving conventional statistical significance. Reporting the effect estimate together with its confidence interval therefore gives a much richer description of the evidence than reporting a P-value alone.
What Is an Effect Size?
An effect size is a numerical measure describing the magnitude of a treatment effect, difference, association, or change. The appropriate measure depends on the endpoint and the scientific question.
For a binary endpoint, the effect can be expressed as an absolute difference in event probabilities or as a relative measure such as a risk ratio or odds ratio. For a continuous endpoint, a raw mean difference may be most informative, while a standardized mean difference can be useful when studies use different measurement scales. For time-to-event outcomes, the hazard ratio is commonly used.
| Endpoint type | Common effect measures | Typical interpretation |
|---|---|---|
| Binary | Risk difference, relative risk, odds ratio, NNT | Absolute or relative change in event probability |
| Continuous | Mean difference, standardized mean difference | Difference in outcome level |
| Time-to-event | Hazard ratio, restricted mean survival time difference | Relative event rate or difference in survival time |
| Repeated/longitudinal | Mean change, treatment difference, model-based contrasts | Difference in change or trajectory |
Effect Size vs. P-Value
A P-value is not an effect size. It depends strongly on sample size, variability, and the statistical model. The same underlying effect can produce different P-values in different studies.
Consider two studies that estimate the same treatment difference. A study with a very large sample may have enough precision to produce a small P-value even when the effect is clinically trivial. Conversely, a clinically important effect may have a wide confidence interval in a small study.
| Quantity | Main question |
|---|---|
| Effect estimate | How large is the observed treatment effect? |
| Confidence interval | How precise is the estimate? |
| P-value | How incompatible are the data with the specified null hypothesis? |
Binary Endpoints: A 2×2 Framework
Suppose a randomized clinical trial compares an experimental treatment with control. Let \(p_T\) be the event probability in the treatment group and \(p_C\) the event probability in the control group.
| Event | No event | |
|---|---|---|
| Treatment | a | b |
| Control | c | d |
The observed event risks are:
Most binary effect measures are transformations of these two risks.
Risk Difference
The risk difference (RD), also called the absolute risk difference, is the difference between the event probabilities in the two groups.
If the event is undesirable, a negative risk difference favors treatment. If the event is desirable, a positive risk difference favors treatment.
For example, suppose the event rate is 20% under control and 12% under treatment. Then:
The treatment reduces absolute event risk by 8 percentage points. This is often highly intuitive for clinicians because it directly describes the change in probability.
Relative Risk
The relative risk (RR), or risk ratio, compares the event probabilities multiplicatively.
Using the same example:
Thus, the estimated risk under treatment is 60% of the risk under control. Equivalently, the relative risk reduction is:
or 40%.
The absolute risk reduction is 8 percentage points, whereas the relative risk reduction is 40%. Both statements are correct, but they emphasize different aspects of the same treatment effect.
Why Absolute and Relative Effects Should Both Be Considered
Relative effects can appear impressive when baseline risk is low. The absolute effect may nevertheless be modest.
| Control risk | Treatment risk | RR | Absolute reduction |
|---|---|---|---|
| 20% | 12% | 0.60 | 8 percentage points |
| 2% | 1.2% | 0.60 | 0.8 percentage points |
The relative effect is identical in the two examples, but the absolute clinical impact is very different. This is why clinical interpretation should consider baseline risk rather than relying on a relative measure alone.
Number Needed to Treat
For a beneficial treatment that reduces an undesirable event, the number needed to treat (NNT) is based on the absolute risk reduction.
With an absolute risk reduction of 0.08:
Because NNT is conventionally reported as a whole number, this would be reported as approximately 13 patients, depending on the prespecified reporting convention.
Number Needed to Harm
The same concept can describe adverse effects. When treatment increases the risk of an undesirable event, the number needed to harm (NNH) can be defined using the absolute risk increase.
For example, if a treatment increases a particular adverse event from 4% to 7%:
So approximately 34 treated patients would correspond to one additional event on this absolute-risk scale, subject to the same assumptions and time horizon.
Odds Ratio
The odds ratio (OR) compares odds rather than probabilities. The odds of an event are the event probability divided by the probability of no event.
Therefore:
For the 12% versus 20% example:
An OR below 1 indicates lower odds under treatment when the endpoint is undesirable. However, an odds ratio of 0.545 should not automatically be described as a 45.5% reduction in risk. Odds and risks are different quantities.
Risk Ratio vs. Odds Ratio
When events are rare, the odds ratio and risk ratio may be numerically similar. As event rates become larger, they can diverge substantially.
| Measure | Uses | Key caution |
|---|---|---|
| Risk ratio | Direct comparison of probabilities | Requires meaningful risk denominators |
| Odds ratio | Logistic regression, case-control studies, some meta-analyses | Can exaggerate the apparent magnitude when events are common |
A Worked Binary Endpoint Example
Suppose a Phase III trial evaluates a treatment for preventing a clinical event. The results are:
| Group | Patients | Events | Risk |
|---|---|---|---|
| Treatment | 500 | 60 | 12.0% |
| Control | 500 | 100 | 20.0% |
The most clinically transparent summary might therefore be: the event risk was 8 percentage points lower with treatment, corresponding to an estimated 40% relative risk reduction and an NNT of approximately 13 over the study's specified time horizon.
Continuous Endpoints: Mean Difference
For a continuous outcome, the simplest effect size is often the mean difference (MD).
Suppose mean systolic blood pressure is 128 mmHg in the treatment group and 135 mmHg in the control group:
The treatment difference is therefore 7 mmHg in the direction of lower blood pressure. The advantage of the raw mean difference is that it retains the original clinical unit.
Standardized Mean Difference
A standardized mean difference (SMD) expresses a mean difference relative to a measure of variability. A common form is Cohen's \(d\).
where \(s_p\) is a pooled standard deviation. For equal or approximately equal variances, a pooled standard deviation can be written as:
For small samples, a bias-corrected standardized mean difference such as Hedges' \(g\) is often preferred.
The SMD is particularly useful when studies assess the same underlying construct using different scales. However, its interpretation is less directly clinical than a raw mean difference because the result is expressed in standard deviation units.
Effect Size for Change From Baseline
Clinical trials frequently measure an outcome at baseline and follow-up. An effect can be expressed as the difference in mean change between treatment groups.
The between-group difference in change is then:
In randomized trials, the analysis should be aligned with the prespecified statistical model and estimand. A simple change-score calculation is not always the most efficient or appropriate analysis, particularly when baseline values are strongly related to follow-up values.
Time-to-Event Outcomes: Hazard Ratio
When the endpoint is time to an event, investigators commonly report a hazard ratio (HR), often from a Cox proportional hazards model.
where \(h_T(t)\) and \(h_C(t)\) are the treatment and control hazard functions at time \(t\).
An HR below 1 indicates a lower instantaneous event rate under treatment, under the model's interpretation. For example, an HR of 0.75 is commonly interpreted as a 25% lower hazard under treatment relative to control, assuming the proportional hazards interpretation is appropriate.
Hazard Ratio vs. Risk Ratio
| Feature | Risk Ratio | Hazard Ratio |
|---|---|---|
| Primary quantity | Cumulative event probability | Instantaneous event rate |
| Time dependence | Defined over a specified time horizon | Uses time-to-event information |
| Typical model | Binomial/log-binomial or related models | Cox proportional hazards model |
| Interpretation | Relative cumulative risk | Relative hazard |
For survival outcomes, it can be useful to report the hazard ratio together with Kaplan-Meier estimates at clinically relevant time points, absolute risk differences at specified times, or restricted mean survival time differences.
Restricted Mean Survival Time Difference
The restricted mean survival time (RMST) is the expected event-free survival time up to a prespecified time horizon \(\tau\). The treatment effect can be summarized as a difference in RMST.
The between-group contrast is:
Unlike a hazard ratio, the RMST difference is expressed directly in units of time. For example, a difference of 1.8 months means that, on average, patients in the treatment group experienced 1.8 additional months of event-free survival within the specified horizon.
Relative Measures Can Depend on Baseline Risk
An important principle in clinical interpretation is that the same relative effect can produce very different absolute effects depending on baseline risk.
If the treatment multiplies risk by a constant \(RR\), then:
and the absolute risk difference is:
Thus, even a strong relative treatment effect can correspond to a small absolute benefit when baseline risk is very low.
Confidence Intervals for Effect Sizes
An effect estimate should generally be accompanied by a confidence interval. The interval communicates uncertainty and helps the reader distinguish precise estimates from imprecise ones.
For a risk difference, the null value is 0:
For ratios such as RR, OR, and HR, the null value is 1:
| Effect measure | Null value | Interpretation of the null |
|---|---|---|
| Risk difference | 0 | No absolute difference |
| Mean difference | 0 | No difference in means |
| Risk ratio | 1 | Equal risks |
| Odds ratio | 1 | Equal odds |
| Hazard ratio | 1 | Equal hazards under the model |
Statistical Significance vs. Clinical Significance
Suppose a large trial estimates a treatment difference of 0.4 units with a very narrow confidence interval. The result may be statistically persuasive but clinically unimportant if a difference of at least 3 units is needed to matter to patients.
Conversely, an estimate of 5 units with a wide confidence interval may be clinically interesting even if the study does not achieve a conventional P-value threshold. The appropriate conclusion may be that the study is imprecise rather than that the treatment has no meaningful effect.
| Scenario | Effect estimate | Precision | Possible conclusion |
|---|---|---|---|
| Small effect, narrow CI | Close to zero | High | Precisely estimated but potentially trivial |
| Large effect, wide CI | Clinically important | Low | Potentially important but uncertain |
| Large effect, narrow CI | Clinically important | High | Strong evidence of a meaningful effect |
Effect Size and the Estimand
The effect measure should correspond to the clinical question being asked. In modern clinical trials, that means thinking in terms of the estimand: the population, treatment conditions, variable or endpoint, handling of intercurrent events, and summary measure that define the treatment effect.
For example, a treatment effect might be summarized as a risk difference at Week 52, a ratio of mean change at Week 12, a difference in RMST through Month 24, or a hazard ratio. These are not interchangeable summaries.
Effect Measures in Common Clinical Trial Settings
| Clinical setting | Useful primary effect measure | Additional measure worth considering |
|---|---|---|
| Binary efficacy endpoint | Risk difference or risk ratio | OR, NNT, event rates by arm |
| Common binary adverse event | Risk difference | RR or NNH |
| Continuous patient-reported outcome | Mean difference | SMD when scales differ |
| Time-to-event endpoint | HR when appropriate | RMST difference and fixed-time risks |
| Rare disease single-arm study | Observed response rate and CI | Absolute response benchmark |
Effect Size in Logistic Regression
In logistic regression, the regression coefficient is commonly interpreted through its exponentiation:
For a binary covariate, \(e^\beta\) is the estimated odds ratio comparing the specified exposure or treatment categories, conditional on the other variables in the model.
The adjusted odds ratio is therefore a model-based effect measure. It should not be assumed to equal an adjusted risk ratio or adjusted risk difference. When absolute risk is important, model-based predicted probabilities or marginal standardization can provide a more directly interpretable summary.
Effect Size in Cox Regression
In a Cox proportional hazards model:
For a one-unit treatment indicator contrast, the hazard ratio is:
Again, the model coefficient and its exponentiation summarize a particular model parameter. They do not directly provide an absolute treatment benefit at a fixed time point.
Effect Size for Subgroup Analyses
Subgroup analyses often report treatment-specific effect estimates within each subgroup. The key statistical question, however, is usually not whether one subgroup has a statistically significant result and another does not. It is whether the treatment effect differs between subgroups.
where \(\theta_1\) and \(\theta_2\) represent treatment effects in two subgroups. This is an interaction question.
Effect Size in Meta-Analysis
Meta-analysis requires a common effect measure or a valid transformation to a common scale. Binary outcomes may be summarized as risk ratios, odds ratios, or risk differences. Continuous outcomes may use mean differences when scales are common or standardized mean differences when they differ.
For ratio measures, meta-analysis is often performed on the logarithmic scale:
The pooled estimate is then transformed back to the original scale for interpretation.
Absolute Effects Are Especially Important for Clinical Decisions
A relative measure is useful for describing transportable treatment effects in some settings, but clinical decisions often require absolute effects.
Suppose a treatment has an RR of 0.80. If baseline risk is 50%, the corresponding risk is approximately 40%, an absolute reduction of 10 percentage points. If baseline risk is 5%, the corresponding risk is approximately 4%, an absolute reduction of only 1 percentage point.
| Baseline risk | RR | Approximate treatment risk | Absolute reduction |
|---|---|---|---|
| 50% | 0.80 | 40% | 10 percentage points |
| 20% | 0.80 | 16% | 4 percentage points |
| 5% | 0.80 | 4% | 1 percentage point |
This is one reason clinical trial reports should generally provide event rates in each treatment group in addition to relative effect measures.
How to Choose the Right Effect Measure
A practical selection process is:
Common Mistakes
- Reporting only the P-value. A P-value does not tell the reader how large the treatment effect is.
- Calling an odds ratio a risk ratio. Odds and risks are different quantities, especially when events are common.
- Reporting only relative effects. Relative measures can obscure the absolute magnitude of benefit or harm.
- Ignoring baseline risk. The same relative effect can produce very different absolute effects in different populations.
- Interpreting an HR as a fixed-time risk ratio. A hazard ratio is not generally equivalent to a ratio of cumulative event risks.
- Using SMD when the raw scale is clinically meaningful. Standardization can improve comparability across scales but can make clinical interpretation less direct.
- Ignoring the time horizon. Risk difference, NNT, and other absolute measures depend on when the outcome is assessed.
- Confusing subgroup significance with interaction. Different P-values within subgroups do not by themselves prove different treatment effects.
- Failing to report uncertainty. An effect estimate without a confidence interval can be difficult to interpret.
A Practical Reporting Example
For a binary clinical endpoint, a strong results statement might report the observed event rates, absolute effect, relative effect, and confidence interval rather than only a significance test.
| Component | Example report |
|---|---|
| Treatment event rate | 12.0% |
| Control event rate | 20.0% |
| Risk difference | −8.0 percentage points |
| Relative risk | 0.60 |
| Relative risk reduction | 40% |
| NNT | Approximately 13 |
In an actual clinical trial report, the effect estimates should be accompanied by their appropriate confidence intervals and clearly labeled with the analysis population and time horizon.
Effect Size and Clinical Importance
The interpretation of an effect size ultimately depends on context. A statistically well-estimated effect is not automatically clinically important.
Clinical interpretation can incorporate:
- Magnitude of benefit
- Magnitude of harm
- Baseline event risk
- Duration of treatment and follow-up
- Patient preferences
- Severity of the outcome
- Burden, cost, and feasibility of treatment
- Prespecified thresholds for clinical importance
Effect Size Does Not Eliminate Study Bias
A precise effect estimate can still be biased if the study design or analysis is biased. Effect-size reporting does not replace careful consideration of randomization, missing data, confounding, measurement error, multiplicity, endpoint definition, and analysis populations.
In randomized clinical trials, the treatment effect is protected by the design and analysis assumptions rather than by the effect-size formula itself. In observational studies, an adjusted effect estimate may still reflect residual confounding.
Summary of Major Effect Measures
| Measure | Formula / basis | Best interpreted as |
|---|---|---|
| Risk difference | \(p_T-p_C\) | Absolute change in event probability |
| Risk ratio | \(p_T/p_C\) | Relative change in risk |
| Odds ratio | \(\frac{p_T/(1-p_T)}{p_C/(1-p_C)}\) | Relative change in odds |
| NNT | \(1/|RD|\) | Patients needed for one additional beneficial event over a specified horizon |
| Mean difference | \(\bar X_T-\bar X_C\) | Difference in original outcome units |
| SMD | Mean difference / SD | Difference in standard deviation units |
| Hazard ratio | \(h_T(t)/h_C(t)\) | Relative hazard under the model |
| RMST difference | \(RMST_T(\tau)-RMST_C(\tau)\) | Difference in mean event-free survival time through \(\tau\) |
Worked Example: Putting the Measures Together
Consider a randomized trial with 1,000 participants, 500 per arm. Suppose the primary undesirable event occurs in 60 treatment patients and 100 control patients. The core effect measures are:
These numbers should not be interpreted as competing answers. They describe the same observed treatment comparison on different scales.
Final Takeaways
Effect size is central to clinical interpretation because it quantifies the magnitude of a treatment difference or association. The correct effect measure depends on the endpoint and the scientific question.
- Use risk difference when absolute changes in event probability are important.
- Use risk ratio to describe relative cumulative risk.
- Use odds ratio when odds are the natural model parameter, while remembering that OR is not RR.
- Use NNT or NNH to translate an absolute risk difference into a clinically intuitive quantity.
- Use mean difference when the original measurement scale is clinically meaningful.
- Use standardized mean difference when comparison across different measurement scales is required.
- Use hazard ratio for an appropriate time-to-event model, but do not interpret it as a risk ratio.
- Consider RMST differences and fixed-time absolute risks as complementary summaries for survival outcomes.
- Always consider confidence intervals, baseline risk, time horizon, and clinical importance.
References
Cummings, P. (2009).
The relative merits of risk ratios and odds ratios.
Archives of Pediatrics & Adolescent Medicine, 163(5), 438–445.
Cochrane Handbook for Systematic Reviews of Interventions.
Effect measures for dichotomous and continuous outcomes.
Altman, D.G. & Bland, J.M. (1998).
How to calculate confidence intervals for a risk ratio.
BMJ, 316, 291.
Sullivan, G.M. & Feinn, R. (2012).
Using effect size—or why the P value is not enough.
Journal of Graduate Medical Education, 4(3), 279–282.
Lakens, D. (2013).
Calculating and reporting effect sizes to facilitate cumulative science.
Frontiers in Psychology, 4, 863.
Royston, P. & Parmar, M.K.B. (2013).
Restricted mean survival time: an alternative to the hazard ratio for the
design and analysis of randomized trials with a time-to-event outcome.
BMC Medical Research Methodology, 13, 152.