Why Reporting Choices Matter
An effect size and a power value can look precise while inviting an interpretation they do not support. An eta-squared value is not a direct measure of a treatment’s practical or causal importance. Two effect sizes with the same numerical value may summarize different fractions of variation. And power calculated after observing the results does not provide independent evidence that a study’s result is reliable.
These distinctions matter because readers use reported numbers to compare findings, assess practical relevance, and plan future studies. A sound report identifies exactly which effect-size measure is being used, what its denominator represents, and how the design and analysis affect the power calculation.
The Model and the Quantities Being Reported
For a one-way ANOVA with \(k\) groups, write the observation in group \(i\) as \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where \(\mu_i\) is the mean for group \(i\). The usual model assumes that the errors are independent, normally distributed with mean zero, and have a common variance \(\sigma^2\). These assumptions support the usual ANOVA calculations; they do not, by themselves, establish that group differences are causal.
Let \(SS_{\text{between}}\) be the between-groups sum of squares, \(SS_{\text{error}}\) the within-groups sum of squares, and \(SS_{\text{total}}=SS_{\text{between}}+SS_{\text{error}}\) the total sum of squares. In a one-way ANOVA, eta-squared and partial eta-squared have the same value:
In models with additional effects, partial eta-squared uses the effect’s sum of squares plus its error sum of squares in the denominator. It excludes other modeled effects. Consequently, eta-squared and partial eta-squared can differ in those settings, even when both are reported for the same effect.
Power is the probability that a specified test will reject \(H_0\) when a specified alternative is true. A planned power calculation therefore needs a chosen significance level \(\alpha\), a design (including group sizes), a model for variation, and an effect size under the alternative. Power is not a property of a test result alone.
Worked Example: Eta-Squared Is Not Causal Importance
Worked Example: Eta-Squared Is Not Causal Importance
Suppose a hypothetical study records scores in three groups. Each group has ten observations: group A has five scores of 8 and five of 12; group B has five scores of 10 and five of 14; group C has five scores of 12 and five of 16. The group means are 10, 12, and 14, and the grand mean is 12. Use the one-way ANOVA sums of squares to calculate eta-squared.
The between-groups sum of squares is the sum of each group size times the squared difference between its mean and the grand mean:
Within each group, five scores are two units below its mean and five are two units above it. Each group therefore contributes \(5(2^2)+5(2^2)=40\) to \(SS_{\text{error}}\). Across three groups, \(SS_{\text{error}}=3(40)=120\), so \(SS_{\text{total}}=80+120=200\). Thus:
The appropriate sample description is that 40.0% of the total variation in these observed scores is between the three group means in this one-way partition. It is not that the groups “caused 40% of the scores,” or that a treatment has 40% causal importance. Eta-squared summarizes a variation breakdown; it does not identify why the group means differ. A causal interpretation requires suitable study design and assumptions beyond the numerical value of eta-squared.
Worked Example: Check the Denominator Before Comparing Effects
Worked Example: Check the Denominator Before Comparing Effects
Consider a hypothetical analysis in an orthogonal design with an effect A, another modeled effect, and residual error. Suppose \(SS_A=40\), the other modeled effect contributes 40, and \(SS_{\text{error}}=120\), giving \(SS_{\text{total}}=200\). Compare eta-squared for A with partial eta-squared for A.
Eta-squared uses total variation in its denominator:
Partial eta-squared for A excludes the other modeled effect from its denominator:
The values differ because the denominators answer different descriptive questions. Eta-squared here expresses A’s sum of squares as a fraction of total variation; partial eta-squared expresses it as a fraction of A’s sum of squares plus its error sum of squares. Reporting “the effect size was 0.25” without naming the measure would conceal this distinction. Comparing 0.25 partial eta-squared from one analysis with 0.25 eta-squared from another does not show that the effects account for the same fraction of total variation.
How Allocation Changes What Power Means
A total sample size does not fully describe a one-way ANOVA design. The number assigned to each group affects the precision of comparisons, and therefore can affect power. The effect-size definition also matters: Cohen’s \(f\) for a one-way design uses group proportions to weight the group means around their weighted grand mean. Changing allocation can change those weights when group means differ.
For a planned comparison with coefficients \(c_i\) satisfying \(\sum_i c_i=0\), the variance of the estimated contrast under independent groups with common variance \(\sigma^2\) is \(\sigma^2\sum_i c_i^2/n_i\). Thus the allocation enters the precision directly. This formula is a useful diagnostic even when the final test uses an estimated error variance.
Worked Example: Same Total Sample, Different Precision
For planning, suppose three hypothetical group means are 0, 2, and 4 outcome units, the common standard deviation is 3 outcome units, and the comparison of interest is the difference between the third and first group. Compare balanced allocation \(n=(10,10,10)\) with unequal allocation \(n=(18,6,6)\). Both designs have 30 observations. The contrast coefficients are \((-1,0,1)\).
For the balanced design, the variance of the estimated difference is:
The standard error is \(\sqrt{1.800}=1.342\) outcome units. The planned difference is \(4-0=4\) units, so its squared signal relative to this variance is \(4^2/1.800=8.889\).
For the unequal design, the first and third groups have sizes 18 and 6; the middle group has coefficient zero for this particular contrast. The corresponding variance and squared signal are:
The contrast is less precise under this unequal allocation, despite the same total sample size and planned group means. This calculation does not establish that balanced allocation is always best for every research question: the comparison of interest and the costs or constraints of each group matter. It does show why a power statement should name the allocation, rather than reporting only the total \(n\) and an effect size.
Why Observed Post Hoc Power Is Not New Evidence
Planned power is calculated before collecting results, using an effect considered meaningful for the study’s purpose. “Observed” or post hoc power is often calculated after the analysis by substituting the estimated effect from the same data into a power formula. This does not independently confirm that the study had an adequate chance of detecting a meaningful effect. When the observed effect is used, the resulting power is closely tied to the observed test statistic and its p-value.
For a one-way ANOVA, Cohen’s \(f\) is related to eta-squared by \(f^2=\eta^2/(1-\eta^2)\). A common noncentrality parameter for a one-way omnibus test is \(\lambda=Nf^2\), where \(N\) is the total sample size. Using an observed eta-squared to calculate \(\lambda\) produces a post hoc power calculation based on the observed result, not an independent assessment.
Worked Example: Identify What an Observed-Power Calculation Uses
Suppose a hypothetical one-way ANOVA has \(k=3\) groups, \(N=30\), and reports \(F=4.000\) with \(df_1=2\) and \(df_2=27\). The corresponding eta-squared is:
The observed Cohen’s \(f^2\) and the noncentrality parameter obtained by substituting that observed effect are:
A post hoc power calculation would then evaluate the upper-tail probability for a noncentral \(F\) distribution with \(df_1=2\), \(df_2=27\), and \(\lambda=8.889\), above the chosen critical value. The calculation reuses the effect size derived from the observed \(F\); it does not supply a second, independent piece of evidence. In reporting this result, give the observed test result and an effect-size estimate, with suitable uncertainty where available, rather than using post hoc power to argue that the result is trustworthy or that a nonsignificant finding proves no effect.
A Short Justification: Why the Two Eta Measures Match in One-Way ANOVA
In a one-way ANOVA, there are only the between-groups and error components of the total sum of squares. Therefore:
\(SS_{\text{total}}=SS_{\text{between}}+SS_{\text{error}}\).
\(\eta^2=SS_{\text{between}}/SS_{\text{total}}=SS_{\text{between}}/(SS_{\text{between}}+SS_{\text{error}})\).
The final expression is the definition of partial eta-squared for the one-way effect, so \(\eta^2=\eta_p^2\) in this setting.
When other effects contribute to total variation, the total sum of squares also includes those components. Eta-squared retains them in its denominator, whereas partial eta-squared excludes them. The denominator difference, not a change in the observed effect’s numerator, explains why the measures can diverge.
Common Reporting Mistakes
- Calling eta-squared “causal importance.” This wrongly treats a variance partition as proof of causation. Report the measure descriptively; make causal claims only when the design and assumptions support them.
- Comparing effect sizes without naming the measure. A value based on total variation and one based on effect-plus-error variation need not mean the same thing. State eta-squared, partial eta-squared, omega-squared, or Cohen’s \(f\) explicitly, as appropriate.
- Reporting power without the design details. Power depends on group sizes and allocation, the effect specified, the variability assumption, the test, and \(\alpha\). A total sample size alone is not enough to reproduce or interpret it.
- Treating observed post hoc power as confirmation. Substituting the observed effect into a power formula does not independently validate a finding. Use a prespecified meaningful effect for planning; after analysis, describe the estimate and its uncertainty.
- Leaving out units or context. A standardized effect size has no outcome units, but a raw mean difference does. Name the outcome and its units when reporting differences in the original measurement scale.
Check Your Understanding
Try these questions before reviewing the explanations above.
- What does a one-way eta-squared value of 0.30 describe, and what does it not establish?
- Why might eta-squared and partial eta-squared differ for the same effect in a model with more than one effect?
- Two studies have the same total sample size but different group allocations. Give one reason their powers for a specified comparison may differ.
- What information is reused when observed post hoc power is calculated from the observed ANOVA effect?
- What details should accompany a planned power statement so that its meaning is clear?