Why the Experimental Unit Matters
A one-way ANOVA compares group means, but the number of recorded measurements is not necessarily the number of independent replicates. If a treatment is applied to a tank, plot, clinic, or specimen and several measurements are then taken within that same unit, those measurements share the conditions of their unit. Counting them as independent treatment replicates can make the evidence for a group difference look stronger than it is.
This problem is called pseudoreplication. The key question is not simply how many values appear in the data set. It is: what unit was independently assigned to a treatment? The assumptions of the usual one-way fixed-effects F test include independent experimental errors. Repeated observations within an experimental unit generally do not satisfy that assumption.
For example, if a hypothetical feed is assigned to each entire tank, the tank is the experimental unit, even if many fish are weighed in each tank. If the feed is instead assigned independently to each fish, each fish may be an experimental unit. The treatment assignment and sampling design determine the answer, not the number of measurements.
What the Usual F Test Assumes
In a one-way fixed-effects model, an observation from group \(i\) is often represented as \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where \(\mu_i\) is the group mean and the errors are independent, have mean zero, and share a common variance. The usual F test uses within-group variation to estimate the error variance. If several observations share a unit-specific influence, their errors are correlated, and the ordinary calculation treats more independent information as available than the design actually provides.
The resulting F statistic can be too large, and the residual degrees of freedom can be too high. The nominal F distribution then need not describe the statistic’s actual null distribution, so the reported p-value may be too small. Positive within-unit correlation often causes this anti-conservative behavior, but the exact effect depends on the design and correlation pattern; pseudoreplication does not guarantee that every calculated F statistic increases.
Why Correlated Measurements Carry Less Information
A simple calculation shows why repeated measurements cannot generally be counted as independent. Suppose a single experimental unit produces \(m\) measurements, each with variance \(\sigma^2\), and every pair has correlation \(\rho\). The variance of their average is obtained by adding the variances and covariances:
If the measurements were independent, their average would have variance \(\sigma^2/m\). With positive correlation, the variance is larger by the factor \(1+(m-1)\rho\), often called the design effect for this equal-correlation setting. Thus, adding measurements within a unit gives less precision than adding the same number of independent units.
This calculation concerns the precision of an average; it does not create a general correction that makes every ordinary ANOVA valid. The treatment was still assigned to the units, so inference must respect the number of independently assigned units. For a simple one-way design with equal numbers of measurements per unit, analyzing the unit averages is one direct approach. A model that explicitly represents unit-level dependence may be appropriate in more complex designs.
Worked Example: Comparing Two Analyses of Repeated Specimen Readings
In a hypothetical laboratory study, a surface treatment is assigned to each specimen. Three specimens receive Treatment A and three receive Treatment B. Two readings are taken from each specimen. The readings are A: \(9,11\); \(11,13\); \(13,15\), and B: \(13,15\); \(15,17\); \(17,19\). Compare the one-way ANOVA based on specimen averages with the calculation that incorrectly treats all readings as independent. Name the method and identify the mistake avoided.
Method. First use each specimen as one experimental unit and analyze its average response. Then calculate the observation-level ANOVA to show what changes when the two readings per specimen are incorrectly counted as independent.
The specimen averages are \(10,12,14\) for A and \(14,16,18\) for B. The group means are \(\bar{Y}_A=12\) and \(\bar{Y}_B=16\), and the overall mean of the six specimen averages is \(14\).
For the unit-level ANOVA, the treatment sum of squares is
The within-group sum of squares among specimen averages is
There are \(6\) experimental units and \(2\) groups, so the treatment and error degrees of freedom are \(2-1=1\) and \(6-2=4\). Therefore \(MS_{\text{treatment}}=24/1=24\), \(MSE=16/4=4\), and the unit-level statistic is \(F=24/4=6\), with degrees of freedom \(1\) and \(4\).
For the incorrect observation-level calculation, each group has six readings. The group means remain \(12\) and \(16\), and the overall mean remains \(14\). The treatment sum of squares is now \(6(12-14)^2+6(16-14)^2=24+24=48\). Within either treatment group, the deviations from its group mean are \(-3,-1,-1,1,1,3\), whose squared values sum to \(9+1+1+1+1+9=22\). Thus the observation-level error sum of squares is \(22+22=44\).
The incorrect residual degrees of freedom are \(12-2=10\), giving \(MSE=44/10=4.4\). The resulting statistic is \(F=48/4.4=10.9091\), with degrees of freedom \(1\) and \(10\). The group means have not changed, but the analysis has counted repeated readings as independent units, altering both the numerator and the reference distribution. In this example the F statistic is larger and the residual degrees of freedom increase from \(4\) to \(10\).
Check. The experimental-unit analysis uses six specimen averages, and its sums of squares satisfy \(24+16=40\), the total sum of squares of those averages. The mistake avoided is treating two readings on one specimen as two independent treatment replicates.
Reading the Design Before Calculating
Before constructing an ANOVA table, establish how treatments were assigned and what was measured afterward. The following sequence helps separate true replication from repeated measurement.
Identify the smallest object, person, or group independently assigned to a treatment. That is the experimental unit.
Use the number of independently assigned units to assess replication. Do not substitute the count of subsamples for this count.
Determine whether a value comes from a different experimental unit or is another reading from an existing unit.
For a simple design with equal subsampling, unit averages can be analyzed as the responses. Otherwise, use a method that accounts for clustering or repeated measurements.
Worked Example: How Correlation Changes Precision
A hypothetical study assigns a program to each of six centers in each of two groups. Four clients are measured in every center. Suppose the variance of an individual measurement is \(15\) and the within-center correlation is \(0.60\). Calculate the design effect and compare the variance of a group average under the correlated and independent calculations. State why the count of clients is not the replication count.
Method. Use the equal-correlation variance formula for four observations within a center, then compare the variance of a group average across six centers with the calculation that treats all \(24\) client measurements as independent. The centers, rather than clients, were independently assigned.
Here \(m=4\) and \(\rho=0.60\). The design effect is
For one center, the variance of the average of its four client measurements is
A group has six independent centers, so its average of the six center averages has variance \(10.5/6=1.75\). If all \(24\) clients were incorrectly treated as independent observations with variance \(15\), the calculated variance of their average would be \(15/24=0.625\). The ratio is \(1.75/0.625=2.8\), matching the design effect.
The count of \(24\) clients per group describes measurements, not independent program assignments. There are six independent centers per group. The design effect quantifies the variance understatement in this particular equal-correlation comparison; it is not a replacement for analyzing the experimental-unit structure.
Check. The correlated variance is larger than the independent calculation, as it must be for positive within-center correlation. The mistake avoided is using the client count as if each client independently received the treatment.
A Full One-Way Illustration With Subsamples
When every experimental unit contributes the same number of measurements, the ordinary observation-level treatment sum of squares is multiplied by that number relative to the analysis of unit averages. The residual sum of squares also includes variation among measurements within a unit. Those arithmetic facts do not make the observation-level F test valid: its degrees of freedom still presume independent experimental units.
Worked Example: Seedlings Measured Within Treated Trays
In a hypothetical greenhouse experiment, one of three growing conditions is assigned to each of five trays per condition. Four seedlings are measured in each tray, so there are \(20\) measurements per condition. The five tray averages are \(4,5,6,7,8\) cm for Condition A, \(6,7,8,9,10\) cm for B, and \(8,9,10,11,12\) cm for C. Within each tray, the four measurements are its average minus \(1\), its average, its average, and its average plus \(1\) cm. Compare the correct tray-average ANOVA with the result of treating all seedlings as independent. State the assignment unit and the error degrees of freedom for the valid analysis.
Method. Because the growing condition is assigned to a tray, analyze the \(15\) tray averages for the treatment comparison. Then calculate the observation-level sums of squares to expose the effect of counting the four seedlings in each tray as independent replicates.
The condition means are \(6,8,10\) cm, and the overall mean of the \(15\) tray averages is \(8\) cm. The treatment sum of squares based on tray averages is
Within each condition, the tray averages differ from their condition mean by \(-2,-1,0,1,2\), whose squared deviations sum to \(4+1+0+1+4=10\). Across three conditions, \(SS_{\text{error}}=3(10)=30\). The degrees of freedom are \(2\) for treatment and \(15-3=12\) for error. Thus \(MS_{\text{treatment}}=40/2=20\), \(MSE=30/12=2.5\), and \(F=20/2.5=8\), with degrees of freedom \(2\) and \(12\).
For the observation-level calculation, every tray average is repeated four times, so the treatment sum of squares is \(4(40)=160\). Within one tray, the four readings have deviations \(-1,0,0,1\) from the tray average; their squared deviations sum to \(1+0+0+1=2\). There are \(15\) trays, so the within-tray sum of squares is \(15(2)=30\). The replicated tray-average deviations contribute \(4(30)=120\) to the observation-level error sum of squares. Therefore that error sum of squares is \(120+30=150\).
The observation-level error degrees of freedom are \(60-3=57\), so its \(MSE=150/57=2.6316\). The resulting statistic is \(F=160/2.6316=60.8\), with degrees of freedom \(2\) and \(57\). This calculation treats \(60\) seedlings as independent treatment replicates, although the treatment was assigned to only \(15\) trays. The valid tray-average analysis has error degrees of freedom \(12\), not \(57\).
Check. For the tray-average analysis, the total sum of squares is \(40+30=70\). For the observation-level analysis, it is \(160+150=310\). The mistake avoided is assigning treatment-level replication to individual seedlings when the tray received the treatment.
Common Mistakes and Practical Consequences
- Counting measurements instead of assignments. Ten readings from one treated plot do not provide ten independent treatment replicates. Count the independently treated plots.
- Assuming a large sample count fixes dependence. More subsamples may estimate a unit’s average more precisely, but they do not create more independently assigned units.
- Using the usual residual degrees of freedom without checking the design. The ordinary formula \(N-k\) is appropriate for independent experimental units under the one-way model. Substituting the total number of clustered measurements for \(N\) can inflate the degrees of freedom.
- Assuming every repeated measurement must be discarded. Subsamples can be useful. For a simple balanced design, unit averages retain the unit-level treatment information; other valid methods can model the dependence directly.
- Claiming the F statistic must always be inflated. Positive correlation often leads to an underestimated standard error and an anti-conservative test, but the consequences depend on the design. The dependable conclusion is that the ordinary independent-observation F reference is not justified when its independence assumption fails.
Check Your Understanding
For each problem, focus first on the assignment unit, then determine what the observation count does and does not tell you.
- A coating is independently assigned to each of eight metal panels in each of two groups. A technician records three thickness readings per panel. How many experimental units are in each group, and are the 24 readings in a group 24 independent treatment replicates? (Answer: 8 experimental units per group; no, the three readings within each panel are subsamples.)
- A hypothetical study assigns a fertilizer to each of seven tanks per group and measures six plants in each tank. If the within-tank correlation is \(0.5\), calculate the equal-correlation design effect for a tank’s six measurements. (Answer: \(1+(6-1)(0.5)=3.5\).)
- Three treatments are independently assigned to four plots each, and five plants are measured per plot. What are the treatment and error degrees of freedom for a one-way ANOVA on plot averages? What error degrees of freedom would the ordinary plant-level calculation use? (Answer: Plot-average analysis: treatment \(2\), error \(12-3=9\); plant-level calculation: error \(60-3=57\), which does not reflect the assignment units.)
- There are five independently assigned centers per group and four observations per center. Each observation has variance \(20\), and every within-center pair has correlation \(0.25\). Find the variance of a center average and the variance of the average across the five centers. (Answer: Design effect \(1+3(0.25)=1.75\); center-average variance \((20/4)(1.75)=8.75\); group-average variance \(8.75/5=1.75\).)
- A program is assigned separately to each participant, with no shared treatment assignment by household. Three outcomes are recorded for each participant over time. Does the participant or each recorded outcome define the independent treatment replicate, and can an ordinary one-way ANOVA automatically treat all outcomes as independent? (Answer: The participant is the assignment unit; no, repeated outcomes from that participant are not automatically independent replicates.)