Why the Unit Assigned to Treatment Matters
An experiment may measure outcomes on many people but assign treatment only a few times. For example, a program may be assigned to classrooms, while quiz scores are recorded for every student. The number of student scores is not automatically the number of independent treatment replicates. To count replication correctly, first ask: what was actually assigned at random?
When whole classrooms, clinics, or households receive the same treatment, they are clusters. People or other observations within a cluster provide information about that cluster's outcome, but they do not represent separate treatment assignments. Confusing these two counts can make an experiment appear to have far more independent evidence than it does.
A Model for Counting Cluster Replicates
Suppose there are \(a\) treatment conditions. Let \(J_i\) be the number of clusters assigned to condition \(i\), and let \(Y_{ij}\) be the outcome recorded for cluster \(j\) in condition \(i\). The outcome could be a cluster-level measurement, such as clinic waiting time, or a summary of individual measurements, such as the mean student score in a classroom. A simple model for these cluster outcomes is
Here, \(\mu+\tau_i\) is the mean cluster outcome for condition \(i\), and \(\varepsilon_{ij}\) is the deviation of cluster \(j\)'s outcome from that mean. For the usual simple cluster-level analysis, assume that cluster outcomes have finite variance and that errors from different clusters are independent, with a common variance if the analysis uses the equal-variance ANOVA model. These are assumptions about the cluster outcomes; they do not assert that people within the same cluster are independent.
Under this setup, the number of treatment replicates in condition \(i\) is \(J_i\). If the clusters contain \(m_{ij}\) measured people each, the total number of people measured in that condition is \(\sum_{j=1}^{J_i}m_{ij}\). Those are different quantities. Increasing the number of people measured within a cluster can improve the precision of that cluster's measured outcome, but it does not increase \(J_i\).
Worked Example: Classrooms Assigned to Two Programs
A hypothetical school district randomly assigns three classrooms to a new reading program and four classrooms to the standard program. Each classroom has 24 students, and the outcome is each student's end-of-term reading score. The classroom mean scores are 72, 75, and 78 for the new program, and 68, 70, 73, and 75 for the standard program. What is the treatment replicate count, and what is the difference between the two average classroom outcomes?
Count the assignments. Classrooms, not students, were randomized. There are three treatment replicates for the new program and four for the standard program. The district measures \(3(24)=72\) students in the new-program classrooms and \(4(24)=96\) in the standard-program classrooms, but those totals are not the cluster replicate counts.
Calculate the average classroom outcomes. Treating each classroom as one cluster outcome, the average for the new program is
The average for the standard program is
The difference between the average classroom outcomes is \(75-71.5=3.5\) score points, with the new-program classrooms higher in this hypothetical example. The comparison is based on three classroom outcomes versus four, not 72 student scores versus 96. The example describes a difference; it does not by itself establish how precisely that difference has been estimated.
Counting Clusters When Cluster Sizes Differ
Cluster sizes do not have to be equal for the assignment count to be clear. A clinic with many measured patients is still one treatment-assigned clinic, just as a smaller clinic assigned to the same condition is one treatment-assigned clinic. However, unequal sizes can affect how a summary is calculated. An unweighted mean of clinic means gives each clinic equal weight; a pooled mean of patient outcomes gives larger clinics more weight. Those summaries answer different questions, so the analysis should specify which outcome or average it is describing.
Worked Example: Clinics With Different Patient Counts
In a hypothetical trial, three clinics are assigned to a new procedure and three to usual care. The average recovery times (in days) are 4, 5, and 6 for the new-procedure clinics, and 3, 4, and 5 for the usual-care clinics. The new-procedure clinics have 12, 18, and 30 measured patients, while the usual-care clinics have 30, 18, and 12. Identify the replicate counts and compare the unweighted average clinic outcomes.
Count the randomized units. Each clinic received one assignment. Thus, each condition has three treatment replicates. There are 60 patients in each condition, but neither condition has 60 independent treatment assignments.
Calculate the unweighted clinic averages. If the outcome for this comparison is the average clinic recovery time, each clinic contributes one value:
The difference in these average clinic outcomes is \(5.00-4.00=1.00\) day. The replicate count remains three per condition. Weighting clinic outcomes by patient count would produce a different summary, because the larger clinics would contribute more; it would not turn the 60 patients in a condition into 60 independently randomized clusters.
Why People in One Cluster Are Not Separate Treatment Replicates
People in the same classroom, clinic, or household often share conditions. Students share a teacher and classroom; patients share clinic staff and procedures; household members share an environment. Their outcomes may therefore be more alike than outcomes from people in different clusters. More importantly for the replicate count, all members of a cluster received the same assigned condition. The treatment was not independently assigned to each person.
A short variance calculation shows why simply counting every person as an independent replicate can overstate the information in the data. Suppose a cluster contains \(m\) people, each outcome has variance \(\sigma^2\), and the correlation between any two different people in that cluster is \(\rho\). The variance of the cluster mean is
The first term inside the brackets comes from the \(m\) individual variances; the second comes from the \(m(m-1)\) covariances between distinct people. If \(\rho=0\), the variance reduces to \(\sigma^2/m\), as it would for independent observations. If \(\rho>0\), the cluster mean has greater variance than that independent-observation calculation suggests. This calculation does not change the assignment count: the cluster was still assigned once. It explains why a standard error that treats all people within a cluster as independent can be too small.
Worked Example: Households Assigned to a Health Message
A hypothetical study randomly assigns five households to receive a health message and six households to receive a comparison message. Four adults in each household complete a questionnaire. What is the number of treatment replicates in each condition, and what would be wrong with reporting the adult counts as the replicate counts?
Count the assignments. The assignment unit is the household. There are five household replicates receiving the health message and six receiving the comparison message.
Count the measurements separately. There are \(5(4)=20\) adults measured in the health-message condition and \(6(4)=24\) in the comparison condition. These are the numbers of adult questionnaires, not the numbers of independent treatment assignments.
Calling the counts 20 and 24 treatment replicates would imply that treatment had been assigned independently to 44 adults. It was instead assigned to 11 households. Household members may also share circumstances that make their questionnaire responses related. A comparison that ignores household clustering can therefore use an inappropriate error calculation as well as the wrong replicate count.
When There Are Too Few Clusters
A large number of individual measurements cannot compensate for having almost no replication at the assignment level. Consider a study that assigns one clinic to a new treatment and one clinic to usual care, then measures 100 patients at each clinic. The treatment replicate count is one clinic per condition, not 100 patients per condition. A difference between the two clinic outcomes could reflect the treatments, differences between the two clinics, or both. With only one cluster in each condition, ordinary cluster-level ANOVA has no within-condition cluster replication from which to estimate between-cluster error.
This is a design issue, not just a matter of choosing a different way to enter the data. Before collecting a large number of measurements within clusters, check that there are enough treatment-assigned clusters for the intended comparison. Adding clusters generally adds treatment-level replication; adding people only within existing clusters adds measurements within those clusters.
Identify the smallest unit that could receive a different treatment assignment from another unit in the design. If whole classrooms were assigned together, the unit is the classroom.
Count how many clusters were assigned to each treatment. Those counts are the treatment replicate counts.
Record how many people or observations were measured within each cluster, but do not substitute those totals for the cluster counts.
Use cluster outcomes or an analysis that accounts for clustering. Do not calculate a standard error as if all people were independently assigned when assignment occurred by cluster.
Common Mistake
Wrong reasoning: “There are 240 patients in each arm, so the experiment has 240 treatment replicates per arm.” This counts measurements as if each patient received an independent assignment. If 240 patients came from 12 clinics, and clinics were assigned to treatment, there are 12 treatment-assigned clusters in that arm, not 240.
Correction: State both counts explicitly: “There are 12 randomized clinics and 240 measured patients in this condition.” The clinic count describes treatment replication; the patient count describes how many outcomes were measured. Keeping the two numbers distinct helps prevent inflated sample-size claims and inappropriate independent-observation analyses.
Check Your Understanding
For each scenario, identify the treatment-assignment unit and the number of treatment replicates requested.
- Four schools are assigned to a new curriculum and four to the standard curriculum. Each school contributes 100 student scores. How many treatment replicates are there in each condition?
- Six clinics are assigned to each of two conditions. One clinic measures 15 patients and another measures 80. Does the larger clinic count as more treatment replicates? Explain.
- Three households receive a reminder and five receive no reminder. Each household has two adults whose outcomes are measured. State the replicate counts and the numbers of measured adults by condition.
- A study assigns one classroom to each of two programs and measures 50 students in each classroom. What is the cluster-level replication in each condition, and what limitation follows?
- Explain why positive correlation between people in the same cluster can make an analysis that treats all people as independent overstate the precision of a treatment comparison.