Why Factorial Experiments Separate Treatment Components
A program may combine two components: for example, a coaching service and a reminder system. A two-factor experiment assigns eligible units to one of four combinations: neither component, coaching only, reminders only, or both. This design can answer more than whether the combined program changes outcomes. It can also estimate the effect of each component and assess whether the effect of one depends on the presence of the other.
The key is to define effects across all four treatment combinations before looking at observed outcomes. In potential-outcomes notation, each unit has an outcome under every combination, although random assignment reveals only the outcome under the combination actually received. The contrasts of those potential outcomes define the causal questions; random assignment supplies a design-based way to estimate them.
This notation presumes a well-defined version of each assigned combination and no interference between units, so that unit \(i\)'s outcome under \((a,b)\) does not depend on other units' assignments. These are aspects of SUTVA. The outcome might be a test score, measured in points, or an employment indicator, measured as zero or one; all contrasts must be interpreted on the chosen outcome scale.
Component Effects and the Interaction Contrast
The effect of component A can first be defined separately at each level of B. At \(B=0\), it is the change from \((0,0)\) to \((1,0)\). At \(B=1\), it is the change from \((0,1)\) to \((1,1)\). Component B has corresponding effects at \(A=0\) and \(A=1\). These conditional effects are distinct quantities; they need not be equal.
The main effects use equal weights for the two levels of the other component by definition. They do not necessarily match a weighted average reflecting the proportion of units in some target population who would receive each level of that component. The interaction is measured on the additive scale of the outcome: it is zero exactly when A's average effect is the same at both levels of B. A zero additive interaction does not rule out interaction on another scale, such as a ratio scale.
Worked Example: Deriving Effects From Four Potential-Outcome Means
Suppose a hypothetical finite population has the following average potential outcomes, measured in test-score points: \(\bar Y(0,0)=50\), \(\bar Y(1,0)=56\), \(\bar Y(0,1)=54\), and \(\bar Y(1,1)=65\). Calculate the component effects and interaction.
A's effect when B is absent: \(56-50=6\) points. A's effect when B is present: \(65-54=11\) points. Therefore,
B's effect when A is absent: \(54-50=4\) points. B's effect when A is present: \(65-56=9\) points. Thus,
Finally, the interaction is \(11-6=5\) points, equivalently \(65-56-54+50=5\) points. On the additive test-score scale, A's average effect is 5 points greater when B is present than when B is absent. The main effects summarize the two conditional effects; they do not imply that A has exactly the same effect in both settings.
Estimating the Contrasts by Random Assignment
Let \(Z_i(a,b)\) equal 1 if unit \(i\) is assigned to combination \((a,b)\), and 0 otherwise. Let \(n_{ab}=\sum_i Z_i(a,b)\) be the number assigned to that cell. When \(n_{ab}>0\), the observed cell mean is the average of the observed outcomes in that treatment combination. Substituting the four observed cell means for the four potential-outcome means gives estimates of the main effects and interaction.
A crucial condition applies to the ordinary, unweighted cell mean. It is unbiased for \(\bar Y(a,b)\) under fixed-count complete random assignment to that cell: exactly \(n_{ab}\) units are assigned there, and every unit has the same probability \(n_{ab}/N\) of being selected into it. More generally, equal inclusion probabilities within the cell are sufficient for this cell-mean argument when the cell size is fixed. Positive assignment probabilities alone are not sufficient. With unequal probabilities, units with different potential outcomes may be overrepresented in the ordinary cell mean.
Record whether cell counts are fixed and whether each unit has the same probability of assignment to each cell. Positivity means each relevant unit can be assigned to the cell, but does not imply equal probabilities.
Under fixed-count complete random assignment, use the ordinary observed cell means. With unequal known assignment probabilities, use an estimator that accounts for those probabilities, such as the Horvitz-Thompson estimator.
Use the estimated cell means in the definitions of \(\tau_A\), \(\tau_B\), and \(\tau_{AB}\). Explain effects and interactions in the outcome's units and on its scale.
Under fixed cell counts and uniform random assignment within each cell, every unit has inclusion probability \(n_{ab}/N\) for that cell. The cell mean is then unbiased for the finite-population mean of potential outcomes in that cell. Because each factorial contrast is a linear combination of the four cell means, its corresponding plug-in estimator is unbiased for the same linear combination of finite-population means.
For unequal assignment probabilities, write \(\pi_i(a,b)=P\{Z_i(a,b)=1\}\), assumed known and positive for the units and cell of interest. An inverse-probability estimator of the finite-population cell mean is
Its expectation is \(\frac{1}{N}\sum_i Y_i(a,b)=\bar Y(a,b)\), because \(E[Z_i(a,b)]=\pi_i(a,b)\) and the observed outcome equals \(Y_i(a,b)\) whenever \(Z_i(a,b)=1\). Contrasts of these Horvitz-Thompson cell-mean estimators are unbiased for the corresponding factorial contrasts. This estimator need not equal the ordinary sample mean, and its estimates can be variable when some assignment probabilities are small.
Worked Example: Computing Factorial Contrasts From Randomized Cell Means
In a hypothetical trial, 20 participants are assigned by complete randomization, with five assigned to each combination. Their observed mean outcome, measured in days of employment over a fixed follow-up period, is 30 days for \((0,0)\), 34 days for \((1,0)\), 33 days for \((0,1)\), and 41 days for \((1,1)\). The assignment within each cell gives all participants equal inclusion probability \(5/20\), and all four cell counts are positive.
The ordinary cell means are design-unbiased for the corresponding finite-population potential-outcome means under this assignment scheme. The estimated effect of A is \(34-30=4\) days when B is absent and \(41-33=8\) days when B is present. Hence the estimated main effect is
For B, the estimated effects are \(33-30=3\) days when A is absent and \(41-34=7\) days when A is present. Thus \(\hat{\tau}_B=(3+7)/2=5\) days. The estimated interaction is \(8-4=4\) days, or \(41-34-33+30=4\) days. In this hypothetical experiment, the estimated average effect of A is 6 additional employment days, averaging equally over B's two levels; its estimated effect is 4 days larger when B is present than when B is absent. These are point estimates, not claims that either effect is statistically distinguishable from zero.
Worked Example: Why Unequal Assignment Probabilities Change the Estimator
Consider a hypothetical finite population of two units. For one treatment combination, their potential outcomes are 0 and 10 units, respectively. Exactly one unit is assigned to that combination, but the first unit has assignment probability 0.9 and the second has probability 0.1. Both probabilities are positive. The finite-population mean potential outcome for this combination is \((0+10)/2=5\) units.
The ordinary cell mean is just the outcome of the selected unit. Its expectation is \(0.9(0)+0.1(10)=1\) unit, not 5. Therefore, positivity does not make the unweighted cell mean unbiased. The Horvitz-Thompson estimate is \(\frac{1}{2}\sum_i Z_iY_i/\pi_i\). If the first unit is selected, this estimate is 0; if the second is selected, it is \(\frac{1}{2}(10/0.1)=50\). Its expectation is \(0.9(0)+0.1(50)=5\) units, the target finite-population mean. This example also shows that an unbiased estimator under unequal probabilities can be highly variable.
Interpretation, Scope, and Common Pitfalls
A factorial main effect is an average over the other component's levels, not a claim that the effect is constant across those levels. When the interaction is substantial, report the conditional effects as well as the averages. For example, saying “A increases the outcome by 6 days” without noting that the estimated effects differ across B's levels can conceal an important feature of the results. State the averaging convention and the units.
The contrasts are defined on the outcome scale used. For an indicator outcome, the additive interaction is a difference of differences in probabilities, interpretable in percentage points; it is not automatically an interaction in odds or risk ratios. Likewise, random assignment supports causal comparisons of assigned combinations, but does not by itself establish that a treatment component operates through a particular mechanism.
- Mistake: treating a positive assignment probability as enough for an unweighted cell mean. The correction is to check inclusion probabilities. The ordinary cell mean has the stated finite-population unbiasedness under equal-probability selection into the cell, as in fixed-count complete random assignment; unequal probabilities require an appropriate weighting estimator.
- Mistake: interpreting the main effect as identical at both levels of the other component. It is the equally weighted average of the two conditional effects. Report the conditional effects and interaction when their difference matters.
- Mistake: calling any nonzero interaction “synergy” without specifying a scale. The contrast above measures departure from additivity on the outcome scale. A different scale defines a different interaction.
- Mistake: interpreting an estimated contrast as proof of a mechanism. Random assignment identifies effects of assigned combinations under the design assumptions; it does not, by itself, identify why those effects occur.
Check Your Understanding
Use the definitions and assignment conditions in this tutorial to answer the following.
- Write the four potential outcomes for a unit in a two-factor experiment and identify which one is observed under assignment \((A_i,B_i)=(1,0)\).
- If A's effect is 3 points when B is absent and 9 points when B is present, what are the factorial main effect of A and the additive interaction?
- Suppose fixed counts are assigned to each of four cells, with every unit equally likely to be assigned to any particular cell. Why is the ordinary mean in a cell unbiased for that cell's finite-population potential-outcome mean?
- Why does positive but unequal assignment probability across units not, by itself, justify an ordinary cell mean? Name an estimator that accounts for known unequal probabilities.
- If the estimated interaction is zero on an additive outcome scale, what does that say about the two conditional effects of A? What does it not establish about interaction on other scales?