Factorial Experiments and Treatment Combinations

Why Combine Factors in One Experiment?

A teaching method may work differently depending on where students study. If researchers want to investigate both questions, they can vary the teaching method and the study setting in the same experiment. The resulting design compares each method in each setting, rather than investigating one factor while leaving the other unexamined.

This is a factorial experiment: an experiment in which two or more factors are varied together, and the treatment is a particular combination of their levels. Factorial designs let researchers ask two kinds of questions: what is the average effect of changing one factor, and does that effect depend on the level of another factor?

For example, let teaching method have two levels, lecture and retrieval practice, and study setting have two levels, quiet and background noise. These factors produce four treatment combinations. A table displays the design directly.

The table describes the combinations, not the results. Before data are collected, we can say which effects the design allows researchers to compare, but we cannot say whether either factor has an effect or whether the factors interact. Those conclusions require outcome data.

What the Design Needs to Support the Comparisons

Suppose students are the experimental units and the response is a quiz score in points. To interpret differences between treatment combinations as effects of the assigned conditions, students should be assigned to those combinations at random. The teaching methods and settings should be sufficiently well-defined that students assigned to the same combination receive comparable versions of it. The same quiz and scoring procedure should be used across combinations.

For the usual causal interpretation, the design should also avoid one student’s assigned combination changing another student’s outcome. This is a no-interference condition; for example, it may be a concern if students in different groups share study materials during the experiment. Random assignment supports causal comparisons among the experimental units, but does not by itself ensure that the participating students represent a broader population. If assignment is not random, observed differences can reflect pre-existing differences as well as the assigned conditions.

In a balanced design, the same number of units is assigned to each treatment combination. Balance makes the comparisons especially transparent, but it is not required for a factorial design. Researchers should plan adequate replication in every combination; assigning only one unit to a combination gives no within-combination replication.

Main Effects and Interaction in a Two-by-Two Design

Consider a two-by-two design with two levels of each factor. Write the mean response in each treatment combination as \(\bar{y}_{ij}\), where \(i\) identifies the level of the first factor and \(j\) the level of the second. In a balanced experiment, the main effect of the first factor can be summarized by comparing its two averages, each formed across the two levels of the second factor.

The main effect of factor B is calculated in the same way, comparing the column averages. These are average comparisons: a main effect does not claim that the difference is identical at every level of the other factor.

To check for interaction, compare the effect of A separately at each level of B. If these two effects differ, there is interaction on the response scale being used. The difference between those two effects is called a difference-in-differences.

A zero difference-in-differences means the effect of A is the same at both levels of B. A nonzero value means the effect changes across the levels. This calculation describes interaction in the observed means; a formal assessment of uncertainty would require additional statistical methods.

Worked Effects from Teaching and Study-Setting Data

A Short Justification: Why Equal Differences Mean No Interaction

The no-interaction condition can be understood by writing the four means in an additive form. Suppose each mean is a baseline plus one contribution from factor A and one contribution from factor B. Denote those contributions by \(a_1,a_2\) and \(b_1,b_2\), respectively:

At level 1 of factor B, the effect of changing A from level 1 to level 2 is \((m+a_2+b_1)-(m+a_1+b_1)=a_2-a_1\). At level 2 of B, that effect is \((m+a_2+b_2)-(m+a_1+b_2)=a_2-a_1\) again. Subtracting these effects gives zero:

If the difference-in-differences is not zero, this additive description does not fit the four means: the change associated with A depends on which level of B is used. This reasoning concerns the scale of the response. A pattern that has no interaction when measured as differences in quiz points need not have no interaction when expressed on a different scale.

Why Factorial Designs Matter

A factorial design studies several factors in one coordinated experiment. It can reveal that a treatment works differently across settings, identify combinations that merit further study, and avoid the misleading impression that one average comparison applies equally everywhere. A treatment-combination table also makes the intended comparisons visible before data collection begins.

The design should match the question. If researchers want to know whether a teaching method’s effect depends on study setting, both factors must be varied so the combinations can be compared. If one method is used only in quiet rooms and another only with background noise, the effects of method and setting cannot be separated from those data.

Common Mistakes

• Calling a factor combination an interaction. Having multiple combinations in the design makes interaction assessable; it does not mean an interaction has been found. Compare outcome differences across levels.
• Treating a main effect as universal. An average advantage does not imply that the advantage is the same in every setting. Check the factor-specific differences.
• Using an unclear average. In a balanced two-by-two design, averaging the two cell means gives an equal-weight main effect. With unequal group sizes, pooled averages may give unequal weight to settings; state which average is intended.
• Claiming causation from the table alone. A table lists combinations, not evidence that outcomes differ. Causal interpretation of comparisons depends on appropriate assignment and implementation, and broader generalization requires separate justification.