One Question, Three Possible Designs
In Matched Pairs Design, you learned how to compare treatments within pairs. Matched pairs are one way to organize an experiment, but they are not the only option. Depending on the experimental units and what is known about them, researchers might instead assign treatments completely at random or form larger blocks before assigning treatments.
The choice is not about finding a design that guarantees a particular result. It is about planning a fair comparison that fits the situation. A useful question is whether the experimental units differ in a known way that could affect the response. If no especially useful grouping is available, a completely randomized design may be appropriate. If a relevant grouping is available, blocking can keep that source of variation from being mixed unevenly across treatment groups. For a two-treatment comparison where units can be paired closely, matched pairs may be especially suitable.
A Practical Way to Choose
Start by naming the treatments, experimental units, and response, as in Experimental Units, Factors, and Treatments and Explanatory and Response Variables in Experiments. Then ask what could make the units differ in their responses before treatment. The answer helps you decide whether to group units before random assignment.
How many treatments are being compared, and what receives each treatment? Consider whether each unit can reasonably receive all treatments or only one.
Is there a known characteristic, such as starting skill level or work shift, that is related to the response? Could units be grouped sensibly using that characteristic?
If there is no useful grouping, consider a completely randomized design. If a useful grouping exists, consider blocks. For two treatments and close pairs, consider matched pairs.
State exactly how chance assigns units to treatments: across all units, separately within blocks, within each pair, or to treatment order for subjects receiving both treatments.
Consider whether treatments can be applied consistently and whether repeated treatments could cause carryover, practice, or other order effects.
A grouping variable is useful when it identifies units likely to have similar responses apart from the treatment. It should be known before the treatments are assigned. Do not form groups using outcomes measured after treatment; doing so would make the design depend on the very results it is meant to compare.
Blocking does not mean that chance is used only to form groups. As in Randomized Block Design, the key is that chance assigns treatments separately within each block. In a completely randomized design, chance assigns treatments across all experimental units without first creating blocks. Both designs use random assignment, a key feature discussed in Why Random Assignment Is the Key to Causation.
How to Compare the Three Designs
| Design | How units are organized | Where chance is used | When it may fit |
|---|---|---|---|
| Completely randomized | All experimental units are considered together. | Chance assigns units directly to treatment groups. | No clearly useful grouping is available, or a simpler assignment plan is appropriate. |
| Randomized block | Units are grouped into blocks based on a characteristic related to the response. | Chance assigns treatments separately within each block. | A relevant characteristic can be used to group similar units, and the treatments can be represented within each block. |
| Matched pairs | Units are organized into pairs, or each subject receives both treatments. | Chance assigns one treatment to each member of a pair, or determines treatment order for each subject. | There are two treatments and a meaningful within-pair comparison is possible. |
In a block design, every treatment should generally be represented within each block when the plan allows it. If there are two treatments, a block might contain two units, one assigned to each treatment, or more units with chance assigning some to each treatment. When every block is a pair and the two treatments are compared within it, the design is matched pairs.
A matched pairs design is not automatically best just because there are two treatments. Pairing needs a sensible basis, or each subject must be able to receive both treatments appropriately. A treatment that has a lasting effect, for example, may make it unsuitable for the same subject to receive both. In that case, pairs of similar subjects may be possible, or a broader randomized block or completely randomized design may fit better.
Worked Example: No Clear Basis for Grouping
Worked Example: Testing Two Inventory Alerts
A fictional online shop wants to compare two versions of an inventory alert for its warehouse staff. Forty staff members will each use one version during a practice shift. The response is the number of stock-count errors made during that shift. The planner has no reliable prior information suggesting that a particular characteristic, such as experience level, predicts errors in this group.
State: The experimental units are the 40 staff members. The treatments are Alert A and Alert B, and the response is the number of stock-count errors per practice shift.
Plan: A completely randomized design is a sensible choice because there is no clear, relevant grouping variable that would support forming blocks. Assigning everyone directly by chance also avoids creating arbitrary groups. The plan uses random assignment, but it does not randomly select staff from a larger population; therefore, random assignment and random selection should not be confused.
Do: Label the staff members 01 through 40. Use a chance process to select 20 distinct labels for Alert A; assign the remaining 20 staff members to Alert B. For example, a random-number generator could choose the 20 labels, with repeated labels skipped. Have both groups use their assigned alert under the same practice-shift procedures, and count errors using the same rules.
Conclude: This is a completely randomized design because chance assigns all 40 staff members directly to the two alert versions without first forming blocks. The design is appropriate given the lack of an identified useful grouping variable. It creates a fair treatment comparison, though it does not guarantee equal groups in every background characteristic.
This example illustrates an important point: a completely randomized design does not mean that every treatment group is guaranteed to be identical in all respects. Random assignment helps make the groups comparable on average, as described in Why Random Assignment Is the Key to Causation. A researcher should not claim that chance eliminates all differences among the groups.
Worked Example: A Relevant Characteristic Suggests Blocks
Worked Example: Comparing Two Cooling Methods
A fictional food-science team wants to compare two cooling methods for prepared meal containers. The experimental units are 24 containers, and the response is the time, in minutes, for a container to reach a target temperature. The team knows that containers placed in different sections of its cooling room may cool at different rates. It can arrange the containers into six groups of four based on their starting positions in the room.
State: The experimental units are the 24 containers. The treatments are Cooling Method A and Cooling Method B. The response is cooling time in minutes. Starting position is a relevant blocking variable because room location may be related to cooling time.
Plan: A randomized block design is appropriate. It groups containers by starting position before treatment assignment, so each treatment can be compared within each location-based block. A completely randomized design would ignore a known source of response variation. Matched pairs would not fit as naturally because there are two treatments but four containers per location group.
Do: Form six blocks of four containers using their starting positions. Within each block, randomly assign two containers to Method A and the other two to Method B. Repeat this independently for all six blocks. This gives 12 containers per method overall, with both methods represented in every block. Apply the methods for the same procedure and measure cooling time consistently.
Conclude: This is a randomized block design because containers are grouped by a characteristic related to cooling time, then chance assigns both methods within each block. The plan makes the treatment comparison within location groups rather than leaving one method potentially concentrated in a particular part of the room.
The main justification is not simply that the team can divide the containers into six groups. It is that starting position is related to the response and therefore gives a meaningful basis for grouping. If the groups were formed using an unrelated characteristic, the extra organization might add complexity without addressing a relevant source of variation.
Worked Example: When Matched Pairs Are the Best Fit
Worked Example: Comparing Two Bicycle Helmets
A fictional product team wants to compare two helmet designs using a comfort rating collected after a short, controlled ride. Twenty-four volunteers can safely test both designs on separate rides. The team expects people’s usual comfort ratings to differ, and it wants a direct comparison for each volunteer. The response is a comfort rating from 1 to 10 after each ride.
State: The experimental units are the 24 volunteers. The treatments are Helmet A and Helmet B. The response is each volunteer’s comfort rating after a ride.
Plan: A matched pairs design is suitable because each volunteer can receive both treatments, and each person can be compared with themself. This is a special form of blocking in which each block consists of one volunteer’s two measurements. A completely randomized design with two unrelated groups could leave differences in people’s usual ratings as part of the between-group comparison.
Do: For each volunteer, use a coin flip or another chance process to decide which helmet is worn on the first ride. Heads means Helmet A first; tails means Helmet B first. The volunteer wears the other helmet on the second ride. Use comparable ride conditions and the same rating question each time. Allow a suitable reset between rides and consider whether familiarity or fatigue could affect the second rating.
Conclude: This is a matched pairs design because each volunteer receives both helmets and chance determines the order. The comparison is the difference between each volunteer’s two comfort ratings. If the first ride influences the second, randomizing order helps prevent one helmet from always being associated with the first or second position, but it may not eliminate every order effect.
If the helmets could not be tested safely by the same volunteer, the team might instead pair similar volunteers using a relevant characteristic and randomly assign one member of each pair to each helmet. If no reasonable pairs could be formed, a completely randomized design could be considered. The design choice follows what is feasible and what structure the units support.
Worked Example: When a Proposed Design Does Not Fit
Worked Example: A Lasting Training Program
A fictional community center wants to compare two intensive, one-day first-aid training programs. Thirty participants will be assigned to a program, and the response is a score on a practical skills assessment the next day. A planner proposes having every participant complete both programs and comparing each person’s two scores.
Identify the concern: The proposal is a repeated-measures matched pairs design, but learning from the first program could carry over into the second. A participant’s second score would reflect both the second program and knowledge gained during the first. Randomizing program order would help balance order across participants, but it would not erase the learning.
Choose a better plan: If the center has a relevant pre-training measure, such as a baseline first-aid skills score, it could form blocks of similar participants and randomly assign each program within each block. If useful blocks cannot be formed, it could use a completely randomized design, assigning 15 participants to each program by chance. In either plan, each participant receives only one program.
Conclude: The proposed repeated-measures matched pairs plan is a poor fit because the first intensive program could affect the response under the second. A block design is preferable if a relevant baseline measure supports meaningful groups; otherwise, a completely randomized design is a reasonable alternative. In both alternatives, chance must determine treatment assignment.
Common Mistakes and AP Exam Tips
- Calling every experiment a completely randomized design. If researchers first group units and then assign treatments within those groups, the design is randomized block, not completely randomized.
- Calling any grouped design matched pairs. Matched pairs requires pairs and a within-pair comparison. Several larger groups are blocks, but not necessarily pairs.
- Choosing blocks without a reason. Name the characteristic used to form blocks and explain why it could be related to the response. “The units were divided into groups” is not a sufficient justification.
- Forgetting where chance is used. For a completely randomized design, explain the assignment across all units. For a block design, state that chance assigns treatments separately within each block. For matched pairs, specify the assignment within each pair or the random order for each subject.
- Assuming blocking guarantees a better result. Blocking can organize a comparison around a relevant source of variation, but it does not guarantee that one treatment will perform better or that every lurking variable is controlled.
- Using the same subjects for treatments that may carry over. Name the possible practice, fatigue, or lasting-treatment effect in context. If the first treatment could influence the response to the second, consider different subjects instead.
- Mixing up random assignment and random selection. A chance-based treatment assignment supports a fair comparison and can support a cause-and-effect conclusion. It does not by itself make a sample representative of a population. As emphasized in Scope of Inference: Four Combinations, selection and assignment answer different questions.
For a strong AP response, identify the design by describing how the units are organized and where chance is used. Then justify the choice using the actual scenario: point to a relevant source of variation if recommending blocks, explain the within-pair comparison if recommending matched pairs, or state why no useful grouping is available if recommending complete randomization.
Check Your Understanding
For each situation, choose a design and give a brief justification that names the chance process.
- A researcher compares three screen layouts using 36 volunteers. The researcher has no prior information about a useful characteristic for grouping the volunteers. Which design is a reasonable choice, and how should chance be used?
- A greenhouse team compares two watering methods. It expects plant location to affect growth and forms blocks of plants by greenhouse bench. What happens within each block?
- A coach compares two warm-up routines by having each athlete try both on separate days. What should be randomized, and what order effect might the coach consider?
- A researcher pairs two students with similar starting reading scores and assigns one student in each pair to each of two tutoring methods. What design is this, and where is the within-pair comparison?
- Why might a completely randomized design be preferable to a matched pairs design when the first treatment could have a lasting effect on a subject?