From a Question to an Experimental Plan
A useful experiment is planned before any treatment is applied. The plan turns a broad question into specific decisions: which units will receive treatments, what those treatments are, how assignment will be randomized, and what response will be recorded. If these decisions are vague, the eventual comparison may be difficult to interpret, even if the arithmetic is correct.
This tutorial develops a completely randomized design for a question comparing two treatments. In this design, the experimental units are assigned to the treatments by chance, with no grouping or matching constraint. The design does not guarantee identical treatment groups. Rather, it makes the assignment procedure impartial and gives a sound basis for comparing outcomes.
A treatment might be a dose, a teaching method, a product formulation, or a defined program of care. A treatment must be described precisely: the name alone may not specify how much is given, how often, for how long, or under what conditions. The response also needs an operational definition. For example, “improvement” is too vague unless the plan states how improvement will be measured and when.
A Planning Sequence
The following sequence keeps the central design decisions connected. The earlier material on explanatory variables, responses, treatments, random assignment, and experimental units provides the underlying terms.
Write a focused question identifying the two treatment conditions and the outcome of interest.
Specify what receives the treatment and define which units can enter the experiment.
Describe both conditions in enough detail that the planned difference between them is clear.
Set the intended number in each treatment group, then use a chance procedure to assign units.
Use the same measurement method and timing for units in both treatment groups.
A two-treatment comparison should make clear what differs between the groups and what is held to a common protocol. If the question is about the effect of a new preparation method, for example, the two treatments should not also differ in handling time or measurement method unless those differences are part of the question. Otherwise, the comparison becomes harder to interpret.
Random Allocation with Fixed Group Sizes
A practical method is to decide in advance how many units will receive each treatment, put all unit labels in a random order, and assign the first specified number to one treatment and the rest to the other. A computer-generated random permutation, a shuffled list, or a well-mixed draw of labeled slips can implement this plan. The method should be recorded so the assignment is reproducible.
Worked Example: Assigning Garden Plots to Two Fertilizers
A hypothetical horticulture class will compare two fertilizers on 12 similar garden plots. The question is whether the mean harvest mass per plot differs between Fertilizer A and Fertilizer B. Plan the allocation and response measurement.
Units and treatments. Each plot is an experimental unit because fertilizer is applied to a plot. Fertilizer A is a specified product applied at 20 g per square metre once at planting. Fertilizer B is the comparison product, also applied at 20 g per square metre once at planting. The remaining watering and growing procedures will be the same for all plots.
Allocation. The plan assigns six plots to each fertilizer. Label the plots 01 through 12 and use a random-number generator to produce a permutation. Suppose the hypothetical permutation is 08, 02, 11, 05, 01, 09, 03, 12, 06, 04, 10, 07. Assign the first six plots (08, 02, 11, 05, 01, 09) to Fertilizer A and the remaining six to Fertilizer B. Every plot is assigned once, and each treatment receives six plots.
Response. At harvest, collect all usable produce from each plot and record its total mass in kilograms using the same scale and harvest rule for every plot. The response is harvest mass per plot, in kilograms; the measurement time is the end of the defined growing period.
This plan makes a treatment comparison possible because the application procedures and response measurement are specified. It does not ensure that the plots have exactly the same soil or growing conditions. Random allocation makes treatment assignment independent of those features in the assignment procedure, but chance differences can remain in a particular experiment.
Why Randomization Helps
Randomization matters because the investigator does not choose which units receive which treatment based on a known or suspected characteristic. With fixed group sizes, each unit has the same chance of being assigned to either treatment when the groups are equal in size. This does not promise a perfect balance in every feature, but it prevents a systematic assignment rule from favoring one treatment group.
The idea can be justified using a measured baseline characteristic. Suppose there are \(N\) units with values \(z_1,\ldots,z_N\), and \(n_A\) of them will be assigned to Treatment A by choosing a uniformly random subset. Let \(I_i=1\) if unit \(i\) is assigned to A and \(I_i=0\) otherwise. Each unit has probability \(n_A/N\) of entering A, so \(E(I_i)=n_A/N\). The mean baseline value in A is \(\bar{z}_A=\frac{1}{n_A}\sum_{i=1}^{N} I_i z_i\). Therefore:
Thus, before the random assignment is carried out, the expected mean baseline value in A equals the mean across all eligible units. The same reasoning applies to Treatment B. This is a result about averages over the possible random assignments, not a guarantee that the observed groups will have equal baseline means. Randomization supports a fair comparison; it does not remove chance variation.
Worked Example: Planning a Two-Method Tutoring Experiment
A hypothetical school wants to compare two short tutoring methods for students who meet a stated eligibility criterion. It will enroll 40 students and assign 20 to each method. Specify a completely randomized plan.
Experimental units. The individual students are the experimental units because each student is assigned a tutoring method. The plan assumes each student can receive the assigned method individually. If methods are assigned only to whole classrooms, classrooms, rather than students, would be the experimental units.
Treatments. Method A consists of four 30-minute sessions using guided practice and immediate feedback over two weeks. Method B consists of four 30-minute sessions over the same period using independent practice and a written answer key. Session duration, number, and schedule are specified so that the intended treatment difference is understandable.
Allocation. Label the 40 eligible students from 01 to 40. Generate a random permutation of those labels. Assign the first 20 labels to Method A and the other 20 to Method B. This produces exactly 20 students per treatment while making the allocation depend on chance rather than the instructor's judgment.
Response. Define the response as each student's score, from 0 to 100 points, on the same assessment administered one week after the final session. Use the same administration conditions and scoring rubric for both groups. The comparison is between the assessment outcomes under the two assigned tutoring methods; it does not, by itself, establish how the methods would perform for students outside the eligible population.
Conditions to Check Before Starting
A well-specified random assignment is only one part of a usable design. Before starting, check that each of the following has been addressed:
- Units are clear. State what receives a treatment. If a treatment is assigned to a classroom, plot, or other group, do not treat the people or objects within it as independently assigned units.
- Treatments are deliverable and distinct. Specify dose, duration, timing, and any important procedures. The comparison should correspond to the question being asked.
- Allocation is genuinely random. Choose the method and group sizes before looking at outcomes, and apply the method to all eligible units.
- Measurement is comparable. Define the response, its units, its measurement time, and the procedure to be used in both groups.
- Interference is considered. For a simple unit-level comparison, one unit's treatment should not substantially change another unit's response. If interaction between units is likely, the unit of assignment or the design may need reconsideration.
It is also important to plan what happens if a unit does not complete its assigned treatment or its response cannot be measured. The design should specify how such cases will be documented rather than silently removing them or measuring them differently by treatment group. The details depend on the setting, but the same follow-up rules should be considered for both groups.
Worked Example: Comparing Two Packaging Processes
A hypothetical manufacturer wants to compare two packaging processes for reducing damage to ceramic mugs during a standard shipping simulation. There are 30 mugs available. Plan a two-treatment experiment and identify a potential design error.
Units and treatments. Each mug is an experimental unit if the packaging process is applied separately to each mug and each mug is tested separately. Treatment A is the current packaging procedure; Treatment B is the proposed procedure, with materials and wrapping steps specified in advance.
Allocation. Label the mugs 01 through 30, generate a random permutation, and assign the first 15 to Treatment A and the remaining 15 to Treatment B. Apply the designated procedure to each mug, then run every package through the same shipping simulation.
Response. Record whether each mug is damaged after the simulation, using the same predefined inspection rule. The response is categorical: damaged or not damaged. Record the number and proportion damaged in each treatment group after collecting the outcomes.
Design error to avoid. It would be a problem to package all mugs in one group on Monday and all mugs in the other group on Friday if storage or shipping conditions might differ by day. The treatment assignment would then be tied to timing. Randomly allocate individual mugs and use the same simulation conditions to reduce this risk.
Common Mistakes
- Randomly selecting participants and calling that random assignment. Selection determines who enters the experiment; assignment determines which treatment each enrolled unit receives. A completely randomized experiment requires random treatment assignment.
- Describing treatments only by name. “New program” does not specify what participants receive. Define the content, amount, timing, and duration relevant to the comparison.
- Measuring different outcomes in the two groups. If one group is assessed after one week and the other after four weeks, the observed difference cannot be attributed to treatment alone with confidence. Use the same response definition and measurement schedule.
- Assuming randomization guarantees equal groups. It does not. Randomization gives a chance-based allocation mechanism; observed imbalances can occur, especially with few units.
- Using the wrong experimental unit. If treatment is assigned to a whole plot or classroom, the plot or classroom is the unit, even when responses are recorded for individual plants or students within it.
Check Your Understanding
Try these questions without looking back.
- A study assigns 24 garden beds to one of two watering schedules. What is the experimental unit, and what information should the treatment descriptions include?
- For 18 units, a researcher plans nine per treatment. Describe one valid completely randomized allocation procedure.
- Why does random allocation support a fair comparison without guaranteeing equal treatment groups?
- A researcher records a response at different times for the two treatment groups. Explain why this weakens the planned comparison.
- A treatment is applied to each classroom, but student scores are recorded. What is the experimental unit, and why?