From a Design Idea to a Complete Description
In Common Errors in Describing Experimental Designs, you practiced checking whether a study has random assignment and a meaningful comparison. Now the goal is to write the plan itself so another person can tell exactly how the comparison will work. A complete description does more than say “split the students randomly.” It identifies who or what receives each method, explains the chance process, makes the conditions comparable, and states what outcome will be measured and in what units.
We will focus on an experiment comparing two study methods. The same structure works for many other treatment comparisons. As introduced in Experimental Units, Factors, and Treatments and Explanatory and Response Variables in Experiments, begin by identifying the experimental units, the factor and its levels, and the response variable. Then describe assignment, control, and replication.
What a Full Description Needs
Suppose a school wants to compare two ways of studying: retrieval practice, in which students try to recall information without looking at notes, and rereading, in which students review the same notes. A strong plan should make each of the following parts explicit.
- Experimental units: Name the individuals or objects to which the methods will be assigned. If students receive the study methods, the students are the experimental units. When the units are people, they may also be called subjects.
- Treatments: Name both methods and describe what students will actually do. “Method A” and “Method B” are not enough if the reader cannot tell how they differ.
- Response and measurement units: State what will be measured and how it will be recorded. For example, the response could be a score on a 20-question quiz, measured in points from 0 to 20. The units here are score points; they are not the students.
- Random assignment: Explain how chance will assign the experimental units to the methods. As discussed in Why Random Assignment Is the Key to Causation, random assignment helps make the groups comparable, on average, with respect to potential lurking variables.
- Control: Hold relevant conditions constant across groups so the study methods are the main planned difference. Specify details such as study time, materials, location, and how the outcome is assessed. Control does not mean that every student will have identical experiences.
- Replication: Apply each method to multiple experimental units. State the number assigned to each method. As explained in Replication in Experiments, repeating a measurement on one student is not a substitute for assigning multiple students to each method.
A study comparing two active methods has a comparison even if neither method is called a “control group.” One method provides the comparison condition for the other. If appropriate, researchers may call one group a control group when it receives a baseline condition, such as usual study practice. What matters in a description is that both conditions and the planned comparison are clear.
Random assignment and random selection also answer different questions. A chance process that assigns recruited students to methods supports a causal comparison for those students. It does not make them a random sample of all students. If participants are volunteers or students from one class, say so rather than implying that the experiment represents every student.
A Reliable Writing Sequence
Use this sequence to turn a scenario into a complete design description. It keeps important details from disappearing into a vague sentence such as “the students will be split into two groups.”
Say what the experiment compares and name what receives the assigned methods.
Describe what participants do in each condition and how the outcome will be measured, including its units.
Give a chance-based procedure that assigns every unit to a treatment, and state how many units receive each method.
Keep relevant conditions consistent, then state which outcomes will be compared and what population the design does or does not represent.
The order can vary in a written response, but all of these ideas should be present. A reader should be able to carry out the plan without having to guess what “randomly split,” “same test,” or “study time” means.
Worked Example: Retrieval Practice Versus Rereading
A teacher wants to compare two study methods for a unit in environmental science. Forty-eight students in the teacher’s classes agree to participate. The teacher plans for each student to study for 30 minutes on four evenings, using either retrieval practice or rereading. At the end, each student will take the same 20-question quiz. The response is the quiz score, measured in points from 0 to 20.
State: The experiment will compare retrieval practice with rereading for the 48 participating students. The experimental units are the individual students. The response variable is each student’s score on the quiz, measured in points from 0 to 20.
Plan: Use a completely randomized design: assign all 48 students to one of the two methods by chance. To make the assignment reproducible, label the students 01 through 48 and use a random number generator to select 24 distinct labels for retrieval practice. Assign the remaining 24 students to rereading. The design includes random assignment because chance, rather than student or teacher choice, determines which method each student receives.
Control conditions by giving both groups the same environmental science materials, the same total study time, the same number of study evenings, and the same study location and schedule as far as practical. Give both groups the same quiz under the same conditions and score it using the same answer key. The planned treatment difference is the way students use the study materials. Replication is present because 24 different students receive each method; the comparison is not based on just one student per method.
Do: Carry out the assignment and study schedule, record each of the 48 quiz scores in points, and compare the score distributions or the groups’ mean scores. No scores have been supplied, so there is no numerical treatment difference to calculate yet.
Conclude: This plan can provide evidence about a cause-and-effect difference between the two study methods for these participating students, because it randomly assigns students to the methods and measures outcomes under controlled conditions. Since the participants are students in these classes who agreed to take part, the results do not automatically generalize to all students.
Why the Details Improve the Comparison
The random assignment procedure in the example names the units being assigned, gives every student one label, and states how many distinct labels go to each method. It also makes clear that the remaining students go to the other method. That is more complete than writing “use a random number generator” without saying what numbers mean or what happens to selected students. The procedure does not guarantee that the groups will be identical. It uses chance to help create groups that are comparable, on average.
Control is also specific, not a promise to eliminate every difference. Students may still vary in prior knowledge, sleep, or motivation. Random assignment helps distribute such potential lurking variables across the groups, while holding the materials and study time constant prevents those planned features from systematically favoring one method. This follows the principle in Controlling Variables Across Treatment Groups: vary the treatment factor while keeping relevant background conditions as consistent as practical.
The response’s units matter because they tell the reader how to understand the recorded values and any later comparison. “Quiz score” names the response, but “score in points on the same 20-question quiz” specifies how it is measured. If the outcome were time to finish the quiz, the response units might instead be minutes. Do not confuse these measurement units with the experimental units: students receive treatments; points or minutes describe measured outcomes.
Finally, replication means that several experimental units receive each method. In this example, the teacher does not assign one class to retrieval practice and the other class to rereading. Doing so would make method and class inseparable: differences could reflect the classes as well as the methods. Assigning individual students to both methods within the participating classes allows each class to contribute students to both treatment groups. This is a practical way to avoid that particular confounding problem.
Worked Example: Adding a Block for Prior Experience
A community learning center recruits 36 adults to compare two methods for learning basic keyboard shortcuts: practicing with printed cards and practicing with an interactive tutorial. Before assignment, the center records whether each participant has little or substantial prior computer experience. There are 18 participants in each experience category. Everyone will practice for 25 minutes on each of three days and then complete the same 15-item shortcut test. The response is the number correct, measured in items correct out of 15.
The center can use a randomized block design because prior experience may be related to test performance. Within the 18 participants who have little experience, use chance to assign 9 to printed cards and 9 to the interactive tutorial. Separately, within the 18 participants with substantial experience, use chance to assign 9 to each method. All 36 participants receive one method, and each method is replicated on 18 participants overall. Each treatment is also represented by 9 participants in each experience block.
Control the practice duration, days of practice, shortcut content, room conditions, test instructions, and scoring procedure across the two methods. The treatment factor is the study method, with printed cards and interactive tutorial as its two levels. The experimental units are the 36 adult participants, and the response is the number of test items answered correctly, measured in items correct out of 15. Compare the test results for the two methods while recognizing that the assignment took place separately within the prior-experience blocks.
This design uses blocking to account for prior experience in organizing treatment assignment; it does not make the recruited adults a random sample of all adults. Random assignment within the blocks supports a causal comparison for these participants, while the recruitment method limits how broadly results can be generalized.
Worked Example: Repairing a Vague Design Description
A library coordinator proposes this plan: “We’ll ask 30 regular visitors to try two study apps. They can choose the app they like, study for a while, and then we’ll see who does better on a quiz.” Rewrite the plan so it describes a fair comparison of two methods.
The original description leaves out a defined assignment process, a clear treatment procedure, a specific study period, and measurement units. Because visitors choose their apps, characteristics such as confidence or prior experience could differ systematically between groups. “Study for a while” does not ensure that study time is comparable, and “who does better” does not define the response precisely.
Revised plan: The experimental units will be the 30 participating library visitors. The study will compare App A, which prompts users to answer recall questions, with App B, which presents the same material for review. Label the visitors 01 through 30 and use a chance process to assign 15 distinct labels to App A; assign the remaining 15 visitors to App B. Each participant will use the assigned app for 20 minutes per day on five days, with the same study topics and access to the same amount of material. On the final day, all participants will complete the same 12-question quiz under the same conditions. The response is the number of correct answers, measured in items correct out of 12. Comparing the quiz results will assess the two apps for these participants.
In the revised description, 15 participants receive each method, so both methods are replicated. The methods differ in a specified way, while the study duration, material, quiz, and testing conditions are held consistent. Random assignment supports a cause-and-effect comparison for the participants, but because the visitors were not randomly selected from a clearly defined population, it does not alone justify generalizing the result to all library users.
Common Mistakes and AP Exam Tips
- Writing “randomly split the group” and stopping there. A full-credit description explains what receives labels, how chance selects assignments, and how many units go to each method.
- Leaving out response units. “Measure learning” is vague. Name a response such as quiz score and report it in points or items correct, with the quiz’s maximum if relevant.
- Confusing experimental units with measurement units. Students are the experimental units; points on a quiz are units for recording the response. State both when clarity requires it.
- Using “control” to mean that nothing changes. An experiment must vary its treatment factor. Control means making relevant non-treatment conditions comparable, not eliminating all differences among participants.
- Calling repeated practice by one student replication. Multiple practice sessions may be part of the treatment, but replication requires multiple experimental units receiving each treatment.
- Claiming that random assignment makes the participants representative. Random assignment helps support a causal conclusion for the experimental units. Random selection is the process relevant to generalizing to a population.
- Claiming a treatment caused an improvement before seeing results. A design description states what the experiment can assess. A conclusion about evidence for a treatment effect must wait until outcomes have been collected and compared.
A strong AP response is concrete and connected to the question. It names the units, treatments, response with measurement units, chance assignment, control measures, and replication. When relevant, it also distinguishes the causal conclusions supported by random assignment from the population generalization supported by random selection.
Check Your Understanding
Use complete design language in your answers. Include the assignment, control, replication, and measurement details when the question asks for a plan.
- A teacher will compare handwritten flashcards with a digital quiz tool for 40 volunteers. Name the experimental units and treatments, and propose a chance assignment with the number assigned to each method.
- A researcher says the response is “how much students learned.” Rewrite this as a measurable response with clear measurement units for a 25-question test.
- For a study comparing two note-taking methods, name three conditions besides the method that could be held consistent across treatment groups.
- Twenty students use one method for five days, and the same students repeat the method five times. How many experimental units receive that method? Explain why the five days do not create five times as many experimental units.
- A study randomly assigns volunteers from one school to two methods. What does random assignment support, and why does it not by itself show that the volunteers represent all students?