Why Misclassification Changes the Conclusion
Calling a factor fixed or random is not a label chosen because of the factor’s name, the study design’s appearance, or the number of levels in the data. The classification describes the role of those levels in the analysis: are they the particular levels whose differences matter, or are they intended to represent a broader population of possible levels? Misclassification can make a conclusion reach beyond the data’s support, or make it narrower than the research question requires.
For example, randomly assigning patients to three named treatments does not mean the treatments are a random sample of treatments. Conversely, observing several clinics does not make them a random factor for inference about all clinics if the clinics were selected for convenience. In each case, ask what the study intends to say and how the factor levels entered the study.
These labels do not by themselves say whether the observations are independent, whether assignment was randomized, or whether the study supports causal conclusions. Those questions require their own design and model assumptions. Here the issue is narrower: what do the factor’s levels represent for the intended inference?
Three Questions That Expose the Error
When a factor seems difficult to classify, check the reasoning in this order. First, state the target of inference in ordinary language. Second, identify how the levels were chosen or assigned. Third, ask whether conclusions are meant to concern those named levels or a defined population from which levels were sampled. The selection mechanism matters because a random-factor interpretation relies on treating the observed levels as informative draws from that population.
- What is the target? Named treatments, sites, or judges point toward conclusions about those specific levels. A defined population of possible sites or judges may motivate population-level conclusions.
- What happened to the levels? Random assignment to study units is not the same as random sampling of factor levels from a population.
- What supports generalization? A claim about a population of levels needs a defensible connection between the observed levels and that population. Merely naming a population after data collection does not create that connection.
A factor’s status can also depend on the question, even when the data do not change. If a study observes six named hospitals and asks only how those six differ, the hospitals are treated as fixed for that question. If its target is variation across a defined population of hospitals, a random-factor interpretation may be appropriate only if the six observed hospitals were selected in a way that supports representing that population. The data table alone does not settle the choice.
Worked Example: Random Assignment Does Not Make Treatments Random
Worked Example: Random Assignment Does Not Make Treatments Random
A hypothetical trial compares three named pain-relief regimens, A, B, and C. Patients are randomly assigned to one of these regimens. The mean reduction in pain, measured in points on the study’s scale, is 10.2 for A, 12.1 for B, and 11.4 for C. A researcher calls the treatment factor random because assignment was randomized. Diagnose and correct the error.
The factor levels are the three specified regimens, and the research question is to compare these particular regimens. The random mechanism assigns patients to the levels; it does not sample the levels from a population of possible regimens. Thus treatment is fixed for this target.
For instance, the observed difference in mean reduction between B and A is:
This descriptive contrast is about the named regimens B and A. Random assignment may support a causal comparison of these treatments for the trial population under the relevant design and analysis assumptions, but it does not turn the three treatment labels into a random sample of treatments. Nor does the contrast alone establish that B is better in every population or setting.
Correction: classify the treatment factor as fixed when the inferential target is the three named regimens. Describe random assignment separately as a feature of how patients were allocated. If the scientific target instead concerns a broader population of possible regimens, that is a different inferential question and would require a design and model that support it.
Convenience Levels and Unsupported Generalization
A second common mistake is to call a grouping factor random simply because its levels are groups, locations, or people, then generalize to a broad population. For a random-factor interpretation, it is not enough that such a population can be imagined. The population must be defined, and the observed levels must have a credible sampling relationship to it. Convenience selection can be useful for a study, but it does not by itself make selected levels representative.
Worked Example: Six Conveniently Chosen Clinics
A hypothetical team studies waiting times at six clinics chosen because they are nearby and agreed to participate. Their mean waiting times are 4, 5, 6, 7, 8, and 10 minutes. The team wants to describe differences among these clinics and also considers claiming that the results represent all clinics in the region.
For the first target, the six clinics are the specific levels of interest. Their equally weighted average of the six reported means is:
This calculation summarizes the six observed clinic means; it does not establish that the average waiting time across all regional clinics is 6.667 minutes. For that broader target, the clinics were not selected through a sampling procedure that gives a defensible basis for representing the region. Calling clinic a random factor would not repair that limitation.
Correction: for conclusions limited to the six named clinics, treat clinic as fixed. To support inference about a population of regional clinics, define that population and use a level-selection process that supports treating the observed clinics as a sample from it. Even then, the model’s assumptions and the study’s other limitations still matter.
The key point is not that convenience samples can never be informative. They can describe the observed levels and may motivate further study. The error is to treat a convenient set as a probability sample from a target population without justification, then use the random-factor label as if it supplied the missing representativeness.
The Number of Levels Is Not Evidence of Sampling
A factor with only a few levels can be random if those levels were sampled from a clearly defined population and the goal concerns that population. A factor with many levels can be fixed if those specific levels are the target. Counting levels cannot reveal the selection mechanism or the inferential target. The following example holds the number of levels constant to show why the classification can change.
Worked Example: Five Judges, Two Different Targets
A hypothetical rating study uses five judges. Their mean ratings, in points, are 6, 5, 7, 8, and 9. Consider two versions of the research question, with the same judges and the same observed values.
Question 1: How do these five named judges differ? The target is the particular judges who took part. Treat judge as fixed for this question. The average of their five means is:
That calculation describes these judges’ mean ratings; it does not show that they represent other judges.
Question 2: What variation is there among qualified judges in a defined population? Suppose instead that the five judges were selected at random from a specified roster of qualified judges, and the aim is to learn about that roster’s judge-to-judge variation. The judge factor is now treated as random for this target: the observed judges are being used as a sample of levels from the defined population. The fact that there are still five levels has not changed; the target and selection process have.
If the same five judges had instead been chosen because they were available, the random-factor interpretation would not follow just from the second question. The sampling process would not support treating them as representative draws from the roster. A population target and a suitable selection basis must be considered together.
Why the Distinction Is Mathematically Meaningful
The difference can be seen by separating two sources of randomness. Let \(Z\) denote the assignment of study units to treatment labels, and let \(L\) denote the factor levels included in a study. In a randomized treatment experiment, \(Z\) can be random even when the set of treatment labels is fixed. In a random-factor setting, the levels \(L\) themselves are treated as sampled from a population of levels. These are different random processes.
The justification is direct: random assignment changes which unit receives which treatment, while leaving the treatment set unchanged. Thus, even if assignment outcomes have probabilities, no probability distribution over possible treatment levels follows from that fact. A random-level interpretation needs its own basis: a specified population and a sampling process that makes the observed levels draws from it. This separation explains why “the study was randomized” is not enough to classify every factor as random.
Likewise, a list of observed groups does not reveal how it was formed. Six groups selected randomly from a roster and six groups picked for convenience may have the same number of levels, but only the first selection can directly support a random-sampling interpretation for that roster. Whether its assumptions are reasonable remains a substantive question.
Common Misclassifications and Corrections
| Wrong reasoning | Why it fails | Correction |
|---|---|---|
| “Treatment assignment was random, so treatment is random.” | Assignment randomizes which units receive the specified treatments, not which treatment levels exist in the study. | Classify treatment from the target: named treatment levels are fixed when those specific comparisons are the goal. |
| “The levels are people or sites, so the factor must be random.” | Names or categories do not establish that observed levels were sampled from a defined population. | Identify the intended population and examine how levels were selected. |
| “We have many levels, so they represent a population.” | A large count does not correct biased or convenient selection. | Assess the selection process and target, not just the count. |
| “We want to generalize, so we can call the factor random.” | A desired generalization is not evidence that the observed levels support it. | State the population and explain what makes the observed levels informative about it. |
A useful diagnostic sentence is: “The factor is treated as fixed/random because the target is ___, and the levels were ___.” If the blanks cannot be filled with a specific target and an accurate account of selection, the classification needs more thought. In particular, do not use “random” as a synonym for “varies,” “was randomized,” or “might generalize.”
Check Your Understanding
For each situation, identify the tempting misclassification and state the correction.
- A study randomly assigns patients to four named therapy programs. Does random assignment alone make the therapy factor random? Explain what is randomized.
- A researcher selects three nearby schools for a study but wants to describe variation across all schools in the state. What information about level selection is missing?
- Five randomly selected laboratories are studied to learn about a defined population of laboratories. Why does the number five not by itself determine the factor’s status?
- The same four named hospitals are compared first as the only hospitals of interest, then as a claimed sample of all hospitals in a country. Explain why the target changes the classification question.
- A report says, “The factor is random because it has many levels.” Identify the error and give a more informative justification for classifying the factor.