Why the Study Design Must Fit the Question
A regression analysis cannot make a research question stronger than the study that produced its data. A carefully calculated relationship may describe only the people who were measured; a prediction may fail when used for a different group; and an observed association does not, by itself, show that changing one variable would change another. The study design determines what was measured, how the cases entered the data, and which interpretations are plausible.
The goal comes first. A descriptive question asks what outcomes or relationships characterize a defined population. An explanatory question asks how an outcome varies with a predictor, or whether a proposed change in a predictor would change the outcome. A predictive question asks how accurately an outcome can be estimated for cases in an intended use population. These goals can involve similar variables, but they require different design priorities.
Match the Design to the Goal
For a descriptive target, first define the population and the quantity to describe. A census measures every unit in that population. When a census is impractical, a sample can be used. A probability sample gives population units known chances of selection; when implemented well, it provides a principled basis for estimating population features. A sample selected by convenience may still describe the observed cases, but it does not automatically represent the wider population.
For an explanatory question, decide whether the target is an association or a causal effect. An observational study records the predictor and outcome as they occur, without assigning the predictor or a treatment. It can estimate and describe an association in the study data and, with suitable sampling, in a target population. It cannot establish that changing the predictor would cause the outcome to change. Other differences between the observed groups could help explain the pattern.
A causal question calls for a design that deliberately assigns a treatment or exposure and compares outcomes across assignment groups. Treatment assignment is central to making a causal interpretation plausible, but assignment alone is not a guarantee: how assignment is carried out and how the study is conducted matter. The design and its implementation must be examined before making a causal claim.
For a predictive question, the design should represent the cases on which predictions will be used and record predictors that would actually be available when a prediction is needed. Prediction performance should be assessed on cases that were not used to construct the prediction rule. Evaluating on the same cases used to build a model can make performance look better than it will be for new cases.
| Research goal | Design priority | Interpretation the design can support | Important limit |
|---|---|---|---|
| Description | Define the target population and obtain a census or an appropriate sample. | Population quantities, when the sampling process supports generalization. | A convenience sample may not represent the target population. |
| Explanatory association | Measure the outcome and predictor in the relevant units. | An association between predictor and outcome. | Observation alone does not establish a causal effect. |
| Causal explanation | Use a design with treatment or exposure assignment appropriate to the causal question. | A causal interpretation may be plausible when assignment and conduct justify it. | The label of the design alone does not validate the causal claim. |
| Prediction | Represent the intended-use cases and evaluate on cases not used to build the prediction rule. | Prediction accuracy for cases like those evaluated. | Performance may not carry over to a different population or use. |
A Short Justification for Probability Sampling
The reason a probability sample can support population description is that random selection connects the sample average to the population average. Consider a finite population of \(N\) units with fixed outcomes \(y_1,\ldots,y_N\), and suppose a simple random sample of size \(n\) is selected. Let \(I_i\) equal 1 if unit \(i\) is selected and 0 otherwise. Each unit has selection probability \(P(I_i=1)=n/N\). The sample mean can be written as \(\bar{y}_{sample}=\frac{1}{n}\sum_{i=1}^{N} I_i y_i\). Taking its expected value over repeated simple random samples gives:
Thus, under this sampling design, the sample mean is unbiased for the finite-population mean: over repeated samples, its average equals the population mean. This result does not say that every particular sample will match the population exactly. Nor does it account for nonresponse, measurement errors, or a mismatch between the sampled population and the population of interest; those issues can undermine a real study’s conclusions.
Worked Example: Estimating a Population Description
Suppose a university wants to describe the mean one-way commute time, in minutes, among all students enrolled this term. In a hypothetical simple random sample of five enrolled students, commute times are 12, 16, 18, 24, and 30 minutes. What does this design and sample say?
Identify the target. The target is the mean commute time for all students enrolled this term. The sample is intended to represent that population, and the stated simple random sampling design gives each student an equal chance of selection.
Calculate the sample estimate. The sample mean is
The sample estimate of the population mean commute time is 20 minutes. Because these data are hypothetical and the sample is very small, the calculation illustrates the interpretation rather than providing a reliable real-world estimate. Even with an actual probability sample, 20 minutes would be an estimate, not a guarantee that the population mean is exactly 20 minutes. A convenience sample of five students, by contrast, would not have the same design-based justification for generalizing to all enrolled students.
Worked Example: An Observed Association Is Not a Causal Effect
Imagine a hypothetical study that records study hours and quiz scores for six students. Researchers do not assign how long anyone studies. For illustration, the three students reporting lower study hours have scores of 62, 68, and 70 points; the three reporting higher study hours have scores of 78, 84, and 88 points. Compare the group means and interpret the design.
Lower study hours:
Higher study hours:
The difference in these sample means is \(83.33-66.67=16.66\) points, or about 16.7 points. In these hypothetical data, the group reporting more study hours also has the higher mean quiz score. Since study time was observed rather than assigned, the result describes an association in these six students. It does not show that increasing study time would cause a 16.7-point increase: other differences between the groups, as well as chance variation in this small sample, could contribute to the observed gap.
Worked Example: Evaluating Predictions for Intended-Use Cases
A hypothetical tutoring program wants predictions of quiz scores for students who will use its service. A prediction rule is built using one set of cases and then evaluated on three separate students from the intended-use population. The observed scores are 70, 82, and 76 points; the corresponding predicted scores are 68, 79, and 80 points. Calculate the mean absolute error for these evaluation cases.
For each case, the absolute prediction error is the absolute difference between the observed and predicted scores. The three errors are \(|70-68|=2\), \(|82-79|=3\), and \(|76-80|=4\) points. The mean absolute error (MAE) is the average of those absolute errors:
On these three hypothetical evaluation cases, predictions differ from observed scores by 3 points on average in absolute value. This is a small illustration, not a stable estimate of performance: three cases provide little information about the range of errors. The design is relevant to the intended use because the evaluation cases are separate from those used to build the rule and are described as coming from the population where predictions will be used. If the cases came from a substantially different population, or if the rule used information unavailable at prediction time, this evaluation would not justify the same interpretation.
A Practical Design-Choice Sequence
The following sequence helps keep the question and design aligned. It does not replace careful scrutiny of sampling, measurement, or data collection; it makes those decisions visible before results are interpreted.
State whether the intended result is a description, an association or causal explanation, or a prediction.
Identify the population and outcome, and state whether the target is a population summary, a relationship, or prediction performance.
For description, plan a census or a suitable sample. For explanation, distinguish observation from an intervention with assignment. For prediction, collect relevant predictors for the intended-use cases and reserve separate cases for evaluation.
State whether the result describes observed cases, supports generalization to a defined population, shows an association, supports a causal interpretation, or evaluates predictions for similar cases.
Common Mistakes and Why Design Matters
A familiar mistake is to see a strong pattern in observational data and describe it as the effect of changing a predictor. The pattern can be real and useful while still falling short of a causal conclusion. The correction is to name the result as an association unless the design and its implementation justify a causal interpretation.
Another mistake is to treat a large sample as automatically representative. A large group recruited through a channel that excludes part of the target population can still give a misleading population description. The number of cases does not repair a mismatch between the study sample and the population the researcher wants to describe.
For prediction, a common error is to report performance measured only on the cases used to build the prediction rule. Those cases have already influenced the rule, so their errors do not provide an independent check of how well it will work for new cases. Use separate evaluation cases and describe the population to which that evaluation is relevant.
Matching design to question matters because a regression result is not a free-standing statement about the world. Its meaning depends on who was measured, how cases were selected, whether predictors were observed or assigned, and whether the evaluated cases resemble those where predictions will be used. A sound interpretation says both what the analysis found and what the design permits the reader to conclude.
Check Your Understanding
For each question, identify a suitable design priority and the kind of conclusion it could support.
- A city wants to estimate the average weekly transit use among its residents. What population and sampling decision should be made before calculating a sample average?
- A study records exercise time and resting heart rate without assigning exercise. What interpretation of a relationship between them is supported, and what stronger interpretation is not established by observation alone?
- A researcher wants to know whether offering a study program changes exam scores. What broad kind of design is relevant to a causal question?
- A clinic builds a rule to predict appointment waiting times for future patients. Why should prediction errors be evaluated on cases not used to build the rule?
- Why might a very large convenience sample give a poor estimate of a population mean?