One Clinical Question, Three Statistical Objects
A trial protocol can name a treatment effect, specify a statistical model, and define which participants enter an analysis. These choices are related, but they are not interchangeable. The estimand states the treatment effect the clinical question targets. The estimator is the rule applied to data to estimate that effect. The analysis set is the collection of participants whose data are used in a specified analysis. Confusing these objects can make a technically correct calculation answer a different clinical question.
For example, suppose the question concerns the average effect of assignment to a new treatment rather than control, with outcomes collected even after treatment discontinuation or rescue medication. This is a treatment-policy question: those intercurrent events do not change the treatment condition being compared, and outcomes after them remain relevant. The analysis may use all observed outcomes, adjust for baseline severity, or model missing outcomes. Those are choices about estimation and data—not, by themselves, changes to the target effect.
The estimand’s target population is a feature of the clinical question; the analysis set is a feature of a particular calculation. They may coincide, but need not. For example, an analysis set may contain only participants with observed endpoint data, while its estimator uses baseline information to account for non-observation and target the population originally specified by the estimand. Conversely, including every randomized participant does not guarantee that the estimator targets the intended treatment effect: the endpoint, intercurrent-event strategy, or statistical model may still be misaligned.
In a randomized trial, an estimand can be expressed using potential outcomes. Let \(Y^1\) and \(Y^0\) denote the endpoint that a participant would have under assignment to treatment and control, respectively, with the specified treatment-policy strategy for intercurrent events. For a mean difference in a target population \(T\), the estimand is
This notation identifies the target quantity, not a particular model or dataset. Its interpretation depends on the attributes: changing the target population, endpoint, intercurrent-event strategy, or summary changes the clinical question and can therefore change \(\Delta_T\). Choosing a regression model to estimate \(\Delta_T\), by contrast, need not change the question if that model and its assumptions support the same target.
Different Estimators Can Target the Same Effect
A useful check is to ask what population each estimator averages over. If the target population has prespecified baseline subgroups, an estimator may estimate a treatment effect within each subgroup and combine those effects using the target population’s subgroup proportions. A regression estimator may instead predict outcomes under each treatment for every target-population participant and average those predictions. Both can target the same marginal effect when correctly specified, even though their calculations differ.
Worked Example: Standardizing to the Target Population
Hypothetical trial data. The target population is 60% mild and 40% severe at baseline. The endpoint is a symptom score at week 12, measured regardless of discontinuation or rescue medication; lower scores are better. The target is the treatment-policy mean difference, treatment minus control. Suppose the observed subgroup means are as follows.
| Baseline severity | Target proportion | Treatment mean (points) | Control mean (points) |
|---|---|---|---|
| Mild | 0.60 | 8 | 10 |
| Severe | 0.40 | 12 | 16 |
One estimator standardizes the treatment and control means to the target population’s severity distribution. The standardized treatment mean is \(0.60(8)+0.40(12)=4.8+4.8=9.6\) points. The standardized control mean is \(0.60(10)+0.40(16)=6.0+6.4=12.4\) points. Thus the estimated treatment-policy mean difference is \(9.6-12.4=-2.8\) points.
A second estimator fits a regression with treatment, severity, and their interaction, then predicts each treatment mean for the target population and averages using weights 0.60 and 0.40. With these subgroup means, it gives the same calculation and the same estimate, \(-2.8\) points. The procedures differ, but both target the severity-standardized marginal mean difference.
A crude comparison of the observed arm means need not do so. Suppose, hypothetically, the treatment analysis set contains 80 mild and 20 severe participants, while the control analysis set contains 40 mild and 60 severe participants. The crude treatment mean is \(0.80(8)+0.20(12)=8.8\); the crude control mean is \(0.40(10)+0.60(16)=13.6\). The crude difference is \(8.8-13.6=-4.8\) points. It uses different severity weights from the target population, so it does not estimate the same standardized contrast in this example. Standardization is needed to align the estimator with the stated target.
The subgroup means must themselves be supported by adequate data and a suitable estimation procedure. In an actual trial, uncertainty in the subgroup estimates and target weights also matters; the arithmetic here illustrates the target, not a claim that the displayed means are known without error.
Why Analysis Sets and Estimands Must Be Kept Apart
An analysis set can be defined for a purpose—for example, all randomized participants for an efficacy analysis, participants who received treatment for a safety analysis, or participants meeting specified adherence criteria for a per-protocol analysis. Such labels do not, on their own, identify the estimand. The question is how the estimator uses the selected data and whether the resulting inference refers to the target population and treatment conditions in the estimand.
A complete-case estimator uses participants with observed endpoint values. If outcome observation is related to the outcome, the observed participants may differ systematically from the target population. A per-protocol analysis that excludes participants who discontinue likewise selects a post-randomization subset. Comparing selected participants can introduce bias because treatment assignment, prognosis, and adherence may be related. These calculations may be informative for other questions, but they do not automatically estimate the treatment-policy effect among the original target population.
This point can be stated directly. Let \(R^a\) indicate whether the endpoint would be observed under assignment \(a\). A complete-case arm mean estimates a mean among those observed in that arm, such as \(E[Y^a\mid R^a=1]\), rather than necessarily the target mean \(E_T[Y^a]\). The two means coincide under suitable assumptions, such as outcome observation being independent of the potential outcome within the relevant population. When observation depends on measured baseline variables, a method that adjusts for those variables may be appropriate under additional assumptions. When observation depends on unobserved outcomes even after conditioning on available information, the data alone generally do not identify the target mean without further assumptions.
Missing Outcomes: Two Procedures, One Target
Missing outcome data are distinct from intercurrent events. Discontinuation or rescue medication is an event in the participant’s treatment course; an endpoint not collected after that event is missing data. Under a treatment-policy estimand, outcomes after the event remain part of the target, so the estimator must address any unobserved values rather than silently redefining the endpoint to stop at the event.
Worked Example: Standardization Under a Missing-at-Random Assumption
Hypothetical data. A trial targets the treatment-policy mean difference in a week-12 score for a population with equal proportions in baseline groups A and B. Some scores are missing. Among participants with observed scores, group means are: treatment, 6 points in A and 10 in B; control, 8 points in A and 12 in B. Suppose the endpoint is missing at random conditional on treatment and baseline group, the response model is correctly specified, and each group has a positive probability of observed follow-up.
A model-based imputation procedure can use the observed group means to predict missing scores within each treatment-by-group cell. Its expected completed-data means are then standardized to the target’s 50%/50% group distribution. For treatment, the standardized mean is \(0.50(6)+0.50(10)=8\) points. For control, it is \(0.50(8)+0.50(12)=10\) points. The estimated target treatment-policy difference is \(8-10=-2\) points.
A response-weighted or stratified estimator can target the same contrast by estimating each treatment-by-group mean from respondents and applying the same target weights. In this simplified example, it also gives \(-2\) points. These procedures are not identical: their uncertainty calculations and sensitivity to model choices can differ. Their shared target follows from the same endpoint, treatment-policy strategy, target population, and standardization weights. The missing-at-random assumption is substantive; it cannot be verified from observed outcomes alone.
For comparison, the hypothetical observed counts are 40 of 50 treatment participants in A, 20 of 50 in B, 45 of 50 control participants in A, and 40 of 50 in B. The unstandardized respondent means are \((40(6)+20(10))/60=440/60=7.333\) points for treatment and \((45(8)+40(12))/85=840/85=9.882\) points for control. Their difference is approximately \(-2.549\) points. That crude respondent contrast uses the respondents’ unequal group mixtures, not the target’s equal mixture. It is not the standardized estimate above.
When Selection Changes the Question
Worked Example: An Adherer-Only Comparison
Hypothetical data. A trial randomizes 100 participants to each treatment condition. The intended estimand is the treatment-policy difference in mean week-12 score for all randomized participants, including outcomes after treatment discontinuation. Suppose a justified missing-data procedure estimates the treatment-arm mean as 5.4 points and the control-arm mean as 7.0 points. The estimated treatment-policy difference is \(5.4-7.0=-1.6\) points.
Now suppose an analysis excludes everyone who discontinued, leaving 90 participants per arm, with mean scores of 5 points among treatment adherers and 7 points among control adherers. The adherer-only difference is \(5-7=-2\) points. This is a numerical contrast among selected participants; it is not automatically an estimate of the all-randomized treatment-policy effect. Discontinuation occurs after randomization, and the adherers in the two arms need not represent comparable groups.
The \(-2\)-point result could be used to address a different target only if that target and the assumptions needed to identify it were explicitly specified. Merely naming the analysis set “per protocol” does not establish that interpretation. For the original treatment-policy question, an estimator must account for outcomes after discontinuation or otherwise justify how the missing information is handled.
A practical alignment check is to write down the estimand before choosing the estimator: who is the target population, what treatment conditions are compared, what endpoint is summarized, how are intercurrent events handled, and what summary is required? Then state the analysis set and the estimator separately. For each restriction or model, ask whether its assumptions connect the observed data back to that target.
| Object | Question it answers | Example |
|---|---|---|
| Estimand | What treatment effect is the clinical question asking for? | Mean week-12 score difference under a treatment-policy strategy in the target population |
| Estimator | What rule turns data into an estimate of that effect? | Regression standardization or response weighting |
| Analysis set | Whose data enter this particular calculation? | All randomized participants, or participants with observed endpoint data |
| Estimate | What numerical result did the rule produce? | A treatment-minus-control difference of \(-2.8\) points |
No estimator is universally best. A crude comparison is simple and may be suitable when its implicit weights match the target and relevant data assumptions hold. Covariate standardization makes the target weights explicit but depends on adequate subgroup data or a suitable model. Likelihood, imputation, and weighting procedures can use different assumptions and have different diagnostics. The preferred choice is the procedure whose assumptions are credible, whose target is explicit, and whose limitations can be examined—not a method chosen solely because its analysis set sounds inclusive.
Diagnostics include checking outcome availability by treatment and prognostic baseline variables, confirming that each target subgroup has observed participants, and comparing crude results with target-standardized analyses. Large differences are a reason to investigate composition, model specification, and missingness assumptions; they do not by themselves identify which estimate is correct. If some target subgroups have no observed outcomes, or if missingness depends on unobserved outcomes, ordinary adjustment may not recover the target without extrapolation. Such limitations should be stated rather than hidden by a label such as “intention to treat.”
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A trial’s target is the mean treatment-policy effect in all randomized participants. Only participants with observed week-12 scores enter a complete-case comparison. Does the analysis set redefine the estimand automatically?
- A target population is 70% group A and 30% group B. Treatment-minus-control subgroup effects are \(-1\) point in A and \(-5\) points in B. What is the standardized target effect?
- State one assumption under which a complete-case arm mean can estimate the target arm mean, and explain why randomization alone does not guarantee it.
- Why can an adherer-only comparison fail to estimate a treatment-policy effect?
Solutions. (1) No. The estimand remains the stated all-randomized treatment-policy effect; the complete-case estimator uses a restricted analysis set and needs assumptions or adjustment to connect observed outcomes to that target. (2) The standardized effect is \(0.70(-1)+0.30(-5)=-0.7-1.5=-2.2\) points. (3) If outcome observation is independent of the potential outcome in the target population and has positive probability, the observed mean equals the target arm mean. Randomization balances treatment assignment, but does not ensure that endpoint observation is independent of outcomes. (4) Adherence is post-randomization. Excluding nonadherers can select participants differently across arms, so the remaining groups need not represent the original target population or support the treatment-policy contrast.