Why an Estimand Can Be Internally Incoherent
A trial can report a precisely calculated treatment effect that answers the wrong clinical question. This can happen when the protocol’s stated question concerns one population, outcome, or treatment condition, but the estimand specifies another. The remedy is not to begin with a preferred analysis method. First, audit whether the estimand’s components fit together and express the intended question.
An estimand is a precise description of the treatment effect of interest. The earlier material on estimand specification tables provides a structured way to record its components. Here the task is to test whether those components are mutually consistent: does the defined population experience the stated treatment conditions, is the outcome meaningful for that population and question, is each intercurrent event handled in a way that matches the question, and does the summary contrast compare the intended conditions?
The audit is about the question before it is about the estimator. An estimator is the procedure used to estimate an estimand from trial data; an estimand defines the target quantity. A convenient observed-data summary cannot, by itself, justify changing the target population or outcome. Likewise, a statistical model does not resolve a contradiction between a stated question and the chosen event strategy.
A Component-by-Component Audit
Begin by writing the clinical question in ordinary language: who is of interest, which treatment conditions are compared, what outcome matters, when it is assessed, and what should happen to the interpretation if an intercurrent event occurs. Then compare that statement with each estimand component. A component is coherent when it has a clear interpretation and agrees with the question; it is not enough for each component to be individually plausible.
- Target population: Are these the patients to whom the clinical question refers? Check for silent changes, such as defining the question for all randomized patients but specifying an estimand only for patients who remain on treatment.
- Treatment conditions: Are the compared conditions the treatments described in the question, including relevant background or post-randomization conditions? Do not substitute a selected subgroup’s treatment experience for the assigned treatment comparison without stating that change.
- Outcome variable and time point: Does the variable represent the clinical outcome of interest, and is it defined at the stated time? Check its scale, direction, and whether it can be meaningfully defined for every person in the target population.
- Intercurrent-event strategy: Is each event handled in a way that matches what the question asks? For example, a treatment-policy strategy includes outcomes regardless of the event, whereas a hypothetical strategy asks about outcomes under a specified scenario in which the event did not occur.
- Population-level summary and contrast: Does the summary measure and effect scale correspond to the intended comparison? A difference in means, ratio of means, or difference in risks answers a question on a particular scale; the scale must be stated and clinically interpretable.
An intercurrent event is an event after treatment initiation that affects either the interpretation or existence of measurements needed for the treatment effect, such as discontinuation or rescue medication. Its strategy is not the same thing as the procedure for handling a missing measurement. If a patient discontinues treatment and the scheduled outcome is not recorded, discontinuation is the event and absence of the measurement is missing data. The event strategy defines the target quantity; missing-data assumptions and methods concern how that quantity can be estimated.
Worked Example: Rescue Medication and a Mismatched Event Strategy
Hypothetical question: A trial asks whether assignment to Drug A rather than placebo improves mean pain score at week 12 in all randomized patients, including those who use rescue medication. Lower scores are better, and the intended treatment-policy question includes outcomes after rescue medication. The protocol instead defines its outcome as week-12 pain score “before any rescue medication” and compares the mean of that outcome between randomized groups.
Audit. The target population and treatment conditions are clear, and mean pain score is a defined outcome. The mismatch is between the clinical question and the event strategy: “including those who use rescue medication” calls for outcomes regardless of rescue medication, while the protocol’s pre-rescue outcome does not represent the stated week-12 outcome for people who use rescue medication. The definition has silently changed the outcome and the question.
Suppose, hypothetically, the randomized groups each contain 100 patients. At week 12, the treatment-policy outcome is observed for 100 patients in each group, with means of 4.2 points for Drug A and 5.1 points for placebo. The stated contrast, Drug A minus placebo, is
This is the hypothetical treatment-policy contrast in mean week-12 pain score. It is not interchangeable with a contrast based on measurements before rescue medication. Those would be different outcome definitions and potentially different time points.
Repair. If the clinical question is to include rescue medication, define the outcome as week-12 pain score regardless of rescue use and specify a treatment-policy strategy for rescue medication. If the intended question is instead the outcome in a scenario without rescue medication, state that hypothetical question and specify the hypothetical strategy. The repair depends on which question clinicians actually intend; one should not pick the strategy just because a particular analysis is easier.
Assumption and check. The arithmetic requires the stated hypothetical data to be week-12 scores measured on the same scale in the two randomized groups and defined for the specified population. The example assumes those data are available for all 100 patients in each group; it does not claim that this would always occur in a real trial.
Deriving a Repair Rather Than Merely Flagging a Problem
A useful audit ends with a revised specification that removes the contradiction while preserving the intended question. The repair should identify the exact component that changes, then check the remaining components again. If the clinical question itself is ambiguous, record the competing interpretations rather than silently selecting one.
Name the population, treatment comparison, outcome, assessment time, and intended meaning of relevant intercurrent events.
Identify where the population, treatment conditions, outcome, event strategy, or summary contrast differs from the plain-language question.
When two components conflict, do not resolve the conflict by statistical convenience. Clarify whether the question or the component should change.
State the repaired component explicitly, then verify that all components can describe one treatment effect for the same target population and outcome.
This process is a consistency check, not a recipe that selects one universally preferred estimand. Different event strategies can be coherent answers to different clinical questions. The problem is claiming that a specification answers one question when its components define another.
Worked Example: Completers Are Not Automatically the Target Population
A hypothetical trial’s clinical question concerns the mean outcome in the full target population, but a report gives only the mean among patients with an observed outcome. Let \(R=1\) indicate that the outcome is observed and \(R=0\) indicate that it is not observed. Suppose \(P(R=1)=0.80\), the mean among observed patients is \(m_1=10\) points, and, for illustration, the mean in the unobserved group under the target outcome definition is \(m_0=6\) points.
When the outcome is defined for both groups, the target-population mean is the mixture of the two conditional means:
Substituting the hypothetical values gives
Thus, the observed-group mean of 10 points is not the target-population mean of 9.2 points in this example. More generally, if \(p=P(R=1)\), the difference between the target mean and the observed-group mean is
When both groups exist, \(0<p<1\), so this difference is zero if and only if \(m_0=m_1\). If everyone has an observed outcome, \(p=1\), the target and observed-group means are also equal. Therefore, a reported completer mean alone does not establish equality with the target-population mean.
Audit lesson. A target-population question cannot be replaced by a completer-only summary merely because only completer outcomes are reported. The mixture identity clarifies the distinction, but it does not supply an unobserved-group mean from the observed data. The example’s assumed \(m_0=6\) is illustrative; it is not inferred from the observed-group mean.
Population, Outcome, and Contrast Must Fit Together
Some incoherencies are less obvious than an event-strategy contradiction. A population may be described as all randomized patients while the outcome is defined only for people who survive to assessment. That can make the outcome undefined for part of the target population. One must clarify the clinical question and define a suitable outcome or population; simply omitting those patients changes what is being summarized.
A treatment contrast can also be internally inconsistent. For example, a question may ask for the difference in mean change from baseline, but the specified contrast compares end-of-study means. Those quantities are not generally identical, even when both use the same scale. The audit should name the exact summary and contrast, including direction. If Drug A minus placebo is the contrast, a negative difference means a lower mean outcome under Drug A, which is favorable only if lower values are clinically better.
Worked Example: Repairing a Population and Contrast Mismatch
Consider a hypothetical protocol that says, “Estimate the difference in mean improvement at week 8 between Treatment B and standard care among all randomized patients.” It then defines the analysis population as patients who completed eight weeks and defines the contrast as the mean week-8 score under Treatment B minus the mean week-8 score under standard care. Improvement is defined as baseline score minus week-8 score, so a positive improvement is favorable.
Population mismatch. “All randomized patients” and “patients who completed eight weeks” are different target populations. If the intended question is about all randomized patients, the target population should remain all randomized patients. Completion status alone does not redefine the clinical target.
Contrast mismatch. The protocol asks about improvement but specifies a contrast in week-8 scores. Suppose, hypothetically, the all-randomized population has mean improvement of 6 points under Treatment B and 4 points under standard care. The intended improvement contrast, Treatment B minus standard care, is
Suppose the corresponding baseline means are 20 points in each condition. Then the week-8 means are 14 and 16 points, respectively, and the week-8 score contrast is
The numerical signs differ because improvement was defined as baseline minus week-8 score, whereas the specified score contrast was week-8 Treatment B minus week-8 standard care. In this particular example, the magnitudes encode the same separation because the baseline means are equal. That coincidence does not make the two definitions interchangeable in general.
Repair. If the question concerns all randomized patients and mean improvement, specify that population and define the contrast as mean improvement under Treatment B minus mean improvement under standard care. If the intended target is completers or the intended outcome is week-8 score, revise the clinical question to say so. Do not retain the all-randomized wording while quietly changing the population or outcome.
Assumption and check. The arithmetic assumes the hypothetical mean improvements and baseline means refer to the same population and treatment conditions, and that improvement is baseline minus week-8 score in both groups. The example does not assert that the two treatment-condition means for each person are simultaneously observed; they are population-level summaries under the stated conditions.
Diagnostics, Alternatives, and Limitations
A practical diagnostic is to ask whether a clinician could read the complete specification and explain, without qualification, whose outcomes are summarized, under which treatment conditions, at what time, and how each relevant event affects the question. A second diagnostic is to paraphrase the estimand in one sentence and compare it with the protocol’s stated objective. If that paraphrase changes “all patients” to “completers,” “regardless of rescue” to “without rescue,” or “improvement” to “final score,” the mismatch should be resolved.
Other warning signs include undefined outcomes for some members of the target population, an event strategy that is not stated for an event mentioned in the clinical question, a treatment contrast with an unstated direction, or a claimed treatment-policy interpretation based only on outcomes collected before an event. These are prompts for clarification, not proof that one particular estimand is correct. Also distinguish an incoherent estimand from an estimand that is coherent but difficult to identify or estimate: the latter raises separate questions about data and assumptions.
The main alternatives are not competing calculation techniques but different clinical questions. A treatment-policy question asks about outcomes regardless of an intercurrent event; a hypothetical question asks about outcomes under a specified scenario in which the event did not occur. Each can be coherent if its population, outcome, conditions, and summary are specified consistently. The earlier material on hypothetical estimands and treatment-policy questions addresses those strategies; the audit here checks whether the selected strategy matches the question.
An audit cannot decide clinical priorities on behalf of investigators, recover unobserved outcomes, or guarantee that the chosen estimand is identifiable from the available data. Nor does a well-aligned specification guarantee a reliable estimator. Its value is narrower and essential: it makes the target question explicit, exposes contradictions before results are interpreted, and ensures that later analysis is judged against the quantity the trial actually meant to estimate.
Check Your Understanding
For each question, identify the relevant coherence issue and compare your reasoning with the solutions.
- A question asks for a treatment effect in all randomized patients regardless of discontinuation, but the estimand is defined among patients who remain on treatment. What is the mismatch, and what must be clarified?
- A protocol asks for mean improvement, defined as baseline score minus follow-up score, but specifies follow-up Treatment A minus follow-up control as its contrast. Why is this a mismatch unless further conditions are stated?
- Suppose \(P(R=1)=0.75\), \(m_1=12\) points, and \(m_0=8\) points. Calculate the target mean and state whether it equals the observed-group mean.
- With \(0<P(R=1)<1\), under what condition does the target-population mean equal the mean among those with observed outcomes?
- Explain why defining a treatment-policy estimand does not, by itself, solve the problem of a missing outcome after an intercurrent event.
Solutions. (1) The stated question includes all randomized patients regardless of discontinuation, whereas the specified population selects those remaining on treatment. Clarify whether the target is all randomized patients or a treatment-persistence subgroup, and align the question and specification. (2) Improvement and follow-up score are different outcome summaries. The follow-up contrast need not equal the improvement contrast unless additional conditions, such as equal baseline population means for the comparison at issue, make them coincide; specify the intended outcome and contrast directly. (3) The target mean is \(0.75(12)+0.25(8)=9+2=11\) points, which differs from the observed-group mean of 12 points. (4) The identity \(\mu-m_1=(1-p)(m_0-m_1)\), with \(p=P(R=1)\), shows that when \(0<p<1\), equality holds exactly when \(m_0=m_1\). (5) The strategy defines the target quantity as including outcomes regardless of the event, but an outcome may still be unobserved. A separate estimation approach and assumptions are needed to address that missing measurement; the strategy does not supply its value.