Why Estimand Documentation Needs Precision
An estimand is the precise treatment-effect question a trial is designed to answer. A protocol may name an outcome and a follow-up time yet leave important choices unstated: which patients the question concerns, what treatment conditions are compared, or how events such as rescue therapy are handled. Such gaps make it possible for the protocol, analysis plan, and report to appear consistent while answering different questions.
Documentation is therefore not a clerical step after the statistical design. It records the target quantity that the design, data collection, analysis, and interpretation are meant to serve. In a Bayesian trial, the model and prior distribution produce a posterior distribution for the target quantity; they do not replace the need to define that quantity. The posterior summary should be reported as evidence about the documented estimand, not as a new estimand introduced by the analysis.
A complete statement makes each attribute interpretable without relying on a reader to infer design choices. The population identifies who the treatment effect concerns; treatment conditions specify what is compared; the outcome attribute states what is measured and when; intercurrent-event strategies specify how events affecting interpretation or existence of an outcome are handled; and the summary defines the contrast or other population-level quantity. Earlier material on estimands, estimators, and analysis sets distinguishes the target quantity from the procedure used to estimate it. Keep that distinction visible in every document.
A Drafting Sequence for the Five Attributes
A reliable draft starts with the scientific question and turns each of its components into an explicit attribute. Name the population in clinical terms and connect it to the planned eligibility criteria. Describe treatment conditions sufficiently to identify the assigned regimens, including dose, route, schedule, and comparator when relevant. Specify the outcome, measurement method, and time point or window. For each intercurrent event that matters, name the event and state the strategy used to define the treatment effect after it occurs. Finally, state the scale and population-level summary, such as a risk difference or a difference in mean outcome.
For example, a statement that “the treatment effect on symptom improvement will be assessed” does not specify a population, comparison, timing, event strategy, or effect scale. A usable statement is longer because it answers those questions directly. It need not be a single unwieldy sentence: a concise lead sentence and a structured table can together document all five attributes.
Write, in plain language, who is being compared, on what outcome, and for what clinical decision.
Specify the population, treatment conditions, outcome, intercurrent-event strategies, and population-level summary. Do not leave an attribute to be inferred from the model or analysis method.
Document the estimand first. Then describe the estimator, analysis population, model, prior, and missing-data assumptions as methods for learning about that target.
Check that the protocol, statistical analysis plan, and results report preserve the same attribute values and effect scale.
Worked Example: Drafting a Binary-Outcome Estimand
Hypothetical trial. A randomized trial compares a new oral medicine with placebo in adults with moderate chronic migraine. The outcome is whether a participant achieves at least a 50% reduction in monthly migraine days from baseline at week 24. Participants may discontinue assigned treatment or use rescue medication before week 24. Draft a complete estimand statement using a treatment-policy strategy for both events.
Population. Adults meeting the protocol’s specified moderate chronic migraine eligibility criteria at randomization. This names the population to which the trial’s target effect applies; it does not automatically imply generalization to patients outside those criteria.
Treatment conditions. Assignment to the specified new oral medicine regimen versus assignment to matching placebo, with the same background-care rules in both groups. Stating assignment clarifies that the contrast concerns treatment strategies as randomized, rather than a comparison limited to participants who adhered to treatment.
Outcome and time. A binary indicator of achieving at least a 50% reduction in monthly migraine days from baseline, assessed at week 24 using the protocol-defined diary and assessment window.
Intercurrent events. Use a treatment-policy strategy for discontinuation of assigned treatment and for rescue medication: the week-24 outcome is included regardless of whether either event occurs. This specification requires the outcome to remain relevant and to be collected after these events where possible. It does not mean that missing week-24 outcomes are automatically observed or that missing-data assumptions can be ignored.
Population-level summary. Define \(Y^a\) as the binary week-24 outcome under assignment to condition \(a\), with the stated treatment-policy handling, where \(a=1\) denotes the new medicine and \(a=0\) denotes placebo. The target is the difference in response probabilities:
The full statement is: In adults meeting the trial’s moderate chronic migraine eligibility criteria, compare assignment to the specified new oral medicine regimen with assignment to matching placebo, and estimate the difference in the probabilities of achieving at least a 50% reduction in monthly migraine days from baseline at week 24. Use a treatment-policy strategy for treatment discontinuation and rescue medication. This identifies all five attributes, with the difference in response probabilities specifying the population-level summary.
Keeping Protocol, Analysis Plan, and Report Aligned
The three documents serve different purposes, so their wording need not be identical. The protocol establishes the planned scientific target and trial procedures. The statistical analysis plan (SAP) explains how the data and model will be used to estimate that target. The results report presents the estimated effect and uncertainty, with enough context to show what quantity the results concern. Consistency means semantic agreement across these roles, not copying a paragraph verbatim.
| Document | What it should make clear | What it should not silently change |
|---|---|---|
| Protocol | The five attributes, clinical rationale, outcome collection, and planned estimand | Population, treatment conditions, outcome definition or time, event strategies, or effect scale |
| SAP | How the analysis estimates the protocol estimand, including model, prior, analysis population, and missing-data approach | The target population or treatment effect merely because a different analysis method is convenient |
| Results report | The estimand being reported, the analysis used, and the estimated effect with appropriate uncertainty | The effect scale, outcome timing, or event strategy when describing the primary result |
A useful audit is to extract the five attributes from each document into a comparison table. For each row, ask whether the same target is described, then separately check whether the SAP and report explain their methods accurately. A model may estimate a marginal treatment effect even if it includes covariates; conversely, a conditional model coefficient does not automatically equal a marginal estimand. The model’s interpretation must support the quantity named in the protocol.
In Bayesian reporting, name the estimand and then state the posterior result for it. For example, report a posterior mean or median and a credible interval on the risk-difference scale, with the outcome, population, and treatment contrast identified. A posterior probability of exceeding a clinically relevant threshold may be a decision summary for the same estimand, but it does not excuse omitting the estimand’s five attributes.
Worked Example: Aligning Three Documents With a Hypothetical Result
Continue the hypothetical migraine trial. Suppose the protocol defines the risk difference in week-24 response probabilities under the treatment-policy strategies above. The SAP specifies a Bayesian model estimating the two response probabilities and their difference in the randomized population. For illustration, suppose the analysis reports posterior mean response probabilities of 0.62 for the new medicine and 0.54 for placebo, and a 95% credible interval for the risk difference of \([-0.01, 0.17]\).
Check the arithmetic. The difference between the reported posterior mean probabilities is \(0.62-0.54=0.08\), or 8 percentage points. This is a posterior mean risk difference if the two probabilities are posterior means under the same model and the difference is computed draw by draw; in general, the posterior mean of a difference equals the difference of posterior means when both summaries refer to the same joint posterior.
Protocol wording. State the five-attribute target as in the preceding example. The protocol should not describe the target only as “response rate” because that would omit the comparison and effect scale.
SAP wording. State that the model estimates the week-24 response probabilities and their risk difference for the specified randomized population and event strategies. Describe the prior, model, missing-outcome assumptions, and computational method separately. These are analysis details, not substitutions for the estimand.
Results wording. A consistent report might say: “For adults meeting the trial eligibility criteria, the posterior mean risk difference in week-24 response probability, comparing assignment to the new medicine with placebo under the stated treatment-policy strategies, was 0.08 (8 percentage points); the 95% credible interval was \(-0.01\) to \(0.17\).” The interval includes zero, and its endpoints correspond to a difference from 1 percentage point lower to 17 percentage points higher for the new medicine. It is a Bayesian credible interval for the risk difference under the specified model and prior, not a guarantee that the effect is beneficial.
Consistency check. If the report instead calls 0.08 a conditional odds ratio, reports a week-12 outcome, or excludes everyone who used rescue medication, it no longer describes the documented estimand. Such a result might answer a different question; it must not be presented as though it were the planned primary estimand.
Justification: Why Each Attribute Must Be Traceable
The need for an explicit five-part statement follows from how a treatment effect is constructed. A population-level summary operates on outcome quantities defined for a population under specified treatment conditions and event strategies. Changing any of these inputs can change the target quantity, even if the trial name and outcome label stay the same.
For example, a risk difference cannot be uniquely interpreted without knowing which population’s response probabilities are being contrasted, which treatment conditions define those probabilities, and what counts as a response at which time. If rescue medication is handled by a treatment-policy strategy in one document and by a hypothetical strategy in another, the outcome distributions being compared are different. Therefore, a mismatch in any attribute is not merely a wording defect: it can change the statistical target. The estimator and posterior distribution are interpretable as evidence about the intended estimand only when their relationship to that target is clear.
Worked Example: Repairing an Incomplete and Contradictory Description
Hypothetical documentation set. The protocol says the primary outcome is “change in pain score at week 12.” The SAP says it will estimate an adjusted difference in mean change among participants who completed assigned treatment. The report says the primary result is the difference in mean week-12 pain score among all randomized participants, regardless of treatment discontinuation. No document specifies the target population beyond “patients with pain,” and rescue medication is not addressed.
Diagnose the gaps. The population is vague; treatment conditions are not fully identified; the outcome needs a scale, direction, assessment method, and exact time definition; the handling of rescue medication is absent; and the population-level summary is not consistently specified. The SAP’s completer restriction also conflicts with the report’s all-randomized description. Thus, the three documents cannot be assumed to report one estimand.
Make a prospective specification. For this hypothetical repair, suppose the intended target is adults meeting the protocol’s chronic-pain eligibility criteria, assigned to a named active regimen or matching placebo, with mean change from baseline in a specified 0–10 pain score at week 12 as the outcome. Specify a treatment-policy strategy for discontinuation and rescue medication, and choose the mean difference in change between assigned groups as the summary.
Write the repaired statement. “In adults meeting the trial’s chronic-pain eligibility criteria, compare assignment to the specified active regimen with assignment to matching placebo. The outcome is change from baseline in the 0–10 pain score at week 12, assessed in the protocol-defined window. Use a treatment-policy strategy for treatment discontinuation and rescue medication. The estimand is the difference between treatment groups in the population mean change.” The hypothetical SAP must then describe a method intended to estimate this population mean difference; a restriction to treatment completers would require an explicit rationale and would not automatically estimate the stated target. The report must label the result as that mean difference and identify the same outcome and event strategies.
Diagnostics, Alternatives, and Limitations
Documentation problems are often detectable before analysis. Look for vague phrases such as “clinical response,” “as treated,” or “at follow-up” without an operational definition. Search all documents for the outcome time point, treatment labels, analysis population, and event strategies; compare their meanings rather than only their terminology. Confirm that outcome collection after treatment discontinuation is planned when the chosen strategy requires it.
A structured five-row table is often the clearest primary record because it makes omissions visible. A prose statement is more compact and may communicate the clinical question more naturally, but it is easier to leave an attribute implicit. The strongest approach commonly uses both: a concise prose statement for readers and a table for operational detail. Neither format alone ensures the analysis matches the target.
Several distinctions help avoid false consistency:
- Intercurrent event versus missing outcome. An intercurrent event is an event after treatment initiation that affects interpretation or existence of the outcome; missing outcome data are unavailable measurements. A strategy for rescue medication does not, by itself, specify how an unobserved week-12 score will be handled.
- Estimand versus estimator. The estimand is the target quantity; the estimator or model is the procedure used to learn about it. A different model does not necessarily change the target, but its assumptions must support estimation of that target.
- Estimand versus analysis set. The set of records included in an analysis is not automatically the target population. Exclusions or availability-based restrictions can affect what is learned and may create a mismatch.
- Outcome scale versus decision threshold. A risk difference, risk ratio, mean difference, and posterior probability of exceeding a threshold answer distinct questions. Name the primary summary explicitly.
A limitation of any written estimand is that documentation cannot guarantee identification or reliable estimation. Outcome measurements may be missing, the model may poorly represent the data, or assumptions needed to connect observed outcomes to the target may be implausible. A complete statement makes those issues assessable; it does not resolve them. Nor should apparent alignment be achieved by leaving the target vague. Precision about the target is what allows readers to judge whether the model, data collection, and interpretation are suitable.
Check Your Understanding
Answer each question by identifying the relevant estimand attribute or documentation issue. Brief solutions follow.
- A protocol names “week-24 response” but gives no response threshold. Which attribute is incomplete, and why does it matter?
- The protocol uses a treatment-policy strategy for rescue medication, while the SAP analyzes only participants who never used rescue medication. Is this automatically consistent? Explain.
- A report gives a posterior mean risk difference and a credible interval but does not identify the population or outcome time. What is missing from the report?
- Explain why collecting no outcomes after treatment discontinuation may create a problem for a treatment-policy strategy.
- Suppose the protocol specifies a mean difference in outcome change, while the report gives an odds ratio for a binary responder outcome. What should a consistency audit conclude?
Solutions. (1) The outcome attribute is incomplete because the threshold determines which participants count as responders and therefore changes the response probabilities and their contrast. (2) No. Restricting to those who did not use rescue medication changes the analyzed group and may not estimate the treatment-policy estimand; the SAP needs a method that targets the stated question or must clearly specify a different target. (3) The report should identify the target population and the time point, and should also make the treatment contrast, event strategies, outcome definition, and effect scale clear enough to link the posterior summaries to the estimand. (4) Treatment policy defines how the event is handled in the target question, but does not create an unobserved outcome. If outcomes after discontinuation are not collected, missing-data assumptions or methods are needed to estimate the target. (5) The reported quantity does not match the stated estimand: both the outcome form and population-level summary differ. The audit should flag a target mismatch rather than treat the odds ratio as another label for the mean difference.