Why Audit a Protocol Revision?
An adaptive trial may need an operational change: eligibility criteria may be clarified, a treatment may be delivered differently, follow-up may be rescheduled, or an endpoint may be collected with a revised instrument. The important question is not simply whether the protocol changed. It is whether the change alters the clinical question represented by the estimand, or instead affects the ability to answer that question.
A useful audit makes that distinction explicit before the revision is implemented. It records the target population, treatment conditions, outcome variable, strategy for intercurrent events, and summary measure, then asks which of those attributes the revision touches. The introductory material on estimand preservation in a seamless multi-stage trial provides the broader setting; here the focus is a practical, revision-by-revision audit.
An intercurrent event is an event after treatment starts that affects either the interpretation or existence of the outcome, such as treatment discontinuation, rescue medication, or switching treatment. Its handling strategy is part of the clinical question: for example, an estimand may compare assignment to treatment regardless of later discontinuation, or may target outcomes under a hypothetical scenario in which discontinuation did not occur. A protocol change that alters what happens after such an event may therefore change the treatment condition or the event strategy, even if the endpoint name remains the same.
A Practical Audit Sequence
Before assessing a proposed revision, write down the current estimand in a short, operational form. Avoid labels such as “the primary endpoint” alone: they do not specify the population, time point, event strategy, or contrast. Then compare the proposed protocol with the current protocol attribute by attribute.
Name the population, treatment conditions, outcome and time point, intercurrent-event strategy, and summary contrast in language that could be applied to hypothetical trial participants.
Record what changes in eligibility, treatment delivery, follow-up, or measurement. Distinguish what the protocol requires from what may happen in practice.
For every attribute, mark “fixed,” “redefined,” or “unchanged but at risk of identification.” Explain the reason rather than relying on a general label.
Ask whether the planned data still represent the target population and measure the same outcome at the same time under the specified event strategy. A preserved estimand may still require new assumptions or data to be estimated reliably.
State whether the revision preserves the estimand, redefines it, or leaves it nominally fixed while creating a population, measurement, or follow-up mismatch.
This audit is a semantic and design check, not a statistical test. It does not establish that the effect can be estimated without bias, nor does a “preserved” label prove that the data satisfy the required assumptions. The earlier material on separating the estimand, estimator, and test is useful here: a changed analysis method is not automatically a changed estimand, and an unchanged estimand does not guarantee that the revised data identify it.
Worked Audits
Worked Example: Narrowing Eligibility
Hypothetical trial. The original protocol enrolls adults with a specified chronic condition across mild and severe baseline disease. The target estimand is the mean difference in symptom score at week 24 between assignment to Drug A and placebo in all adults meeting the original criteria, regardless of treatment discontinuation. Lower scores indicate fewer symptoms. A proposed revision excludes people with severe disease because recruitment is slower than planned.
Audit the population. The original target includes mild and severe disease; the revised eligibility rule enrolls only mild disease. If the intended target population is also formally changed to mild disease, the population attribute is redefined. The clinical question is now the effect in mild disease, not the effect in all adults meeting the original criteria.
The distinction matters even if treatment effects within severity groups are unchanged. Suppose, purely hypothetically, the mean Drug A-minus-placebo difference is \(-4\) score points in mild disease and \(-8\) points in severe disease. Suppose the original target population is 60% mild and 40% severe. Its target mean difference is
If the revised target is mild disease only, its mean difference is \(-4\) score points. The difference between \(-5.6\) and \(-4\) is not an arithmetic inconsistency; it reflects different population weights and therefore different target populations.
If the protocol excludes severe disease from enrollment but still claims the original all-severity target, the estimand has not necessarily been redefined on paper. However, the revised sample no longer directly represents the full target. The audit should mark the population as unchanged but at risk of identification, and require a credible plan and assumptions for learning about severe disease. Merely retaining the original wording does not restore information about a group no longer enrolled.
Worked Example: Revising Treatment Delivery
Hypothetical trial. Participants are randomized to Drug B at a fixed dose or placebo for 12 weeks. The estimand compares mean change in a disease activity score at week 12 under assignment to these treatment conditions, regardless of subsequent discontinuation. The proposed revision replaces the fixed dose with a dose-titration algorithm based on tolerability.
Audit the treatment condition. A fixed dose and a rule-based titration regimen are different treatment conditions: they prescribe different interventions, even if they use the same drug. If the intended comparison is now titration versus placebo, the treatment-condition attribute has been redefined. The result would answer a question about the titration strategy, not the original fixed-dose regimen.
The event strategy also needs review. Suppose the original rule is to include outcomes after discontinuation when they are available. If the revised titration protocol permits dose reduction but still follows participants and includes their week-12 outcomes, discontinuation handling may remain unchanged. If it instead censors outcomes at dose reduction or treats dose reduction as a failure, the strategy for handling that event has also changed.
A protocol could preserve the original estimand if titration is only an operational means of delivering the same treatment condition as that condition was already defined. That conclusion requires a precise definition of the condition; it cannot be inferred from the fact that the active ingredient is unchanged. The audit therefore records the actual dose rules and event handling, not just the treatment’s name.
Worked Example: Changing Follow-Up and Endpoint Collection
Hypothetical trial. The target is the mean difference between two assigned regimens in a validated symptom score at week 24, with outcomes included regardless of rescue medication. A site revision proposes collecting the score at week 26 instead, using a shorter questionnaire, and continuing to contact participants after rescue medication.
Follow-up time. Week 26 is not automatically equivalent to week 24. If the target remains the week-24 score, the protocol must still obtain or validly derive that outcome. Replacing the week-24 target with a week-26 outcome changes the assessment-time attribute and therefore changes the estimand.
Endpoint collection. A shorter questionnaire may preserve the outcome only if it measures the same defined variable on a sufficiently comparable scale. For example, suppose a prespecified validation establishes that the short-form score equals the full-scale score plus a constant offset of 2 points for every participant and treatment group. If the short-form means are 18 and 22 points, then the corresponding full-scale means are \(18-2=16\) and \(22-2=20\), so their difference is \(16-20=-4\) points, the same as \(18-22=-4\). Under this hypothetical calibration, the treatment contrast is preserved, although the measurement procedure changed.
If instead the short form has a treatment-dependent offset, say 1 point in one group and 3 points in the other, then the contrast may change. With the same short-form means of 18 and 22, the calibrated full-scale means would be 17 and 19, giving a difference of \(17-19=-2\) points rather than \(-4\). The instrument change would then affect the treatment contrast and cannot be treated as harmless without further justification.
Rescue medication. Continuing to collect outcomes after rescue medication is consistent with the stated strategy of including those outcomes regardless of rescue. If the revision instead stopped collection at rescue, it would create missing outcomes precisely after an intercurrent event and could undermine estimation of the stated estimand. The audit outcome is therefore: week-26-only follow-up redefines the target time; the short form requires evidence of measurement comparability; and post-rescue collection preserves the stated event strategy.
What a Complete Audit Record Should Show
A concise record can use a table, but each entry should be specific enough that a reviewer can tell what changed and why. For example:
| Attribute | Audit question | Possible finding |
|---|---|---|
| Target population | Does the target group change, or does the revised sample merely cover it less well? | Redefined; or unchanged but at risk of identification |
| Treatment conditions | Does the assigned regimen or comparison strategy change? | Fixed-dose comparison replaced by titration comparison |
| Outcome and time | Is the same variable targeted at the same time point? | Same construct with supported calibration; or new time-point outcome |
| Intercurrent-event strategy | Are outcomes after discontinuation, rescue, or switching handled in the same way? | Preserved, changed, or no longer observed as required |
| Summary measure | Is the same contrast summarized over the same target population? | Mean difference retained, but target-population weights changed |
The summary measure deserves explicit attention. A mean difference, risk difference, or hazard-based contrast can remain the same named measure while being applied to a different population, outcome time, or treatment condition. The meaning of the quantity depends on all the estimand attributes together, not on the contrast label alone.
Justifying a Preservation Decision
A compact way to reason about population changes is to express the target mean treatment effect as an average of subgroup effects. Let \(S\) denote a prespecified population subgroup, let \(\tau_s\) be the treatment effect in subgroup \(s\), and let \(w_s\) be that subgroup’s proportion in the target population. Then the marginal effect is
This identity follows by partitioning the target population into mutually exclusive subgroups: the overall mean outcome under each treatment is the weighted sum of the subgroup mean outcomes, and subtracting those two overall means gives the weighted sum of the subgroup differences. Thus, when treatment effects differ across subgroups, changing the target weights can change the estimand even if each subgroup-specific effect remains fixed. When effects are identical across subgroups, changing weights does not change this particular average contrast, but the target population still differs and should be documented as such.
For treatment conditions, the corresponding justification is definitional: the treatment effect compares the specified potential outcomes under the specified interventions. Changing the intervention changes what those potential outcomes refer to. For measurement revisions, preservation requires that the revised observation still represent the same target variable at the same time, or that a defensible transformation recovers it. For event strategies, preservation requires the same rule for interpreting outcomes after the event. These checks are more informative than asking whether the primary endpoint’s name stayed the same.
Diagnostics, Alternatives, and Limits
Several warning signs indicate that an audit may be overlooking estimand drift:
- Only the endpoint label is compared. A “symptom score” can refer to different instruments, times, or event-handling rules.
- Enrollment is confused with the target population. A narrower sample may leave the target estimand written as before but weaken its identification; a formally narrowed target changes the estimand.
- Delivery is treated as a drug name. Dose, schedule, titration rules, and allowed concomitant treatment can define materially different treatment conditions.
- Missing follow-up is treated as a neutral scheduling issue. If collection stops after an intercurrent event, the observed data may no longer support the event strategy in the estimand.
- Measurement equivalence is assumed from convenience. A shorter or remote assessment needs evidence that it captures the same outcome in a comparable way.
The main alternative to preserving the original estimand is not an informal claim that the revised protocol is “close enough.” It is to define a new estimand transparently, explain why it answers a clinically relevant question, and align the protocol and analysis plan to that question. If retaining the original estimand is essential, the alternative is to keep the relevant protocol features fixed or to add design and analysis provisions that make the original target estimable under explicit assumptions. Neither route is universally preferable; the audit clarifies the trade-off.
An audit cannot prove that a revision has no effect on bias, precision, recruitment, or participant experience. Nor does a plausible calibration or population adjustment automatically establish validity. The audit should therefore flag assumptions for separate scientific and statistical review, particularly when the revised design provides little or no direct information about an intended subgroup, time point, or post-event outcome.
Check Your Understanding
For each question, decide what the revision does to the estimand and compare your reasoning with the solutions.
- A trial keeps its original broad target population in the protocol, but a recruitment revision enrolls only younger adults. Is the population attribute necessarily redefined, and what must the audit flag?
- A trial changes from a fixed daily dose to a prespecified titration algorithm but keeps the same drug and endpoint. Which attribute is most directly changed?
- A week-12 endpoint is collected by telephone instead of in person, using the same validated instrument. What must be checked before calling this estimand-preserving?
- A trial plans to include outcomes after rescue medication, but the revised protocol stops contacting participants when rescue begins. Which estimand feature is at risk?
Solutions. (1) Not necessarily: the written target may remain broad, but the enrolled sample no longer covers it directly. The audit should flag an unchanged target at risk of identification and specify what assumptions or data would support inference for older adults. If the target is formally narrowed to younger adults, it is redefined. (2) The treatment condition changes because the assigned regimen now includes a titration rule rather than a fixed dose. (3) Check that telephone administration measures the same outcome at the same target time with comparable measurement properties; a changed mode alone does not prove equivalence. (4) Outcome collection after rescue is at risk. The protocol may no longer provide the outcomes required by the stated strategy, even if the estimand wording remains unchanged.