Why Outcome Measurement Needs Its Own Temporal Audit
In a platform trial, an outcome difference between earlier and later participants may reflect a change in patients, treatment, or standard care. It may also arise because a protocol version changed what counted as the outcome, when it was assessed, or how it was measured. Those possibilities are not interchangeable: a change in measurement can produce a temporal difference even when the underlying patient outcomes have not changed.
This tutorial focuses on comparing outcome measurement across protocol versions. It does not assume that a statistical adjustment for calendar period can correct a changed endpoint. First establish whether the same construct was measured in a comparable way; only then interpret temporal outcome contrasts.
A useful representation for a quantitative outcome is
where \(Y^{obs}_{iv}\) is the recorded measurement for patient \(i\) under protocol version \(v\), \(Y^*_{i}\) is the underlying outcome at the specified target time, \(b_v\) is a version-specific measurement offset, and \(\epsilon_{iv}\) is residual measurement error. This additive representation is a working model, not a universal law. A changed endpoint construct or a changed scale may require a more complex mapping—or may make direct comparison inappropriate.
What to Compare Across Protocol Versions
An audit should compare the operational details of the endpoint, not just its name in the protocol. A primary outcome may retain the same label while its meaning changes. For every version, record the endpoint definition, assessment time relative to a common time origin, permitted visit window, measurement instrument, administration procedure, assessor training, scoring rules, and any transformations or thresholds used to create the analysis variable.
For a binary endpoint, a threshold change can alter event rates even if the underlying continuous measurements are identical. For a continuous endpoint, a change of instrument, scale, assessor, or timing can shift the mean or variability. For an outcome based on clinical adjudication, revised criteria or access to different information can change classification. The audit should identify whether source data still permit all versions to be recoded to one prespecified definition.
A comparison also requires a common target: what outcome is intended, measured at what time, and in what units or categories? A scheduled assessment at week 8 is not automatically equivalent to one at week 10. When outcomes naturally change over time, changing the assessment time changes the quantity being measured. That is distinct from merely recording the same quantity with a different instrument.
Worked Example: A Revised Event Threshold
Suppose a hypothetical protocol version defines a binary event as a laboratory score of at least 10, while the next version lowers the threshold to at least 8. The same stored measurements are available for 100 patients: 24 scores are at least 10, and 39 are at least 8.
Under the earlier definition, the event proportion is \(24/100=0.24\), or 24%. Under the revised definition, it is \(39/100=0.39\), or 39%. The difference caused solely by applying the two rules to the same measurements is
or 15 percentage points. In this hypothetical reclassification, 15 patients with scores from 8 up to (but not including) 10 become events under the revised rule. This is an exact consequence of the supplied counts, not an estimate of a biological change. If earlier-period patients were classified using the first threshold and later-period patients using the second, the raw event-rate contrast could therefore include this 15-point definitional shift. A valid comparison would require a common endpoint definition, if the underlying scores are available and comparable.
Quantifying a Measurement Shift With Bridge Data
When both old and new procedures can be applied to the same eligible specimens, images, or patients at the same target time, a bridge study can estimate their systematic difference. Let \(d_i\) be the paired difference between the new and old measurements for bridge unit \(i\). The paired approach removes between-unit variation from the comparison: each unit serves as its own comparison. It does not, by itself, show that the bridge sample represents every patient or every site in the trial.
For a small bridge study, a conventional interval for the mean paired difference uses the paired \(t\) method. Its assumptions are that paired differences are independent across sampled units, are representative of the target measurement process, and are approximately normally distributed (especially important when the sample is small). If these conditions are implausible, the reported interval may not be reliable; inspect the differences and consider a larger or differently designed calibration study.
Worked Example: Estimating an Instrument Offset
In a hypothetical bridge study, six specimens are measured using both an old and a new instrument, in the same units. Define each difference as new minus old. The paired differences are \(2, 1, 3, 0, 2,\) and \(1\) units. Assume the six specimens are independent and representative of the specimens to which the calibration is intended to apply, and that the paired differences are reasonably consistent with a normal distribution.
The mean difference is
The deviations from 1.50 are \(0.50,-0.50,1.50,-1.50,0.50,-0.50\). Their squared sum is 5.50, so the sample standard deviation and standard error are
With \(df=5\), \(t_{0.975,5}\approx2.571\). The 95% confidence interval is
Under the stated assumptions, the interval estimates the population mean new-minus-old measurement difference as approximately 0.399 to 2.601 units. The estimated offset is positive: on average, the new instrument records higher values. This result supports investigating a calibration or harmonization rule; it does not establish that subtracting 1.50 units will make the methods interchangeable for every specimen.
Assessment Schedules Can Change the Measured Outcome
A schedule change can create an apparent difference without any instrument change. If the outcome changes naturally between two assessment times, the recorded value at the later time is a different time-specific outcome. A bridge assessment at both times can reveal the size and direction of that change, but it does not make the two scheduled endpoints identical. In particular, an observed within-person change does not prove that every period or treatment group would have the same change.
Worked Example: Moving the Primary Assessment Time
In a hypothetical schedule-bridging substudy, 40 participants have symptom scores recorded at both week 6 and week 8 using the same scale, with lower scores indicating fewer symptoms. The mean week-8 minus week-6 paired difference is \(-1.50\) points, and the sample standard deviation of those paired differences is 3.00 points. Assume independent participants, a representative substudy, and approximately normal paired differences.
The standard error is
With \(df=39\), \(t_{0.975,39}\approx2.023\). A 95% confidence interval for the mean week-8 minus week-6 change is
Under the assumptions, the mean symptom score is lower at week 8 than at week 6 in the target population represented by the substudy. If one protocol version measures the endpoint at week 6 and another at week 8, the resulting period contrast may partly reflect this assessment-time difference. The interval quantifies average within-person change; it is not a correction that can be applied automatically to every treatment group or calendar period.
Diagnosing and Responding to Drift
A practical investigation proceeds from the protocol and raw data toward statistical adjustment, rather than starting with an adjustment model. The following sequence helps keep the measurement question explicit.
Compare endpoint definitions, time origins and target assessment times, instruments, procedures, scoring, and assessor training. Record exactly when each rule applied.
Determine whether archived source measurements can be consistently rescored or reclassified under one definition. Verify that the needed raw fields are available and comparable.
Where feasible, apply both procedures to the same units, or have blinded assessors score the same source material. Examine paired differences and whether they vary by site, severity, or other relevant features.
If one version is used only in an early period and another only later, a version effect and a real temporal change may be inseparable without common measurements or defensible external information.
Plotting outcomes by protocol version and calendar period can expose a sudden step near a version change. Also inspect measurement distributions, missing or invalid values, assessor or site patterns, and the raw-to-derived endpoint rules. Such patterns are diagnostic clues, not proof of drift: a true change in the patient population or care could coincide with the same transition.
Three broad responses are possible. Harmonization re-derives the endpoint from common raw data and is generally preferable when feasible. Calibration uses bridge information to estimate a mapping between procedures, but relies on the bridge applying to the trial measurements. Sensitivity analysis evaluates how conclusions change under plausible measurement shifts or endpoint definitions when direct harmonization is unavailable. Any adjustment should be reported with its assumptions and uncertainty.
| Approach | Useful when | Main limitation |
|---|---|---|
| Common re-scoring or reclassification | Raw data contain the fields needed for one endpoint definition | Unavailable when source information was not retained or methods are not comparable |
| Paired bridge or calibration | Old and new procedures can be applied to the same units | Requires representative bridge data; average calibration may not fix individual disagreement |
| Model adjustment for version or period | There is enough overlap to estimate distinct effects under a suitable model | Cannot separate version from period when they are perfectly aligned; adjustment cannot repair a changed outcome construct by itself |
| Sensitivity analysis | Measurement shift is uncertain but plausible bounds can be specified | Shows robustness to chosen assumptions, not which assumed shift is true |
Common Mistake and Practical Limits
Wrong reasoning: “The later version has a higher event rate, so the outcome worsened over calendar time; we can include calendar period in a regression model to account for it.” This conclusion is not justified if the event threshold, assessment time, or scoring changed at the same time. A period term can describe an observed difference, but it cannot identify whether that difference came from patient outcomes or measurement when the two changes have no independent variation.
Correction: establish endpoint comparability first. If a common definition can be applied to raw measurements, use it and document the re-derivation. If a bridge study is available, assess whether its results are relevant to the trial population and procedures. If version and period are inseparable, state that limitation and present sensitivity analyses rather than claiming the model has resolved the ambiguity.
A measurement offset model is also limited when procedures differ in more than a constant amount. Instruments may agree at low values but diverge at high values; assessors may classify borderline cases differently; a revised definition may change which clinical events count. In those settings, a single average offset is inadequate. Report the dimensions of disagreement and avoid interpreting a calibrated value as though it were directly observed under the common procedure.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A protocol changes a continuous endpoint to a binary endpoint by applying a threshold to stored raw scores. What is the first analytic check to make?
- In a paired calibration study, the mean new-minus-old difference is \(-2\) units. Interpret the direction of the estimated offset.
- Why does adding a protocol-version indicator to a regression not identify measurement drift when each version occurs in only one calendar period?
- What does a paired confidence interval for week-8 minus week-6 scores estimate, and what does it not establish?
- Name one situation in which a single additive calibration offset would be inadequate.
Solutions. (1) Check whether the stored raw scores permit every participant to be classified using one common threshold and verify that the underlying measurement was comparable; then re-derive the endpoint consistently if possible. (2) The new procedure records values 2 units lower on average than the old procedure, for the population represented by the bridge study. (3) Version and period are perfectly aligned, so their separate effects cannot be distinguished from those data alone. (4) It estimates the mean paired change in the represented population between the two assessment times under the paired-analysis assumptions; it does not show that all participants or periods change by the same amount. (5) For example, if the methods differ more at high than at low values, a constant offset will not adequately map one to the other.