Negative-Control Outcomes for Detecting Calendar-Time Bias

Why Use a Negative-Control Outcome?

In a platform trial, a new arm may have concurrent controls enrolled during the same calendar period and nonconcurrent controls enrolled earlier. The earlier material on calendar time as a confounder and sensitivity analysis for unmeasured temporal confounding establishes why differences between enrollment periods can compromise a comparison. A negative-control outcome provides an additional diagnostic: it is an outcome that the study treatment is not expected to affect, but whose distribution may respond to the same time-related changes that threaten the primary comparison.

For example, changes in patient mix, background care, referral patterns, or outcome ascertainment may alter an outcome over calendar time. If a suitably chosen negative-control outcome differs between historical and current control participants, that difference is evidence that the periods are not interchangeable in at least one relevant respect. It is a warning signal, not a direct estimate of bias in the primary treatment-effect estimate.

Definition: A negative-control outcome is an outcome that, based on substantive knowledge, is not causally affected by the study treatment over the specified follow-up period. It is used as a diagnostic because its distribution may still reflect calendar-time changes, participant differences, or measurement processes relevant to the primary analysis. A negative-control period contrast compares its distribution across the calendar periods or control groups being assessed.

The diagnostic is most interpretable when the outcome is measured in both periods in the same way, over the same follow-up window, and in comparable populations. The outcome should be common or measured precisely enough to detect meaningful differences. Its causal irrelevance to treatment must be plausible: merely observing no statistically significant treatment association does not establish that it is unaffected.

What a Period Contrast Can and Cannot Show

Let \(Y^{NC}\) denote a binary negative-control outcome, such as an event expected not to respond to the study treatment. Let \(H\) denote the historical control period and \(C\) the current period. A simple diagnostic is the difference in observed risks:

\widehat{\Delta}_{NC} = \hat{p}_{C}-\hat{p}_{H}, \qquad \hat{p}_{C}=\frac{x_C}{n_C}, \qquad \hat{p}_{H}=\frac{x_H}{n_H}.

Here \(x_C\) and \(x_H\) are the numbers of participants with the negative-control outcome, and \(n_C\) and \(n_H\) are the corresponding group sizes. A nonzero contrast may reflect calendar-time changes, differences in participant composition, chance variation, or changes in how the outcome was observed. It does not identify which explanation applies.

The treatment-effect estimand remains the effect on the primary outcome in its defined target population. A negative-control contrast does not replace that estimand, nor does it automatically supply a correction factor. Even a large negative-control difference cannot be translated into a primary-outcome bias amount without additional assumptions connecting the two outcomes and their causes.

Interpretation: A difference in a credible negative-control outcome is evidence against treating the compared periods as fully comparable on all relevant features. A small or statistically inconclusive difference is not proof of exchangeability: the outcome may be insensitive to the relevant bias, measured imprecisely, or affected by different time mechanisms than the primary outcome.

Unadjusted Binary Outcome Contrast

For independent samples and a binary outcome, the estimated variance of the risk difference is obtained by adding the estimated variances of the two sample proportions. A large-sample confidence interval uses the estimated standard error and an approximately normal sampling distribution. This calculation describes uncertainty in the negative-control contrast; it does not resolve confounding.

SE(\widehat{\Delta}_{NC}) = \sqrt{ \frac{\hat{p}_{C}(1-\hat{p}_{C})}{n_C} + \frac{\hat{p}_{H}(1-\hat{p}_{H})}{n_H} }, \qquad \widehat{\Delta}_{NC}\ \pm\ 1.96\,SE(\widehat{\Delta}_{NC}).

Worked Example: Comparing Historical and Current Controls

Suppose, hypothetically, that a platform trial records a binary negative-control event in 200 historical controls and 300 current controls. The event occurs in 12 historical controls and 30 current controls. Assume participants are independent and the event definition and follow-up are the same in both periods.

The observed risks are \(\hat{p}_{H}=12/200=0.060\) and \(\hat{p}_{C}=30/300=0.100\). The current-minus-historical risk difference is:

\widehat{\Delta}_{NC}=0.100-0.060=0.040.

Thus, the negative-control event is 4.0 percentage points more frequent in the current controls. The estimated standard error is:

SE = \sqrt{\frac{0.100(0.900)}{300}+\frac{0.060(0.940)}{200}} = \sqrt{0.000300+0.000282} = 0.0241.

An approximate 95% confidence interval is \(0.040\pm1.96(0.0241)\), or \((-0.0073,\ 0.0873)\). A two-sided normal-approximation test gives \(z=0.040/0.0241=1.66\), with \(p\approx0.097\). The data do not provide strong evidence of a period difference at the 0.05 level, but the interval is compatible with a current-period increase as large as about 8.7 percentage points. The result is not evidence that the periods are equivalent.

Adjustment for Measured Composition Differences

A raw period contrast can arise because the proportions of participant subgroups differ, even if risks are stable within each subgroup. To make the periods comparable on measured covariates, estimate the negative-control risk within covariate strata and average each period's risks using the same target-population weights. This is direct standardization. It addresses only the measured variables included in the strata and requires data support in each stratum for both periods.

\hat{p}^{\,std}_{g} = \sum_{k=1}^{K} w_k\hat{p}_{gk}, \qquad \sum_{k=1}^{K}w_k=1, \qquad \widehat{\Delta}^{\,std}_{NC} = \hat{p}^{\,std}_{C}-\hat{p}^{\,std}_{H}.

Here \(g\) indexes period, \(k\) indexes covariate strata, and \(w_k\) is the chosen target-population proportion in stratum \(k\), held fixed across periods. A standardized contrast answers a different, conditional question from the crude contrast: how do the periods compare after placing them on the same measured covariate distribution?

Worked Example: Standardizing to the Same Risk-Group Mix

In a hypothetical comparison, participants are classified into lower- and higher-risk groups. Historical controls have 20 events among 200 lower-risk participants and 20 among 100 higher-risk participants. Current controls have 30 events among 100 lower-risk participants and 30 among 200 higher-risk participants. Use a target population that is 50% lower-risk and 50% higher-risk.

The stratum-specific risks are \(0.10\) and \(0.20\) historically, and \(0.30\) and \(0.15\) currently. Standardizing each period to the same 50:50 mix gives:

\hat{p}^{\,std}_{H}=0.5(0.10)+0.5(0.20)=0.150, \qquad \hat{p}^{\,std}_{C}=0.5(0.30)+0.5(0.15)=0.225.

The standardized current-minus-historical contrast is \(0.225-0.150=0.075\), or 7.5 percentage points. For an approximate standard error, treating stratum-specific samples as independent, use the variance of each weighted risk:

\widehat{\mathrm{Var}}(\hat{p}^{\,std}_{H}) = 0.5^2\frac{0.10(0.90)}{200} + 0.5^2\frac{0.20(0.80)}{100} =0.0005125,

\widehat{\mathrm{Var}}(\hat{p}^{\,std}_{C}) = 0.5^2\frac{0.30(0.70)}{100} + 0.5^2\frac{0.15(0.85)}{200} =0.0006844.

The standard error of the difference is \(\sqrt{0.0005125+0.0006844}=0.0346\). The approximate 95% interval is \(0.075\pm1.96(0.0346)=(0.0072,\ 0.1428)\). Under these assumptions, the standardized contrast suggests a higher negative-control risk in the current period. Standardization has aligned the two periods on the specified risk-group mix; it has not controlled unmeasured differences or established that this contrast is the bias in a primary treatment comparison.

Choosing and Checking the Negative Control

A useful negative-control outcome requires substantive justification before looking at the results. Ideally, investigators explain why the treatment cannot plausibly affect it, why it may respond to the same calendar-time processes as the primary outcome, and why its measurement and follow-up are comparable across periods. These are distinct requirements: an outcome can be treatment-unaffected but irrelevant to the temporal bias of concern, or time-sensitive but affected by treatment.

Diagnostics should examine more than a single p-value. Plot period-specific risks or means with uncertainty intervals, inspect event counts and missingness, and check whether definitions, surveillance, and follow-up windows remained stable. For adjusted comparisons, examine covariate overlap and stratum-specific estimates; a standardized result can be unstable if a period has few or no participants in important strata. If several negative controls are examined, report the set and their results rather than highlighting only the most striking contrast.

Worked Example: A Continuous Negative-Control Outcome

Suppose a hypothetical negative-control measurement, in the same units and at the same follow-up visit, has mean 2.4 and sample standard deviation 1.5 among 150 historical controls, and mean 3.0 and standard deviation 1.8 among 180 current controls. Assume independent participants and use a large-sample normal approximation.

The estimated current-minus-historical mean difference is \(3.0-2.4=0.60\) measurement units. Its standard error is:

SE = \sqrt{\frac{1.8^2}{180}+\frac{1.5^2}{150}} = \sqrt{0.018+0.015} = 0.182.

The approximate 95% confidence interval is \(0.60\pm1.96(0.182)=(0.244,\ 0.956)\) units, and the normal-approximation statistic is \(z=0.60/0.182=3.30\), with a two-sided \(p\)-value of about 0.001. If the outcome is genuinely unaffected by treatment and measured comparably, this is a clear signal that its distribution differs between periods. It still does not establish whether the cause is participant mix, background conditions, or measurement, and it does not quantify bias in the primary treatment effect.

How This Diagnostic Relates to Other Approaches

Negative-control outcomes complement, rather than replace, methods for addressing calendar time. Regression adjustment, period stratification, and standardization use measured information to define or adjust comparisons. A negative-control outcome probes whether an additional outcome displays a period association that could signal remaining differences. Sensitivity analysis examines how conclusions change under specified assumptions about unmeasured temporal confounding. Concurrent controls remain especially informative because treatment and control participants are enrolled under more similar calendar conditions.

These approaches answer different questions. Adjustment may reduce measured imbalance but cannot guarantee that unmeasured factors are controlled. A negative-control signal can reveal concern but does not identify the source or size of bias. Conversely, a null negative-control contrast can occur even when the primary comparison is biased, for example if the negative control is insensitive to the relevant time-varying factor. No single diagnostic establishes exchangeability.

Key limitation: A negative-control outcome is not a substitute for a defensible treatment-effect analysis. Use it to assess the plausibility of period comparability, alongside design knowledge, covariate diagnostics, and sensitivity analyses. Do not “correct” the primary estimate by subtracting the negative-control contrast unless a separate, explicit model justifies that operation.

Common Mistakes

Choosing an outcome because its result looks convenient. Select and justify it based on causal and measurement knowledge, not on whether its observed contrast is null.
Treating a nonsignificant test as evidence of no period difference. Inspect the confidence interval and ask whether it excludes differences that would matter; low precision can conceal substantial contrasts.
Assuming a nonzero contrast identifies temporal confounding. It may also reflect composition, selection, chance, or measurement changes. Investigate these explanations rather than assigning a single cause automatically.
Using the contrast as a numerical correction to the primary effect. The negative-control and primary outcomes need not share causes or bias magnitudes. A correction requires additional, explicit assumptions.
Ignoring outcome definition and follow-up. Different ascertainment or observation time across periods can create a contrast even if underlying outcome risks are stable.

Specify the diagnostic.
State the treatment-unaffected rationale, target follow-up, periods, and population for the negative-control outcome.

Check comparability of measurement.
Confirm that definitions, surveillance, and missing-data handling are sufficiently similar across periods.

Estimate and display the contrast.
Report period-specific summaries, the contrast, uncertainty, and any standardized analysis with its target weights.

Interpret as a diagnostic signal.
Discuss plausible explanations and how the result affects confidence in period comparability; do not claim it proves or quantifies primary-outcome bias.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What two substantive properties should support the choice of a negative-control outcome?
A risk is 8% in historical controls and 9% in current controls, with a wide confidence interval for the difference. Does this establish period equivalence? Explain.
Why might a standardized negative-control contrast differ from the crude contrast?
A negative-control outcome differs across periods. Can its risk difference be subtracted from the primary treatment-effect estimate as a bias correction?

Solutions. (1) The treatment should not plausibly cause the outcome, and the outcome should be measured comparably and be relevant to time-related processes that could affect the primary comparison. (2) No. A wide interval may include substantial period differences; failure to reject a null difference is not evidence of equivalence. (3) The periods may have different covariate distributions, so standardization reweights their stratum-specific risks to a common target mix. (4) Not without additional assumptions linking the negative-control and primary outcomes and their sources of bias; the contrast is ordinarily a diagnostic, not a correction factor.