Negative Control Checks In A Propensity-Matched Analysis

Why Use a Negative Control Outcome?

Propensity-score matching can improve comparability on measured baseline covariates, but a good balance table cannot show whether all relevant confounding has been removed. A negative control outcome provides a separate diagnostic: it is an outcome that the treatment cannot causally affect, yet is chosen because it may be subject to some of the same confounding or measurement processes as the primary outcome.

For example, suppose a matched cohort compares patients who start a medication with similar patients who do not. The outcome of interest is hospital admission during the following year. A prespecified measure of health-care use during the year before treatment could serve as a negative control outcome: future medication use cannot cause past visits. If the matched groups still differ on that past outcome, the difference cannot be interpreted as a treatment effect. It may instead point to residual differences between the groups, chance variation, or a problem with measurement or design.

Definition: A negative control outcome is an outcome for which the causal effect of the treatment is known, by design or subject-matter knowledge, to be zero over the relevant period. It is analyzed using the matched cohort to check for an observed treatment association that could signal residual confounding, selection, or other systematic differences. The check is informative only to the extent that the negative control shares relevant sources of bias with the primary outcome.

Let \(A=1\) denote treatment and \(A=0\) denote comparison. Write \(Y^{NC}(1)\) and \(Y^{NC}(0)\) for a unit’s potential negative-control outcomes under treatment and comparison. The key causal condition is \(Y^{NC}(1)=Y^{NC}(0)\) for each unit; for an average-effect interpretation, the average causal effect must also be zero for the population and estimand represented by the matched comparison, not merely in a broader target population. A baseline outcome measured before treatment often satisfies this condition because the later treatment cannot change the past. For a future outcome, the no-effect claim requires substantive justification: it must not be affected directly or indirectly by treatment.

Designing and Estimating the Check

Choose the negative control before examining its treatment association. It should have a defensible zero causal effect, be measured with adequate quality, and plausibly share some causes or recording processes with the primary outcome. A control outcome with no connection to the confounding structure can be uninformative even when it is truly unaffected by treatment. Conversely, a control outcome that treatment can affect is not a valid negative control: a nonzero association could then be a real treatment effect.

Use the matched sample and its design as specified. For one-to-one matched pairs, let \(Y_{Tj}^{NC}\) and \(Y_{Cj}^{NC}\) be the negative-control outcomes for the treated and comparison members of pair \(j\), and define \(d_j=Y_{Tj}^{NC}-Y_{Cj}^{NC}\). The estimated matched association is the mean paired difference. Its interpretation is an observed treatment-group contrast in the matched cohort, not a causal effect of treatment on a valid negative control.

\bar d=\frac{1}{n}\sum_{j=1}^{n}d_j,\qquad SE(\bar d)=\frac{s_d}{\sqrt{n}}

Here \(s_d\) is the sample standard deviation of the pair differences. A paired \(t\) interval or test uses \(df=n-1\), assuming pairs are independent and the distribution of differences is adequately compatible with the \(t\)-based procedure. For binary outcomes, the matched risk difference is the difference in the two matched proportions. In paired data, the discordant pairs carry the information about that difference; McNemar’s test, including its exact binomial version, can test the null of no matched association. For variable-ratio matching or matching with weights, the estimator and uncertainty calculation must respect the actual matched-set structure and weights rather than treating all observations as independent.

The diagnostic logic is simple. If the negative control is truly unaffected, systematic differences in its observed outcome are not explained by a causal treatment effect. A nonzero estimate may therefore reveal residual confounding or another failure of comparability. But it can also arise by chance. Even a very small estimate cannot establish that the primary outcome is free of residual bias: the control may not share the relevant unmeasured causes, and positive and negative sources of bias may cancel.

Worked Example: A Continuous Baseline Negative Control

In a hypothetical matched cohort, eight treated patients are paired with eight comparison patients. The negative control is the number of health-care visits in the year before treatment began. The observed treated-minus-comparison pair differences, in visits per year, are \(0.5, 1.5, 1.5, 1.5, 3.5, 3.5, 3.5, 4.5\). Because the visits occurred before treatment, the later treatment cannot have caused them. We use a paired \(t\) analysis, while recognizing that eight pairs provide limited evidence for assessing the distribution of differences.

The mean difference is \(20/8=2.5\) visits per year. Deviations from this mean are \(-2,-1,-1,-1,1,1,1,2\), with squared deviations summing to 14. Therefore,

s_d=\sqrt{\frac{14}{8-1}}=\sqrt{2}=1.4142,\qquad SE(\bar d)=\frac{1.4142}{\sqrt{8}}=0.5000

The test of \(H_0:\mu_d=0\) against \(H_a:\mu_d\ne0\) gives \(t=2.5/0.5=5.000\), with \(df=7\). The two-sided \(p\)-value is approximately \(0.001565\). Under the null hypothesis and the paired \(t\)-test assumptions, a statistic at least this extreme in either direction would occur with probability about 0.1565%. This is strong evidence of a matched association in pre-treatment visits; it is not evidence that treatment caused those past visits.

A 95% \(t\) confidence interval uses \(t_{0.975,7}\approx2.3646\):

2.5\ \mathbin{\pm}\ 2.3646(0.5000) =[1.3177,\ 3.6823]\text{ visits per year}

The interval estimates a positive treated-minus-comparison difference in this matched cohort. Since these are pre-treatment visits, the result flags a comparability concern worth investigating, such as differences in prior health-care use or access. It does not quantify how much that concern biases the primary treatment-effect estimate. The small sample also makes the normality assumption for the pair differences important and difficult to diagnose.

Binary Outcomes and Matched Discordances

For a binary negative control, the matched risk difference is easy to interpret in percentage points. In paired data, pairs in which both members have the same outcome do not determine which group has the higher proportion; the contrast comes from discordant pairs. Under the null of no matched association, the two directions of discordance have equal probability, which motivates McNemar’s test. An exact version conditions on the total number of discordant pairs and uses a binomial distribution with probability \(1/2\).

Worked Example: A Binary Negative Control

In a hypothetical study there are 40 matched pairs. The negative control is a pre-treatment indicator of whether the patient had a specified health-care event. The pair counts are: 10 pairs with the event for both members, 22 with neither member having it, 6 with the event only for the treated member, and 2 with the event only for the comparison member. The treated proportion is \((10+6)/40=0.400\); the comparison proportion is \((10+2)/40=0.300\). Thus the matched risk difference is \(0.400-0.300=0.100\), or 10 percentage points.

There are \(6+2=8\) discordant pairs. Under the null, the number favoring the treated member has a \(\mathrm{Binomial}(8,0.5)\) distribution. The exact two-sided \(p\)-value is

2P\{X\ge6\} =2\frac{\binom{8}{6}+\binom{8}{7}+\binom{8}{8}}{2^8} =2\frac{28+8+1}{256} =0.2891

This result does not provide strong evidence against the no-association null at conventional significance levels. It also does not show that the groups are equivalent or that residual bias is absent: there are only eight discordant pairs, and the point estimate is still a 10-percentage-point difference. The control’s timing makes a causal treatment explanation implausible, so the estimate and its uncertainty should be considered with other diagnostics.

Reading the Check Alongside Balance Diagnostics

Balance diagnostics and negative control outcomes answer related but different questions. Standardized differences compare measured baseline covariates between matched groups; small differences support measured-covariate balance, not the absence of all confounding. A negative control tests for a treatment association in an outcome that should not respond causally to treatment. Its usefulness depends on whether it reflects relevant unmeasured differences as well as on its precision. Neither diagnostic replaces the other.

Worked Example: Good Measured Balance, but a Control Association

Consider a hypothetical analysis with 30 matched pairs. After matching, the age difference between groups is 0.4 years with a pooled standard deviation of 8 years, and the severity-score difference is 0.12 points with a pooled standard deviation of 1.5 points. The absolute standardized differences are \(0.4/8=0.050\) and \(0.12/1.5=0.080\), respectively. These are small measured imbalances for these two covariates; they do not demonstrate balance on unmeasured factors.

A prespecified negative control, prior-year visit count, was not used to construct the matches. The mean paired difference is 0.70 visits per year, and the standard deviation of pair differences is 1.20. Its standard error is \(1.20/\sqrt{30}=0.2191\). With \(df=29\), \(t_{0.975,29}\approx2.045\), so the 95% paired \(t\) interval is

0.70\ \mathbin{\pm}\ 2.045(0.2191) =0.70\ \mathbin{\pm}\ 0.4481 =[0.2519,\ 1.1481]\text{ visits per year}

The interval is above zero, indicating a positive matched association in the prior-year outcome despite small standardized differences for age and severity. Because treatment begins after that year, this association cannot be a treatment effect. The analyst should investigate whether prior utilization, access to care, or another related factor was inadequately represented in the matching design. The result does not prove that any particular variable caused residual bias, and it does not give the bias in the primary outcome in visits, admissions, or any other units. A defensible response might be to incorporate an appropriate baseline utilization measure in the design, rematch, and reassess the diagnostics, while preserving the prespecified nature of the check.

Diagnostics, Alternatives, and Limitations

A negative-control analysis is most useful when its role is limited and explicit. Prespecification reduces the temptation to report only controls that support a preferred conclusion. Analyze it with the matched-set structure, account for repeated or clustered observations where relevant, and report an estimate and uncertainty interval rather than relying only on a thresholded \(p\)-value. Examine whether the control outcome has adequate variation and whether missingness or measurement may differ by treatment group. A control outcome recorded differently for treated and comparison patients can produce an association that reflects ascertainment rather than confounding.

A statistically detectable association is a warning, not a calibrated correction. The control’s causes may overlap only partly with those of the primary outcome, so its association can be larger or smaller than the bias in the primary estimate. A non-significant result may simply be imprecise. It may also miss bias if the relevant unmeasured confounder affects the primary outcome but not the control, or if opposing mechanisms cancel in the control contrast. Avoid selecting many candidate controls after seeing their results and treating one favorable result as validation.

Compared with balance diagnostics, a negative control can probe consequences of differences not captured by the listed covariates, but it is more dependent on substantive knowledge about the control outcome. Compared with the earlier material on Rosenbaum sensitivity analysis, a negative control uses an observed outcome as an empirical check; sensitivity analysis asks how conclusions might change under specified levels of hidden bias. These approaches complement each other but rely on different information and assumptions. Neither makes unmeasured confounding disappear.

The central interpretation is therefore asymmetric: an association in a credible negative control is a reason to question the design or investigate systematic differences; a result near zero is not proof that the primary comparison is unbiased. The check is strongest when the zero-effect condition is secure, the outcome is measured well, and the control plausibly shares important sources of bias with the primary outcome.

Key takeaway: A prespecified negative control outcome should have no causal response to treatment. Its matched association is a diagnostic signal to interpret alongside balance, uncertainty, and design knowledge—not an estimate of the primary outcome’s residual bias and not proof that bias is absent when the signal is small.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A treatment starts in January. Could the number of clinic visits in the preceding year be a negative control outcome? State the key condition.
In 25 matched pairs, the mean paired negative-control difference is 0.30 units and the standard deviation of differences is 1.00. Calculate the standard error.
For a binary negative control, there are 5 discordant pairs favoring treated patients and 5 favoring comparison patients. What is the matched risk difference attributable to these discordances among 40 pairs?
All measured covariates have small standardized differences and a negative-control estimate is near zero with a wide interval. What can and cannot be concluded?

Solutions. (1) Yes, if the visits were measured before treatment and later treatment cannot alter their recording; the zero causal effect condition must be credible. (2) \(SE=1.00/\sqrt{25}=0.200\) units. (3) The discordance contribution is \((5-5)/40=0\), so the paired risk difference is zero. (4) The measured covariates appear balanced, but balance on measured variables does not establish exchangeability. The negative-control estimate shows no precise evidence of an association, but its wide interval leaves substantial uncertainty; neither finding proves that residual confounding is absent.