Power and Equivalence for Negative Control Tests

Why Power and Equivalence Matter

A negative control test is a diagnostic: it asks whether a chosen exposure and outcome that should not be causally related nevertheless show an association. A detected association can flag a problem with the design or analysis. But a result that is not statistically significant is harder to interpret. It may reflect a genuinely small association, or it may reflect limited information. Power calculations and equivalence tests help distinguish those possibilities.

The central question is not simply whether the test rejects an exact null association. It is also whether the data are precise enough to rule out associations that would be important for the diagnostic. That requires defining, before interpreting the results, what size of association would matter and using a confidence interval or a power calculation on the same scale.

For example, \(\theta\) might be an adjusted mean difference in symptom-score points, a risk difference, or a log odds ratio. The meaning of \(\Delta\) depends on that scale. A margin is not chosen merely because it makes an interval fit; it should represent the largest association considered unimportant for the diagnostic question. Negative controls probe residual bias only under their validity conditions, established in the earlier material on negative controls. Neither a small association nor equivalence proves that the primary analysis is unbiased.

Power for a Test of No Association

Suppose the estimator is approximately normally distributed with standard error \(SE\), and the standard error adequately represents its sampling uncertainty. For a two-sided test at significance level \(\alpha\), reject \(H_0:\theta=0\) when the absolute standardized statistic exceeds \(z_{1-\alpha/2}\), the \(1-\alpha/2\) quantile of the standard normal distribution. Power is the probability of rejection at a specified true value of \(\theta\), under the assumed sampling model. It is not a property of the test alone: it depends on the true association, its standard error, and the significance level.

Here \(\Phi\) is the standard normal cumulative distribution function. The approximation uses \(\hat{\theta}\sim N(\theta,SE^2)\), treating \(SE\) as fixed. For a positive association, the second term is usually the main contribution; the first accounts for rejection in the opposite tail. This expression makes clear that a test may have low power to detect a meaningful association when \(SE\) is large relative to that association.

A useful planning summary is the smallest association a proposed design can detect with a chosen power, sometimes called a minimum detectable effect. It is found by solving the power expression for \(|\theta|\) at the desired power, given \(SE\) and \(\alpha\). It should be compared with a subject-matter threshold for a consequential association, not treated as a universal benchmark.

Equivalence: Ruling Out Associations That Matter

A conventional test asks whether the data are inconsistent with exactly zero. Equivalence asks a different question: are the data sufficiently precise to rule out associations outside the interval \((-\Delta,\Delta)\)? Its null and alternative hypotheses are

This is commonly tested with two one-sided tests (TOST), each at level \(\alpha\). Both component null hypotheses must be rejected: the lower-bound test must support \(\theta>-\Delta\), and the upper-bound test must support \(\theta<\Delta\). If either test fails, equivalence has not been established at that level.

Under the normal approximation, reject the lower-bound null if \(T_L>z_{1-\alpha}\), and reject the upper-bound null if \(T_U<-z_{1-\alpha}\). This is equivalent to requiring the \(100(1-2\alpha)\%\) two-sided confidence interval to lie wholly inside the equivalence margins. In particular, at \(\alpha=0.05\), the interval is a 90% confidence interval. The confidence level is not 95% because equivalence requires two one-sided tests, each with a 5% one-sided error rate.

The reason for this equivalence can be seen directly from the interval endpoints. The lower endpoint exceeds \(-\Delta\) precisely when \(\hat{\theta}+\Delta>z_{1-\alpha}SE\), which is the lower-bound rejection condition. The upper endpoint is below \(\Delta\) precisely when \(\hat{\theta}-\Delta<-z_{1-\alpha}SE\), which is the upper-bound rejection condition. Both inequalities must hold, so the whole interval must be inside the margins.

The equivalence margin must be justified independently of the observed estimate. If the margin is too wide, equivalence may be easy to establish but uninformative; if it is narrower than the study can resolve, the result may be inconclusive even when the estimated association is close to zero. Its units and substantive interpretation should be reported.

Power for Equivalence and What a Null Result Can Say

Equivalence also has power: the probability that both one-sided tests establish equivalence at a particular true value of \(\theta\). Under the normal approximation and fixed \(SE\), this is the probability that \(\hat{\theta}\) falls between the two values that make the confidence interval fit within the margins. Writing \(c=z_{1-\alpha}\), equivalence is concluded when \(-\Delta+cSE<\hat{\theta}<\Delta-cSE\). Therefore,

Assumptions, Diagnostics, and Limits

These calculations require a defensible estimate of uncertainty. The normal approximation should be reasonable for the estimator, and the standard error should account for the design and dependence in the data. For example, if observations are clustered, an independence-based standard error can be too small; the resulting power can look too high and confidence intervals too narrow. Sparse outcomes, model misspecification, influential observations, or unstable estimates can also make a simple normal approximation unreliable. Use an appropriate model-based, robust, or resampling-based uncertainty procedure when the design requires it, and base power and equivalence calculations on that same procedure.

A diagnostic conclusion also depends on the negative control being suitable. The control must satisfy the relevant causal and measurement conditions for its association to be informative about the concern being probed. A precise, near-zero association for a control that does not share the relevant bias pathway may still be uninformative about residual bias in the primary analysis. Conversely, a nonzero association is a warning signal, not a direct estimate of the bias in the primary treatment effect.

Distinguish the main approaches:

Prefer design-stage power calculations to “observed power” computed after seeing a nonsignificant result. After data collection, the estimate, its standard error, and its confidence interval directly show the range and precision of associations compatible with the data. Report the chosen margin, its rationale, the scale and units, the interval level, and the uncertainty method. Keep the diagnostic conclusion proportionate: “equivalent within the prespecified margins” is more precise than “there is no association.”

Common Pitfalls

• Treating \(p>0.05\) as proof of no meaningful association. A large standard error can make a consequential association hard to detect. Inspect the interval and assess whether it excludes the prespecified margins.
• Choosing margins after seeing the estimate. This makes the equivalence conclusion depend on the result. Define and justify \(\Delta\) on the scientific scale before interpreting the data.
• Using the wrong interval level. At \(\alpha=0.05\), the standard TOST equivalence criterion corresponds to a 90% two-sided interval, not a 95% interval. Apply the stated procedure consistently.
• Claiming equivalence because the point estimate is near zero. Equivalence depends on both estimate and precision. The entire required interval must fit inside the margins.
• Interpreting a null control result as proof that the primary analysis is unbiased. A control result is informative only insofar as the control satisfies its validity conditions and probes the relevant concern.