Comparing Control Associations on Compatible Scales

Why Scale Compatibility Matters

A treatment association and a negative-control association can look very different simply because they have been reported on different scales. A risk difference is measured in percentage points, an odds ratio is a ratio of odds, and a regression coefficient may be expressed in the units of an outcome or on a model’s link scale. Comparing their numerical sizes without accounting for those differences can mislead. Even estimates reported as odds ratios may correspond to different absolute changes when the outcomes have different baseline prevalences.

The purpose of scale alignment is narrower than establishing that the two associations represent the same bias. It makes their magnitudes interpretable on a common, explicit scale. Whether a negative-control association informs the likely bias in a treatment analysis depends on additional substantive and design assumptions. A scale conversion cannot supply those assumptions.

Definition: Two associations are on compatible scales when they are expressed using the same effect measure and reference population or covariate distribution, with clearly stated units or reference values. Compatibility supports a meaningful numerical comparison; it does not establish that the associations have the same cause or are equally informative about bias.

Choose a Scale Before Comparing Estimates

For a binary outcome, common effect measures include the risk difference, risk ratio, and odds ratio. The risk difference is \(p_1-p_0\), where \(p_1\) and \(p_0\) are the risks under the two exposure or treatment conditions being compared. It is an absolute change in probability. The risk ratio is \(p_1/p_0\), and the odds ratio is \(\{p_1/(1-p_1)\}/\{p_0/(1-p_0)\}\). Ratios are unitless, but their values do not directly state an absolute change in risk.

For a continuous outcome, a mean difference retains the outcome’s original units. It is often useful to report that difference alongside a standardized mean difference, which divides it by a stated reference standard deviation. The standard deviation used for this purpose should be chosen and reported, rather than selected after seeing which one makes estimates look most alike.

The model’s link scale matters too. A logistic-regression coefficient is a change in log odds, so exponentiating it gives a conditional odds ratio. A log-link coefficient exponentiates to a ratio on the modeled mean scale, such as a risk ratio when that model is appropriate. Neither coefficient is automatically a risk difference. To compare effects on an absolute scale, obtain predicted risks or means and calculate their difference in a common target population.

Convert Binary Associations at a Stated Baseline Risk

Suppose a binary-outcome association is summarized by an odds ratio \(OR\), and the reference risk is \(p_0\). The odds in the reference group are \(p_0/(1-p_0)\). Multiplying those odds by \(OR\) gives the comparison-group odds. Converting the resulting odds back to a probability gives \(p_1\), and then the risk difference is \(p_1-p_0\).

\frac{p_1}{1-p_1}=OR\frac{p_0}{1-p_0} \quad\Longrightarrow\quad p_1=\frac{OR\,p_0}{1-p_0+OR\,p_0}, \qquad RD=p_1-p_0

This conversion requires a reference risk appropriate to the comparison. If baseline risk varies across populations, a single odds ratio can correspond to different risk differences in those populations. The formula also assumes that the odds ratio and reference risk refer to compatible comparisons—for example, both are marginal, or both describe the same conditional stratum. A conditional odds ratio should not simply be combined with a marginal baseline risk and treated as a marginal risk difference.

Worked Example: The Same Odds Ratio, Different Risk Differences

Suppose, hypothetically, a treatment-outcome association has \(OR=1.50\) and a reference risk of \(p_0=0.10\). A negative-control association also has \(OR=1.50\), but its reference risk is \(p_0=0.50\). Calculate the implied risk difference for each association.

For the treatment association:

p_1=\frac{1.50(0.10)}{1-0.10+1.50(0.10)} =\frac{0.15}{1.05}=0.1429

Thus \(RD=0.1429-0.10=0.0429\), or about 4.29 percentage points. For the negative-control association:

p_1=\frac{1.50(0.50)}{1-0.50+1.50(0.50)} =\frac{0.75}{1.25}=0.6000

Thus \(RD=0.6000-0.5000=0.1000\), or 10.0 percentage points. The odds ratios match, but the absolute changes do not. It would be incorrect to conclude from the equal odds ratios that these associations have equal absolute magnitude. These calculations describe associations; they do not by themselves establish causal effects or identify bias.

Standardize When the Covariate Distributions Differ

Even if two estimates use the same effect measure, they can refer to different covariate distributions. For a descriptive comparison, one way to align them is to calculate both associations in a shared target distribution. For each covariate stratum \(k\), obtain predicted risks \(\hat p_{1k}\) and \(\hat p_{0k}\), then average each set of risks using common weights \(w_k\), where \(w_k\geq0\) and \(\sum_k w_k=1\). The standardized risk difference is the difference between these two weighted averages.

\widehat{RD}_{\mathrm{std}} = \sum_k w_k\hat p_{1k} - \sum_k w_k\hat p_{0k}

The shared weights give the comparison a common reference population. The predictions still depend on the fitted models. This approach requires adequate data to estimate the relevant stratum-specific risks, overlap for the comparisons being made, and sufficiently well-specified prediction models. If the standardized quantity is to be interpreted causally, additional causal assumptions—such as consistency, conditional exchangeability, and positivity—are needed. Without them, describe it as a standardized association.

Worked Example: Comparing Standardized Risk Differences

Consider hypothetical binary outcomes and two equally weighted covariate strata. In the treatment analysis, the reference and comparison risks are 0.10 and 0.20 in stratum 1, and 0.40 and 0.60 in stratum 2. In the negative-control analysis, the corresponding risks are 0.05 and 0.10 in stratum 1, and 0.20 and 0.30 in stratum 2. Use weights \(w_1=w_2=0.50\) for both.

For the treatment association, the standardized comparison risk is \(0.50(0.20)+0.50(0.60)=0.40\), and the standardized reference risk is \(0.50(0.10)+0.50(0.40)=0.25\). Therefore, \(\widehat{RD}_{\mathrm{std}}=0.40-0.25=0.15\), or 15 percentage points.

For the negative-control association, the standardized comparison risk is \(0.50(0.10)+0.50(0.30)=0.20\), and the standardized reference risk is \(0.50(0.05)+0.50(0.20)=0.125\). Therefore, \(\widehat{RD}_{\mathrm{std}}=0.20-0.125=0.075\), or 7.5 percentage points.

On this common target distribution and risk-difference scale, the treatment association is twice the negative-control association in absolute magnitude. That numerical comparison does not establish that the treatment association contains twice as much bias: the two outcomes may differ in how they respond to shared causes, and the negative control may have other sources of association.

Account for Outcome Variance

For continuous outcomes with different units, raw mean differences cannot be compared directly. If a treatment association is 4 mmHg and a negative-control association is 2 mg/dL, the numbers alone do not establish which association is larger. A standardized mean difference expresses each difference in units of a stated reference standard deviation:

d_{\mathrm{std}}=\frac{\widehat{\Delta}}{s_{\mathrm{ref}}}

Here \(\widehat{\Delta}\) is the estimated mean difference, and \(s_{\mathrm{ref}}\) is a standard deviation chosen to represent the outcome’s reference variability. A baseline or untreated-group standard deviation is often useful when the aim is to describe a difference relative to a reference population. The choice must be made consistently and reported. Dividing by each study’s observed post-treatment standard deviation can make the scale depend on treatment-related variance changes.

Worked Example: Standardizing Continuous Associations

Suppose, hypothetically, the treatment-outcome association is a mean difference of 4 mmHg, with a reference standard deviation of 10 mmHg. The negative-control association is a mean difference of 2 mg/dL, with a reference standard deviation of 4 mg/dL.

For the treatment outcome, \(d_{\mathrm{std}}=4/10=0.400\) reference standard deviations. For the negative-control outcome, \(d_{\mathrm{std}}=2/4=0.500\) reference standard deviations. The raw values cannot be compared as 4 versus 2 because their units differ. On the stated standardized scale, the negative-control association is larger in magnitude: 0.500 versus 0.400 reference standard deviations.

This comparison assumes the reference standard deviations are meaningful and estimated comparably. A standardized difference is not automatically a universal measure of importance: different outcomes can have different measurement reliability, substantive meaning, or sensitivity to confounding.

Diagnostics, Alternatives, and Limitations

Before comparing estimates, check that the effect measure, reference group, covariate adjustment, target population, and time point are aligned. For model-based estimates, verify that both associations have been converted from their link scales to the intended marginal scale using compatible prediction and standardization procedures. Inspect the range of predicted risks or means and check for sparse strata or poor overlap. If predictions rely heavily on extrapolation, the standardized comparison may be unstable even when its formula is correct.

A practical alternative is to report more than one scale. For binary outcomes, a risk difference can communicate absolute change while a risk ratio or odds ratio communicates relative change. For continuous outcomes, report the original-unit difference alongside the standardized difference. Showing both makes scale dependence visible rather than hiding it behind a single summary. If no shared reference risk, standard deviation, or target distribution is substantively defensible, present the estimates separately and explain why a direct numerical comparison is limited.

Comparison approach	Useful when	Main caution
Compare ratios on the same ratio scale	Relative effects are the intended target and comparisons are defined consistently	Equal ratios can imply different absolute changes at different baseline risks
Convert to risk differences at stated reference risks	Absolute changes are meaningful and appropriate reference risks are available	Results depend on those reference risks
Standardize to a common covariate distribution	The analyses concern different covariate mixes and stratum-specific predictions are supported by data	Model misspecification, sparse strata, or poor overlap can undermine the comparison
Scale continuous differences by reference standard deviations	Outcomes have different units and a reference variability is defensible	Standardized values can obscure clinical meaning and depend on the chosen standard deviations

Scale alignment has important limits. A negative-control outcome may share some confounding mechanisms with the treatment outcome but not others. It may have different measurement error, prevalence, variance, or sensitivity to those mechanisms. Conversely, an association on a negative-control outcome may arise for reasons other than the bias of interest. Therefore, do not multiply, subtract, or otherwise use a negative-control association as a bias correction solely because the two estimates have been expressed on the same scale. Such a correction requires a substantive calibration model and its own assumptions.

Common pitfall: “Both logistic models gave an odds ratio near 1.5, so the treatment and negative-control associations are equally large.” This reasoning ignores baseline risks and may also compare conditional odds ratios with marginal ones. Convert estimates to a clearly stated common scale, using compatible reference populations, before comparing magnitudes. Then keep the separate question—whether the negative control diagnoses or calibrates the relevant bias—explicitly open.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

An odds ratio is 2.00 at a reference risk of 0.20. What comparison risk and risk difference does it imply?
Why can identical odds ratios for two outcomes correspond to different risk differences?
A mean difference is 3 units and the stated reference standard deviation is 6 units. What is the standardized difference, and what does it mean?
Why might estimates calculated for different covariate distributions need standardization before a risk-difference comparison?
Does putting a treatment association and a negative-control association on the same scale prove that the latter measures the bias in the former? Explain.

Solutions. 1. \(p_1=\{2.00(0.20)\}/\{1-0.20+2.00(0.20)\}=0.40/1.20=0.3333\). Thus \(RD=0.3333-0.20=0.1333\), or 13.33 percentage points. 2. An odds ratio maps a reference risk to a comparison risk; changing the reference risk changes the absolute difference produced by the same ratio. 3. \(d_{\mathrm{std}}=3/6=0.500\) reference standard deviations. The estimated difference is half of the stated reference standard deviation. 4. Different covariate mixes can produce different averages of stratum-specific risks. Applying common weights compares the associations in the same stated target distribution, provided the predictions are adequately supported. 5. No. Scale alignment makes the magnitudes interpretable on a shared metric; it does not show that the associations share the same bias-generating mechanisms or satisfy a calibration model.