Why Look for a Tipping Point?
An estimate from an observational study can be sensitive to a confounder that was not measured. A tipping-point analysis asks a focused question: how strong would that confounder have to be, under specified assumptions, to change the conclusion? Rather than selecting one set of assumptions and reporting one adjusted estimate, the analyst lays out a grid of plausible values and checks what the estimate would be in each case.
The grid is a way to make sensitivity assumptions visible. It does not establish that an unmeasured confounder exists, nor does it identify its true associations from the observed data. Its value is in showing whether the conclusion changes under assumptions that subject-matter experts regard as plausible.
A Binary-Confounder Bias Model
Let \(A\) be a binary treatment, \(Y\) a binary adverse outcome, and \(U\) a binary unmeasured confounder. Use the risk difference scale, with the treatment effect written as treated minus untreated risk, so a negative effect indicates benefit when \(Y\) is adverse. Suppose the observed, unadjusted risk difference is \(\widehat{\Delta}_{\mathrm{obs}}\).
For this formula, the outcome model can be written as \(E[Y\mid A=a,U=u]=\alpha+\tau a+\gamma u\), with risks in the range zero to one. It assumes that this model is appropriate and that the treatment effect does not vary with \(U\). Interpreting \(\tau\) as the causal risk difference additionally requires consistency, conditional exchangeability of treatment given \(U\), and positivity of treatment within relevant \(U\) strata. The bias formula is for this stated model and scale; it is not a universal adjustment formula for every estimator or outcome.
Often it is more natural to specify the overall prevalence of \(U\) and how \(U\) is associated with treatment. Let \(q=P(U=1)\), \(a_1=P(A=1\mid U=1)\), and \(a_0=P(A=1\mid U=0)\). The treatment association can be summarized by \(a_1-a_0\). Bayes’ rule gives the corresponding prevalences in the treatment groups:
These quantities must be coherent: for example, \(q\), \(a_1\), and \(a_0\) jointly determine \(p_1\) and \(p_0\). Varying them independently without checking this relationship can produce an impossible scenario. Values of \(a_1\) and \(a_0\) should also be strictly between zero and one when the scenario is meant to represent treatment groups in which both levels of \(U\) can occur.
Why the Bias Formula Works
Under the additive model, the expected outcome risk in each observed treatment group is found by averaging over that group’s distribution of \(U\). For the treated group, this is \(\alpha+\tau+\gamma p_1\); for the untreated group, it is \(\alpha+\gamma p_0\). Subtracting the second risk from the first gives:
In this model the causal risk difference is \(\tau\), because the effect is constant across levels of \(U\). Thus the observed risk difference equals the causal effect plus \(B=\gamma(p_1-p_0)\). Subtracting that bias yields the adjustment formula. This argument also clarifies the sign: if \(U\) raises adverse-outcome risk and is more prevalent among treated people, then \(B\) is positive and the observed treated-minus-untreated difference is shifted upward.
Worked Example: Confounder More Common Among Controls
Suppose a hypothetical study reports adverse-outcome risks of 12% in the treated group and 22% in the untreated group. The observed risk difference is \(0.12-0.22=-0.10\), or 10 percentage points fewer adverse outcomes among treated people. Suppose a binary adverse-outcome risk factor is present in 20% of treated people and 60% of untreated people, and its conditional outcome association is \(\gamma=0.25\), a 25-percentage-point increase in risk.
Here \(p_1=0.20\) and \(p_0=0.60\), so the bias is:
The adjusted estimate is:
Under these assumptions, the adjusted risk difference is zero. Because the adverse-outcome confounder is more prevalent among controls, the bias is negative: it shifts the observed treated-minus-untreated difference downward. The observed estimate therefore exaggerates the apparent treatment benefit; removing the modeled bias eliminates that apparent benefit.
From Prevalence and Treatment Association to Bias
A grid can be built either from \(p_1-p_0\) directly or from assumptions about \(q\), \(a_1\), and \(a_0\). The second approach is often easier to discuss with subject-matter experts: they may be able to judge how common a risk factor is overall and whether it makes treatment more or less likely. The formulas above then translate those assumptions into a treatment-group prevalence contrast.
Worked Example: Calculating a Scenario from Treatment Probabilities
Consider a hypothetical analysis with observed risk difference \(-0.08\), or 8 percentage points fewer adverse outcomes in the treated group. Suppose \(U\) has overall prevalence \(q=0.30\), and the treatment probabilities are \(a_1=0.60\) for people with \(U=1\) and \(a_0=0.30\) for people with \(U=0\). Assume \(U\) raises outcome risk by \(\gamma=0.20\), or 20 percentage points.
First, the overall treatment probability is \(0.30(0.60)+0.70(0.30)=0.39\). Therefore:
Among untreated people, the probability of \(U=1\) is \(0.30(1-0.60)=0.12\), and the overall probability of being untreated is \(1-0.39=0.61\). Hence:
The bias and adjusted estimate are:
In this scenario, the adverse-outcome confounder is more prevalent among treated people and raises outcome risk, so it creates positive bias. The observed estimate is shifted upward toward zero; under the model, removing that bias gives a more negative risk difference, about 13.3 percentage points fewer adverse outcomes. This is a bias-adjusted point estimate, not a claim that the causal effect is known exactly.
Building and Reading a Tipping-Point Grid
A tipping point is a combination of assumptions at which a chosen interpretation changes. For a point estimate, the sign changes when the adjusted estimate reaches zero. If \(\widehat{\Delta}_{\mathrm{obs}}=-0.08\), then:
The same estimate might be considered clinically important only if it is at most \(-0.05\), representing at least a 5-percentage-point reduction in adverse outcomes. The boundary for that decision is different:
Thus a grid should state which decision it examines. A sign change, loss of a clinically meaningful effect, and loss of statistical significance are distinct events. The following hypothetical grid uses an observed risk difference of \(-0.08\) and treats values at or below \(-0.05\) as clinically meaningful benefit. The columns \(p_1\) and \(p_0\) are calculated from each row’s prevalence and treatment probabilities.
Worked Example: A Grid That Changes the Decision
For each row, use \(B=\gamma(p_1-p_0)\) and \(\widehat{\Delta}_{\mathrm{adj}}=-0.08-B\). For example, in the first row the overall prevalence is \(q=0.25\), and treatment probabilities are \(a_1=0.50\) and \(a_0=0.30\). These give \(p_1=0.125/(0.125+0.225)=0.3571\) and \(p_0=0.125/(0.125+0.525)=0.1923\). The bias is \(0.20(0.3571-0.1923)=0.0330\), so the adjusted estimate is \(-0.1130\).
| \(q\) | \(a_1\) | \(a_0\) | \(p_1-p_0\) | \(\gamma\) | \(B\) | Adjusted estimate | Interpretation |
|---|---|---|---|---|---|---|---|
| 0.25 | 0.50 | 0.30 | 0.1648 | 0.20 | 0.0330 | -0.1130 | Clinically meaningful benefit |
| 0.25 | 0.30 | 0.50 | -0.1515 | 0.20 | -0.0303 | -0.0497 | Just above the stated threshold |
| 0.40 | 0.20 | 0.50 | -0.3056 | 0.10 | -0.0306 | -0.0494 | Just above the stated threshold |
| 0.40 | 0.10 | 0.60 | -0.5000 | 0.15 | -0.0750 | -0.0050 | Does not meet clinical-benefit threshold |
For the final row, the calculations are \(p_1=0.04/(0.04+0.36)=0.1000\) and \(p_0=0.36/(0.36+0.24)=0.6000\). Thus \(B=0.15(0.1000-0.6000)=-0.0750\), and \(-0.08-(-0.0750)=-0.0050\). This scenario does not cross the zero boundary. It does cross the clinical threshold: the adjusted estimate is no longer at or below \(-0.05\).
The middle rows illustrate a different tipping point. Their adjusted estimates are close to, but not at or below, the prespecified \(-0.05\) threshold. The assumptions do not reverse the sign, but they do change whether the estimate meets this particular clinical criterion. A complete analysis would also justify whether the confounder prevalence, treatment probabilities, and outcome association in each row are plausible for the application.
Common Pitfalls and Interpretation
The direction of bias is determined by the product \(\gamma(p_1-p_0)\), not by the outcome association alone. If \(U\) raises adverse-outcome risk, then \(\gamma>0\). When \(U\) is more prevalent among treated people, \(p_1-p_0>0\), so the bias is positive: it makes a beneficial negative estimate less negative. When \(U\) is more prevalent among controls, \(p_1-p_0<0\), the bias is negative and makes the observed estimate more negative than the modeled causal effect. In that case, the observed estimate exaggerates apparent benefit.
- Reversing the adjustment sign. The observed estimate equals the causal effect plus bias. Therefore, subtract \(B\) from the observed estimate; do not add it.
- Using incompatible assumptions. Overall prevalence and treatment probabilities determine \(p_1\) and \(p_0\). Calculate the implied group prevalences before calculating bias.
- Calling a point-estimate threshold a significance threshold. A grid of adjusted point estimates does not by itself recompute a standard error, confidence interval, or p-value. A claim about statistical significance requires an appropriate uncertainty analysis.
- Interpreting the grid as evidence about the true confounder. The grid shows consequences of assumed values. It does not estimate those values from the observed data.
- Ignoring the outcome scale and model. The product formula here is for a binary confounder, a risk difference, and the stated additive model. Interactions, other outcome scales, or a different estimator may require a different bias calculation.
A useful report names the observed estimate and its scale, gives the confounder parameters varied, shows how each combination maps to bias, and states the exact tipping criterion. It should then distinguish conclusions that are stable across plausible scenarios from those that depend on a narrow or extreme set of assumptions.
Check Your Understanding
Work through these questions using the risk-difference model stated above.
- If \(\gamma=0.18\), \(p_1=0.25\), \(p_0=0.50\), and the observed estimate is \(-0.06\), calculate the bias and adjusted estimate. Does the adjustment make the estimate more or less beneficial?
- Explain why \(q\), \(a_1\), and \(a_0\) should be used to calculate \(p_1\) and \(p_0\), rather than treated as unrelated grid entries.
- For an observed estimate of \(-0.07\), what value of bias places the adjusted point estimate at zero?
- Suppose an adverse-outcome confounder is more common among treated people and has a positive outcome association. What is the sign of the bias, and in which direction does it shift the observed estimate?
- Why does a tipping point for a clinically meaningful point estimate not automatically show where statistical significance is lost?