Robustness Values for Losing Statistical Significance

Why Ask How Robust Statistical Significance Is?

A statistically significant estimate can still be sensitive to an omitted confounder. Sensitivity analysis makes that concern quantitative: it asks how strongly an unmeasured covariate would have to be associated with both treatment and outcome, conditional on the measured covariates, to make the confidence interval include zero. The result is a robustness value, not a probability that a confounder exists or that the causal conclusion is correct.

This tutorial concerns the robustness value for losing statistical significance. It builds on partial \(R^2\) measures and bias-adjusted estimates. The calculations below use a linear regression setting and conventional OLS standard errors. In particular, they account for the residual degrees of freedom used by the adjusted model, which change when additional covariate columns are included.

Model, Strength Measures, and Robustness Value

Suppose the fitted regression includes an outcome \(Y\), a treatment or exposure \(D\), and measured covariates \(X\). Let \(\hat{\tau}\) be the coefficient of \(D\), \(SE(\hat{\tau})\) its conventional standard error, and \(df\) the residual degrees of freedom in this original regression. An omitted covariate block \(U\) consists of \(q\) columns. We compare the original regression with the regression that also includes those \(q\) columns.

The partial \(R^2\) strengths describe how much of the remaining variation is associated with \(U\) in two different regressions. The treatment-side strength concerns how well \(U\) predicts \(D\) after adjusting for \(X\). The outcome-side strength concerns how well \(U\) predicts \(Y\) after adjusting for both \(D\) and \(X\).

This equal-strength definition gives one summary number. The two strengths need not be equal in a particular sensitivity analysis; an unequal pair \((R_D,R_Y)\) can also be assessed. The robustness value is a threshold under a specified model and inference procedure, not an estimate of the actual strength of hidden confounding.

For the formulas and examples here, assume the same observations are used in both regressions, the relevant design matrices have full rank, and the usual OLS standard errors are used. For \(t\)-based confidence intervals, use the adjusted model's residual degrees of freedom \(df-q\). Exact small-sample \(t\) inference additionally relies on the usual linear-model error assumptions, including independent, homoskedastic normal errors; other standard-error or inference procedures require corresponding adjustments.

Bias Bound and Adjusted Standard Error

Given \(R_D\) and \(R_Y\), the largest omitted-variable bias in the direction that moves \(\hat{\tau}\) toward zero, expressed in units of the original standard error, is bounded by

The bound allows the omitted block's associations to align in the most damaging way consistent with the two partial \(R^2\) values. It is not a claim that every omitted variable with those strengths produces that much bias. The adjusted standard error also changes when \(U\) is added:

The factor \(df/(df-q)\) is needed for the conventional OLS standard error: the residual sum of squares changes as outcome variation is explained by \(U\), and the residual variance estimate is divided by \(df-q\), rather than \(df\). If \(t=|\hat{\tau}|/SE(\hat{\tau})\), the smallest adverse adjusted absolute \(t\)-statistic allowed by the bias bound is

At a two-sided significance level \(\alpha\), the adjusted confidence interval includes zero when \(t_{\mathrm{adj}}\le t_{1-\alpha/2,\,df-q}\). The maximum with zero reflects that, if the permitted bias can reach the estimate, an adverse bias can bring the adjusted estimate to zero rather than force it past zero.

Why These Formulas Have This Form

A short geometric argument explains the bias bound. After removing the linear association with \(X\), the treatment has a residual component not explained by \(X\). The partial \(R^2\), \(R_D\), is the fraction of this residual treatment variation explained by the component of \(U\) not explained by \(X\). After removing \(D\) and \(X\) from the outcome, \(R_Y\) is the fraction of the remaining outcome variation explained by the corresponding residualized \(U\).

The omitted-variable bias is determined by the product of the treatment's projection onto the residualized \(U\) and the outcome association with that residualized \(U\). Cauchy-Schwarz bounds the magnitude of this product by the product of the two projection lengths. Expressing those lengths as fractions of residual sums of squares yields the factor \(R_YR_D/(1-R_D)\). Scaling by the original residual degrees of freedom and standard error gives \(B=\sqrt{dfR_YR_D/(1-R_D)}\). This is a worst-direction bound; using it asks how damaging an omitted block could be, not what its bias must be.

For the standard-error ratio, adding \(U\) reduces the outcome residual sum of squares by the fraction \(R_Y\), and reduces the residual variation in \(D\) by the fraction \(R_D\). The coefficient variance is proportional to the outcome residual variance divided by that treatment residual variation. The variance estimate also changes from division by \(df\) to division by \(df-q\), giving the stated ratio. Thus, the degrees-of-freedom adjustment depends on the number of omitted-variable columns.

Equal-Strength Threshold

For the robustness value, set \(R_D=R_Y=r\). The standard-error ratio simplifies, but it does not become exactly one when \(q>0\):

Consequently, the equal-strength threshold is the smallest \(r\) satisfying

When the threshold is below the point estimate, as it is in the examples, the maximum can be dropped while solving. Define \(a=\{t-t_{1-\alpha/2,\,df-q}\sqrt{df/(df-q)}\}/\sqrt{df}\). Then \(r/\sqrt{1-r}=a\), and the nonnegative solution is

This expression applies when \(a>0\). If the original statistic is already no larger than the adjusted critical value, no positive confounding strength is needed to lose significance. If a different inference method, such as a robust variance estimator, is used, this particular \(t\)-threshold calculation should not be substituted without adapting the standard error and reference distribution.

Interpretation and Common Pitfalls

A robustness value is a scale for sensitivity to omitted confounding. It is useful for asking whether the strength required to alter the significance conclusion seems small or substantial in the context of the measured covariates and subject-matter knowledge. It does not establish that an estimate is unbiased, that an omitted confounder is present, or that a causal effect has been identified. Statistical significance and causal identification are different questions.

The robustness value concerns whether an interval includes zero, not whether an effect becomes clinically unimportant. An estimate can lose statistical significance while remaining clinically important, and an estimate that remains statistically significant can still be too small to matter clinically. It also concerns the zero-effect threshold specifically; other targets require different calculations.

• Ignoring the extra columns. Using \(df\) rather than \(df-q\) for the adjusted model misses a change in the usual OLS standard error and critical value. State \(q\) and use the adjusted degrees of freedom.
• Using only one partial \(R^2\). A confounder must relate to treatment and outcome to generate omitted-variable bias. The bias bound depends jointly on \(R_D\) and \(R_Y\).
• Calling the robustness value a probability. A value such as 19.97% is a partial \(R^2\) strength threshold, not a 19.97% probability that the result is confounded or false.
• Confusing significance with causal validity. A confidence interval excluding zero does not rule out confounding; a robustness value only quantifies one specified sensitivity model.