Wald Inference After Firth Bias Reduction

From a Finite Estimate to an Interval

Firth bias reduction can produce finite logistic regression estimates even when ordinary maximum likelihood fails because of separation. A finite estimate, however, is not itself an uncertainty statement. Wald inference adds an estimated standard error and uses a normal approximation to construct intervals or tests. This tutorial develops that calculation and explains what its covariance matrix represents.

The central distinction is between the method used to obtain the point estimate and the information used for its usual model-based Wald standard error. The estimate solves Firth's adjusted score equations, whereas the standard covariance calculation uses the ordinary logistic model's Fisher information, evaluated at that estimate. The resulting intervals are approximate; finiteness alone does not make them exact or guarantee good coverage.

Information-Based Covariance

Consider independent binomial responses \(Y_i\), with \(m_i\) trials and success probability \(p_i\), and a logistic linear predictor \(\eta_i=x_i^\mathsf{T}\beta\). Here \(x_i\) is a column vector of covariates and \(\beta\) is the coefficient vector. For Bernoulli data, \(m_i=1\). The ordinary Fisher information at a coefficient vector \(\beta\) is

If the design matrix \(X\) has full column rank and all fitted probabilities lie strictly between zero and one, this information is positive definite. For a Firth estimate \(\hat\beta_F\), the usual model-based covariance estimate and standard error for coefficient \(j\) are

The estimate in these expressions is the Firth estimate, but \(I\) is the ordinary model Fisher information—not the Hessian of the Jeffreys-penalized log-likelihood. This is a common model-based Wald convention. A different covariance convention based on penalized curvature would require its own definition and interpretation; it should not be silently substituted into these formulas.

Why the Information Formula Has This Form

The information calculation follows from the variance of the ordinary logistic score. For the binomial model, the score is \(U(\beta)=X^\mathsf{T}(Y-mp)\), where \(mp\) denotes the vector with entries \(m_i p_i\). Independence gives a diagonal covariance matrix for \(Y\), with diagonal entries \(m_i p_i(1-p_i)\). Consequently, the score covariance equals \(I(\beta)\). When the model is locally identifiable, the inverse information is the usual first-order covariance for estimating \(\beta\). After Firth fitting, evaluating it at \(\hat\beta_F\) gives the stated plug-in estimate.

The same calculation also gives uncertainty for any linear contrast. Let \(c\) be a fixed coefficient vector, and let \(\psi=c^\mathsf{T}\beta\). Its estimate is \(c^\mathsf{T}\hat\beta_F\), and its estimated variance is

One way to see the approximation is to write \(I_n(\beta_0)/n\to J\), where \(J\) is positive definite and \(\beta_0\) is the true coefficient vector. Assume, in addition, that the covariates are bounded, the true probabilities stay away from zero and one, and the Firth adjustment \(A(\beta)\) and its derivative are bounded in probability in a neighborhood of \(\beta_0\). The adjusted score is \(U(\beta)+A(\beta)\). Expanding it about \(\beta_0\), the score equation at a consistent Firth estimate gives

The ordinary score has mean zero and covariance \(I_n(\beta_0)\); boundedness and the score central limit theorem therefore give \(U(\beta_0)/\sqrt n\xrightarrow{d}N(0,J)\). Since \(A(\beta_0)=O_p(1)\), its contribution divided by \(\sqrt n\) vanishes. Rearranging the expansion yields

Also, \(I_n(\hat\beta_F)/n\xrightarrow{p}J\), so standardizing by the plug-in information gives the stated asymptotic Wald approximation. This argument concerns regular large-sample behavior. In a small or sparse sample, the normal approximation may be poor even when the Firth estimate and its information matrix are finite.

Worked Wald Calculations

Wald Intervals and Tests Are Two Views of the Same Calculation

For a scalar contrast \(\psi=c^\mathsf{T}\beta\), let \(\hat\psi=c^\mathsf{T}\hat\beta_F\) and \(s^2=c^\mathsf{T}I(\hat\beta_F)^{-1}c\). Its Wald interval is \([\hat\psi-z_{1-\alpha/2}s,\hat\psi+z_{1-\alpha/2}s]\). The two-sided test of \(H_0:\psi=\psi_0\) rejects when

Rearranging the non-rejection condition gives \(|\hat\psi-\psi_0|\leq z_{1-\alpha/2}s\), which is equivalent to \(\psi_0\) lying inside the interval. Thus, for the same contrast and critical value, the Wald interval and test are equivalent ways to report the calculation. For a vector of \(q\) restrictions \(C\beta=d\), the corresponding Wald statistic is

which is compared approximately with a chi-squared distribution with \(q\) degrees of freedom under the same large-sample conditions.

Interpretation and a Common Pitfall

Ordinary maximum-likelihood Wald inference evaluates information at the maximum-likelihood estimate. After Firth bias reduction, the estimate changes, so the fitted probabilities and the information-based standard errors change as well. The inferential framework remains a model-based normal approximation; the bias correction does not turn it into an exact procedure. In particular, when separation has made the ordinary maximum-likelihood estimate diverge, the Firth estimate can make a finite Wald calculation possible, but the Wald approximation may still be unreliable in a small sample.

• Pitfall: treating finite as precise. A finite Firth coefficient can have a very large standard error. Check the information matrix and interval width rather than concluding that finiteness guarantees stability.
• Pitfall: using penalized curvature without saying so. The information-based covariance described here is \(I(\hat\beta_F)^{-1}\). The Hessian of the penalized log-likelihood is a different matrix and does not give the same calculation by definition.
• Pitfall: interpreting the coefficient interval on the wrong scale. Logistic coefficients are log odds ratios. Exponentiate both endpoints to obtain an odds-ratio interval; do not exponentiate only the estimate or the standard error.
• Pitfall: dropping covariance terms for contrasts. For a linear combination of coefficients, use \(c^\mathsf{T}\widehat Vc\), including off-diagonal terms.