Why Profile an Interval?
After Firth bias reduction, a coefficient estimate is the maximizer of a penalized log-likelihood, not the ordinary logistic log-likelihood. A Wald interval summarizes uncertainty using the estimate and a standard error; its endpoints are symmetric on the coefficient scale. A profile interval instead asks which fixed coefficient values remain compatible with the data after the other coefficients have been reoptimized. That reoptimization can produce an asymmetric interval, which matters when data are sparse or a fitted probability is near zero or one.
The construction is an inversion of a penalized likelihood-ratio comparison. It is useful to distinguish this procedure from exact conditional inference: the cutoff used here is an asymptotic chi-square calibration, not an exact finite-sample guarantee.
Definition and Construction
For logistic regression with coefficient vector \(\beta\), write the Firth penalized log-likelihood as \(\ell_F(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|\), where \(\ell\) is the ordinary log-likelihood and \(I(\beta)\) is the expected Fisher information. Terms that are constant in \(\beta\) do not affect maximization or likelihood-ratio differences. Let \(\hat\beta_F\) maximize \(\ell_F\). To form an interval for coefficient \(\beta_j\), collect the remaining coefficients in \(\lambda\), fix \(\beta_j=b\), and maximize the full penalized objective over \(\lambda\).
The supremum is over nuisance coefficients at the fixed value \(b\); it is not obtained by simply substituting \(b\) into the unrestricted fit. The Jeffreys-information penalty is part of every constrained optimization, and it is evaluated for the full model at each candidate parameter value. If the resulting set has multiple components, report that fact rather than silently treating all solutions as one ordinary interval; in routine applications, the interval is usually the connected component containing \(\hat\beta_{F,j}\).
Find \(\hat\beta_F\) and the maximized value \(\ell_F(\hat\beta_F)\).
Set \(\beta_j=b\), retain all other terms in the model, and maximize \(\ell_F\) over the nuisance coefficients.
Compare the constrained maximum with the unrestricted maximum using \(R_F(b)\).
Locate the values of \(b\) where \(R_F(b)\) reaches the chosen chi-square cutoff. Values below the cutoff are retained.
Why the Test and Interval Match
For each fixed \(b\), the constrained model is the model used to test the hypothesis \(H_0:\beta_j=b\). The profile statistic measures how much the best penalized fit worsens under that constraint. Inverting these tests gives the interval directly.
This equivalence follows from the test definition, not from a further approximation. For any candidate \(b\), the test rejects precisely when \(R_F(b)>c\). Therefore it does not reject precisely when \(R_F(b)\leq c\). Collecting all nonrejected values gives the displayed set. The choice \(c=\chi^2_{1,1-\alpha}\) supplies the usual approximate confidence calibration.
Under standard regularity conditions, that calibration is asymptotically justified: the model is identifiable and smooth, the true parameter is interior, the information is nonsingular, the parameter dimension is fixed, and the sample size increases. The Firth penalty is of lower order than the ordinary log-likelihood in this regime; locally, its contribution to the likelihood-ratio comparison is asymptotically negligible. Thus the usual one-parameter likelihood-ratio limit is chi-square with one degree of freedom. This argument does not make the interval an exact confidence interval in a small sample.
Worked Examples
Worked Example: A One-Group Firth Profile Interval
Suppose \(Y\sim\operatorname{Binomial}(4,p)\) and the observed count is \(Y=0\). Use the logistic parameter \(\gamma=\log\{p/(1-p)\}\). For an intercept-only binomial model, the Firth estimate is \(\hat p_F=(Y+0.5)/(n+1)=0.5/5=0.1\), so \(\hat\gamma_F=\log(0.1/0.9)\approx-2.1972\).
Up to constants, the penalized objective as a function of \(p\) is \(g(p)=0.5\log(p)+4.5\log(1-p)\). Its maximum occurs at \(p=0.1\). With a 95% cutoff of \(3.841\), an endpoint solves \(2\{g(0.1)-g(p)\}=3.841\), or equivalently \(0.5\log(p/0.1)+4.5\log\{(1-p)/0.9\}=-3.841/2\).
Numerically, the two roots are approximately \(p=0.000838\) and \(p=0.5099\). Using the displayed rounded endpoints, substitution into the displayed equation gives a left side of approximately \(-1.9206\) at the lower endpoint and \(-1.9205\) at the upper endpoint, both close to the target \(-1.9205\). Transforming the limits to the logit scale gives \(\log(0.000838/0.999162)\approx-7.084\) and \(\log(0.5099/0.4901)\approx0.0396\). Thus the profile interval for \(\gamma\) is approximately \([-7.08,\,0.04]\).
For comparison, the expected information for \(\gamma\) at the Firth estimate is \(4(0.1)(0.9)=0.36\). The usual information-based Wald standard error is \(\sqrt{1/0.36}\approx1.6667\). The Wald interval is \(-2.1972\pm1.96(1.6667)\), or approximately \([-5.46,\,1.07]\). It is symmetric around the estimate, whereas the profile limits are far apart and asymmetric. Both are approximations; the difference illustrates why the profile shape can be informative when the data are sparse.
Worked Example: Reoptimizing a Nuisance Coefficient
Consider two independent binomial groups, each of size 4, with observed counts \(y_1=0\) and \(y_2=3\). Write \(\operatorname{logit}(p_1)=\alpha\) and \(\operatorname{logit}(p_2)=\alpha+\beta\). The parameter \(\beta\) is the group log-odds contrast. The Firth fitted probabilities are \(p_1=(0+0.5)/5=0.1\) and \(p_2=(3+0.5)/5=0.7\), giving \(\hat\beta_F=\operatorname{logit}(0.7)-\operatorname{logit}(0.1)=\log(21)\approx3.0445\).
To profile at the trial value \(\beta=\log(3)\), the nuisance parameter \(\alpha\) must be reoptimized. Up to constants, the full penalized objective is \(g(p_1,p_2)=0.5\log(p_1)+4.5\log(1-p_1)+3.5\log(p_2)+1.5\log(1-p_2)\). At fixed \(\beta\), \(p_2=3p_1/(1+2p_1)\). Differentiating with respect to \(\alpha\), which shifts both group logits, gives the constrained score \((0.5-5p_1)+(3.5-5p_2)=4-5(p_1+p_2)\).
The maximizing nuisance value therefore satisfies \(p_1+p_2=0.8\). Substituting the fixed-\(\beta\) relationship gives \(p_1+3p_1/(1+2p_1)=0.8\), hence \(5p_1^2+6p_1-2=0\). The admissible root is \(p_1=(-3+\sqrt{19})/5\approx0.2718\), and then \(p_2\approx0.5282\). The second derivative with respect to \(\alpha\) is \(-5p_1(1-p_1)-5p_2(1-p_2)<0\), so this stationary point is the unique constrained maximum. This calculation shows what “profile” requires: the nuisance intercept is refitted at the trial contrast, not held at its unrestricted estimate.
Worked Example: Testing No Group Contrast
For the same two-group data, test \(\beta=0\) by profiling over \(\alpha\). Under this constraint both groups share probability \(p\). The ordinary log-likelihood contributes \(3\log(p)+5\log(1-p)\), since the groups have 3 total successes and 5 total failures. The full-model Jeffreys penalty under the constraint contributes \(\log(p)+\log(1-p)\), up to a constant. Thus the constrained objective is \(4\log(p)+6\log(1-p)\), maximized at \(p=4/(4+6)=0.4\).
At the unrestricted Firth fit, the objective is \(0.5\log(0.1)+4.5\log(0.9)+3.5\log(0.7)+1.5\log(0.3)\approx-4.6797\). At the constrained maximum it is \(4\log(0.4)+6\log(0.6)\approx-6.7301\). Using the unrounded objective values, \(R_F(0)=2[\{0.5\log(0.1)+4.5\log(0.9)+3.5\log(0.7)+1.5\log(0.3)\}-\{4\log(0.4)+6\log(0.6)\}]\approx4.1008\).
Since \(4.1008>3.841\), zero is excluded by this nominal 95% profile test. This is a test of the contrast \(\beta\), not an interval calculation for the intercept or either group probability. Keeping the target coefficient clear is essential when profiling.
Profile and Wald Intervals
A Wald interval uses a local quadratic approximation around the fitted coefficient. In the usual information-based form it is \(\hat\beta_{F,j}\pm z_{1-\alpha/2}\sqrt{[I(\hat\beta_F)^{-1}]_{jj}}\). It is convenient to compute, but its symmetry is imposed by construction. A profile interval instead follows the change in the optimized penalized objective as the target coefficient moves; nuisance parameters can adapt differently on either side of the estimate.
Profile intervals can therefore be strongly asymmetric, and their endpoints need not be close to Wald endpoints in sparse data. They also require repeated constrained maximizations, so they can be more computationally demanding. The asymmetry is not itself evidence of an error: it can reflect the uneven information in the data. Conversely, a profile interval is not guaranteed to have better coverage in every finite sample. Its calibration still depends on the approximation and on the model being appropriate.
Common Pitfall
- Holding nuisance coefficients fixed. Substituting a candidate coefficient into the unrestricted estimate does not produce a profile likelihood. Refit every nuisance coefficient at each candidate value.
- Dropping the penalty in the constrained fit. Both the unrestricted and constrained objectives must be Firth penalized log-likelihoods, with the information penalty evaluated for the full model.
- Calling the interval exact. The chi-square cutoff gives asymptotic calibration. It does not guarantee nominal coverage for a particular small or sparse sample.
- Assuming profile limits must be symmetric. Symmetry belongs to the Wald construction, not to the profile definition.
Check Your Understanding
Use the definitions and examples above to answer these questions.
- In a model with target coefficient \(\beta_j\) and nuisance vector \(\lambda\), what is held fixed and what is reoptimized to calculate \(\ell_{F,p}(b)\)?
- Why must the information penalty be evaluated in the constrained full model rather than omitted during profiling?
- What is the relationship between a profile interval and the corresponding family of penalized likelihood-ratio tests?
- Give one reason a Firth profile interval can differ substantially from a Wald interval.
- Does using a chi-square cutoff make a profile penalized-likelihood interval exact in a small sample? Explain.