Finite Estimates Are Not Necessarily Reliable Estimates
A sparse logistic table can present two different problems that should not be conflated. Separation can make the maximum-likelihood estimate (MLE) fail to exist as a finite coefficient. But even when the MLE exists and every coefficient is finite, a small sample can make the estimate strongly biased and unstable. This tutorial isolates the second point using the simplest setting in which it can be seen exactly: a logistic regression with a binary group indicator.
Consider independent binomial samples from two groups. Let \(Y_0\) and \(Y_1\) be the numbers of successes among \(n_0\) and \(n_1\) observations, respectively, with success probabilities \(p_0\) and \(p_1\). In a logistic model with group indicator \(x\in\{0,1\}\), \(\operatorname{logit}(p_x)=\alpha+\beta x\). Thus \(\alpha=\operatorname{logit}(p_0)\), and \(\beta=\operatorname{logit}(p_1)-\operatorname{logit}(p_0)\). The coefficient \(\beta\) is the difference between the groups’ log odds.
Deriving the Finite MLE
The binomial likelihood, apart from factors not involving the probabilities, is \(p_0^a(1-p_0)^b p_1^c(1-p_1)^d\). The two probabilities can vary independently over \((0,1)\), because the transformation between \((\alpha,\beta)\) and \((p_0,p_1)\) is one-to-one: \(\alpha=\operatorname{logit}(p_0)\) and \(\beta=\operatorname{logit}(p_1)-\operatorname{logit}(p_0)\).
Proof. The log-likelihood, omitting binomial coefficients that do not depend on the parameters, is
It separates into one term for each group. For group 0, differentiating with respect to \(p_0\) gives \(a/p_0-b/(1-p_0)\); setting this equal to zero gives \(p_0=a/(a+b)\). The second derivative is \(-a/p_0^2-b/(1-p_0)^2<0\) throughout \((0,1)\), so this stationary point is the unique maximum for group 0. The same calculation gives the unique maximum \(p_1=c/(c+d)\) for group 1. Both are strictly between zero and one because all four counts are positive. The one-to-one transformation to \((\alpha,\beta)\) therefore gives a unique finite maximum there. Finally,
This also identifies the estimated odds ratio comparing group 1 with group 0: it is \(\exp(\hat\beta)=bc/(ad)\). \(\square\)
Worked Example: A Finite Estimate From Small Counts
Suppose group 0 has 1 success and 2 failures, while group 1 has 2 successes and 1 failure. Here \(a=1\), \(b=2\), \(c=2\), and \(d=1\). Every cell is positive, so the theorem applies.
The fitted probabilities are \(\hat p_0=1/(1+2)=1/3\) and \(\hat p_1=2/(2+1)=2/3\). The intercept is \(\hat\alpha=\log(1/2)=-\log 2\), and the group coefficient is \(\hat\beta=\log((2)(2)/((1)(1)))=\log 4\), approximately \(1.3863\). The fitted log odds in group 1 are \(-\log 2+\log 4=\log 2\), as required by \(\hat p_1=2/3\).
The estimate exists and is finite, but it is based on only six observations. Its magnitude alone should not be read as strong evidence of a stable, large population effect.
Exact Conditional Calculation of Small-Sample Bias
A single finite estimate cannot establish bias: bias is a repeated-sampling property. In sparse data, however, the ordinary MLE may fail to be finite on some possible samples. We can still calculate exactly how much bias remains among samples for which it is finite, provided we state the conditioning event clearly.
Suppose each group has \(n=3\) observations, with true probabilities \(p_0=0.2\) and \(p_1=0.8\). The true coefficient is \(\beta=\operatorname{logit}(0.8)-\operatorname{logit}(0.2)=\log 4-(-\log 4)=\log 16=4\log 2\). Let \(Y_0\) and \(Y_1\) be the success counts. Condition on the event \(A=\{Y_0\in\{1,2\},\,Y_1\in\{1,2\}\}\). On \(A\), all four cells are positive, so the fitted group logits and their difference are finite.
Worked Example: Conditional Bias With Three Observations Per Group
For group 1, \(Y_1\sim\operatorname{Binomial}(3,0.8)\). The two probabilities needed under the conditioning event are \(P(Y_1=1)=3(0.8)(0.2)^2=0.096\) and \(P(Y_1=2)=3(0.8)^2(0.2)=0.384\). Therefore, conditional on \(Y_1\in\{1,2\}\), the probabilities of 1 and 2 successes are \(0.096/0.48=0.2\) and \(0.384/0.48=0.8\), respectively. The sample logit is \(-\log 2\) for one success and \(+\log 2\) for two successes. Hence
For group 0, \(Y_0\sim\operatorname{Binomial}(3,0.2)\). Now \(P(Y_0=1)=3(0.2)(0.8)^2=0.384\) and \(P(Y_0=2)=3(0.2)^2(0.8)=0.096\), so the conditional probabilities of 1 and 2 successes are \(0.8\) and \(0.2\). Consequently,
The group samples are independent, and the event \(A\) is the intersection of separate conditions on each group. Their conditional distributions therefore remain independent. Since \(\hat\beta=\operatorname{logit}(Y_1/3)-\operatorname{logit}(Y_0/3)\) on \(A\), linearity of expectation gives \(E[\hat\beta\mid A]=0.6\log 2-(-0.6\log 2)=1.2\log 2\approx0.8318\). The conditional bias relative to the true \(\beta=4\log 2\) is \(1.2\log 2-4\log 2=-2.8\log 2\approx-1.9408\). Thus, among these interior samples, the finite estimate is biased substantially downward.
This is a conditional bias calculation, not an unconditional bias claim for a finite-valued MLE. The conditioning event has probability \(0.48\times0.48=0.2304\). Outside it, at least one group has zero successes or zero failures, and the unrestricted two-group MLE is not finite. The calculation isolates the bias among finite fits rather than treating nonexistence as an ordinary finite estimate.
Why Sparse Interior Tables Still Demand Caution
The previous example separates two issues. On the event \(A\), every cell is positive and the MLE exists; nevertheless, its conditional expectation is far from the true coefficient. Outside \(A\), the failure is different: a group has only successes or only failures, and the likelihood is maximized at a boundary probability rather than at a finite logistic coefficient. A finite estimate is therefore not a certificate of negligible bias, and nonexistence should not be described as merely “very large bias.”
Small counts also make estimates sensitive to individual observations. In the two-group model, changing one cell by one observation changes \(\hat\beta=\log(bc/(ad))\) directly. Large-sample standard-error approximations can be especially fragile in this setting. For reference, when all group counts are sufficiently large, a first-order approximation for the variance of the estimated log odds ratio is \(1/a+1/b+1/c+1/d\). Its justification uses large-sample approximations for the log of the sample odds in each group; with cells of size one or two, it should not be treated as a reliable small-sample guarantee.
Worked Example: A Sparse Estimate With a Large Approximate Standard Error
Suppose group 0 has 1 success and 9 failures, and group 1 has 2 successes and 8 failures. All four cells are positive, so the MLE is finite. The fitted probabilities are \(1/10=0.1\) and \(2/10=0.2\), and
The first-order variance expression gives \(1/1+1/9+1/2+1/8=1+0.1111+0.5+0.125=1.7361\), so its square root is approximately \(1.3176\). That approximate standard error is larger than the estimated coefficient. It signals substantial uncertainty, but because the table is sparse, the approximation itself may be poor; it does not repair the bias or guarantee accurate inference.
Common Mistake: Treating Finiteness as a Bias Correction
- Wrong reasoning: “Every cell is positive, so the estimate is finite; therefore the estimate is not seriously biased.”
- Correction: Positive counts establish a finite MLE in this saturated two-group model. They do not establish that its sampling expectation is close to the true coefficient. Bias depends on repeated sampling, and the conditional calculation above exhibits substantial bias among finite fits.
- Another wrong conclusion: “An estimate that would diverge under separation is just an especially biased finite estimate.”
- Correction: These are distinct situations. With an interior table the likelihood has a finite maximizer; with an extreme group count, the unrestricted MLE may not be finite at all. State which situation applies before interpreting the estimate.
Check Your Understanding
Use the two-group table notation from this tutorial.
- For \(a=3\), \(b=5\), \(c=4\), and \(d=6\), calculate \(\hat p_0\), \(\hat p_1\), and \(\hat\beta\).
- Why does positivity of all four cells guarantee a finite estimate in this two-group model?
- In the conditional bias example, why is \(E[\hat\beta\mid A]\) not an unconditional bias calculation for a finite-valued MLE?
- Explain why a finite estimate from a sparse table can still be strongly biased.
- What is the difference between a boundary/nonexistence problem and a finite but unstable estimate?