Separation and Boundary Predictions in Logistic Targeting

Why Logistic Targeting Can Misbehave

In logistic targeting, an initial outcome regression is updated along a specified fluctuation submodel. The update is often a one-parameter logistic regression with the initial prediction as an offset and a clever covariate as its predictor. Even when the model is deliberately simple, its likelihood can have no finite maximizer. In other cases, a finite update exists but the starting predictions are so close to zero or one that computation is sensitive to rounding and optimization choices.

These are related but distinct problems. Separation is a property of the observed outcomes and the fluctuation covariate: the likelihood can keep increasing as the fluctuation parameter diverges. Boundary prediction describes an initial or updated probability close to zero or one. It can make computations fragile, but it does not by itself imply separation. The earlier material on positivity failures concerns a different issue: limitations in the data-generating treatment or censoring probabilities, not simply a logistic likelihood with an infinite estimate.

The Offset-Logistic Fluctuation

For a binary outcome, let $Y_i\in\{0,1\}$ be the observed outcome contribution for row $i$, $Q_i$ its initial predicted probability, and $H_i$ the fixed clever covariate for the chosen fluctuation. Assume $0<Q_i<1$ and finite $H_i$. The one-dimensional fluctuation keeps the offset fixed and varies a scalar $\epsilon$:

\operatorname{logit} Q_i(\epsilon) = \operatorname{logit} Q_i+\epsilon H_i, \qquad Q_i(\epsilon) = \frac{\exp\{\operatorname{logit} Q_i+\epsilon H_i\}} {1+\exp\{\operatorname{logit} Q_i+\epsilon H_i\}}.

The empirical Bernoulli log-likelihood, omitting no terms that depend on $\epsilon$, is

\ell(\epsilon) = \sum_{i=1}^{n} \left[ Y_i\{\operatorname{logit} Q_i+\epsilon H_i\} - \log\{1+\exp(\operatorname{logit} Q_i+\epsilon H_i)\} \right].

Differentiating gives the targeting score equation:

S(\epsilon) = \ell'(\epsilon) = \sum_{i=1}^{n}H_i\{Y_i-Q_i(\epsilon)\} = 0.

A finite solution to this equation is the empirical targeting update for this fluctuation. If the fluctuation is defined with positive observation weights, the corresponding weighted log-likelihood and weighted score must be used consistently; the discussion below assumes unit weights.

Definition: In this one-parameter offset-logistic model, separation occurs when there is a direction $d\in\{-1,1\}$ such that $(2Y_i-1)H_i d\geq 0$ for every row and the inequality is strict for at least one row. Complete separation has strict inequality on every row with $H_i\ne 0$; quasi-complete separation has at least one informative equality. Along the separating direction, the likelihood has no finite maximizer.

Two Results for Diagnosing the Update

Result: For finite offsets and covariates, $\ell(\epsilon)$ is concave, and $$ S'(\epsilon) = -\sum_{i=1}^{n}H_i^2Q_i(\epsilon)\{1-Q_i(\epsilon)\} \leq 0. $$ If at least one $H_i\ne0$, the inequality is strict for every finite $\epsilon$. Thus the score is strictly decreasing, and there can be at most one finite solution.

Proof. Each $Q_i(\epsilon)$ is differentiable, with derivative $H_iQ_i(\epsilon)\{1-Q_i(\epsilon)\}$. Differentiating $S(\epsilon)=\sum_iH_i\{Y_i-Q_i(\epsilon)\}$ therefore yields the displayed expression. For finite offsets, finite $\epsilon$, and finite $H_i$, each $Q_i(\epsilon)$ lies strictly between zero and one. Every summand in $S'$ is nonpositive, and any row with $H_i\ne0$ contributes a strictly negative term. Hence the score is strictly decreasing when at least one covariate is nonzero, which also proves uniqueness of any root. $\square$

Strict decrease gives a practical existence check. Define the one-sided limits $S(-\infty)$ and $S(+\infty)$ by taking the limits of each fitted probability. If $S(-\infty)>0$ and $S(+\infty)<0$, continuity and the intermediate value theorem give a finite root; strict decrease makes it unique. If a separating direction exists, there is no finite maximizer, even if an optimizer returns a very large number.

Result: Under the same finite-data assumptions, a separating direction makes the log-likelihood strictly increase along that direction at every finite parameter value. Consequently, it has a finite supremum approached at infinity but no finite maximum.

Proof. Write $z_i=2Y_i-1$ and $\eta_i(\epsilon)=\operatorname{logit} Q_i+\epsilon H_i$. The $i$th log-likelihood contribution is $-\log\{1+\exp(-z_i\eta_i(\epsilon))\}$. Its derivative in direction $d$ is

d\,\frac{d\ell_i(\epsilon)}{d\epsilon} = \frac{z_iH_i d}{1+\exp\{z_i\eta_i(\epsilon)\}}.

When $z_iH_i d\geq0$, this derivative is nonnegative. It is strictly positive for every finite $\epsilon$ whenever $z_iH_i d>0$, because the denominator is positive and finite. The separation condition supplies at least one such row, so summing the contributions gives a strictly positive directional derivative at every finite value. The likelihood therefore cannot attain its supremum at a finite value along that direction. $\square$

Worked Example: Complete Separation With a Clever Covariate

Suppose two rows have $Q_1=Q_2=0.2$, $H_1=H_2=1$, and $Y_1=Y_2=1$. The common offset is $\operatorname{logit}(0.2)=\log(0.25)$, and the fitted probability on either row is $q(\epsilon)=\operatorname{expit}\{\log(0.25)+\epsilon\}$.

The log-likelihood and score are

\ell(\epsilon)=2\log q(\epsilon), \qquad S(\epsilon)=2\{1-q(\epsilon)\}>0 \quad\text{for every finite }\epsilon.

As $\epsilon$ increases, $q(\epsilon)$ tends to one and the log-likelihood tends to zero, its supremum for these two observations. But $q(\epsilon)<1$ at every finite $\epsilon$, so the supremum is not attained and the score never equals zero. Here $(2Y_i-1)H_i=1$ for both rows: the data are completely separated in the fluctuation direction. Reporting a large numerical estimate as though it were a finite targeting solution would be incorrect.

Boundary Predictions Are Not the Same as Separation

A probability near zero or one has a large-magnitude logit: for example, $\operatorname{logit}(0.001)$ is about $-6.907$. This can cause overflow in a naive calculation of $\exp(\eta)$, loss of precision when probabilities are rounded, or nearly flat likelihood curvature when fitted probabilities approach the endpoints. Yet a boundary-near starting prediction does not establish that the likelihood is separated. The outcome pattern and the signs of the clever covariate determine whether a finite maximizer exists.

Worked Example: A Finite Update With Boundary-Near Starting Values

Take two rows with $H_1=H_2=1$, outcomes $Y_1=1,Y_2=0$, and initial predictions $Q_1=0.01,Q_2=0.9$. Their offsets are $o_1=\log(0.01/0.99)\approx-4.59512$ and $o_2=\log(0.9/0.1)\approx2.19722$. The score equation is

1-\operatorname{expit}(o_1+\epsilon) -\operatorname{expit}(o_2+\epsilon)=0.

For any real $x$, $\operatorname{expit}(x)+\operatorname{expit}(-x)=1$. Set $o_1+\epsilon=-(o_2+\epsilon)$, which gives

\hat\epsilon = -\frac{o_1+o_2}{2} \approx 1.19895.

The updated logits are approximately $-3.39617$ and $3.39617$; the updated probabilities are approximately $0.0324$ and $0.9676$. Their sum is one, so the score is $1-0.0324-0.9676=0$, up to rounding. The predictions remain near the boundaries, but the mixed outcomes make the score cross zero at a finite value. Boundary-near predictions warrant numerical care; they are not, by themselves, proof of separation.

Diagnostics and Remedies That Keep the Model Intact

A useful diagnostic examines the score and likelihood, not just the optimizer’s convergence flag. Evaluate $S(\epsilon)$ at a sequence of increasingly large positive and negative values. A score that approaches zero on one side without changing sign, together with a likelihood that keeps increasing, is consistent with separation. Check the rows with large $|H_i|$, the outcomes on those rows, the range of initial logits, and whether any probabilities were rounded to exactly zero or one before targeting. For a scalar update, a bracketed root finder is often more informative than an optimizer that reports only an extreme parameter estimate.

The following practices can improve computation while preserving the specified fluctuation:

Use stable arithmetic. Compute the logistic function with a numerically stable implementation and evaluate the log-likelihood with a stable log-sum-exp calculation. Avoid first forming a potentially overflowing exponential.
Retain full precision. Do not round $Q_i$ to zero or one before computing its offset. If an initial prediction is exactly at an endpoint, the ordinary finite-offset model is not defined for that row; that is not cured by pretending the offset is finite.
Reparameterize without changing the model. If $H_i$ is rescaled to $cH_i$, where $c\ne0$, use the parameter $\epsilon/c$. Then the product $\epsilon H_i$, and therefore every prediction in the submodel, is unchanged. This can improve optimizer scaling without changing the score equation’s target.
Check for a finite root before optimizing. For the scalar score, inspect its direction and search for a sign-changing bracket. If separation is present, no choice of optimizer can create a finite unpenalized maximum.

Other apparent remedies change the statistical procedure. Clipping an initial prediction changes its offset; bounding $\epsilon$ changes the allowed fluctuation; and adding a ridge or Firth-type penalty changes the objective and generally the score equation. These may be considered as explicitly modified procedures, but their output should not be described as the exact solution of the original empirical targeting equation. Similarly, altering or truncating the clever covariate changes the fluctuation direction. The important distinction is whether a step merely stabilizes computation or changes the model being targeted.

Worked Example: Clipping Changes the Offset and the Update

Let $H_1=H_2=1$, $(Y_1,Y_2)=(1,0)$, and $(Q_1,Q_2)=(0.001,0.5)$. The offsets are $o_1=\log(0.001/0.999)\approx-6.90675$ and $o_2=0$. As in the preceding example, the score is zero when the two updated probabilities sum to one. Thus

\hat\epsilon = -\frac{o_1+o_2}{2} \approx3.45338.

The updated logits are approximately $-3.45338$ and $3.45338$, giving probabilities about $0.0307$ and $0.9693$. Now suppose the first initial prediction is clipped from $0.001$ to $0.01$. Its offset becomes $\log(0.01/0.99)\approx-4.59512$, so the solution becomes $\hat\epsilon\approx2.29756$, with updated probabilities about $0.0913$ and $0.9087$. Both calculations are finite, but they are solutions for different offsets. Clipping may be a deliberate modification, but it does not preserve the original fluctuation model.

How the Alternatives Differ

Approach	What it does	Does it preserve the original fluctuation equation?
Stable calculation and root finding	Improves evaluation or solution of the same likelihood and score.	Yes, if inputs and model are unchanged.
Rescaling the clever covariate with matching parameter transformation	Changes numerical scale but leaves every linear predictor unchanged.	Yes.
Clipping predictions or truncating the clever covariate	Changes offsets or the direction of fluctuation.	No.
Penalization or a finite bound on the parameter	Changes the objective or restricts the allowed parameter values.	No; it defines a modified procedure.

A direct inverse-probability-weighted estimator is an alternative g-method, not a numerical repair to this likelihood. An untargeted sequential regression also does not solve the same empirical score equation. Those methods may have different strengths and failure modes, but switching to one does not establish that the original logistic targeting update existed.

Common Pitfalls and Limits

Calling every extreme estimate separation. A large finite estimate can arise from weak information or poor scaling. Check the score and likelihood along the relevant direction before diagnosing separation.
Treating an optimizer warning as proof that no solution exists. Numerical failure can occur even when a finite root exists. Stable evaluation and a bracketed score search help distinguish computational failure from nonexistence.
Assuming clipping is purely numerical. Clipping changes the offset, as the worked calculation shows. It can also affect the targeted predictions and must be reported as a modeling choice.
Interpreting an extreme but finite result as evidence of reliable targeting. A near-zero curvature, $ -S'(\epsilon)$, means the likelihood may be nearly flat. Small numerical or data changes can then produce a large change in $\hat\epsilon$.

These diagnostics concern the specified finite-sample logistic fluctuation. They do not establish identification, cure lack of support in the treatment process, or guarantee good statistical performance. If the unpenalized likelihood is separated, the exact empirical score equation has no finite solution; a finite answer requires changing the data, the fluctuation, or the estimation criterion.

Check Your Understanding

Work through each question, then compare with the solutions.

For one row with $H=2$, $Y=0$, and finite initial prediction, which direction separates the row, and why is there no finite likelihood maximum?
Can initial probabilities of $0.001$ and $0.999$ alone establish separation? Explain.
What happens to the fluctuation if $H_i$ is replaced by $cH_i$ and the parameter is replaced by $\epsilon/c$, for nonzero $c$?
Why does a ridge penalty not solve the original empirical targeting score equation in general?

Solutions. (1) Here $2Y-1=-1$, so $(2Y-1)H=-2$; choose $d=-1$. Along that direction the linear predictor moves toward $-\infty$, the probability of the observed zero tends to one, and the likelihood increases toward its supremum without attaining it at a finite value. (2) No. Separation depends on the outcomes and clever covariates as well as the offsets; boundary-near starting probabilities can occur with a finite score root, as the two-row example demonstrates. (3) The linear predictor is unchanged because $(\epsilon/c)(cH_i)=\epsilon H_i$, so the predictions and likelihood are unchanged. (4) A ridge penalty adds a derivative term to the likelihood score. Its penalized optimum generally solves a penalized equation, not $\sum_iH_i\{Y_i-Q_i(\epsilon)\}=0$.