Weak Identification in Selection-Model Likelihoods

Why a Good Outcome Fit May Not Identify the Response Effect

Selection models describe both an outcome distribution and the probability that the outcome is observed. The observed outcomes can strongly constrain the outcome distribution while saying much less about how response depends on an unobserved outcome. The distinction matters: a model can fit the observed outcomes and its optimizer can report convergence, yet the likelihood may still be nearly flat in a response parameter.

Here the aim is to make that possibility precise in a small model. The example isolates a response slope and derives its local information. It also clarifies why weak identification is not the same thing as nonconvergence: the former is a property of the information in the observed data under the model, while the latter concerns whether a numerical algorithm has successfully located a maximum.

Definition: A parameter is weakly identified by the observed data when the observed-data likelihood changes little across a substantial range of its values, after accounting for the other model parameters. Equivalently, the likelihood has little curvature in at least one parameter direction near its maximum. Weak identification concerns the information supplied by the data and model; it does not, by itself, mean that a maximum cannot be computed.

A Normal-Outcome Selection Model

Let \(Y_i\) be an outcome and let \(R_i=1\) indicate that it is observed; \(R_i=0\) means it is missing. For a transparent calculation, suppose the outcome follows a normal distribution with known mean zero and variance \(\sigma^2\), and response follows a logistic model:

Y_i\sim N(0,\sigma^2), \qquad P(R_i=1\mid Y_i=y)=\operatorname{expit}(\alpha+\gamma y), \qquad \operatorname{expit}(u)=\frac{1}{1+\exp(-u)}.

The intercept \(\alpha\) controls the baseline response probability, and \(\gamma\) is the response slope per outcome unit. The mean is fixed at zero here to isolate the information in the response slope; it is not being estimated alongside \(\gamma\). The variance and mean are specified as part of this illustrative model, rather than inferred from nonresponders.

Assume independent individuals, a correctly specified normal outcome distribution, and the stated logistic response mechanism. The observed-data likelihood uses the outcome value for each responder and integrates the outcome out for each nonresponder:

L(\alpha,\gamma,\sigma^2) = \prod_{i:R_i=1} f(y_i;\sigma^2)\operatorname{expit}(\alpha+\gamma y_i) \prod_{i:R_i=0} \int_{-\infty}^{\infty} f(y;\sigma^2)\{1-\operatorname{expit}(\alpha+\gamma y)\}\,dy,

where \(f(y;\sigma^2)\) is the density of \(N(0,\sigma^2)\). This likelihood is essential to the interpretation: a nonresponder contributes the probability of nonresponse averaged over the unobserved outcome distribution, not a response probability evaluated at an invented outcome.

Deriving the Local Information for the Response Slope

Consider the point \(\gamma=0\), where response is independent of the outcome. Write \(r=\operatorname{expit}(\alpha)\), so the response probability at this point is \(r\). For a responder with observed value \(y\), the score for \(\gamma\), meaning the derivative of the log likelihood with respect to \(\gamma\), is \((1-r)y\). For a nonresponder, the likelihood contribution is the integral in the preceding equation. At \(\gamma=0\), its derivative with respect to \(\gamma\) is proportional to \(E(Y)=0\), so its score for \(\gamma\) is zero.

The score for \(\alpha\) is \(1-r\) for a responder and \(-r\) for a nonresponder. The expected information is the expected product of scores. Since \(E(Y)=0\), the cross-information between \(\alpha\) and \(\gamma\) is zero at this point. For one individual, the information for \(\gamma\) is therefore:

I_{\gamma\gamma} = P(R=1) E\{(1-r)^2Y^2\} = r(1-r)^2\sigma^2.

For \(n\) independent individuals, information adds. Thus the local expected information for \(\gamma\), with \(\alpha\) treated as a nuisance parameter, is:

I_{\gamma\gamma\cdot\alpha} = n r(1-r)^2\sigma^2.

The calculation shows two sources of limited information. If outcomes vary little, changes in response probability across the outcome range are hard to detect. Also, information grows with sample size, but a large sample may be needed to compensate for very small outcome variability. This is local expected information at \(\gamma=0\) under the stated model; it is not a guarantee that the likelihood is globally quadratic or that a Wald interval will perform well.

Worked Example: Small Local Information Despite Many Observations

Suppose a hypothetical study has \(n=200\) independent participants, with \(160\) outcomes observed. For illustration, take \(r=0.80\) and a specified outcome standard deviation of \(\sigma=0.20\) outcome units. Evaluate the local information for \(\gamma\) at \(\gamma=0\).

Check the setup. The observed response fraction is \(160/200=0.80\), consistent with the chosen local response probability. The information calculation also assumes the normal outcome model and logistic response model above, with mean zero and variance fixed for this illustration.

Substitute and calculate. Since \(1-r=0.20\) and \(\sigma^2=(0.20)^2=0.04\),

I_{\gamma\gamma\cdot\alpha} = 200(0.80)(0.20)^2(0.20)^2 = 200(0.80)(0.04)(0.04) = 0.256.

The local curvature-based standard-error scale is \(1/\sqrt{0.256}=1.976\) inverse outcome units. This is large on the scale of a response slope per outcome unit. It signals that the local likelihood may be quite uninformative about \(\gamma\), even though 160 outcome values are available. It does not establish that a particular fitted model has this exact standard error: the result is an expected-information calculation at the specified parameter point.

Outcome Information and Response Information Need Not Match

In the same illustration, the observed outcomes can estimate the outcome variance much more sharply than they estimate the response slope. At \(\gamma=0\), responders are a random sample from the specified normal outcome distribution. If its mean is fixed at zero, the information about \(\sigma^2\) from \(m=160\) observed outcomes is \(m/(2\sigma^4)\). With \(\sigma=0.20\), this is \(160/[2(0.20)^4]=50{,}000\) per squared variance unit. The units and parameter meanings differ from those for \(\gamma\), so these raw information values should not be compared as if they were on one common scale. The substantive point is that the observed outcome distribution can be well characterized while the outcome-dependence of response remains uncertain.

The contrast is especially clear when the observed outcomes occupy a narrow range. A response slope describes change in response probability over outcome differences. If almost all observed outcomes lie close together, the data contain little leverage for estimating that change. Merely having many outcomes does not ensure that their values span a range informative about the response mechanism.

Worked Example: Narrower Outcomes Weaken Slope Information

Keep \(n=200\) and \(r=0.80\), but suppose the specified outcome standard deviation is \(0.05\) outcome units rather than \(0.20\). Assume the same independence, normality, logistic selection model, and evaluation at \(\gamma=0\).

The information is:

I_{\gamma\gamma\cdot\alpha} = 200(0.80)(0.20)^2(0.05)^2 = 200(0.80)(0.04)(0.0025) = 0.016.

The information is one-sixteenth of \(0.256\), because the outcome variance has fallen from \(0.04\) to \(0.0025\), also a factor of sixteen. The corresponding local curvature-based standard-error scale is \(1/\sqrt{0.016}=7.906\) inverse outcome units. Under these assumptions, the narrower range makes the response slope much less precisely determined. This comparison concerns information per slope in the model's outcome units; changing the outcome's units also changes the numerical value of \(\gamma\).

Diagnosing Weak Identification Versus Nonconvergence

A convergence message alone does not show that the data identify every parameter well. Conversely, a large reported standard error can arise from a poorly scaled or inaccurate computation, so it should not automatically be attributed to weak identification. Check whether the fit and the information tell a coherent story.

Evidence consistent with weak identification: the optimizer meets its convergence criteria, but small changes in \(\gamma\) produce little change in the observed log likelihood; the estimated curvature in that direction is small; or estimates change substantially under modest changes in model assumptions or supplied information.
Evidence consistent with nonconvergence: the algorithm stops with a large gradient, fails to meet its stopping rules, or gives materially different likelihood values across runs that should locate the same maximum. Such evidence indicates an optimization problem that must be addressed before interpreting curvature.
Check the scale and computation: verify that the likelihood uses the correct integrated contribution for nonresponders, and check numerical integration where it is needed. Inspect parameter units and the observed Hessian only after the likelihood evaluation and optimization are credible.

A flat likelihood can make estimates sensitive to small perturbations even when optimization is successful. That sensitivity is a warning about what the observed data determine, not proof of algorithmic failure. Likewise, a stalled optimizer does not prove weak identification: the cause could instead be poor starting values, numerical accuracy, or scaling.

Worked Example: Reading a Converged but Flat Fit

Suppose a hypothetical optimizer reports convergence for a selection-model fit. To illustrate the local scale, approximate the log likelihood near \(\hat{\gamma}=0\), after accounting for the intercept, by a quadratic with information \(0.256\), as in the first calculation. This is a local approximation, not a claim that the full likelihood is exactly quadratic.

Under this approximation, the log-likelihood difference at a displacement \(d\) is \(-\tfrac12(0.256)d^2\). At \(d=1\) inverse outcome unit, the difference is \(-0.128\); at \(d=2\), it is \(-0.512\):

\ell(\hat{\gamma}+d)-\ell(\hat{\gamma}) \ \approx\ -\frac{1}{2}(0.256)d^2, \qquad d=1:\ -0.128,\qquad d=2:\ -0.512.

These small changes illustrate why a range of slope values could have similar likelihood in this local approximation. If the optimizer's gradient is small and its likelihood evaluation is reliable, the appropriate diagnosis is weak local information, not nonconvergence. If instead the gradient remains large or repeated runs give inconsistent maxima, the fit has not yet established a reliable optimum, so the curvature should not be interpreted as a settled diagnostic.

What the Diagnosis Does—and Does Not—Say

Weak identification has substantive consequences. The observed data may support a narrow range of outcome-distribution values but permit a broad range of response slopes. Since the response slope governs how unobserved outcomes relate to response, uncertainty about it can materially affect inferences involving missing outcomes. A precise outcome fit alone does not eliminate that uncertainty.

The main alternatives address different sources of information and assumptions:

Approach	What it contributes	Important qualification
Selection-model likelihood	Uses the specified outcome distribution and response mechanism jointly.	The observed data may contain little information about the response slope.
Missing-at-random restriction	Removes outcome dependence from the response model by assumption.	It changes the model assumption; it does not demonstrate that the unrestricted response slope was identified.
Pattern-mixture specification	Describes outcome distributions by response pattern.	It expresses assumptions in a different factorization and does not automatically supply information absent from the observed data.
External information or sensitivity analysis	Can show how conclusions change under supplied restrictions or plausible response effects.	Results then depend partly on that external information or sensitivity range, which should be stated clearly.

The information calculation is conditional on a particular model, parameter point, and known mean. It is not a general numerical threshold for declaring a selection parameter weakly identified. If the mean is estimated, if the response mechanism is misspecified, or if the data depart from the assumed outcome distribution, the information structure can change. The diagnostic should therefore be interpreted alongside model assumptions and the observed likelihood, not used as a universal rule.

Key takeaway: A well-estimated outcome distribution does not guarantee a well-estimated response mechanism. Assess whether the observed-data likelihood has meaningful curvature for the response parameter, and keep that question separate from whether the optimizer converged.

Check Your Understanding

Answer each question using the assumptions and calculations in this tutorial. Solutions follow the questions.

For \(n=100\), \(r=0.80\), and \(\sigma=0.20\), calculate the local information for \(\gamma\) at zero.
For the first worked example, why does reducing \(\sigma\) from \(0.20\) to \(0.05\) reduce information by a factor of sixteen?
What is the key distinction between weak identification and nonconvergence?
If an optimizer converges but the observed log likelihood changes little over a broad range of \(\gamma\), what is the appropriate concern?
Does fitting the outcome distribution precisely establish that the response slope is precisely estimated? Explain.

Solutions. (1) \(100(0.80)(0.20)^2(0.20)^2=100(0.80)(0.04)(0.04)=0.128\). (2) Information is proportional to \(\sigma^2\); the variance changes from \(0.04\) to \(0.0025\), and \(0.0025/0.04=1/16\). (3) Weak identification means the observed-data likelihood contains little information in a parameter direction; nonconvergence means the numerical algorithm has not reliably located a maximum. (4) The fit may be weakly identifying \(\gamma\), provided likelihood evaluation and convergence diagnostics are credible. (5) No. Observed outcomes may characterize their distribution well while providing little information about response dependence on the unobserved outcome.