Why Diagnose Exclusions?
A selection model may satisfy its formal identification conditions yet estimate the outcome equation imprecisely in a particular sample. This can happen when an excluded variable barely changes selection, when predicted selection probabilities leave little common support, or when the selection correction is hard to distinguish from the included outcome regressors. In these cases, the data provide limited practical information about the selection correction, even if the model is identified in principle.
The goal of a practical diagnostic is not to certify an exclusion restriction as valid. Its validity depends on substantive reasoning: an excluded variable must affect selection while having no direct effect on the outcome, conditional on the included covariates. Diagnostics instead show whether the proposed exclusion supplies useful variation and whether the resulting outcome estimates depend heavily on modeling choices.
What Variation Identifies the Correction?
Let \(D_i=1\) indicate that outcome \(Y_i^*\) is observed, and let \(X_i\) denote the outcome regressors. Write the selection equation as \(D_i=1[Z_i'\gamma+u_i>0]\), where \(Z_i\) contains \(X_i\) and any excluded variables. Under the usual Heckman model, the outcome equation is \(Y_i^*=X_i'\beta+\varepsilon_i\), and the standardized selection error and outcome error are jointly normal with correlation \(\rho\). Let \(\sigma_\varepsilon\) be the standard deviation of \(\varepsilon_i\). The outcome is observed only when \(D_i=1\).
Under these assumptions, the conditional expectation of the outcome among selected observations is
where \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution functions. The inverse Mills ratio \(\lambda(Z_i'\gamma)\) is the selection correction. The excluded variables help identify the outcome equation when they generate useful variation in this correction that is not simply a linear combination of the variation in \(X_i\).
The relevant question is therefore more specific than “Is the excluded-variable coefficient statistically significant?” A coefficient can be estimated precisely but induce only small changes in predicted selection probabilities over the observed covariate range. Conversely, a noisy coefficient may produce substantial estimated variation but leave that variation uncertain. Inspect both the magnitude and uncertainty of the first-stage variation, and then examine what happens to the outcome estimates.
First-Stage Variation and Probability Overlap
Estimate the selection equation using the specified selection model, and examine fitted probabilities \(\widehat{P}(D_i=1\mid Z_i)\) across meaningful values of the excluded variables and other covariates. For a probit selection equation, these probabilities are \(\Phi(Z_i'\hat{\gamma})\). Report changes in probabilities over realistic covariate contrasts, not just the scale or significance of a probit coefficient.
Next, compare the distributions of fitted probabilities for selected and unselected observations. If the groups occupy largely separate probability ranges, the model may rely on predictions near zero or one for many observations. This can signal limited overlap and sensitivity to the functional form of the selection equation. Overlap is a diagnostic, not a requirement that every observation have the same probability, and it does not establish that the exclusion is substantively valid.
For the outcome equation, also inspect the correction itself among selected observations. When \(\widehat{\lambda}_i\) barely varies, or varies almost exactly with the columns of \(X_i\), it is difficult to separate the correction coefficient from the outcome coefficients. A useful summary can include the range and quantiles of \(\widehat{\lambda}_i\), along with its relationship to the outcome regressors. No single summary or universal cutoff determines whether an exclusion is sufficiently strong.
Worked Example: Interpreting a Modest First-Stage Shift
Suppose a hypothetical probit selection equation includes a binary excluded variable \(Z\). At a representative value of the other covariates, the estimated index is \(0.20\) when \(Z=0\). The estimated coefficient on \(Z\) is \(0.18\), with standard error \(0.14\). Assess the estimated probability change and what the coefficient test says.
The fitted probabilities at the two values of \(Z\) are
Thus, at these covariates, changing \(Z\) from zero to one raises the fitted selection probability by \(0.6480-0.5793=0.0687\), or 6.87 percentage points. The coefficient’s Wald statistic is \(0.18/0.14=1.29\), which gives a two-sided normal-reference \(p\)-value of about \(0.20\). This is limited evidence against a zero coefficient in this sample. The probability contrast is also fairly modest at the stated covariates.
Neither result alone settles practical identification. The contrast may differ at other covariate values, and uncertainty in \(\hat\gamma\) matters. The exclusion’s substantive validity is a separate question: these calculations do not show whether \(Z\) has a direct effect on the outcome.
Worked Example: Reading Predicted-Probability Overlap
In a hypothetical sample of 500 observations, 350 outcomes are selected and observed, and 150 are not selected. The table groups observations by their fitted selection probabilities.
| Fitted probability range | Selected | Not selected |
|---|---|---|
| Below 0.20 | 2 | 25 |
| 0.20 to below 0.40 | 18 | 40 |
| 0.40 to below 0.60 | 55 | 45 |
| 0.60 to below 0.80 | 125 | 30 |
| 0.80 to 1.00 | 150 | 10 |
| Total | 350 | 150 |
There is meaningful overlap in the middle ranges: for example, both selected and unselected observations occur between 0.40 and 0.60. But the tails are uneven. Most observations below 0.20 are unselected, while most observations above 0.80 are selected. The diagnostic therefore points to some probability regions with limited representation of one selection group; it does not imply that all observations lack overlap.
For selected observations, the corresponding correction values can vary substantially. For illustration, if fitted probabilities are 0.40, 0.60, 0.80, and 0.95, the probit indices are approximately \(-0.253\), \(0.253\), \(0.842\), and \(1.645\). The associated inverse Mills ratios are approximately \(0.966\), \(0.644\), \(0.350\), and \(0.109\), respectively. This range provides potentially useful correction variation, but the table alone does not establish that it is independent of the outcome regressors. The analyst should check that relationship among the selected observations and consider whether estimates depend on observations in thinly represented tails.
A Short Justification for the Correction
The inverse Mills ratio arises from truncation of the selection error. Let \(\eta=Z_i'\gamma\), and let the selection error \(u_i\) have a standard normal distribution. Selection occurs when \(u_i>-\eta\). Under joint normality, the conditional mean of the outcome error given \(u_i\) is \(E(\varepsilon_i\mid u_i)=\rho\sigma_\varepsilon u_i\). Therefore,
The last equality is the mean of a standard normal variable truncated below at \(-\eta\). Adding this conditional error mean to \(X_i'\beta\) gives the selected-sample expectation stated above. This also explains why a weak exclusion is consequential: if changes in the excluded variables scarcely move \(\eta\), or if the resulting correction is redundant with \(X_i\), the data have little distinct variation with which to estimate its contribution.
Outcome-Equation Sensitivity
A practical assessment should trace how the outcome estimates respond to defensible changes in the exclusion set or selection specification. Compare the coefficient of substantive interest, its standard error or interval, and the estimated selection-correction contribution. Changes that are large relative to the estimates’ uncertainty warrant careful reporting. They may indicate that the data provide little information to distinguish the outcome relationship from the modeled selection process.
Such comparisons must be designed carefully. A variable should not be added to or removed from the exclusion set merely to obtain a preferred result. Each proposed exclusion requires a substantive argument for its absence from the outcome equation. Likewise, changing a functional form can alter estimates for reasons other than exclusion strength. Sensitivity analysis reveals dependence on assumptions; it does not identify which specification is correct.
Worked Example: Comparing Outcome Estimates Across Specifications
Suppose a hypothetical study models hourly wages in dollars as the outcome and years of education as an outcome regressor. A baseline selection specification gives an education coefficient of \(2.40\) dollars per hour per additional year, with standard error \(0.70\). Removing a weakly shifting exclusion gives \(1.55\), with standard error \(0.95\). A separately motivated selection specification gives \(3.05\), with standard error \(1.25\). For comparison, calculate approximate 95% normal-reference intervals as estimate \(\pm 1.96\) standard errors.
For the baseline specification,
For the specification omitting the weak exclusion,
For the separately motivated specification,
The point estimates range from \(1.55\) to \(3.05\) dollars per hour per additional year of education, and the intervals differ in width and in whether they include zero. This is sensitivity to specification, not proof that one estimate is correct or that the selection model is false. The intervals are approximate and are meaningful only if their standard errors properly account for the estimation procedure. The investigator should report the variation, justify the specifications, and avoid presenting one preferred result as if the data had ruled out the alternatives.
Weak Identification Is Not a Model Test
A weak exclusion concerns how much the available data can distinguish parameters under a model. Evidence against the model concerns whether its assumptions or restrictions fit the data. These are different questions. A broad interval or unstable coefficient can arise because the correction is weakly identified; it does not establish that selection dependence is absent, that the exclusion is invalid, or that the selection equation is misspecified. Conversely, a strong first-stage shift does not validate the exclusion restriction or the assumed error distribution.
There is no universal first-stage significance threshold or probability-overlap rule that settles the issue for every selection model. The useful evidence is a coherent set of diagnostics: plausible and substantively defensible exclusions, appreciable and reasonably precise probability shifts over relevant covariate values, adequate overlap to avoid reliance on extreme predictions, correction variation distinct from the outcome regressors, and transparent outcome sensitivity.
Common Pitfall
- Wrong reasoning: “The excluded-variable coefficient has a small \(p\)-value, so the exclusion is valid and identification is strong.”
- Correction: Statistical significance addresses evidence about a coefficient under the specified first-stage model. It does not establish the exclusion restriction’s substantive validity, the size of the probability changes, overlap, or independent variation in the correction term.
- Wrong reasoning: “The outcome estimate changes across specifications, so the selection model is disproved.”
- Correction: Instability is evidence that conclusions depend on assumptions or weakly informative variation. It is a reason to qualify and investigate the result, not by itself a test that selects between models.
Check Your Understanding
Answer in terms of the diagnostic and the inference it does, and does not, support.
- Why is a statistically significant excluded-variable coefficient not enough to establish a strong, valid exclusion?
- What does substantial separation between selected and unselected fitted-probability distributions suggest, and why is it not a definitive model test?
- Why should the analyst examine variation in the inverse Mills ratio among selected observations in relation to the outcome regressors?
- An education coefficient changes substantially when a weakly shifting exclusion is removed. What conclusion is warranted, and what conclusion is not warranted?
- Give one reason why a strong first-stage probability shift cannot establish the substantive validity of an exclusion restriction.