Why the Usual Reference Distribution Can Fail
A likelihood-ratio test compares the best fit allowed by an alternative model with the best fit allowed by a null model. In regular problems, a large-sample theorem supplies a convenient reference distribution: under the null, the likelihood-ratio statistic is approximately chi-square, with degrees of freedom equal to the number of restrictions. Selection-model likelihoods can violate the conditions behind that result. Weak identification can make likelihoods unusually flat in some directions, and a parameter constraint can place the null value on the edge of the permitted parameter space. In either case, using the usual chi-square reference without checking its assumptions can misstate the test's tail probability.
Here the focus is calibration: how to obtain a reference distribution for a selection-effect test when the standard one is doubtful. A parametric bootstrap does this by simulating data under the fitted null model and repeating the same test on each simulated data set. It adapts the reference distribution to the assumed model, design, sample size, and fitting procedure, but it does not repair a model that is unidentified or incorrectly specified.
Wilks' theorem gives an approximate \(\chi^2_q\) reference distribution when the null is true, the model is correctly specified, the relevant parameters are identifiable and lie in the interior of the parameter space, and the log-likelihood has suitable smoothness and curvature near the true parameter. Here \(q\) is the number of independent restrictions imposed by the null. The approximation is an asymptotic result, not an automatic property of every likelihood-ratio statistic.
In selection models, weak identification can occur when different combinations of outcome and response parameters yield very similar observed-data likelihoods. The likelihood may then have a shallow ridge, so a large sample may be needed before the usual local quadratic approximation is useful. A boundary constraint presents a different problem: the null value may lie at an edge rather than in the interior of the allowed parameter space. The resulting limiting distribution need not be the ordinary chi-square distribution. In some simple, regular boundary problems it is a mixture of chi-square distributions, but that particular mixture is not a universal correction for selection models.
A Selection-Effect Test and Its Assumptions
For a concrete setup, let \(Y_i\) be a continuous outcome, \(X_i\) observed covariates, and \(R_i\) an indicator equal to 1 when \(Y_i\) is observed and 0 otherwise. Suppose the outcome model has density \(f(y\mid x;\eta)\), and the response model is \(P(R_i=1\mid Y_i=y,X_i=x)=\pi(y,x;\alpha,\gamma)\). The parameter \(\gamma\) measures the outcome dependence of response in this specified model. Testing for no outcome-dependent selection means comparing \(H_0:\gamma=0\) with an alternative that permits \(\gamma\ne 0\).
For an observed outcome, the observed-data likelihood contribution includes both the outcome density and the probability of response. For a missing outcome, it integrates over the unobserved value:
The test uses the observed-data log-likelihood formed from these contributions. The null and alternative must use the same outcome model, covariates, records, and likelihood calculation; they differ only in whether \(\gamma\) is fixed at zero. This is a model-based test of a specified selection effect. It does not establish that the model is correct, or give the selection coefficient a causal interpretation.
Before using either a chi-square approximation or a bootstrap, check that both fits are genuine maxima of the intended likelihood, that the alternative contains the null, and that the likelihood is evaluated consistently. Weak identification can make these checks consequential: a reported optimizer solution may lie on a ridge or depend materially on starting values. Reliable calibration cannot compensate for an incorrect likelihood or failed optimization.
Worked Example: A Regular-Case Chi-Square Benchmark
Suppose, hypothetically, a selection-effect test has one restriction, \(H_0:\gamma=0\), and the fitted log-likelihoods are \(\ell(\hat{\theta}_{A})=-412.80\) and \(\ell(\hat{\theta}_{0})=-414.72\). Assume for this illustration that the model is correctly specified, the relevant parameters are identifiable and interior, and the likelihood is sufficiently regular for Wilks' approximation.
The observed statistic is $$ T_{\mathrm{obs}}=2\{(-412.80)-(-414.72)\} =2(1.92)=3.84. $$
With one restriction, the regular reference is \(\chi^2_1\). Its upper-tail probability at 3.84 is approximately 0.050. At significance level \(\alpha=0.05\), this is just at the conventional rejection threshold: the chi-square approximation provides borderline evidence against \(H_0\), rather than a result that is clearly beyond the threshold.
This calculation is a benchmark, not evidence that the assumptions hold in a selection-model application. If the selection effect is weakly identified or constrained at a boundary, the nominal \(\chi^2_1\) tail probability may not describe the null distribution of the statistic.
Null Parametric-Bootstrap Calibration
A parametric bootstrap estimates the null distribution by reproducing the data-generating and analysis steps under the fitted null. For a selection model, this means generating outcomes and response indicators according to the null model and then applying the same missingness pattern to the outcomes. Each simulated data set is analyzed with both the constrained and unrestricted models, and its likelihood-ratio statistic is recorded. The simulated statistics provide a model-based approximation to the distribution of the test statistic under \(H_0\).
Estimate the outcome and response parameters subject to \(\gamma=0\), retaining the observed covariates and design features required by the model.
For each replicate, generate outcomes from the fitted null outcome model and response indicators from the fitted null response model. Treat outcomes with \(R_i=0\) as missing in the simulated analysis data.
Fit both null and alternative models to that simulated observed data, using the same restrictions and fitting procedure as for the original data. Save \(T_b=2\{\ell(\hat{\theta}_{A,b})-\ell(\hat{\theta}_{0,b})\}\).
With \(B\) replicates and \(K\) statistics at least as large as \(T_{\mathrm{obs}}\), use \(\hat{p}_{\mathrm{boot}}=(K+1)/(B+1)\). Report the number of replicates and the simulation uncertainty as appropriate.
The null fit is used to generate the replicates because the question is how unusual the observed statistic would be if the null model were true. Refitting both models in every replicate is essential: it reproduces the variability and optimization steps involved in calculating the observed statistic. Holding the fitted parameters fixed during each simulated alternative fit, or comparing simulated data only with a fixed estimate, would not repeat the likelihood-ratio test.
The plus-one estimate avoids reporting a zero p-value when none of the finite number of simulated statistics exceeds the observed statistic. It is a Monte Carlo estimate, so its precision depends on \(B\), especially when the p-value is small. The generated reference distribution is conditional on the fitted null model and its parameter estimates; it is not an assumption-free distribution.
Worked Example: Estimating a Bootstrap P-Value
Suppose the observed selection-effect statistic is \(T_{\mathrm{obs}}=5.20\). Under the fitted null, \(B=999\) replicate data sets are generated and both models are refitted for each. In 37 replicates, the statistic is at least 5.20. Assume the simulated fits converged and all statistics were computed with the same likelihood definition as the observed statistic.
Here \(K=37\), so $$ \hat{p}_{\mathrm{boot}}=\frac{K+1}{B+1} =\frac{37+1}{999+1} =\frac{38}{1000}=0.038. $$
At \(\alpha=0.05\), this bootstrap estimate is below the significance level, so the test rejects \(H_0\) using this calibration. In context, the fitted-null simulations produced a likelihood-ratio statistic of at least 5.20 in an estimated 3.8% of replications. This is not the probability that \(H_0\) is true. Because only 999 replicates were used, 0.038 also has Monte Carlo error; more replicates would improve the precision of the estimated tail probability.
What the Bootstrap Can and Cannot Resolve
The bootstrap is particularly useful when the shape of the null distribution is hard to specify analytically. It can reflect a finite sample, the missingness pattern generated by the model, and the fitting procedure's behavior. But its validity still depends on the null model being an adequate generator for the data and on the refitting procedure being stable. If the null itself leaves important parameters unidentified, or if the fitted null estimates are poor representations of the relevant parameter values, a plug-in bootstrap may also give misleading calibration.
A useful diagnostic is to inspect the entire bootstrap distribution, not just its estimated tail probability. A pronounced pile-up at zero, a long right tail, or substantial disagreement with the nominal chi-square reference signals that ordinary calibration may be inappropriate. Such patterns need interpretation in light of the constraints and the likelihood geometry; they do not, by themselves, identify one correct replacement distribution. Also inspect failed or extreme fits. If a material number of replicates fail to converge, silently dropping them can change the estimated tail probability.
Increasing \(B\) reduces Monte Carlo noise but does not fix model misspecification, weak identification in the null, or numerical errors. Likewise, agreement between a bootstrap and chi-square p-value is reassuring only about the observed calibration in that setup; it does not prove that the selection model is scientifically adequate.
| Calibration approach | When it is justified | Main caution |
|---|---|---|
| Ordinary \(\chi^2_q\) reference | Regular large-sample conditions hold, including interior identification and suitable likelihood curvature. | Can miscalibrate under weak identification, boundary constraints, or limited samples. |
| Analytic boundary mixture | A specific result applies to the model and constraints under consideration. | A mixture derived for one boundary problem is not a general selection-model correction. |
| Null parametric bootstrap | The fitted null is a credible generator and the test can be reliably refitted on simulated data. | Model-dependent; can fail if the null is poorly identified, misspecified, or numerically unstable. |
Common Mistake
Wrong reasoning: “The alternative adds one selection coefficient, so the likelihood-ratio statistic must have a \(\chi^2_1\) distribution.” The number of restrictions determines the degrees of freedom in the regular Wilks approximation, but it does not establish that the approximation's regularity conditions hold.
Correction: First assess whether the null is interior and whether the tested parameter is adequately identified with a well-behaved likelihood. If constraints or weak identification undermine the standard argument, use a calibration justified for the model, such as a carefully implemented null parametric bootstrap. Describe its model dependence and numerical limitations rather than treating it as a universal remedy.
Check Your Understanding
Answer each question, then compare with the solutions.
- A test imposes two independent restrictions and satisfies the regularity conditions. What is its usual asymptotic reference distribution?
- In a null bootstrap with \(B=499\), 9 simulated statistics are at least as large as the observed statistic. Calculate the plus-one bootstrap p-value.
- Why must both the null and alternative models be refitted for every bootstrap replicate?
- Does a bootstrap p-value of 0.03 mean there is a 3% probability that the null hypothesis is true? State the correct interpretation.
Solutions. (1) The reference is \(\chi^2_2\), because there are two independent restrictions and the regularity conditions are assumed to hold. (2) \(\hat{p}_{\mathrm{boot}}=(9+1)/(499+1)=10/500=0.020\). (3) Refitting both models repeats the likelihood-ratio analysis and accounts for estimation variability under the simulated null data; not refitting would produce a different statistic. (4) No. The estimate 0.03 is the simulated null-tail proportion, with the plus-one adjustment, for statistics at least as large as the observed one. It is not the probability that the null is true.