Parametric Bootstrap Calibration Of A Likelihood Ratio

Why Calibrate a Likelihood Ratio by Simulation?

A likelihood-ratio statistic measures how much better the alternative model fits than the null model. Its reference distribution is sometimes approximated by a chi-square distribution, but that approximation can be unreliable in small samples, with sparse data, or when parameter constraints affect the statistic’s behavior. A parametric bootstrap offers a direct way to estimate a reference distribution for the sample size and fitted null model at hand.

The key is to simulate under the null, not under the alternative. Then, for every simulated dataset, repeat the fitting and comparison procedure used on the observed data. In particular, refitting both models matters: comparing each simulated alternative fit with the original observed null fit would not reproduce the statistic’s sampling behavior.

Definition: For nested models fitted to the same observed data, let $\ell_0$ be the maximized log-likelihood under the null hypothesis and $\ell_1$ the maximized log-likelihood under the alternative. The likelihood-ratio statistic is $$ T_{\mathrm{obs}}=2(\ell_1-\ell_0). $$ The parametric-bootstrap calibration estimates the null distribution of this statistic by generating datasets from the null model fitted to the observed data, refitting both models to each generated dataset, and comparing the resulting statistics with $T_{\mathrm{obs}}$.

For a nested comparison, the alternative includes every parameter value allowed by the null. Therefore, when both models are maximized correctly on the same data, $\ell_1\geq \ell_0$, so $T_{\mathrm{obs}}\geq 0$. A negative value is a warning to check the likelihoods, constraints, optimization, and data used. If models are non-nested, a statistic formed from their likelihoods does not automatically have the usual likelihood-ratio test interpretation; its null distribution and comparison procedure must be justified separately.

The Bootstrap Procedure

Suppose the null model has a nuisance parameter $\theta$, and its maximum-likelihood estimate from the observed data is $\hat{\theta}_0$. The bootstrap uses this fitted null distribution as the generating model. Each bootstrap dataset is then analyzed as if it were new observed data.

Fit the observed data.
Maximize the null and alternative likelihoods on the same data, and calculate $T_{\mathrm{obs}}=2(\ell_1-\ell_0)$.

Generate under the fitted null.
Use the null model with nuisance parameters set to their observed-data estimates. Preserve the sample size, exposures, covariates, truncation, censoring, or other relevant features of how the data were observed.

Refit both hypotheses.
For each simulated dataset, separately maximize the null and alternative likelihoods. Re-estimate null nuisance parameters rather than carrying their observed-data estimates into the replicate.

Recalculate the statistic.
Record $T_b=2(\ell_{1,b}-\ell_{0,b})$ for bootstrap replicate $b$, using the same statistic and fitting conventions as for the observed data.

Estimate the upper-tail probability.
Count bootstrap statistics at least as large as the observed statistic and apply the finite-simulation adjustment shown below.

Formula: With $B$ bootstrap replicates and $r=\sum_{b=1}^{B} I(T_b\geq T_{\mathrm{obs}})$, the commonly used Monte Carlo estimate is $$ \hat p_{\mathrm{boot}}=\frac{r+1}{B+1}. $$ The comparison includes ties. The plus-one form avoids reporting a zero p-value merely because none of a finite number of simulated statistics exceeded the observed value.

This is a plug-in calibration: the generating null parameters are estimated rather than known. Thus the result is generally an approximation to the finite-sample null tail probability, not an exact test for every composite null. Increasing $B$ reduces Monte Carlo noise but does not remove approximation arising from estimating the null parameters or from a misspecified model.

Why the Procedure Reproduces the Test

The statistic is a function of the data and of the fitting procedure. Under the null, new data would vary, and so would both maximized log-likelihoods. Simulating new datasets under the fitted null approximates that variation. Refitting both hypotheses in every replicate applies the same data-to-statistic rule to simulated data as to observed data.

More formally, write $T(D)$ for the statistic obtained by applying the complete fitting procedure to dataset $D$. If $D^*$ is generated from the fitted null model, the bootstrap values $T(D^*_1),\ldots,T(D^*_B)$ approximate the distribution of $T(D)$ when the null is true at the fitted parameter. The fraction at least as large as $T(D_{\mathrm{obs}})$ estimates the corresponding upper-tail probability. This is why the null fitting and alternative fitting must both be repeated: the target is the distribution of the whole statistic, not just the variation in one likelihood.

Worked Example: A Binomial Test With a Small Sample

A hypothetical quality audit records six independent units, each classified as passing or failing. Five pass. Test the simple null hypothesis $p=0.5$ against an alternative in which $p$ is unrestricted. Find the observed likelihood-ratio statistic, then illustrate bootstrap calibration. The method is a likelihood-ratio comparison followed by simulation under the null; the main trap is confusing the one-sided observed count with a one-sided test statistic.

Under the null, the log-likelihood, omitting the common binomial coefficient, is $\ell_0=5\ln(0.5)+\ln(0.5)=6\ln(0.5)$. Under the unrestricted alternative, the maximum-likelihood estimate is $\hat p=5/6$, giving $\ell_1=5\ln(5/6)+\ln(1/6)$.

The log-likelihood improvement can be written as $5\ln[(5/6)/(1/2)]+\ln[(1/6)/(1/2)]$. Therefore,

T_{\mathrm{obs}} =2\left[5\ln(5/3)+\ln(1/3)\right] =2(2.5541281188-1.0986122887) \approx 2.911032.

Now generate bootstrap samples of six independent Bernoulli outcomes with success probability $0.5$. For each sample, re-estimate the alternative probability as its observed success proportion and compare its maximized log-likelihood with the null log-likelihood for that replicate. Suppose one illustrative run uses $B=2{,}000$ replicates and records $r=434$ statistics at least $2.911032$. Then

\hat p_{\mathrm{boot}}=\frac{434+1}{2{,}000+1} =\frac{435}{2{,}001}\approx 0.2174.

As a check for this simple null, the exact tail probability can also be enumerated. For six fair Bernoulli trials, $T\geq2.911032$ occurs with 0, 1, 5, or 6 successes. There are $1+6+6+1=14$ such outcomes among $2^6=64$, giving $14/64=0.21875$. The illustrative bootstrap estimate is close, with the difference attributable to simulation variation. The common mistake would be to count only samples with at least five successes; the likelihood-ratio statistic is two-sided here, so unusually few successes can also produce a large value.

Refitting a Composite Null

When the null contains an unknown parameter, the fitted value is used to generate bootstrap samples. But the null parameter is estimated anew within each replicate. This distinction is essential. The generating value is held fixed at the observed-data estimate for the simulation experiment; it is not treated as known when calculating each replicate’s likelihood-ratio statistic.

Worked Example: Comparing Two Binomial Groups

A hypothetical monitoring study observes four units in each of two groups. Group 1 has three passes and group 2 has one. The null model has one common pass probability $p$; the alternative has separate probabilities $p_1$ and $p_2$. Compute the observed statistic and illustrate a parametric bootstrap. The method is to fit a pooled probability under the null and group-specific probabilities under the alternative. The trap is to simulate from the alternative estimates rather than the fitted null.

The null estimate pools all eight outcomes: $\hat p_0=(3+1)/8=0.5$. The alternative estimates are $\hat p_1=3/4=0.75$ and $\hat p_2=1/4=0.25$. The binomial coefficients cancel from the log-likelihood difference. The improvement for each group is $3\ln(0.75/0.5)+\ln(0.25/0.5)=3\ln(1.5)+\ln(0.5)$. The two groups have the same improvement, so

T_{\mathrm{obs}} =4\left[3\ln(1.5)+\ln(0.5)\right] =4(1.2163953243-0.6931471806) \approx 2.092993.

For each bootstrap replicate, generate four Bernoulli outcomes in each group using the fitted null probability $0.5$. If the replicate counts are $x_1$ and $x_2$, refit the null with $\hat p_0^*=(x_1+x_2)/8$ and refit the alternative with $\hat p_1^*=x_1/4$ and $\hat p_2^*=x_2/4$. Compute the replicate likelihoods using those estimates, then record whether $T_b\geq2.092993$.

Suppose an illustrative run of $B=2{,}000$ replicates gives $r=975$. Its plus-one estimate is $(975+1)/(2{,}000+1)=976/2{,}001\approx0.4878$. This value is a Monte Carlo estimate, not a universal tail probability for all samples with the same observed statistic. The critical procedural point is that the pooled null probability is re-estimated within each replicate. Keeping it fixed at $0.5$ when calculating the replicate null likelihood would not reproduce the composite-null fitting procedure.

Preserve the Data-Generation and Observation Process

A bootstrap replicate must resemble a possible dataset under the null, including how observations enter the data. For example, if the observed sample is left-truncated, generating an untruncated sample and analyzing it as though it were truncated changes the likelihood experiment. If observations are censored, the replicate should preserve the censoring design or simulate censoring according to the model, and the likelihood contributions for censored observations must be recalculated in every fit. The same principle applies to exposures, group sizes, covariates, and fixed sample sizes.

The earlier material on truncated and censored likelihoods supplies the likelihood appropriate to those observations. Bootstrap calibration adds a simulation layer; it does not replace that likelihood. If the original comparison uses a conditional likelihood for the observed sampling scheme, each replicate must be fitted with the corresponding conditional likelihood as well.

Worked Example: Two Poisson Rates With Equal Exposures

A hypothetical claims audit has two portfolios, each observed for exposure 2 years. The counts are 7 and 1. Test a common Poisson rate against separate rates. The method is a likelihood-ratio comparison with exposure included in each Poisson likelihood, followed by simulation under the fitted common-rate null. A common mistake is to treat the counts as rates without dividing by exposure.

Under the null, the total count is 8 over total exposure 4, so $\hat\lambda_0=8/4=2$ per year. Under the alternative, $\hat\lambda_1=7/2=3.5$ and $\hat\lambda_2=1/2=0.5$ per year. In the log-likelihood difference, the exposure terms cancel because the fitted total expected count is 8 under either model. Thus

\begin{aligned} T_{\mathrm{obs}} &=2\left[7\ln(3.5/2)+1\ln(0.5/2)\right]\\ &=2\left[7\ln(1.75)+\ln(0.25)\right]\\ &=2(3.9173105164-1.3862943611)\\ &\approx 5.062032. \end{aligned}

For each bootstrap replicate, generate independent counts $Y_1^*$ and $Y_2^*$ from $\operatorname{Poisson}(4)$, since each exposure is 2 and the fitted null rate is 2. Refit the common rate as $(Y_1^*+Y_2^*)/4$, and the separate rates as $Y_1^*/2$ and $Y_2^*/2$. Recalculate the likelihood ratio from those refitted models. If an illustrative run with $B=2{,}000$ yields $r=213$ values at least as large as the observed statistic, then $(r+1)/(B+1)=214/2{,}001\approx0.1069$.

The result is an estimated finite-sample tail probability under the fitted common-rate model. It does not establish that the Poisson model is appropriate; model adequacy remains a separate question. The unit check is also useful: the generating Poisson mean is rate times exposure, $2\times2=4$, not simply the fitted rate 2.

Practical Checks and Limitations

A sound implementation uses identical fitting rules for the observed data and every replicate. In particular, use the same parameter constraints, treatment of boundary estimates, constants in the likelihood, and convergence checks. If an optimizer fails for a replicate, do not silently assign it a convenient statistic: diagnose the failure and choose a defensible, consistently applied fitting procedure.

Check nesting and data comparability. The usual nonnegative likelihood-ratio statistic relies on a properly nested comparison fitted to the same observations. A change in sample or likelihood contributions can invalidate the comparison.
Use the correct tail. For the usual likelihood-ratio test, count $T_b\geq T_{\mathrm{obs}}$, including ties. Do not substitute a tail for a parameter estimate unless the test statistic and alternative are explicitly one-sided.
Do not omit refitting. A replicate’s statistic must use the null and alternative fits to that replicate, not a likelihood evaluated at observed-data estimates.
Report simulation uncertainty. A larger $B$ improves Monte Carlo precision. It does not fix a poor null model, an incorrect likelihood, or the plug-in nature of a composite-null bootstrap.
Do not interpret zero exceedances as zero probability. The plus-one estimate remains positive; with $r=0$, it is $1/(B+1)$, which also signals that more simulations may be useful if a more precise tail estimate is needed.

Key takeaway: Parametric-bootstrap calibration simulates under the fitted null, then refits both hypotheses and recomputes the same likelihood-ratio statistic in every replicate. Its reliability depends on preserving the null model, the observation process, and the complete fitting procedure.

Check Your Understanding

For each problem, state the fitting or simulation step that matters and check the arithmetic before rounding.

A hypothetical sample has eight independent Bernoulli outcomes, with six successes. Test $p=0.5$ against unrestricted $p$. Find $T_{\mathrm{obs}}$. (Answer: $2[6\ln(1.5)+2\ln(0.5)]\approx2.0930$.)
Two hypothetical Poisson portfolios each have exposure 2 years and counts 5 and 3. Under a common-rate null, find the fitted rate and the likelihood-ratio statistic against separate rates. (Answer: $\hat\lambda_0=2$ per year; $T=2[5\ln(1.25)+3\ln(0.75)]\approx0.5053$.)
A bootstrap uses $B=999$ replicates, of which 37 statistics are at least the observed statistic. Calculate the plus-one p-value estimate. (Answer: $38/1{,}000=0.0380$.)
Observed severities are left-truncated at a fixed threshold, and the fitted null is used to generate bootstrap data. State what must be preserved and what must be refitted in each replicate. (Answer: Preserve the truncation/sampling mechanism and sample design; refit both null and alternative using the corresponding truncated-data likelihood.)
A composite null has a nuisance parameter estimated from the observed data. In a bootstrap replicate, should the null likelihood use the original estimate or a new estimate? Explain briefly. (Answer: Re-estimate the null nuisance parameter from that replicate; the original estimate is used to generate bootstrap data, not held fixed in the replicate’s likelihood fit.)

What You'll Learn