Bootstrap Calibration of Profile Likelihood Cutoffs

Why Calibrate a Profile Likelihood Cutoff?

A profile likelihood confidence set retains target values whose profile likelihood-ratio statistic is not too large. In regular, sufficiently large samples, a chi-square cutoff is often used. In finite samples, however, the statistic's distribution can differ materially from that approximation. This can happen with sparse counts, strong nuisance-parameter effects, or parameter constraints. A parametric bootstrap offers a way to estimate the statistic's null distribution using repeated data sets generated from a fitted model.

The essential feature is that calibration is performed under the candidate value being tested. For each candidate, fit the model subject to that value, simulate from the constrained fit, and recompute the statistic after both constrained and unrestricted refitting. The resulting cutoff can depend on the candidate value; it need not be a single constant across the profile.

Statistic and Bootstrap Procedure

Definition: Let $\theta$ index a parametric model with log-likelihood $\ell(\theta;y)$, and let the scalar target be $\psi=g(\theta)$. Write $\hat{\theta}$ for an unrestricted maximum likelihood estimate and $\hat{\theta}_{\psi}$ for a maximum likelihood estimate subject to $g(\theta)=\psi$. The profile likelihood-ratio statistic for testing $\psi$ is $$ T_{\mathrm{obs}}(\psi)=2\{\ell(\hat{\theta};y)-\ell(\hat{\theta}_{\psi};y)\}. $$ It is nonnegative when both maximizations are performed over the stated parameter spaces and the constrained space is contained in the unrestricted one.

To calibrate this statistic at a candidate $\psi$, generate bootstrap data $Y^{*(b)}$ from the model at $\hat{\theta}_{\psi}$. For each generated data set, refit both models, obtaining $\hat{\theta}^{*(b)}$ without the constraint and $\hat{\theta}_{\psi}^{*(b)}$ with $g(\theta)=\psi$. Calculate

T^{*(b)}(\psi) =2\{\ell(\hat{\theta}^{*(b)};Y^{*(b)}) -\ell(\hat{\theta}_{\psi}^{*(b)};Y^{*(b)})\}.

For $B$ replicates, let $\hat{c}_{1-\alpha}(\psi)$ be the empirical $1-\alpha$ quantile of these $B$ statistics. One explicit convention is the order statistic of rank $\lceil B(1-\alpha)\rceil$, after sorting from smallest to largest. The calibrated confidence set is

\mathcal{C}_{1-\alpha} =\{\psi:T_{\mathrm{obs}}(\psi)\leq \hat{c}_{1-\alpha}(\psi)\}.

A parametric bootstrap cutoff is not automatically an exact finite-sample cutoff. It is a plug-in estimate of a null quantile: the nuisance parameters in the null model are estimated, and only finitely many bootstrap data sets are generated. Its performance therefore depends on whether the fitted model and the bootstrap approximation represent the sampling distribution adequately.

Worked Example: A Binomial Profile Statistic and Its Bootstrap Cutoff

Suppose $X\sim\operatorname{Binomial}(10,p)$, and the observed count is $x=8$. Consider testing the candidate value $p_0=0.5$. The unrestricted maximum likelihood estimate is $\hat p=8/10=0.8$, while under the candidate the fitted value is $0.5$. The statistic is $$ T_{\mathrm{obs}}(0.5) =2\left\{8\log\left(\frac{0.8}{0.5}\right) +2\log\left(\frac{0.2}{0.5}\right)\right\} =2\{8\log(1.6)+2\log(0.4)\} \approx 3.8549. $$

Under the candidate null, bootstrap counts are generated from $\operatorname{Binomial}(10,0.5)$. For any generated count $x^*$, the unrestricted estimate is $x^*/10$, and the constrained estimate remains $0.5$. For example, if $x^*=7$, $$ T^*=2\{7\log(1.4)+3\log(0.6)\} \approx 1.645657568, $$ which rounds to $1.646$. If $x^*=4$, $$ T^*=2\{4\log(0.8)+6\log(1.2)\} \approx 0.402710. $$

For a small arithmetic illustration, take ten bootstrap counts $4,5,6,7,3,5,4,6,7,5$. Their statistics are three zeros from the counts of 5, four values of approximately $0.403$ from counts of 4 or 6, and three values of approximately $1.646$ from counts of 3 or 7. The sorted eighth value is $1.646$; with the empirical-quantile convention above, this is the 80th-percentile cutoff for $B=10$. The observed statistic, $3.8549$, exceeds it, so this illustrative bootstrap calibration rejects $p=0.5$ at that nominal tail level. Ten replicates are far too few for reliable practice; the calculation demonstrates the mechanics, not a recommended simulation size.

Why Both Refits Are Necessary

The likelihood-ratio statistic compares two optimized likelihoods on each data set. Using the observed-data estimates in the bootstrap likelihoods, or holding the original nuisance estimates fixed during the bootstrap refits, generally does not reproduce that statistic's sampling behavior. The generated data must be analyzed in the same way as the observed data: unrestricted maximization for the numerator fit and constrained maximization for the candidate null.

Fit the observed data.
For each candidate $\psi$, compute the unrestricted and constrained fits and then $T_{\mathrm{obs}}(\psi)$.

Generate under the candidate null.
Simulate each replicate from the fitted model at $\hat{\theta}_{\psi}$, which satisfies $g(\hat{\theta}_{\psi})=\psi$.

Repeat both optimizations.
For every replicate, maximize without the constraint and with the same target constraint.

Invert the calibrated tests.
Estimate the desired null quantile and retain candidates whose observed statistic does not exceed it.

Worked Example: Bootstrap Calibration in a Poisson Rate Model

Let $Y_1,\ldots,Y_4$ be independent $\operatorname{Poisson}(\lambda)$ observations with observed values $0,1,2,3$. The unrestricted estimate is $\hat{\lambda}=(0+1+2+3)/4=1.5$. Test the candidate $\lambda_0=1$. The constrained estimate is $1$, and the log-likelihood ratio simplifies in terms of the total $S=\sum_iY_i$: $$ T_{\mathrm{obs}}(1) =2\{S\log(1.5)-4(1.5-1)\} =2\{6\log(1.5)-2\} \approx 0.86558. $$

For each bootstrap replicate, generate four independent $\operatorname{Poisson}(1)$ counts. If their total is $S^*$, then the unrestricted estimate is $S^*/4$, and $$ T^*(1)=2\{S^*\log(S^*/4)-S^*+4\}, $$ with the convention $0\log 0=0$. For example, the replicate $(0,1,0,1)$ has $S^*=2$ and statistic $2\{2\log(0.5)+2\}\approx1.22741$. The replicate $(1,1,2,1)$ has $S^*=5$ and statistic $2\{5\log(1.25)-1\}\approx0.23144$.

Consider five illustrative replicate totals $2,4,5,3,6$. Their statistics, in that order, are approximately $1.22741,0,0.23144,0.27391,0.86558$. Sorted, they are $0,0.23144,0.27391,0.86558,1.22741$. At the empirical 80th percentile with $B=5$, the rank is $\lceil5(0.8)\rceil=4$, giving a cutoff of $0.86558$. The observed statistic is approximately equal to that cutoff, so it is retained under the stated rule. With so few replicates, this boundary comparison is only illustrative and is sensitive to Monte Carlo variation.

Two Properties of the Calibration

Result: For a fixed candidate value, suppose bootstrap statistics are independent draws from a distribution with cumulative distribution function $F$. Let $q=1-\alpha$, and let $c_q$ satisfy $F(c_q)=q$. If $F$ is continuous and strictly increasing in a neighborhood of $c_q$, then the empirical quantile $\hat c_q$, defined by order-statistic rank $\lceil Bq\rceil$, converges almost surely to $c_q$ as $B$ tends to infinity.

Proof. Fix any $\varepsilon>0$. Choose $0<\delta<\varepsilon$ small enough that $c_q-\delta$ and $c_q+\delta$ lie in the neighborhood where $F$ is strictly increasing. Continuity and strict increase there give $F(c_q-\delta)<q<F(c_q+\delta)$. By the strong law of large numbers, the empirical proportions of bootstrap statistics at or below each of these two fixed values converge almost surely to the corresponding values of $F$. Eventually, the empirical proportion at $c_q-\delta$ is below $q$, and the proportion at $c_q+\delta$ is above $q$. Thus the empirical quantile lies between these two values eventually. Since $\delta<\varepsilon$, this proves $|\hat c_q-c_q|<\varepsilon$ eventually. The argument applies to every $\varepsilon>0$, and hence $\hat c_q\to c_q$ almost surely. $\square$

Result: Suppose, under a true candidate value, $T_n$ converges in distribution to a random variable with continuous cumulative distribution function $F_0$. Suppose the bootstrap quantile $\hat c_n$ converges in probability to $c_q$, where $F_0(c_q)=q$ and $F_0$ is continuous at $c_q$. Then $P(T_n\leq\hat c_n)\to q$.

Proof. For any $\varepsilon>0$, $$ P(T_n\leq c_q-\varepsilon)-P(|\hat c_n-c_q|>\varepsilon) \leq P(T_n\leq\hat c_n) \leq P(T_n\leq c_q+\varepsilon)+P(|\hat c_n-c_q|>\varepsilon). $$ The error probabilities involving $\hat c_n$ tend to zero by convergence in probability. Convergence in distribution and continuity of $F_0$ imply that the other probabilities tend to $F_0(c_q-\varepsilon)$ and $F_0(c_q+\varepsilon)$. Letting $\varepsilon$ decrease to zero, continuity at $c_q$ makes both limits equal $F_0(c_q)=q$. The lower and upper bounds therefore converge to $q$, proving the result. $\square$

This result states what calibration needs to accomplish: the bootstrap cutoff must estimate the relevant null quantile, and the statistic must have a suitable limiting distribution. The result does not establish bootstrap validity for every GLM or every parameter configuration. That validity is a substantive modeling and asymptotic question.

Worked Example: Calibrating a Gaussian Mean Profile

Suppose four independent observations follow $N(\mu,\sigma^2)$, with both parameters unknown, and the observed data are $1,2,3,4$. Test the candidate mean $\mu_0=2$. The unrestricted estimate is $\hat\mu=2.5$, and its residual sum of squares is $5$. Under $\mu=2$, the constrained residual sum of squares is $6$. For the normal likelihood with unknown variance maximized in each fit, the statistic is $$ T_{\mathrm{obs}}(2)=4\log(6/5)\approx0.729286. $$ The constrained maximum likelihood estimate of the variance is $6/4=1.5$. Bootstrap samples are generated from $N(2,1.5)$, and each replicate's mean and variance are refitted both freely and with $\mu=2$.

For an arithmetic illustration, consider five rounded bootstrap samples $(0,1,2,3)$, $(1,2,3,5)$, $(0,1,3,5)$, $(1,1,2,4)$, and $(0,2,3,5)$. For a replicate with constrained residual sum of squares $A=\sum_i(y_i^*-2)^2$ and unrestricted residual sum of squares $R=\sum_i(y_i^*-\bar y^*)^2$, its statistic is $4\log(A/R)$. The five $(A,R)$ pairs are respectively $(6,5)$, $(11,8.75)$, $(15,14.75)$, $(6,6)$, and $(14,13)$, giving statistics approximately $0.7293, 0.9154, 0.0672, 0,$ and $0.2964$. The sorted fourth value, the empirical 80th percentile for five replicates, is $0.7293$. The observed statistic is at the cutoff and is retained. In actual computation, the samples are randomly generated; the listed rounded samples simply make the refitting and quantile arithmetic explicit.

Constructing and Interpreting Endpoints

To construct an interval-like confidence set, evaluate $T_{\mathrm{obs}}(\psi)$ and its bootstrap cutoff across candidate values, retaining those satisfying $T_{\mathrm{obs}}(\psi)\leq\hat c_{1-\alpha}(\psi)$. Endpoints occur where the inequality changes from true to false, and can be located by refining a grid and solving the boundary equation when the relevant functions behave continuously. The cutoff itself may change with $\psi$, so replacing it with one cutoff computed at a single candidate is not generally justified.

The retained set need not be a single bounded interval. It may have multiple components or extend without bound; these possibilities should not be removed merely to force a conventional interval form. Near a boundary or under weak identification, simulation can also expose unstable fits or a poorly behaved statistic, but it does not guarantee reliable inference. In particular, bootstrap calibration cannot create information absent from the data or repair a misspecified sampling model.

Common Pitfalls

Simulating from the unrestricted fit. The null distribution for a candidate is generated under its constrained fit, not under the unrestricted estimate, unless the candidate happens to agree with that fit.
Skipping refitting. The bootstrap statistic requires new unrestricted and constrained maximizations for every replicate; reusing observed-data estimates changes the statistic.
Treating a Monte Carlo cutoff as exact. Finite $B$ gives a noisy quantile estimate. Increase $B$ when precision matters and report the simulation size and quantile convention.
Assuming a single cutoff applies everywhere. Candidate-specific null fits can yield candidate-specific bootstrap distributions and cutoffs.
Ignoring failed fits. Convergence failures, boundary estimates, or nonfinite likelihoods should be diagnosed and handled transparently, not silently discarded.

Key takeaway: Bootstrap calibration of a profile likelihood ratio is a test-inversion procedure: simulate under each candidate's constrained fit, refit both models in every replicate, estimate the null quantile, and retain candidates whose observed statistic falls below that cutoff.

Check Your Understanding

Use these questions to check whether the simulation and inversion logic is clear.

For a candidate $\psi$, which fitted parameter value determines the bootstrap data-generating distribution?
Why must each bootstrap replicate be refitted both with and without the target constraint?
What does the bootstrap empirical quantile estimate, and why is it not generally an exact finite-sample cutoff?
If the bootstrap cutoff changes across candidate values, how should the confidence set be constructed?
Name two reasons a parametric bootstrap may still produce unreliable profile inference.