Likelihood-Ratio Cutoffs and Small-Sample Calibration

Why the Likelihood-Ratio Cutoff Matters

A profile-likelihood interval is obtained by comparing the best fit at each candidate parameter value with the unrestricted best fit. That comparison produces a likelihood-ratio statistic, but it does not by itself specify which candidate values to retain. A cutoff turns the statistic into a confidence set. With sparse or noisy observations, the usual asymptotic cutoff may not achieve its intended coverage, so the choice of cutoff can materially affect conclusions about parameter precision.

The focus here is the cutoff and its calibration. The introductory material on profile-likelihood confidence intervals in QSP establishes how profiling produces a curve and how its crossings define an interval. We make the cutoff explicit, state when its standard reference distribution is appropriate, and show how parametric simulation can assess finite-sample performance.

Definition: Let $\psi$ be a scalar parameter of interest and let $\lambda$ denote nuisance parameters. Write the full parameter vector as $(\psi,\lambda)$, the log-likelihood as $\ell(\psi,\lambda)$, and its unrestricted maximum as $\ell(\hat{\psi},\hat{\lambda})$. The profile log-likelihood at a fixed $\psi$ is $\ell_p(\psi)=\max_{\lambda}\ell(\psi,\lambda)$. The profile likelihood-ratio statistic is $$ R(\psi)=2\left\{\ell(\hat{\psi},\hat{\lambda})-\ell_p(\psi)\right\}. $$ For a chosen cutoff $c$, the profile-likelihood confidence set is the set of candidate values satisfying $R(\psi)\leq c$.

The statistic is nonnegative because the unrestricted fit cannot be worse than the fit constrained to a particular $\psi$. If the objective function is the deviance $D=-2\ell$, then $R(\psi)=D_p(\psi)-D_{\min}$. Thus a difference in deviance is already on the likelihood-ratio scale; it should not be multiplied by two again.

When the Chi-Square Cutoff Is Appropriate

For a regular, identifiable scalar parameter in the interior of its allowed range, Wilks' theorem gives the large-sample reference distribution $R(\psi_0)\mathrel{\dot\sim}\chi^2_1$ when the candidate value $\psi_0$ is the true value. A nominal 95% two-sided profile interval therefore retains values with $R(\psi)\leq\chi^2_{1,0.95}$, approximately 3.841.

This approximation is a large-sample result, not a universal property of likelihood ratios. It relies on a correctly specified likelihood, sufficient information in the data, local identifiability, a smooth likelihood near the true parameter, and nonsingular information. The true parameter must be an interior point for the ordinary result to apply. In a QSP application, the likelihood must also represent the observation structure appropriately, including any modeled error dependence. A small number of observations, high measurement noise, weak identification, or parameters constrained to a boundary can make the reference distribution inaccurate.

Interpretation: A 95% profile-likelihood interval using the 3.841 cutoff is an asymptotic, repeated-sampling procedure. Its intended coverage is 95% under the model and regularity conditions; it does not mean that there is a 95% probability that the fixed parameter lies in the computed interval.

A short justification explains the approximation. Near a regular maximum, a Taylor expansion of the log-likelihood has a leading quadratic term. After scaling by the information, the estimate's local error is approximately normal. For one scalar constraint, the loss in fit is approximately the square of a standard normal quantity:

R(\psi_0)\ \mathrel{\dot\sim}\ Z^2,\qquad Z\sim N(0,1), \qquad Z^2\sim\chi^2_1.

The chi-square cutoff is consequently a convenient approximation to the distribution of the statistic under the candidate value. The argument is asymptotic: it does not guarantee accurate calibration in a particular small data set.

Worked Example: Finding Profile Crossings

Suppose a hypothetical QSP analysis estimates a clearance parameter $\psi$, measured in L/h. The unrestricted estimate is 1.00 L/h. The computed profile likelihood-ratio values are 4.4 at 0.55 L/h, 3.6 at 0.60 L/h, 3.5 at 1.20 L/h, and 4.2 at 1.25 L/h. Use the asymptotic 95% cutoff and linearly interpolate the two crossings.

The candidate values are retained when $R(\psi)\leq 3.841$. On the lower side, the cutoff lies between 0.55 and 0.60 L/h. Linear interpolation gives

\psi_L=0.55+ \frac{4.4-3.841}{4.4-3.6}(0.60-0.55) =0.5849\ \text{L/h}.

On the upper side, it lies between 1.20 and 1.25 L/h:

\psi_U=1.20+ \frac{3.841-3.5}{4.2-3.5}(1.25-1.20) =1.224\ \text{L/h}.

The approximate interval is therefore 0.585 to 1.224 L/h. This calculation uses the 3.841 cutoff as an asymptotic approximation. Interpolation only estimates the location of the crossing between computed profile points; it does not correct a poorly calibrated cutoff or an inaccurately optimized profile.

Calibrating a Cutoff by Parametric Simulation

Parametric simulation can estimate the finite-sample distribution of $R(\psi_0)$ under a candidate value $\psi_0$. The null model fixes $\psi=\psi_0$, while nuisance parameters are set to estimates obtained under that constraint. Each simulated data set is generated from this fitted null model. For every replicate, fit both the constrained model and the unrestricted model, then calculate the likelihood-ratio statistic using the same definition and fitting procedure as for the observed data.

If $B$ replicate statistics are $R_1,\ldots,R_B$, an estimated 95% cutoff is their empirical 95th percentile. For a confidence set, this can be done at candidate values: retain a candidate when its observed statistic is no greater than its simulated cutoff. Since the distribution can depend on $\psi_0$ and nuisance parameters, one simulated cutoff need not be valid for every candidate. This procedure uses a plug-in estimate of nuisance parameters, so it is a practical approximation rather than an exact guarantee of coverage.

Fit the constrained model.
Fix the candidate value $\psi_0$ and estimate the nuisance parameters under that constraint.

Generate replicate data.
Simulate $B$ data sets from the constrained fitted model, preserving the design and modeled observation-error structure.

Refit both models.
For each replicate, maximize the likelihood with $\psi=\psi_0$ and without that constraint.

Estimate the cutoff and compare.
Calculate each replicate's $R(\psi_0)$, take the desired empirical quantile, and compare the observed statistic with that cutoff.

Worked Example: A Candidate-Specific Simulated Cutoff

Suppose a hypothetical analysis has only eight informative observations. At a candidate value $\psi_0=1.00$ L/h, the observed profile statistic is 4.40. To assess the asymptotic cutoff, the analyst simulates $B=1{,}999$ replicate data sets under the constrained fit and refits both models for each replicate. The ordered simulated statistics have value 5.02 at rank 1,900.

Using the empirical quantile convention $k=\lceil 0.95(B+1)\rceil$, the target rank is

k=\lceil 0.95(2{,}000)\rceil=1{,}900.

The simulated 95% cutoff is therefore 5.02. The observed statistic, 4.40, is below this cutoff, so this candidate is not rejected by the simulated calibration. By contrast, it exceeds the asymptotic cutoff 3.841. The two calibrations thus lead to different decisions for this candidate. This result does not establish that 5.02 is a universal cutoff: it is an estimate for this candidate and this fitted null model. Monte Carlo variability also remains because the cutoff is estimated from a finite number of simulations.

Assessing Coverage and Diagnosing Failure

A simulated cutoff answers how a statistic behaves under a specified fitted null model. A complementary assessment asks how often an interval procedure contains a known value across repeated simulated data sets. For a coverage study, choose a plausible parameter vector, simulate data from the model, construct an interval from each data set, and count the fraction of intervals containing the generating value. This checks the entire interval procedure, including how its cutoff and endpoints are computed, at the chosen simulation settings. It is not proof of coverage across all parameter values or designs.

Worked Example: Comparing Simulated Coverage

Consider a hypothetical sparse, noisy QSP design with six observation times. The generating clearance is 1.00 L/h. In a simulation study of 500 independent data sets, intervals using the 3.841 cutoff contain 1.00 L/h in 446 cases; intervals using a candidate-specific parametric calibration contain it in 472 cases.

The estimated coverage rates are

\hat{C}_{\mathrm{asym}}=\frac{446}{500}=0.892,\qquad \hat{C}_{\mathrm{sim}}=\frac{472}{500}=0.944.

The approximate Monte Carlo standard error of an estimated coverage $\hat{C}$ from $M$ independent data sets is $\sqrt{\hat{C}(1-\hat{C})/M}$. For the asymptotic procedure it is

SE(\hat{C}_{\mathrm{asym}}) \approx\sqrt{\frac{0.892(1-0.892)}{500}} =0.0139.

For the calibrated procedure it is

SE(\hat{C}_{\mathrm{sim}}) \approx\sqrt{\frac{0.944(1-0.944)}{500}} =0.0103.

In this hypothetical study, the asymptotic procedure undercovers at the tested parameter setting, while the simulated procedure is closer to the nominal 95% target. The difference is informative, but its size should be interpreted in light of Monte Carlo error and the particular model, parameter values, and design used.

Diagnostics should be applied before interpreting a cutoff comparison. Check that simulated data follow the intended model and design, and that constrained and unrestricted fits converge reliably across replicates. A substantial fraction of failed fits, implausible parameter estimates, or statistic values that change with optimizer settings can contaminate an empirical quantile. Increase the simulation count to reduce Monte Carlo noise, and inspect whether the conclusion changes across plausible parameter settings. If the parameter is at a boundary, the usual chi-square reference may not apply; the distribution can change under constraints, so calibration should reflect the actual constraint and null model.

The main alternatives answer different questions. The Wald approach uses local curvature and a standard error to form an interval; it can be convenient but may be inaccurate when the likelihood is asymmetric or the parameter is weakly estimated. The asymptotic profile-likelihood approach uses the shape of the profile and a chi-square cutoff, but still depends on the large-sample approximation. Parametric calibration keeps the likelihood-ratio statistic and estimates its reference cutoff by simulation under a specified model; it is more computationally demanding and depends on that model and its plug-in nuisance estimates. None of these approaches can compensate for a misspecified likelihood or uninformative data.

Common Pitfalls and Practical Limits

Applying 3.841 automatically. It is a 95% asymptotic cutoff for a regular, scalar, interior parameter, not a universal threshold for every profile.
Confusing the likelihood-ratio and log-likelihood scales. The statistic is twice the log-likelihood difference. If values are already differences in deviance, use that difference directly.
Treating a simulated cutoff as exact. It is estimated with Monte Carlo error and typically uses fitted nuisance parameters. Its calibration is specific to the candidate and simulation model.
Ignoring failed fits. Failed or inconsistent optimizations can distort the simulated distribution. Resolve or report failures rather than silently treating them as valid replicates.
Reading coverage from one interval. Coverage is a long-run property of a procedure, assessed over repeated data sets, not a probability statement about a single computed interval.

Key takeaway: Use the $\chi^2_1$ cutoff of 3.841 for a nominal 95% scalar profile interval only when the regular large-sample conditions are plausible. With sparse or noisy data, simulate the likelihood-ratio distribution under candidate constrained fits and, when useful, check interval coverage over repeated simulated data sets.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A profile statistic is 3.70 at a candidate value. Under the regular scalar asymptotic 95% rule, is the candidate retained?
In a simulation with $B=999$, what order statistic estimates the 95th percentile using $k=\lceil 0.95(B+1)\rceil$?
Why can a candidate-specific simulated cutoff differ from 3.841?
In a coverage study, 90 of 100 intervals contain the generating value. What is the estimated coverage, and what does it describe?

Solutions. (1) Yes. Since $3.70\leq3.841$, the candidate is retained by this rule. (2) $k=\lceil0.95(1{,}000)\rceil=950$, so use the 950th ordered statistic. (3) The finite-sample distribution under the fitted constrained model may differ from the asymptotic chi-square distribution; the simulated estimate also has Monte Carlo variability. (4) The estimated coverage is $90/100=0.90$, or 90%. It estimates the repeated-sampling fraction of intervals containing the generating value under the specified simulation settings; it is not the probability that the value lies in any one computed interval.