Profiling Versus Integrating Out Nuisance Parameters

Two Ways to Handle a Nuisance Parameter

When a model has a parameter of interest and a nuisance parameter, inference about the target still requires a decision about the nuisance parameter. One approach asks how well the model can fit at each fixed target value: maximize the likelihood over the nuisance parameter. Another approach averages the likelihood over nuisance values, using a specified measure or prior distribution. These operations can produce different functions of the target, even when they start from the same likelihood.

The distinction matters because maximization needs no probability distribution over nuisance values, whereas integration does. Thus an integrated likelihood generally reflects both the data and the chosen weighting of nuisance values. It is not obtained merely by rewriting a profile likelihood.

Definition: Let $L(\theta,\lambda)$ be a likelihood, where $\theta$ is the scalar parameter of interest and $\lambda$ is a nuisance parameter. The profile likelihood for $\theta$ is $$ L_p(\theta)=\sup_{\lambda\in\Lambda}L(\theta,\lambda). $$ Given a probability measure $\Pi_\theta$ on the nuisance-parameter space, the integrated likelihood for $\theta$ is $$ L_I(\theta)=\int_{\Lambda}L(\theta,\lambda)\,\Pi_\theta(d\lambda). $$ If $\Pi_\theta$ has density $\pi(\lambda\mid\theta)$ with respect to a specified measure, this is $L_I(\theta)=\int L(\theta,\lambda)\pi(\lambda\mid\theta)\,d\lambda$. When the measure comes from a Bayesian prior, this is often called a marginal likelihood for the target, or an integrated likelihood.

The conditional notation $\Pi_\theta$ allows the nuisance distribution to depend on the target. If a joint prior is specified, the relevant nuisance distribution at a fixed $\theta$ is its conditional distribution given $\theta$, when that conditional is defined. Different choices of measure can give different integrated likelihoods. In contrast, profiling selects the likelihood value at the best-fitting nuisance value for each $\theta$.

A Finite Example: Maximizing Is Not Averaging

Worked Example: A Two-Value Nuisance Parameter

Suppose the nuisance parameter has values $\lambda=0,1$, and the likelihood at the observed data is as follows:

Target value	$L(\theta,0)$	$L(\theta,1)$
$\theta=A$	0.8	0.1
$\theta=B$	0.7	0.6

For profiling, take the larger likelihood in each row: $L_p(A)=\max(0.8,0.1)=0.8$, while $L_p(B)=\max(0.7,0.6)=0.7$. The profile likelihood favors $A$.

Now integrate using the nuisance distribution $P(\lambda=0)=0.1$, $P(\lambda=1)=0.9$. The two integrated likelihoods are $L_I(A)=0.1(0.8)+0.9(0.1)=0.17$ and $L_I(B)=0.1(0.7)+0.9(0.6)=0.61$. This integrated likelihood favors $B$, because the specified distribution puts most of its weight on $\lambda=1$, where $B$'s likelihood is larger. The reversal is not a contradiction: maximizing and averaging answer different questions.

Normal Mean With Unknown Variance

Consider independent observations $X_1,\ldots,X_n$ from $N(\theta,\sigma^2)$, where $\theta$ is the target and $\sigma^2>0$ is nuisance. Write $S(\theta)=\sum_{i=1}^n(X_i-\theta)^2$. For observed data that are not all equal to the fixed value $\theta$, the likelihood is

L(\theta,\sigma^2) =(2\pi\sigma^2)^{-n/2} \exp\!\left(-\frac{S(\theta)}{2\sigma^2}\right).

For fixed $\theta$, maximizing this likelihood over $\sigma^2$ gives $\hat{\sigma}^2(\theta)=S(\theta)/n$. Substitution yields the profile likelihood

L_p(\theta) =\left(\frac{2\pi S(\theta)}{n}\right)^{-n/2}e^{-n/2}, \qquad S(\theta)>0.

Thus the profile likelihood is proportional, as a function of $\theta$, to $S(\theta)^{-n/2}$. The nuisance variance is not averaged over: it is set to its maximizing value separately at each candidate $\theta$.

Worked Example: Profiling and Integrating in the Normal Model

Take the observed values $x_1=0$, $x_2=2$, so $n=2$. Then $S(\theta)=\theta^2+(2-\theta)^2=2(\theta-1)^2+2$. At $\theta=1$, $S=2$ and the maximizing variance is $2/2=1$. At $\theta=0$, $S=4$ and the maximizing variance is $4/2=2$.

Because the profile likelihood is proportional to $S^{-1}$, its relative value at $\theta=0$, compared with its value at $\theta=1$, is $2/4=1/2$. This comparison removes the common likelihood constant.

For integration, choose an inverse-gamma prior on $\sigma^2$, with shape $\alpha>0$ and scale $\beta>0$:

\pi(v)=\frac{\beta^\alpha}{\Gamma(\alpha)} v^{-\alpha-1}\exp(-\beta/v),\qquad v>0.

Here $v$ denotes the variance. Multiplying the likelihood by this density and integrating gives

\begin{aligned} L_I(\theta) &=\frac{\beta^\alpha}{\Gamma(\alpha)}(2\pi)^{-n/2} \int_0^\infty v^{-(\alpha+n/2)-1} \exp\!\left[-\frac{\beta+S(\theta)/2}{v}\right]\,dv\\ &=\frac{\beta^\alpha}{\Gamma(\alpha)}(2\pi)^{-n/2} \frac{\Gamma(\alpha+n/2)} {\left(\beta+S(\theta)/2\right)^{\alpha+n/2}}. \end{aligned}

For the second equality, set $u=(\beta+S(\theta)/2)/v$. Then $v=(\beta+S(\theta)/2)/u$, and the integral becomes $(\beta+S(\theta)/2)^{-(\alpha+n/2)} \int_0^\infty u^{\alpha+n/2-1}e^{-u}\,du$, which equals the displayed gamma-function expression.

Set $\alpha=2,\beta=1$. At $\theta=1$, the target-dependent denominator is $1+2/2=2$; at $\theta=0$, it is $1+4/2=3$. The integrated likelihood ratio is therefore $L_I(0)/L_I(1)=(2/3)^3=8/27$, approximately $0.296$. This is not the profile ratio $1/2$: integration with this prior gives a different function of $\theta$.

What Can Be Proved About the Comparison?

A useful general comparison follows directly when the nuisance distribution is a probability measure. Assume $L(\theta,\lambda)$ is measurable in $\lambda$, nonnegative, and has a finite supremum for the fixed $\theta$ under consideration. Then the integrated likelihood cannot exceed the profile likelihood at that $\theta$.

Result: If $\Pi_\theta$ is a probability measure, then $$ L_I(\theta)\leq L_p(\theta). $$ Equality holds if and only if $L(\theta,\lambda)=L_p(\theta)$ for $\Pi_\theta$-almost every $\lambda$.

Proof. By definition of the supremum, $L(\theta,\lambda)\leq L_p(\theta)$ for every $\lambda$. Integrating this pointwise inequality against the nonnegative probability measure $\Pi_\theta$ gives

\int L(\theta,\lambda)\,\Pi_\theta(d\lambda) \leq \int L_p(\theta)\,\Pi_\theta(d\lambda) =L_p(\theta),

because $\Pi_\theta(\Lambda)=1$. For the equality condition, the nonnegative function $L_p(\theta)-L(\theta,\lambda)$ has integral zero exactly when it is zero almost everywhere under $\Pi_\theta$. This proves both claims. $\square$

This pointwise bound does not say that the integrated likelihood must be smaller by the same factor at every target value, or that the two methods must rank target values identically. The finite example showed that the rankings can differ. Also, an improper measure is not a probability measure, so this bound does not apply to an integral using an improper prior without additional normalization and justification.

A Case Where the Two Calculations Are Proportional

Worked Example: The Scale-Invariant Variance Measure

Return to the normal likelihood, but formally integrate with respect to the improper measure $dv/v$, rather than a proper inverse-gamma prior. For $S(\theta)>0$, the integral is finite:

\begin{aligned} \int_0^\infty L(\theta,v)\,\frac{dv}{v} &=(2\pi)^{-n/2}\int_0^\infty v^{-n/2-1}\exp\!\left[-\frac{S(\theta)}{2v}\right]\,dv\\ &=(2\pi)^{-n/2}\Gamma(n/2) \left(\frac{S(\theta)}{2}\right)^{-n/2}. \end{aligned}

The substitution $u=S(\theta)/(2v)$ gives the second line. This integrated function is proportional to $S(\theta)^{-n/2}$, just like the profile likelihood. In the two-observation data example, its ratio at $\theta=0$ versus $\theta=1$ is consequently $2^{-1}=1/2$, equal to the profile ratio.

This agreement is a special feature of this model and measure, not a general identity between profiling and integration. Since $dv/v$ is improper, the integral is not a marginal likelihood based on a normalized prior; it is a relative integrated-likelihood calculation. The proper inverse-gamma example shows how a proper prior can instead change the shape as well as the scale of the target function.

Why the Distinction Matters

Profiling is a likelihood operation: at each target value it retains the best fit available from the nuisance parameter. Integration is an averaging operation: nuisance values with more probability weight contribute more to the result, even if they do not maximize the likelihood. Consequently, integrated likelihoods require a defensible measure and can be sensitive to its choice. A prior that is diffuse on one parameter scale need not remain diffuse after a change of scale.

In GLM work, dispersion parameters and other model parameters can be nuisance parameters when attention is on a coefficient or a function of coefficients. Profile likelihood handles them by constrained maximization; integrating them out instead requires a probability distribution or another explicitly stated measure. The two procedures may sometimes be proportional, but that must be established for the model and measure in use, not assumed from their shared starting likelihood.

Common Pitfall

Wrong reasoning: “The nuisance parameter is not the target, so I can remove it by integrating the likelihood over all its possible values.” There is no unique such integral until a measure, including its scale and parameterization, has been specified.
Correction: If the intended operation is to retain the best nuisance fit, use the supremum defining the profile likelihood. If the intended operation is averaging, state the nuisance distribution or measure and perform the integral. Do not treat the resulting functions as interchangeable without checking their dependence on the target.

Key takeaway: Profiling maximizes over nuisance values; integrating averages over them using a specified measure. The former is a likelihood operation, while the latter depends on an additional weighting choice. They can yield different relative likelihoods and different conclusions about target values.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In one sentence, distinguish the operation defining $L_p(\theta)$ from the operation defining $L_I(\theta)$.
Why is a nuisance-parameter measure needed to define an integrated likelihood, but not a profile likelihood?
For the observations $0,2$, calculate $S(\theta)$ and the maximizing variance at $\theta=2$.
In the inverse-gamma calculation, which part of the integrated likelihood depends on the chosen shape and scale parameters?
Why does the bound $L_I(\theta)\leq L_p(\theta)$ require a probability measure?