Profile Likelihood for a Nuisance Parameter

Why Profile a Likelihood?

A likelihood with several parameters can be difficult to maximize all at once. Often the main interest is in one parameter, while the others are needed to describe the model but are not themselves the target. These additional parameters are called nuisance parameters. Profiling reduces the problem: for each fixed value of the target parameter, maximize the likelihood over the nuisance parameter, then compare the resulting maximized likelihoods across target values.

This is a maximization procedure, not an assumption that the nuisance parameter is known. Its key practical benefit is that it turns a joint optimization into a sequence of conditional optimizations followed by a one-parameter optimization. It is aligned with the SOA VEE Mathematical Statistics syllabus.

Definition: Let $L(\theta,\eta)$ be the likelihood for a model with target parameter $\theta$ and nuisance parameter $\eta$. For each fixed $\theta$, the profile likelihood is $$ L_p(\theta)=\sup_{\eta\in\mathcal{E}(\theta)} L(\theta,\eta), $$ where $\mathcal{E}(\theta)$ is the set of nuisance-parameter values feasible at that $\theta$. If the supremum is attained, a value attaining it is a conditional maximum likelihood estimate of $\eta$, denoted $\widehat{\eta}(\theta)$.

The feasible set can depend on $\theta$, particularly when the parameter space imposes joint constraints. For each $\theta$, first optimize only over its feasible nuisance values. If the likelihood is positive, profiling the log-likelihood gives the same maximizing values because the logarithm is strictly increasing: $ \ell_p(\theta)=\sup_{\eta\in\mathcal{E}(\theta)}\ell(\theta,\eta) $, where $\ell=\log L$.

How Profiling Locates a Joint MLE

The connection between a profile maximum and a joint maximum follows directly from the definition. For every feasible pair $(\theta,\eta)$, the profile satisfies $L_p(\theta)\geq L(\theta,\eta)$. Consequently, if a feasible pair attains a global maximum of the joint likelihood, its target coordinate must maximize the profile. Conversely, a profile maximum gives a joint MLE only when its nuisance-parameter supremum is attained.

Theorem: If $\widehat{\theta}$ maximizes $L_p(\theta)$ over the allowed target values, and there is a feasible $\widehat{\eta}\in\mathcal{E}(\widehat{\theta})$ such that $L(\widehat{\theta},\widehat{\eta})=L_p(\widehat{\theta})$, then $(\widehat{\theta},\widehat{\eta})$ is a joint MLE. Indeed, for every feasible $(\theta,\eta)$, $$ L(\theta,\eta)\leq L_p(\theta)\leq L_p(\widehat{\theta}) =L(\widehat{\theta},\widehat{\eta}). $$

This argument also identifies an important limitation: a supremum need not be attained. A profile can have a maximum even though no feasible parameter pair achieves that likelihood value. In that case there is no joint MLE at the profile maximum.

Hold the target fixed.
Write the likelihood as a function of the nuisance parameter, respecting the feasible parameter space.

Maximize conditionally.
Find the nuisance maximizer or supremum for each fixed target value. Check boundaries as well as any interior score solution.

Form and maximize the profile.
Substitute the conditional maximizer into the likelihood when the supremum is attained, then optimize over the target parameter.

Verify attainment and feasibility.
Confirm the resulting pair is allowed and actually attains the conditional supremum at the profile maximum.

Worked Applications

Worked Example: Two Normal Samples With Shared Nuisance Mean

Suppose independent observations $X_1,X_2,X_3\sim N(\theta+\eta,1)$ are $2,4,6$, and independent observations $Y_1,Y_2\sim N(\eta,1)$ are $-1,1$. All five observations are mutually independent; the variances are known to be 1, and both parameters are unrestricted real numbers. Treat $\theta$ as the target. Find the profile likelihood maximizer and the joint MLE.

Method. Since the variances are known, maximizing the likelihood is equivalent to minimizing the sum of squared residuals. The sample means are $ \bar{x}=(2+4+6)/3=4 $ and $ \bar{y}=(-1+1)/2=0 $. For fixed $\theta$, the log-likelihood, apart from a constant, is

\ell(\theta,\eta)=-\frac12\left\{\sum_{j=1}^3(x_j-\theta-\eta)^2+ \sum_{j=1}^2(y_j-\eta)^2\right\}.

Differentiate with respect to $\eta$ and set the derivative to zero. Equivalently, minimize the squared residual sum:

3(\theta+\eta-\bar{x})+2(\eta-\bar{y})=0, \qquad \widehat{\eta}(\theta)=\frac{3(\bar{x}-\theta)+2\bar{y}}{5} =\frac{12-3\theta}{5}.

This conditional estimate exists for every real $\theta$. To see how the profile varies, separate the within-group sums of squares from the differences between the fitted means and observed sample means. The within-group terms are $ (2-4)^2+(4-4)^2+(6-4)^2=8 $ and $ (-1-0)^2+(1-0)^2=2 $. The remaining sum is $3(\theta+\eta-4)^2+2\eta^2$. Substituting $\widehat{\eta}(\theta)$, or minimizing this quadratic in $\eta$, gives

\operatorname{SSE}_p(\theta)=10+\frac65(\theta-4)^2.

Because the profile likelihood is proportional to $\exp\{-\operatorname{SSE}_p(\theta)/2\}$, it is maximized by minimizing this sum. The unique maximum occurs at $\widehat{\theta}=4$. Then $ \widehat{\eta}=\widehat{\eta}(4)=(12-12)/5=0 $. At this pair, the fitted means are $\theta+\eta=4$ for the $X$ sample and $\eta=0$ for the $Y$ sample, matching their sample means. The conditional supremum is attained, so this is a joint MLE. Check: the joint estimates are $(4,0)$, not $(4,2)$; the latter would give fitted means 6 and 2 rather than 4 and 0.

Common mistake avoided: the nuisance estimate must be calculated from both samples’ contributions to the likelihood. Using only the $X$ sample would not correctly account for the information about $\eta$ in the $Y$ observations.

Worked Example: Profiling Two Poisson Means

Independent counts have distributions $X\sim\operatorname{Poisson}(\theta\lambda)$ and $Y\sim\operatorname{Poisson}(3\lambda)$, with observed values $x=8$ and $y=12$. Assume $\theta>0$ and $\lambda>0$. Treat $\theta$ as the target and find both MLEs and the fitted means.

Method. Ignoring factors that do not depend on the parameters, the likelihood is $ L(\theta,\lambda)\propto e^{-\lambda(\theta+3)}\theta^8\lambda^{20} $. For fixed $\theta$, its log-likelihood in $\lambda$, apart from a constant, is $ -\lambda(\theta+3)+20\log\lambda+8\log\theta $. The derivative with respect to $\lambda$ is $ -(\theta+3)+20/\lambda $, which is zero at

\widehat{\lambda}(\theta)=\frac{20}{\theta+3}.

The conditional maximum is attained: the derivative is positive below this value and negative above it. Substitution into the likelihood gives, up to a positive constant independent of $\theta$, $ L_p(\theta)\propto \theta^8/(\theta+3)^{20} $. The profile log-likelihood derivative is

\frac{d}{d\theta}\log L_p(\theta)=\frac8\theta-\frac{20}{\theta+3} =\frac{24-12\theta}{\theta(\theta+3)}.

For $\theta>0$, the denominator is positive. The derivative is positive for $0<\theta<2$ and negative for $\theta>2$, so the profile has its maximum at $\widehat{\theta}=2$. The nuisance estimate is $ \widehat{\lambda}=20/(2+3)=4 $. The fitted means are $\widehat{\theta}\widehat{\lambda}=2(4)=8$ and $3\widehat{\lambda}=3(4)=12$, equal to the observed counts. The estimates are feasible and attain the joint maximum. Check: the two fitted means sum to $20$, the total observed count.

Common mistake avoided: after finding $\widehat{\theta}=2$, substitute it into the conditional nuisance estimate. Do not report a nuisance estimate or fitted means from an earlier, unprofiled calculation.

Worked Example: Normal Mean With Unknown Variance as Nuisance

A hypothetical instrument produces independent readings $1,3,5,7$, modeled as $X_i\sim N(\theta,\sigma^2)$, where both $\theta\in\mathbb{R}$ and $\sigma^2>0$ are unknown. Treat the mean $\theta$ as the target and profile out the variance. Find the joint MLE.

Method. For fixed $\theta$, write $ \operatorname{SSE}(\theta)=\sum_{i=1}^4(x_i-\theta)^2 $. The likelihood is proportional to $ (\sigma^2)^{-2}\exp\{-\operatorname{SSE}(\theta)/(2\sigma^2)\} $. Differentiating its log with respect to $\sigma^2$ gives $ -2/\sigma^2+\operatorname{SSE}(\theta)/(2(\sigma^2)^2) $. Setting this equal to zero yields the conditional estimate

\widehat{\sigma^2}(\theta)=\frac{\operatorname{SSE}(\theta)}{4}.

The observed mean is $\bar{x}=(1+3+5+7)/4=4$. Expand the squared residual sum around this mean: $ \operatorname{SSE}(\theta)=\sum_i(x_i-4)^2+4(\theta-4)^2 $, because $\sum_i(x_i-4)=0$. The first term is $9+1+1+9=20$, so $ \operatorname{SSE}(\theta)=20+4(\theta-4)^2 $. For each fixed $\theta$, this is positive and the conditional variance estimate is an attained maximum. Substitution shows that the profile likelihood is proportional to $\operatorname{SSE}(\theta)^{-2}$, since the exponential factor becomes the constant $e^{-2}$. It is therefore maximized when $\operatorname{SSE}(\theta)$ is minimized, at $\widehat{\theta}=4$. Then $ \widehat{\sigma^2}=20/4=5 $. Thus the joint MLE is $(\widehat{\theta},\widehat{\sigma^2})=(4,5)$. Check: the fitted mean is the observed sample mean, and the fitted variance is the average squared deviation from it.

Common mistake avoided: the likelihood estimate of the variance uses the divisor $n=4$, not $n-1$. The latter is used in a familiar unbiased sample-variance formula, not in this likelihood maximization.

When a Profile Maximum Is Not a Joint MLE

A conditional maximum that does not exist is a distinct issue from a conditional maximum that exists but is located at a boundary. Always check the parameter space and whether the conditional supremum is actually achieved. The following example illustrates why the attainment condition matters.

Worked Example: A Profile With No Attained Joint Maximum

One Bernoulli observation is a success, with success probability $p=\eta/(1+\eta)$, where $\eta>0$. Let $\theta$ be an unrestricted parameter that does not affect the probability model. Find the profile likelihood and determine whether a joint MLE exists.

The likelihood for the observed success is $ L(\theta,\eta)=\eta/(1+\eta) $. For each fixed $\theta$, this increases toward 1 as $\eta$ increases, so $ L_p(\theta)=\sup_{\eta>0}\eta/(1+\eta)=1 $. The profile is constant, so every allowed $\theta$ maximizes it. However, no finite $\eta>0$ gives likelihood 1: for every such $\eta$, $ \eta/(1+\eta)<1 $. Therefore the conditional supremum is not attained, and no feasible pair is a joint MLE. Check: taking $\eta$ arbitrarily large approaches 1 but never equals 1.

Common mistake avoided: it is not enough to find a maximizing target value of the profile. One must also verify that a feasible nuisance value attains the profile there.

Common Traps and a Practical Check

Profiling is reliable when the conditional optimization and the final comparison are both done over the correct parameter space. In an unconstrained, smooth interior problem, differentiating the log-likelihood with respect to the nuisance parameter often gives a conditional score equation. But a score solution alone is not proof of a maximum: compare it with feasible boundaries and check that it is a maximum. Under constraints linking $\theta$ and $\eta$, the conditional feasible set may change with $\theta$, so an unconstrained nuisance estimate can be invalid.

Do not confuse a conditional estimate with a final estimate. $\widehat{\eta}(\theta)$ depends on the fixed target value. Evaluate it at the profile-maximizing $\widehat{\theta}$ to obtain the joint estimate.
Do not assume a profile maximum is attained. A supremum can be approached without being reached, as the Bernoulli example shows.
Do not maximize the original likelihood in only one parameter while freezing the nuisance at an arbitrary value. Profiling requires re-maximizing over the nuisance parameter for each target value.
Check the whole parameter space. An interior score solution may not be the global maximum, especially with boundaries or parameter constraints.

A useful verification is to substitute the final estimates back into the model. Check that each parameter is feasible, that the nuisance value attains the conditional maximum at the selected target value, and that the profile is no larger at other allowed target values. When the model has a clear fitted-value interpretation, compare fitted quantities with the observations as an additional arithmetic check.

Key takeaway: Profile by maximizing over the nuisance parameter at each fixed target value, then maximize the resulting profile. A profile maximizer corresponds to a joint MLE only if the conditional supremum there is attained by a feasible nuisance value.

Check Your Understanding

For each problem, identify the conditional nuisance estimate before using the profile. Answers are provided for checking.

Independent Poisson counts have means $\theta\lambda$ and $2\lambda$, with observed counts $6$ and $8$, where $\theta,\lambda>0$. Find the profile MLEs and fitted means. (Answer: $\widehat{\theta}=1.5,\ \widehat{\lambda}=14/3$; fitted means $7$ and $28/3$.)
Independent observations $0,2,4$ come from $N(\theta,\sigma^2)$, with $\theta\in\mathbb{R}$ and $\sigma^2>0$. Profile out the variance and find the joint MLE. (Answer: $\widehat{\theta}=2,\ \widehat{\sigma^2}=8/3$.)
Independent Poisson counts have means $\theta\lambda$ and $4\lambda$, with observed counts $3$ and $12$, where both parameters are positive. Find the profile maximizer, nuisance estimate, and fitted means. (Answer: $\widehat{\theta}=1,\ \widehat{\lambda}=3$; fitted means $3$ and $12$.)
Independent readings $X_1,X_2,X_3\sim N(\theta+\eta,1)$ have sample mean $2$, while two independent readings $Y_1,Y_2\sim N(\eta,1)$ have sample mean $0$. For fixed $\theta=1$, find the conditional estimate of $\eta$, then find the joint MLE. (Answer: $\widehat{\eta}(1)=3/5;\ \widehat{\theta}=2,\ \widehat{\eta}=0$.)
A single Bernoulli observation is a success with probability $\eta/(1+\eta)$, for $\eta>0$, and an unrestricted parameter $\theta$ does not affect the model. What is the profile likelihood, and does a joint MLE exist? (Answer: $L_p(\theta)=1$ for every $\theta$; no joint MLE exists because the supremum over $\eta>0$ is not attained.)

What You'll Learn