Conditional Likelihood for Eliminating Nuisance Parameters

Why Condition to Eliminate a Nuisance Parameter?

A likelihood for a parameter of interest can be difficult to interpret when it also depends on an unknown nuisance parameter. One way to remove that nuisance parameter is to condition on a statistic that captures the data’s dependence on it. In suitable discrete exponential families, the conditional distribution within the observed statistic’s fiber depends on the parameter of interest but not on the nuisance parameter.

The method is exact in the sense that it constructs a genuine conditional probability model, not an approximation to the original likelihood. Its cost is that conditioning can discard information about the parameter of interest as well as remove the nuisance parameter. The key is therefore to identify when the conditional distribution has the desired parameter-free property and then examine how much information remains.

Conditional Likelihood in a Discrete Exponential Family

Definition: Let \(X\) have a discrete sample space \(\mathcal X\), with parameter of interest \(\theta\) and nuisance parameter \(\lambda\). For an observed value \(x\), the conditional likelihood given \(S=s\) is \(L_c(\theta;x\mid S=s)=P_{\theta,\lambda}(X=x\mid S=s)\), where \(s=S(x)\), viewed as a function of \(\theta\). It is a nuisance-free conditional likelihood when this conditional probability does not depend on \(\lambda\).

A useful sufficient condition is a factorization of the joint probability mass function of the form \(p_{\theta,\lambda}(x)=b(x)\exp\{q_\theta(x)+\eta(\lambda)^\mathsf{T}S(x)-A(\theta,\lambda)\}\). Here \(b(x)\) is parameter-free, \(q_\theta(x)\) depends only on \(\theta\) and \(x\), \(S(x)\) is a statistic, and \(\eta(\lambda)\) is a vector of nuisance natural parameters. Assume a common discrete support and that the sums below are finite or converge. For a fixed value \(s\), the fiber is \(\mathcal X_s=\{y\in\mathcal X:S(y)=s\}\). We also assume the conditioning event has positive probability under the parameter values considered.

Theorem: Under the factorization above, for \(x\in\mathcal X_s\), \(P_{\theta,\lambda}(X=x\mid S=s)=\dfrac{b(x)e^{q_\theta(x)}}{\sum_{y\in\mathcal X_s}b(y)e^{q_\theta(y)}}\). Thus the conditional distribution, and its likelihood for \(\theta\), do not depend on \(\lambda\).

Proof. For any \(x\in\mathcal X_s\), the numerator of the conditional probability is \(b(x)e^{q_\theta(x)+\eta(\lambda)^\mathsf{T}s-A(\theta,\lambda)}\). The probability of the conditioning event is the sum of the same expression over \(y\in\mathcal X_s\): \(\sum_{y\in\mathcal X_s}b(y)e^{q_\theta(y)+\eta(\lambda)^\mathsf{T}s-A(\theta,\lambda)}\). The factors \(e^{\eta(\lambda)^\mathsf{T}s-A(\theta,\lambda)}\) are common to every term in this sum and to the numerator. They cancel, leaving the stated ratio. The remaining expression contains no \(\lambda\). \(\square\)

This is also a way to understand the role of sufficiency: at each fixed \(\theta\), the conditional distribution given \(S\) is free of \(\lambda\). In an exponential-family representation, the nuisance parameter contributes through the statistic being conditioned on. The condition is sufficient, not necessary: other factorizations may also yield a nuisance-free conditional distribution.

Worked Discrete Models

Worked Example: A Rate Ratio from Two Poisson Counts

Suppose \(X\sim\operatorname{Pois}(\lambda\theta)\) and \(Y\sim\operatorname{Pois}(\lambda)\) independently, where \(\theta>0\) is the rate ratio of interest and \(\lambda>0\) is a nuisance rate. Find the conditional likelihood for \(\theta\) given \(T=X+Y\), and evaluate its maximum for observed counts \(x=4,y=2\).

For nonnegative integers \(x,y\), the joint mass is \(P(X=x,Y=y)=e^{-\lambda(\theta+1)}\lambda^{x+y}\theta^x/(x!y!)\). On the fiber \(x+y=t\), the factor \(e^{-\lambda(\theta+1)}\lambda^t\) is constant across possible values of \(x\). Summing over that fiber gives \(\sum_{u=0}^t e^{-\lambda(\theta+1)}\lambda^t\theta^u/(u!(t-u)!)\).

Cancel the common factor and use \(\sum_{u=0}^t\theta^u/(u!(t-u)!)=(1+\theta)^t/t!\). Therefore \(P(X=x\mid T=t)=\binom{t}{x}\theta^x/(1+\theta)^t\), for \(x=0,\ldots,t\). This is a binomial distribution with size \(t\) and success probability \(\theta/(1+\theta)\).

The observed total is \(t=6\), so the conditional likelihood is \(L_c(\theta)=\binom{6}{4}\theta^4/(1+\theta)^6\). Its log-likelihood, up to a constant, is \(4\log\theta-6\log(1+\theta)\). Differentiating gives \(4/\theta-6/(1+\theta)\), which is zero when \(4(1+\theta)=6\theta\), or \(\hat\theta=2\). The second derivative is \(-4/\theta^2+6/(1+\theta)^2\), which at \(\theta=2\) equals \(-1+2/3=-1/3<0\). The conditional likelihood has a maximum there.

Worked Example: A Conditional Odds-Ratio Likelihood

Let \(X_1\sim\operatorname{Bin}(3,p_1)\) and \(X_2\sim\operatorname{Bin}(2,p_2)\) independently. Parameterize the odds by \(p_1/(1-p_1)=\lambda\theta\) and \(p_2/(1-p_2)=\lambda\), where \(\theta>0\) is the odds ratio and \(\lambda>0\) is a nuisance parameter. For observed counts \(X_1=2,X_2=1\), derive and maximize the likelihood conditional on \(T=X_1+X_2=3\).

For possible counts \(X_1=u,X_2=v\), the joint probability is \(\binom{3}{u}\binom{2}{v}(\lambda\theta)^u\lambda^v/ \{(1+\lambda\theta)^3(1+\lambda)^2\}\). When \(u+v=3\), the factors \(\lambda^3\), \((1+\lambda\theta)^{-3}\), and \((1+\lambda)^{-2}\) are common across the fiber. The remaining weights are \(\binom{3}{u}\binom{2}{3-u}\theta^u\). The possible values of \(u\) are 1, 2, and 3, with weights \(3\theta\), \(6\theta^2\), and \(\theta^3\), respectively.

The conditional probability of the observed \(u=2\) is therefore \(L_c(\theta)=6\theta^2/(3\theta+6\theta^2+\theta^3)\). The log-likelihood derivative is \(2/\theta-(3+12\theta+3\theta^2)/(3\theta+6\theta^2+\theta^3)\). Setting it to zero and multiplying by the positive denominator gives \(2(3\theta+6\theta^2+\theta^3)=3\theta+12\theta^2+3\theta^3\), hence \(\theta(3-\theta^2)=0\). Since \(\theta>0\), the only stationary point is \(\sqrt{3}\). The likelihood tends to 0 as \(\theta\) tends to 0 or infinity, so this stationary point is its maximum.

Worked Example: Conditioning a Multinomial Model

Each of \(n\) trials falls in one of three categories, with probabilities proportional to \(1,\lambda,\lambda\theta\), respectively. Thus the probabilities are \(1/(1+\lambda+\lambda\theta)\), \(\lambda/(1+\lambda+\lambda\theta)\), and \(\lambda\theta/(1+\lambda+\lambda\theta)\). The parameter \(\lambda>0\) controls the overall chance of falling in categories 1 or 2, while \(\theta>0\) compares category 2 with category 1. Suppose the counts are \((n_0,n_1,n_2)=(5,3,4)\).

Let \(M=n_1+n_2\). Given \(M=m\), the probability of \(n_2=k\) is obtained by dividing the multinomial probability by the probability of the observed total \(M=m\). Equivalently, among the \(m\) trials that fell in categories 1 or 2, each has conditional probability \(\{\lambda\theta/(1+\lambda+\lambda\theta)\}/\{\lambda/(1+\lambda+\lambda\theta)+\lambda\theta/(1+\lambda+\lambda\theta)\}=\theta/(1+\theta)\) of being in category 2. Consequently, \(P(n_2=k\mid M=m)=\binom{m}{k}\theta^k/(1+\theta)^m\).

Here \(m=3+4=7\) and \(k=4\), so \(L_c(\theta)=\binom{7}{4}\theta^4/(1+\theta)^7\). Its log-likelihood derivative is \(4/\theta-7/(1+\theta)\), which vanishes when \(4(1+\theta)=7\theta\), giving \(\hat\theta=4/3\). The nuisance parameter has disappeared because, conditional on entering one of the two nonbaseline categories, the relative probability of category 2 depends only on \(\theta\).

What the Conditional Score Tells Us

A conditional likelihood is an ordinary likelihood for the conditional distribution on the observed fiber. Its score therefore compares the observed contribution to the average contribution under that conditional distribution. The next result makes this precise in a finite-fiber model of the form in the theorem.

Result: Suppose the fiber \(\mathcal X_s\) is finite and \(q_\theta(x)\) is differentiable in a scalar \(\theta\). Define \(u_\theta(x)=\partial q_\theta(x)/\partial\theta\). Then \(\dfrac{\partial}{\partial\theta}\log P_\theta(X=x\mid S=s)=u_\theta(x)-E_\theta[u_\theta(X)\mid S=s]\). The conditional Fisher information, defined as the conditional expected square of this score, is \(\operatorname{Var}_\theta(u_\theta(X)\mid S=s)\).

Proof. Write \(Z_s(\theta)=\sum_{y\in\mathcal X_s}b(y)e^{q_\theta(y)}\). The conditional log-likelihood is \(\log b(x)+q_\theta(x)-\log Z_s(\theta)\). Because the fiber is finite, differentiation commutes with this finite sum, giving \(Z_s'(\theta)=\sum_{y\in\mathcal X_s}b(y)e^{q_\theta(y)}u_\theta(y)\). Dividing by \(Z_s(\theta)\) shows that \(Z_s'(\theta)/Z_s(\theta)=E_\theta[u_\theta(X)\mid S=s]\). Differentiating the conditional log-likelihood gives the stated score. Its conditional expectation is zero, since it is \(u_\theta(X)\) minus its conditional mean. Its conditional expected square is therefore the conditional variance of \(u_\theta(X)\), as claimed. \(\square\)

This result also identifies a limitation: if \(u_\theta(X)\) is constant over the observed fiber, the conditional score and conditional information are zero. In that case conditioning has left no information about \(\theta\) for this observation, even though the unconditional data may contain information.

When the Method Helps—and a Common Pitfall

Conditioning is especially useful when the nuisance parameter enters through a low-dimensional statistic and the conditional distribution retains a clear interpretation for the target. The Poisson example turns a rate comparison into a binomial problem; the binomial and multinomial examples yield likelihoods for odds ratios. These conditional likelihoods support direct estimation and, when desired, inference based on their exact discrete distributions.

Conditional likelihood is not the same as full likelihood or profile likelihood. The full likelihood uses the joint probability of the observed data and involves all parameters. A profile likelihood maximizes the full likelihood over nuisance parameter values for each fixed value of the target parameter. A conditional likelihood instead uses the probability of the observation given a statistic. The three likelihoods need not have the same shape or yield the same estimate.

Common mistake: “The statistic is sufficient for the nuisance parameter, so conditioning cannot lose information about the target.” Sufficiency for \(\lambda\) at fixed \(\theta\) ensures that conditioning removes \(\lambda\); it does not ensure that the conditional distribution retains all information about \(\theta\).
Correction: Check the conditional distribution on the observed fiber. If the fiber has only one possible data value, the conditional likelihood is constant in \(\theta\), and it provides no information about the target.
Common mistake: “Any sufficient statistic can be conditioned on to eliminate any nuisance parameter.” The cancellation requires the nuisance parameter’s data dependence to be absorbed by the conditioned statistic, as in the stated factorization. Sufficiency for some other parameter or parameterization does not establish that cancellation.
Correction: Write the joint probability mass function and identify which parameter-dependent factors are constant over the chosen fiber. Only then claim that the conditional law is nuisance-free.

Key takeaway: To construct a nuisance-free conditional likelihood, find a statistic whose fixed-value fiber makes the nuisance-dependent part of the joint mass common to all points in that fiber. Divide by the probability of the fiber, verify that the nuisance parameter cancels, and assess how much information about the target remains.

Check Your Understanding

For each question, identify the conditional structure or explain the relevant limitation.

In the independent Poisson model \(X\sim\operatorname{Pois}(\lambda\theta)\), \(Y\sim\operatorname{Pois}(\lambda)\), what is the conditional distribution of \(X\) given \(X+Y=t\), and which parameter has been eliminated?
In the binomial odds-ratio example, why are the possible values of \(X_1\), given \(X_1+X_2=3\), restricted to 1, 2, and 3?
For the three-category multinomial example, explain why conditioning on \(M=n_1+n_2\) removes \(\lambda\) but not \(\theta\).
What does the conditional score result imply if \(u_\theta(x)\) is constant over the observed fiber?
Why does a nuisance-free conditional distribution not imply that the conditional likelihood retains all information about the parameter of interest?