Conditioning Away Nuisance Parameters

Why Condition on a Statistic?

In sparse-data models, likelihoods can be irregular or estimates can be strongly affected by nuisance parameters. Conditional inference offers a different route: rather than use the distribution of the entire observed data, it uses the distribution given a statistic that captures the nuisance parameter. When that statistic is sufficient for the nuisance parameter, the conditional distribution can retain information about the parameter of interest while no longer depending on the nuisance parameter.

The distinction is important. Conditioning does not change the original model or make a nuisance parameter disappear from its unconditional likelihood. It changes the probability distribution used for inference by restricting attention to outcomes that share the observed value of a statistic.

Definition: Let \(Y\) be data with distribution depending on a parameter of interest \(\psi\) and a nuisance parameter \(\lambda\). A statistic \(T(Y)\) is sufficient for \(\lambda\) conditional on \(\psi\) if, for every value \(t\) with positive probability, the conditional distribution of \(Y\) given \(T(Y)=t\) may depend on \(\psi\) but does not depend on \(\lambda\). Conditional inference uses this conditional distribution, evaluated at the observed value of \(T(Y)\).

The phrase “conditional on \(\psi\)” matters: the distribution of \(T(Y)\) itself may depend on both parameters. The requirement is that, once \(\psi\) is held fixed, conditioning on \(T(Y)\) removes \(\lambda\) from the distribution of the remaining data. The set of possible outcomes with \(T(Y)=t\) is called the conditional sample space or fiber at \(t\).

A Factorization Criterion

For finite sample spaces, a simple likelihood factorization makes the cancellation precise. Assume the possible data values form a fixed finite set, and that the probability mass function is positive on that set for the parameter values under consideration.

Theorem: Suppose that for every data value \(y\), the mass function can be written as \(f_{\psi,\lambda}(y)=a_{\psi,\lambda}(T(y))b_\psi(y)\), where \(a_{\psi,\lambda}\) depends on \(y\) only through \(T(y)\), and \(b_\psi(y)\) does not depend on \(\lambda\). Then the conditional distribution of \(Y\) given \(T(Y)=t\) does not depend on \(\lambda\).

Proof. For any \(y\) such that \(T(y)=t\), the conditional probability is the joint mass at \(y\) divided by the total mass of all outcomes in the fiber:

P_{\psi,\lambda}(Y=y\mid T(Y)=t) = \frac{f_{\psi,\lambda}(y)} {\displaystyle\sum_{z:T(z)=t} f_{\psi,\lambda}(z)}.

Substitute the factorization. For every \(z\) in the sum, \(T(z)=t\), so the common factor \(a_{\psi,\lambda}(t)\) cancels:

\frac{a_{\psi,\lambda}(t)b_\psi(y)} {\displaystyle\sum_{z:T(z)=t}a_{\psi,\lambda}(t)b_\psi(z)} = \frac{b_\psi(y)} {\displaystyle\sum_{z:T(z)=t}b_\psi(z)}.

The resulting probability can depend on \(\psi\), but it does not depend on \(\lambda\). The denominator is positive because \(t\) has positive probability. \(\square\)

This criterion is a useful way to verify nuisance-parameter elimination: identify a statistic \(T\), factor the mass function, and check that every factor involving \(\lambda\) depends on the data only through \(T\). In finite, positive-support models, the converse can also be expressed through likelihood ratios: within each fiber, the ratio \(f_{\psi,\lambda}(y)/f_{\psi,\lambda}(z)\) must be independent of \(\lambda\) for any \(y,z\) with \(T(y)=T(z)\). Indeed, conditional probabilities have these ratios; conversely, if these ratios are independent of \(\lambda\), choose a reference outcome in each fiber and factor out its mass. Fixed support and positivity ensure that these ratios and reference values are defined.

Worked Examples of Conditioning

Worked Example: A Sparse Logistic Model With an Unknown Intercept

Suppose \(Y_1,Y_2,Y_3\) are independent Bernoulli outcomes with \(\operatorname{logit}\{P(Y_i=1)\}=\alpha+\beta x_i\), where the covariates are \(x=(0,0,1)\). The intercept \(\alpha\) is a nuisance parameter, and \(\beta\) is the parameter of interest. Consider the observed vector \(y=(0,0,1)\).

Write \(T=\sum_{i=1}^3Y_i\) for the total number of successes and \(U=\sum_{i=1}^3x_iY_i\) for the number of successes at covariate value 1. Independence and the Bernoulli logistic model give the likelihood

L(\alpha,\beta;y) = \frac{\exp(\alpha T+\beta U)} {\prod_{i=1}^3\{1+\exp(\alpha+\beta x_i)\}}.

For the observed data, \(T=1\) and \(U=1\). Given \(T=1\), there are exactly three possible outcome vectors: \((1,0,0)\), \((0,1,0)\), and \((0,0,1)\). Their respective \(U\)-values are \(0,0,1\). In particular, two outcomes, \((1,0,0)\) and \((0,1,0)\), have \(U=0\); neither is the only outcome with that value.

The denominator of the likelihood is the same for all three vectors, and their likelihood weights are proportional to \(1,1,\exp(\beta)\). The factor involving \(\alpha\) is also common because \(T=1\). Therefore,

P_{\alpha,\beta}\{Y=(0,0,1)\mid T=1\} = \frac{\exp(\beta)}{2+\exp(\beta)}.

The conditional probability depends on \(\beta\), but not on \(\alpha\). At \(\beta=0\), it is \(1/3\), as all three possible locations for the single success have equal conditional probability. This illustrates how conditioning on the total number of successes can remove an unknown intercept while preserving information about the covariate effect.

Worked Example: Two Binomial Counts With a Shared Probability

Suppose \(X\sim\operatorname{Bin}(2,p)\) and \(Y\sim\operatorname{Bin}(1,p)\) independently, where the common success probability \(p\) is unknown. Let \(S=X+Y\), and suppose the observed total is \(S=1\). The nuisance parameter \(p\) affects the unconditional distribution of both counts.

The event \(S=1\) can occur in two ways. If \(X=1,Y=0\), its probability is \(\binom{2}{1}p(1-p)(1-p)=2p(1-p)^2\). If \(X=0,Y=1\), its probability is \((1-p)^2p\). Thus

P(X=1\mid S=1) = \frac{2p(1-p)^2} {2p(1-p)^2+p(1-p)^2} = \frac{2}{3}, \qquad 0<p<1.

The other conditional probability is \(P(X=0\mid S=1)=1/3\), so the full conditional distribution is free of \(p\). The calculation relies on the two counts having the same success probability. If they instead had unrelated probabilities, the relative probabilities of the two ways to obtain \(S=1\) would generally depend on both; conditioning on the total would not in itself remove those nuisance parameters.

Worked Example: Removing a Shared Poisson Rate

Let \(A\sim\operatorname{Poisson}(\lambda)\) and \(B\sim\operatorname{Poisson}(\lambda\theta)\) be independent, where \(\lambda>0\) is a nuisance rate and \(\theta>0\) is the parameter of interest. Set \(S=A+B\). For \(A=k\) and \(B=s-k\), the joint mass is

P(A=k,B=s-k) = \frac{e^{-\lambda}\lambda^k}{k!} \frac{e^{-\lambda\theta}(\lambda\theta)^{s-k}}{(s-k)!}.

For fixed \(s\), the terms involving \(\lambda\) are common across the possible values \(k=0,\ldots,s\). Dividing by the total probability of \(S=s\), or equivalently normalizing the weights over \(k\), gives

P(A=k\mid S=s) = \binom{s}{k}\frac{\theta^{s-k}}{(1+\theta)^s}, \qquad k=0,\ldots,s.

For example, if \(\theta=2\) and the observed total is \(s=3\), then

P(A=1\mid S=3) = \binom{3}{1}\frac{2^{3-1}}{(1+2)^3} = \frac{3\cdot4}{27} = \frac{4}{9}.

The shared rate \(\lambda\) has cancelled, while the relative rate \(\theta\) remains. The conditional distribution is useful precisely because it distinguishes the parameter of interest from the common scale nuisance parameter.

What Conditioning Achieves—and What It Does Not

The examples share a pattern: outcomes in the same fiber have nuisance-parameter factors in common. Conditioning normalizes their relative probabilities, so that common factor cancels. This can be particularly attractive in sparse samples, where relying on a large-sample approximation may be undesirable. The conditional distribution is a finite-sample distribution under the model, not an approximation based on a large-sample limit.

Nevertheless, conditioning has a cost. It discards variation in \(T(Y)\), including any information about \(\psi\) that the distribution of \(T(Y)\) might contain. It can also leave a very small conditional sample space. If the observed fiber contains only one possible outcome, then its conditional probability is 1 and the conditional distribution carries no information about \(\psi\). A nuisance-free distribution is not automatically an informative one.

Key takeaway: Conditioning removes a nuisance parameter only when the conditional distribution given the chosen statistic is genuinely free of that parameter. Check the factorization or calculate the conditional probabilities; do not assume that any convenient-looking total will suffice.

Common Pitfalls

Conditioning on a statistic that is not nuisance-sufficient. Fixing a total does not guarantee cancellation. Verify that nuisance-dependent factors are constant within each fiber.
Claiming that the nuisance parameter has vanished from the model. It remains in the unconditional model and may affect the distribution of the conditioning statistic. It is absent only from the specified conditional distribution.
Confusing the number of outcomes with the number of distinct statistic values. In the logistic example, when \(T=1\), three outcome vectors are possible, and two of them have \(U=0\). Count the vectors explicitly before normalizing their weights.
Assuming conditioning preserves all information. Conditioning restricts the sample space and can reduce information about the parameter of interest, or leave no conditional variation at all.
Ignoring support and positivity. A conditional distribution is defined only for conditioning values with positive probability. Factorization arguments using ratios also require that the relevant probabilities are positive and the support is fixed.

Check Your Understanding

Use the definitions and calculations above to answer the following questions.

In a model with parameter of interest \(\psi\) and nuisance parameter \(\lambda\), what must be true of the conditional distribution given \(T(Y)=t\) for \(T\) to be sufficient for \(\lambda\) conditional on \(\psi\)?
In the three-observation logistic example, list all outcome vectors with \(T=1\), and give the \(U\)-value for each.
Why does conditioning on \(S=X+Y\) remove \(p\) in the shared-probability binomial example? Which assumption about the two counts is essential?
In the Poisson example, which parameter remains in the conditional distribution of \(A\) given \(S=s\), and which parameter cancels?
Give one reason a nuisance-free conditional distribution might still be unhelpful for inference about the parameter of interest.