Deriving Conditional Laws in Exponential Families

Why Derive a Conditional Law?

In a parametric model, a statistic may carry the dependence on a nuisance parameter without carrying all the information about a parameter of interest. Conditioning on that statistic can remove the nuisance parameter from the distribution of the remaining data. The key is not merely that the model is an exponential family: the nuisance parameter must enter through a function that is fixed once the conditioning statistic is fixed, and the conditional support must not reintroduce it.

The derivation has two parts. First, write the joint law so that its parameter-dependent factors are visible. Then restrict attention to data sets with the same value of the nuisance statistic and normalize the remaining weights. The normalizing constant is usually parameter-dependent; it is essential to the conditional probability law, even when the nuisance parameter has disappeared.

Exponential-Family Form and Conditioning

Definition: A dominated family has the required canonical form if, relative to a common measure $\mu$, its density is $$ f_{\lambda,\psi}(x) = h(x)\exp\left\{\lambda^{\mathsf T}S(x)+\psi^{\mathsf T}R(x)-A(\lambda,\psi)\right\}. $$ Here $h$ is a parameter-independent nonnegative function, $S$ and $R$ are statistics, $\lambda$ is a nuisance parameter, and $\psi$ is the parameter of interest. The function $A$ makes the density integrate to one. Conditioning on $S(X)$ can eliminate $\lambda$ when the support is common and $\lambda$ enters the data-dependent part of the exponent only through $\lambda^{\mathsf T}S(x)$.

This form permits $A(\lambda,\psi)$ to depend on both parameters. On the set $S(x)=s$, the factors $\exp\{\lambda^{\mathsf T}s-A(\lambda,\psi)\}$ are constant in $x$. They therefore cancel when probabilities within that set are normalized. By contrast, the factor involving $R(x)$ need not be constant and generally remains in the conditional law.

For discrete data, write $\mathcal X_s=\{x:S(x)=s\}$. Provided the sum of weights is positive and finite, the conditional probability is

P_{\lambda,\psi}(X=x\mid S(X)=s) = \frac{h(x)\exp\{\psi^{\mathsf T}R(x)\}} {\displaystyle\sum_{u\in\mathcal X_s}h(u)\exp\{\psi^{\mathsf T}R(u)\}}, \qquad x\in\mathcal X_s.

Thus the conditional normalizer depends on $s$ and $\psi$, but not on $\lambda$. In continuous models, the event $S(X)=s$ may have probability zero. The same cancellation is made using a regular conditional distribution rather than by dividing probabilities of that event.

Conditional-Law Result and Proof

Theorem: Suppose the family has the displayed exponential-family form on a common support. Let $H(dx)=h(x)\mu(dx)$, and assume $H$ admits a disintegration over $S$: $H(dx)=\int K_s(dx)\nu(ds)$, where $\nu$ is the distribution measure of $S$ under $H$ and $K_s$ is supported on $\{x:S(x)=s\}$. For each parameter value under consideration, assume $$ 0<M_\psi(s):=\int \exp\{\psi^{\mathsf T}R(x)\}\,K_s(dx)<\infty $$ for $\nu$-almost every relevant $s$. Then a version of the conditional law is $$ P_{\lambda,\psi}(X\in B\mid S=s) = \frac{\displaystyle\int_B \exp\{\psi^{\mathsf T}R(x)\}\,K_s(dx)} {\displaystyle M_\psi(s)}. $$ In particular, this conditional law does not depend on $\lambda$.

Proof. Under the model, integration against the density is integration against $H$ weighted by $\exp\{\lambda^{\mathsf T}S(x)+\psi^{\mathsf T}R(x)-A(\lambda,\psi)\}$. For any measurable set $B$, disintegrating $H$ and using $S(x)=s$ on the support of $K_s$ gives

P_{\lambda,\psi}(X\in B,\ S\in C) = \int_C e^{\lambda^{\mathsf T}s-A(\lambda,\psi)} \left[\int_B e^{\psi^{\mathsf T}R(x)}K_s(dx)\right]\nu(ds).

Taking $B$ to be the whole sample space gives the marginal law of $S$:

P_{\lambda,\psi}(S\in C) = \int_C e^{\lambda^{\mathsf T}s-A(\lambda,\psi)}M_\psi(s)\,\nu(ds).

For $\nu$-almost every $s$, the proposed conditional probability is nonnegative and equals one when $B$ is the whole space, because its numerator then equals $M_\psi(s)$. Multiplying that conditional probability by the marginal density of $S$ relative to $\nu$ gives the joint expression above. This verifies the defining integral identity for a regular conditional law. The factors involving $\lambda$ have canceled, proving the claim. $\square$

The hypotheses matter. The theorem is an almost-everywhere statement for $s$ under the relevant marginal law; it does not define a conditional law at every mathematically possible value of $s$. For discrete models the corresponding requirement is that the conditioning event have positive probability and that the conditional normalizer be finite.

Worked Examples

Worked Example: Conditioning Two Poisson Counts on Their Total

Let $X_1$ and $X_2$ be independent, with $X_1\sim\operatorname{Poisson}(\lambda)$ and $X_2\sim\operatorname{Poisson}(\lambda\theta)$, where $\lambda>0$ is nuisance and $\theta>0$ is of interest. For nonnegative integers $x_1,x_2$, their joint mass is

P(X_1=x_1,X_2=x_2) =e^{-\lambda(1+\theta)} \frac{\lambda^{x_1+x_2}\theta^{x_2}}{x_1!x_2!}.

Set $S=X_1+X_2$, and condition on $S=n$. The possible data are $(n-k,k)$, for $k=0,\ldots,n$. Their weights, after omitting factors constant in $k$, are $\theta^k/((n-k)!k!)$. The sum of these weights is $(1+\theta)^n/n!$, since

\sum_{k=0}^n\frac{\theta^k}{(n-k)!k!} =\frac{1}{n!}\sum_{k=0}^n {n\choose k}\theta^k =\frac{(1+\theta)^n}{n!}.

Therefore $X_2\mid S=n\sim\operatorname{Binomial}(n,\theta/(1+\theta))$. The nuisance rate $\lambda$ has disappeared, while the rate ratio parameter $\theta$ remains. For example, if $n=4$ and $\theta=2$, then $P(X_2=3\mid S=4)={4\choose3}(2/3)^3(1/3)=32/81$.

Worked Example: A Conditional Binomial Law With a Nuisance Intercept

Let $Y_1\sim\operatorname{Binomial}(2,p_1)$ and $Y_2\sim\operatorname{Binomial}(1,p_2)$ independently, with $\operatorname{logit}(p_i)=\alpha+\beta z_i$, where $z_1=0,z_2=1$. Here $\alpha$ is a nuisance intercept and $\beta$ is the parameter of interest. The joint mass, up to its parameter-independent binomial-coefficient factor, is

\exp\left\{\alpha(y_1+y_2)+\beta y_2 -2\log(1+e^\alpha)-\log(1+e^{\alpha+\beta})\right\}.

Condition on $S=Y_1+Y_2=1$. The possible outcomes are $(1,0)$ and $(0,1)$. Their full weights, including the binomial coefficients, are $2$ and $e^\beta$ times a common factor. Normalizing gives

P((Y_1,Y_2)=(1,0)\mid S=1)=\frac{2}{2+e^\beta}, \qquad P((Y_1,Y_2)=(0,1)\mid S=1)=\frac{e^\beta}{2+e^\beta}.

For $\beta=\log 2$, these probabilities are $1/2$ and $1/2$, respectively. The intercept disappears because both outcomes have the same total $S$; the effect remains because $y_2$ differs between them. The coefficient 2 in the first weight is required by the binomial sampling model.

Worked Example: Conditioning Normal Data on Their Sum

Let $X_1,X_2$ be independent $N(\mu,\sigma^2)$ observations, with nuisance mean $\mu$ and interest parameter $\sigma^2>0$. Their joint density is proportional to

\exp\left\{\frac{\mu}{\sigma^2}(x_1+x_2) -\frac{x_1^2+x_2^2}{2\sigma^2} -\frac{\mu^2}{\sigma^2}\right\}.

Set $S=X_1+X_2$ and $D=X_1-X_2$. Given $S=s$, write $x_1=(s+d)/2$ and $x_2=(s-d)/2$. Then $x_1^2+x_2^2=(s^2+d^2)/2$. In the conditional density, the terms involving $s$, including the term involving $\mu s$, are constant in $d$. The remaining kernel is

\exp\left\{-\frac{d^2}{4\sigma^2}\right\}.

Its integral over the real line is $\sqrt{4\pi\sigma^2}$. Thus $D\mid S=s\sim N(0,2\sigma^2)$, for every $s$, independently of $\mu$. The conditional law retains the variance parameter; conditioning has not removed all parameter dependence.

When Does Nuisance Elimination Work?

The derivation identifies the conditions directly. A nuisance parameter is eliminated if its data-dependent contribution is constant on each conditioning set and the conditional support and base weights do not otherwise depend on it. In the canonical form, $\lambda^{\mathsf T}S(x)$ is constant when $S(x)=s$, and the global factor $\exp\{-A(\lambda,\psi)\}$ cancels in normalization. The residual conditional weights contain $h(x)\exp\{\psi^{\mathsf T}R(x)\}$, so the parameter of interest can remain.

The conditional normalizer is not optional. In the Poisson example it is what turns the weights into a binomial distribution; in the grouped-binomial example it is $2+e^\beta$. Dropping it may still leave a kernel useful for identifying parameter dependence, but it does not give a probability law.

A parameter-dependent support is a warning sign. If the possible values of $X$ given $S=s$ vary with the nuisance parameter, then even a canceled exponential factor cannot make the conditional law nuisance-free. Likewise, if nuisance dependence occurs through a statistic not fixed by conditioning, it generally remains in the conditional weights. Parameter restrictions can also matter: the formula removes explicit nuisance dependence for fixed allowable parameter values, but it does not make distinct parameter values equivalent or guarantee that every conditional sample point is attainable.

Common Mistake

Common mistake: “The nuisance parameter appears only in a factor that is common to the observed data, so I can discard it and use the remaining expression as the conditional distribution.” A common factor can be discarded when comparing conditional weights, but the remaining weights must still be divided by their sum or integral over the conditioning set. Also check that the support and base measure are common; otherwise nuisance dependence may remain through which outcomes are possible.

The distinction is between a conditional likelihood kernel and a conditional probability law. The kernel displays which parameter terms remain; the conditional normalizer makes the result sum or integrate to one. Both are needed for a complete derivation.

Check Your Understanding

For each question, identify which factors are constant on the conditioning set before deciding what remains.

In a canonical family with exponent $\lambda S(x)+\psi R(x)-A(\lambda,\psi)$, what terms cancel when conditioning on $S=s$, and what is the conditional normalizer?
For independent Poisson counts with means $\lambda a$ and $\lambda b$, identify the total-count statistic and state which parameter combination appears in the conditional law.
In the grouped-binomial example, why does the outcome $(1,0)$ have conditional weight 2 rather than 1?
Why is it not generally valid to define a continuous conditional law by dividing by $P(S=s)$?
Give one support-related reason why conditioning might fail to eliminate a nuisance parameter, even if an exponential factor cancels.