Affine Recodings of Canonical Statistics

Why Recode Canonical Statistics?

An exponential-family representation depends on a choice of coordinates for its canonical statistic. Those coordinates may be convenient for algebra, computation, or interpretation, but they are not themselves the probability model. If we replace a statistic by an invertible affine transformation, the natural parameter and normalizing function must change in a coordinated way. Done correctly, this recoding changes the representation but not the distribution.

The central bookkeeping issue is the translation in the affine map. A linear transformation changes the coordinates paired with the natural parameter. A translation adds a parameter-dependent constant to the exponent, so it must be included in the log-partition function. The carrier term, by contrast, remains unchanged when the sample space and reference measure are unchanged.

Definition and Transformation Rule

Definition: Let $X$ have density or mass function, with respect to a fixed measure $\nu$, $$ f_\eta(x)=h(x)\exp\{\eta^T T(x)-A(\eta)\}. $$ Here $T(x)\in\mathbb{R}^p$ is a measurable canonical statistic, $\eta\in\Theta\subseteq\mathbb{R}^p$ is the natural parameter, $h(x)\geq 0$ is a parameter-independent carrier, and $A(\eta)$ is finite on the parameter domain $\Theta$ and normalizes the density. For an invertible $p\times p$ matrix $B$ and a fixed vector $b\in\mathbb{R}^p$, the affine recoding is $U(x)=BT(x)+b$.

Write $\lambda=B^{-T}\eta$, where $B^{-T}=(B^{-1})^T$. Equivalently, $\eta=B^T\lambda$. Substituting $T(x)=B^{-1}(U(x)-b)$ into the exponent gives

\eta^T T(x) =(B^T\lambda)^T B^{-1}(U(x)-b) =\lambda^T U(x)-\lambda^T b.

Therefore the same density can be written using $U$ as the canonical statistic. The precise domain and normalizer are as follows.

Theorem: Suppose $f_\eta(x)=h(x)\exp\{\eta^TT(x)-A(\eta)\}$ is a normalized family with a common reference measure and parameter-independent carrier. For an invertible matrix $B$ and fixed $b$, define $U=BT+b$, $\lambda=B^{-T}\eta$, and $$ \Theta'=\{\lambda\in\mathbb{R}^p:B^T\lambda\in\Theta\}. $$ Then the same family has representation $$ f_\lambda(x)=h(x)\exp\{\lambda^TU(x)-A'(\lambda)\}, \qquad A'(\lambda)=A(B^T\lambda)+\lambda^Tb,\quad \lambda\in\Theta'. $$ The carrier and support are unchanged. The map between natural parameters is one-to-one and onto $\Theta'$.

To verify normalization, integrate the transformed expression and substitute the formula for $A'$:

\begin{aligned} \int h(x)\exp\{\lambda^TU(x)-A'(\lambda)\}\,d\nu(x) &=\int h(x)\exp\{\eta^TT(x)-\lambda^Tb-A'(\lambda)\}\,d\nu(x)\\ &=\exp\{-\lambda^Tb-A'(\lambda)\} \int h(x)\exp\{\eta^TT(x)\}\,d\nu(x)\\ &=\exp\{-\lambda^Tb-A'(\lambda)+A(\eta)\}\\ &=1. \end{aligned}

In the final equality, $\eta=B^T\lambda$ and $A'(\lambda)=A(\eta)+\lambda^Tb$. This also establishes the theorem’s claim that the recoded expression is the original density, not merely a proportional function. The parameter map is invertible because its inverse is $\eta=B^T\lambda$; consequently, the recoding neither removes nor adds distributions.

Worked Examples

Worked Example: Reversing the Bernoulli Statistic

Let $X\sim\operatorname{Bernoulli}(p)$, with $0<p<1$. Its canonical representation uses $T(X)=X$, natural parameter $\eta=\log\{p/(1-p)\}$, carrier $h(x)=1$ on $\{0,1\}$, and $A(\eta)=\log(1+e^\eta)$. Recode the statistic as $U=1-X$.

Here $B=-1$ and $b=1$, so $\lambda=-\eta$. The new log-partition function is

A'(\lambda)=A(-\lambda)+\lambda =\log(1+e^{-\lambda})+\lambda =\log(1+e^\lambda).

For $X=0$, $U=1$, and the recoded mass is $e^\lambda/(1+e^\lambda)$. For $X=1$, $U=0$, and the mass is $1/(1+e^\lambda)$. These sum to $1$. Since $\lambda=-\eta$, the recoding simply describes the complement of the original Bernoulli variable in the reversed canonical coordinate; it does not change the family of distributions.

Worked Example: An Affine Recoding of a Poisson Statistic

Let $X\sim\operatorname{Poisson}(\mu)$, where $\mu>0$. Its canonical form is $$ f_\eta(x)=\frac{1}{x!}\exp\{\eta x-e^\eta\}, \qquad x=0,1,2,\ldots,\quad \eta=\log\mu. $$ Thus $T(x)=x$, $h(x)=1/x!$, and $A(\eta)=e^\eta$. Use $U=3X-2$.

The affine map has $B=3$ and $b=-2$, so $\lambda=\eta/3$ and $\eta=3\lambda$. The recoded log-partition function is $$ A'(\lambda)=A(3\lambda)-2\lambda=e^{3\lambda}-2\lambda. $$ The natural-parameter domain is $\mathbb{R}$, since $3\lambda$ can be any real number. Substitution gives

\lambda(3x-2)-A'(\lambda) =3\lambda x-2\lambda-e^{3\lambda}+2\lambda =3\lambda x-e^{3\lambda}.

This is exactly the original exponent with $\eta=3\lambda$. In particular, the factor $1/x!$ remains the carrier. Although $U$ takes values $-2,1,4,\ldots$, the sample-space support for $X$ and the probabilities assigned to its points have not changed.

Worked Example: Recoding a Normal Location Statistic

Suppose $X\sim N(\mu,\sigma^2)$, with fixed $\sigma^2>0$ and unrestricted $\mu\in\mathbb{R}$. With respect to Lebesgue measure, take $T(x)=x$, $\eta=\mu/\sigma^2$, $$ h(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left\{-\frac{x^2}{2\sigma^2}\right\}, \qquad A(\eta)=\frac{\sigma^2\eta^2}{2}. $$ Recode as $U=-2X+4$.

Here $B=-2$, $b=4$, and $\lambda=-\eta/2$, so $\eta=-2\lambda$. The new log-partition function is $$ A'(\lambda)=A(-2\lambda)+4\lambda=2\sigma^2\lambda^2+4\lambda. $$ For every $x$, $$ \lambda(-2x+4)-A'(\lambda) =-2\lambda x+4\lambda-2\sigma^2\lambda^2-4\lambda =-2\lambda x-2\sigma^2\lambda^2. $$ The original exponent, after setting $\eta=-2\lambda$, is $$ \eta x-A(\eta)=-2\lambda x-\frac{\sigma^2(-2\lambda)^2}{2} =-2\lambda x-2\sigma^2\lambda^2. $$ The two forms agree exactly. The carrier $h(x)$ is unchanged, and $\lambda$ ranges over all of $\mathbb{R}$, just as $\eta$ does.

Means, Covariances, and the New Coordinates

The transformation also gives immediate relationships among moments of the canonical statistic. Where the log-partition functions are differentiable, the earlier log-partition results identify their gradients with means and their Hessians with covariance matrices. Applying the chain rule to $A'(\lambda)=A(B^T\lambda)+\lambda^Tb$ gives

\nabla A'(\lambda)=B\nabla A(B^T\lambda)+b, \qquad \nabla^2 A'(\lambda)=B\{\nabla^2 A(B^T\lambda)\}B^T.

These identities agree with the direct moment calculations. Since $U=BT+b$, linearity of expectation yields $E[U]=B E[T]+b$. For covariance, constants disappear, and bilinearity gives

\operatorname{Cov}(U) =\operatorname{Cov}(BT) =B\operatorname{Cov}(T)B^T.

Thus the translation changes the mean but not the covariance; the linear part transforms the covariance by the usual matrix rule. These statements require the relevant first and second moments to exist. For interior natural parameters where the log-partition derivatives exist, they follow equivalently from the gradient and Hessian identities.

Result: Under an invertible affine recoding $U=BT+b$, the mean parameter becomes $B E[T]+b$, and the covariance becomes $B\operatorname{Cov}(T)B^T$. The carrier remains $h$, while the log-partition function becomes $A(B^T\lambda)+\lambda^Tb$.

Common Mistake: Dropping the Offset from the Normalizer

A translation in the statistic contributes a constant in $x$, but that constant depends on the natural parameter. It therefore cannot be discarded as if it were a parameter-free carrier term. The Bernoulli recoding above makes the normalization failure visible.

For the recoded statistic $U=1-X$, the correct normalizer is $A'(\lambda)=\lambda+\log(1+e^{-\lambda})=\log(1+e^\lambda)$. Suppose instead one keeps only $A_{\mathrm{wrong}}(\lambda)=\log(1+e^{-\lambda})$, omitting the offset contribution $\lambda$. For $U\in\{0,1\}$, the resulting purported masses are

P_{\mathrm{wrong}}(U=0)=\frac{1}{1+e^{-\lambda}}, \qquad P_{\mathrm{wrong}}(U=1)=\frac{e^\lambda}{1+e^{-\lambda}}.

Their sum is $$ \frac{1+e^\lambda}{1+e^{-\lambda}}=e^\lambda, $$ which is not $1$ unless $\lambda=0$. At $\lambda=0$ the error happens to be hidden because the sum is $1$; that single value does not make the proposed normalizer correct over the family. The proper partition function for weights $1$ and $e^\lambda$ is $1+e^\lambda$, giving log-partition function $\log(1+e^\lambda)$.

A reliable check is to sum or integrate the transformed expression for a fixed parameter value. A second check is to substitute $T=B^{-1}(U-b)$ into the original exponent and collect every term that depends on the natural parameter. Both checks expose a missing offset.

What Changes, and What Does Not

Affine recoding is useful when a different coordinate system makes a model or a calculation simpler. It can center or rescale statistics, reverse their orientation, or combine them through an invertible linear transformation. But the calculation must respect the distinction between the data-space carrier and parameter-space normalization:

Natural parameter: changes from $\eta$ to $\lambda=B^{-T}\eta$.
Log-partition function: changes to $A'(\lambda)=A(B^T\lambda)+\lambda^Tb$.
Carrier and support: remain unchanged under this recoding with the same reference measure.
Probability distribution: remains unchanged at corresponding parameter values $\eta=B^T\lambda$.

The carrier could be written differently if one independently changed the reference measure or multiplied and divided by a fixed, parameter-independent function. That is a separate representational choice. It is not a consequence of the affine recoding itself. Keeping these operations distinct makes it easier to verify which parts of an exponential-family representation encode the sample space and which parts encode normalization.

Check Your Understanding

Use the transformation rule to answer each question, and check any proposed representation by verifying normalization.

For $U=BT+b$, solve for the old natural parameter $\eta$ in terms of $\lambda$, and state the transformed log-partition function.
Why does an affine translation alter the log-partition function even though it does not alter the carrier?
If $U=2T+5$ in a one-parameter family with log-partition function $A$, express the transformed parameter and log-partition function.
For $U=BT+b$, derive $E[U]$ and $\operatorname{Cov}(U)$ from $E[T]$ and $\operatorname{Cov}(T)$.
Explain why a proposed transformed expression that normalizes at one parameter value need not be a valid representation of the full family.