Equivalent Canonical Representations Under Rescaling

Why a Canonical Representation Can Change Without Changing the Model

An exponential-family distribution can have more than one canonical representation. The statistic in the exponent is not uniquely determined: multiplying it by a nonzero constant and adjusting the natural parameter accordingly leaves the distribution intact. The log-partition function must change as well, because it normalizes the representation in the new parameterization.

This is an algebraic change of coordinates, not a change to the probability model. It is useful when comparing formulas that appear different, or when choosing a scale for the statistic that is convenient for a particular application. The essential requirement is to transform all parts of the representation consistently.

Definition: Suppose a distribution has density or mass function, with respect to a fixed measure \(\nu\), of the form \(f_\eta(y)=h(y)\exp\{\eta T(y)-A(\eta)\}\). Here \(h\) is a nonnegative carrier function, \(T\) is a measurable statistic, \(\eta\) is the natural parameter, and \(A\) is the log-partition function. Its natural-parameter space is \(H=\{\eta\in\mathbb R:\int h(y)\exp(\eta T(y))\,d\nu(y)<\infty\}\), and \(A(\eta)=\log\int h(y)\exp(\eta T(y))\,d\nu(y)\) for \(\eta\in H\).

The fixed measure and carrier make explicit what is being held constant in this tutorial. For a discrete distribution, the integral is a sum; for a continuous distribution, it is an integral against the chosen dominating measure. The argument below applies when these objects do not change with the parameter.

The Rescaling Rule

Let \(c\) be any nonzero real number and define a new statistic \(T_c(y)=cT(y)\). To preserve the coefficient of the original statistic in the exponent, set \(\eta_c=\eta/c\). The new log-partition function is determined by the same normalization integral, expressed in terms of \(\eta_c\).

Theorem: If \(f_\eta(y)=h(y)\exp\{\eta T(y)-A(\eta)\}\), then the same family can be written using \(T_c=cT\) as \(f_{\eta_c}(y)=h(y)\exp\{\eta_cT_c(y)-A_c(\eta_c)\}\), where \(H_c=\{\eta_c:c\eta_c\in H\}\) and \(A_c(\eta_c)=A(c\eta_c)\). The corresponding natural parameters are related by \(\eta_c=\eta/c\). This holds for every \(\eta\in H\), provided \(c\ne0\).

Proof. For \(\eta_c\in H_c\), the definition of \(A_c\) gives

\begin{aligned} \int h(y)\exp\{\eta_cT_c(y)\}\,d\nu(y) &=\int h(y)\exp\{c\eta_cT(y)\}\,d\nu(y)\\ &=\exp\{A(c\eta_c)\}. \end{aligned}

Taking logarithms shows that the log-partition function for \(T_c\) is \(A_c(\eta_c)=A(c\eta_c)\), and that the transformed natural-parameter space is exactly \(H_c\). Now take any \(\eta\in H\) and set \(\eta_c=\eta/c\). Then

\eta_cT_c(y)-A_c(\eta_c) =\frac{\eta}{c}cT(y)-A\left(c\frac{\eta}{c}\right) =\eta T(y)-A(\eta).

The carrier is unchanged, so the two formulas give the same density or mass at every \(y\). Since \(c\ne0\), the mapping \(\eta\mapsto\eta/c\) is one-to-one and onto \(H_c\). Thus the two canonical representations describe the same family. \(\square\)

If \(c\) is negative, the rule is unchanged: the natural parameter changes sign as needed, and \(H_c\) is still defined by the condition \(c\eta_c\in H\). The transformed space need not have the same numerical endpoints or orientation as the original space.

Worked Example: Rescaling a Bernoulli Statistic

Let \(Y\sim\operatorname{Bernoulli}(p)\), with \(0<p<1\). The canonical representation uses \(T(y)=y\), \(\eta=\log\{p/(1-p)\}\), \(h(y)=1\) for \(y\in\{0,1\}\), and \(A(\eta)=\log(1+\exp(\eta))\). Rescale the statistic to \(T_3(y)=3y\).

The transformed natural parameter is \(\eta_3=\eta/3\), so \(p=\exp(3\eta_3)/(1+\exp(3\eta_3))\). The transformed log-partition function is \(A_3(\eta_3)=A(3\eta_3)=\log(1+\exp(3\eta_3))\).

For example, take \(p=0.2\). Then \(\eta=\log(0.2/0.8)=\log(0.25)\), \(\eta_3=\log(0.25)/3\), and \(A_3(\eta_3)=\log(1.25)\). At \(y=1\), the rescaled probability is

\exp\{3\eta_3-A_3(\eta_3)\} =\exp\{\log(0.25)-\log(1.25)\} =\frac{0.25}{1.25}=0.2.

At \(y=0\), it is \(\exp\{-\log(1.25)\}=1/1.25=0.8\). Both probabilities agree with the original Bernoulli distribution; the statistic changed scale, but the distribution did not.

Affine Changes and the Normalizing Term

A shift can be included as well as a scale change. Let \(T_{c,d}(y)=cT(y)+d\), where \(c\ne0\) and \(d\) is a fixed real constant. Setting \(\eta_{c,d}=\eta/c\) gives

\eta T(y)=\eta_{c,d}T_{c,d}(y)-d\eta_{c,d}.

The shift contributes a term involving the natural parameter alone. It must therefore be included in the log-partition function; it cannot be discarded as though it were a carrier term independent of the parameter.

Result: Under the affine transformation \(T_{c,d}=cT+d\), the equivalent log-partition function and natural-parameter space are \(A_{c,d}(\eta_{c,d})=A(c\eta_{c,d})+d\eta_{c,d}\) and \(H_{c,d}=\{\eta_{c,d}:c\eta_{c,d}\in H\}\), respectively.

To verify the normalization, integrate the exponential term using the new statistic:

\begin{aligned} \int h(y)\exp\{\eta_{c,d}T_{c,d}(y)\}\,d\nu(y) &=\exp(d\eta_{c,d})\int h(y)\exp\{c\eta_{c,d}T(y)\}\,d\nu(y)\\ &=\exp\{d\eta_{c,d}+A(c\eta_{c,d})\}. \end{aligned}

Taking the logarithm gives the stated \(A_{c,d}\). In the density itself, subtracting this log-partition function cancels the added shift term:

\eta_{c,d}T_{c,d}(y)-A_{c,d}(\eta_{c,d}) =c\eta_{c,d}T(y)-A(c\eta_{c,d}).

This establishes the equivalent representation and shows why an affine shift changes the formula for the log-partition function even though it does not change the distribution.

Worked Example: Rescaling a Poisson Count

Suppose \(Y\sim\operatorname{Poisson}(\lambda)\), where \(\lambda>0\). Its mass function is \(f(y)=\exp\{y\log\lambda-\lambda-\log(y!)\}\) for \(y=0,1,2,\ldots\). Thus \(T(y)=y\), \(\eta=\log\lambda\), \(A(\eta)=\exp(\eta)\), and \(h(y)=1/y!\).

Choose \(T_2(y)=2y\). The transformed parameter is \(\eta_2=\eta/2\), and \(A_2(\eta_2)=A(2\eta_2)=\exp(2\eta_2)\). To check the formula numerically, take \(\lambda=4\), so \(\eta_2=\log(4)/2=\log(2)\) and \(A_2(\eta_2)=4\). For \(y=2\), the transformed mass is

\frac{1}{2!}\exp\{2\eta_2(2)-4\} =\frac12\exp\{4\log(2)-4\} =\frac12\exp\{\log(16)-4\} =8e^{-4}.

The original Poisson formula gives \(\exp\{2\log(4)-4-\log(2!)\}=16e^{-4}/2=8e^{-4}\), the same value. The equality holds for every nonnegative integer \(y\), since \(2\eta_2y=\eta y\) and the log-partition term is still \(\lambda=4\).

Worked Example: Shifting and Rescaling an Exponential Statistic

For an exponential distribution with rate \(\lambda>0\), the density is \(f(y)=\lambda e^{-\lambda y}\) for \(y>0\). Its canonical form has \(h(y)=1\) on \(y>0\), \(T(y)=y\), \(\eta=-\lambda<0\), and \(A(\eta)=-\log(-\eta)\). Now use \(T_{2,1}(y)=2y+1\).

The transformed parameter is \(\eta_{2,1}=\eta/2=-\lambda/2\), whose space is \(\eta_{2,1}<0\). The affine rule gives \(A_{2,1}(\eta_{2,1})=-\log(-2\eta_{2,1})+\eta_{2,1}\). Take \(\lambda=2\). Then \(\eta_{2,1}=-1\) and \(A_{2,1}(-1)=-\log(2)-1\). At \(y=0.5\), \(T_{2,1}(y)=2\), so the transformed density is

\exp\{-2-(-\log(2)-1)\} =\exp\{\log(2)-1\} =2e^{-1}.

The original density at \(y=0.5\) is \(\lambda e^{-\lambda y}=2e^{-2(0.5)}=2e^{-1}\). The added \(+1\) in the statistic is accounted for by the \(+\eta_{2,1}\) term in the log-partition function.

What Is Preserved, and Why the Distinction Matters

The rescaling rule changes the coordinates used to write the family, not the family’s probabilities. More formally, if \(\eta=c\eta_c\), the density at every observation is identical in the two parameterizations. For independent observations \(y_1,\ldots,y_n\), the log-likelihood is also unchanged after this parameter mapping. Indeed, the original and transformed expressions are

\begin{aligned} \ell(\eta) &=\sum_{i=1}^n\log h(y_i)+\eta\sum_{i=1}^nT(y_i)-nA(\eta),\\ \ell_c(\eta_c) &=\sum_{i=1}^n\log h(y_i)+\eta_c\sum_{i=1}^n cT(y_i)-nA(c\eta_c) =\ell(c\eta_c). \end{aligned}

Thus, if a likelihood maximizer exists in one parameterization, its corresponding value in the other parameterization gives the same maximized likelihood. This is a consequence of the one-to-one parameter mapping, not evidence that the two numerical parameter values are equal.

The formulas matter in practice because different presentations may use different scales for the same statistic. Comparing only the coefficient in the exponent can make equivalent representations look unrelated. To check equivalence, compare the product of the natural parameter and statistic, then verify that the log-partition function and parameter space have been transformed consistently.

Common Mistakes

Scaling the statistic but not the parameter. Replacing \(T\) by \(cT\) while keeping \(\eta\) fixed changes the exponent in general. To preserve it, use \(\eta_c=\eta/c\).
Keeping the old log-partition function unchanged. For pure rescaling, the correct function is \(A_c(\eta_c)=A(c\eta_c)\), not \(A(\eta_c)\).
Ignoring the shift contribution. For \(T_{c,d}=cT+d\), the log-partition function includes \(d\eta_{c,d}\). Without this term the density will generally not equal the original one.
Assuming the parameter space stays numerically the same. The transformed space is \(H_c=\{\eta_c:c\eta_c\in H\}\). When \(c\) is negative, its orientation reverses.
Calling two representations different distributions just because their formulas look different. Substitute the parameter mapping and compare the complete exponent, including the log-partition term.

Key takeaway: Under \(T_c=cT\), use \(\eta_c=\eta/c\) and \(A_c(\eta_c)=A(c\eta_c)\). Under \(T_{c,d}=cT+d\), add \(d\eta_{c,d}\) to the transformed log-partition function. These adjustments preserve the distribution.

Check Your Understanding

Use the transformation rules to answer the following questions.

If \(T_c=5T\), what are the transformed natural parameter and log-partition function in terms of the original ones?
If the original natural-parameter space is \(H=(-\infty,0)\) and \(T_c=-2T\), determine the transformed natural-parameter space.
For \(T_{c,d}=3T+4\), write the transformed log-partition function and explain the role of its final term.
In a Poisson representation, why does replacing \(T(y)=y\) by \(T_2(y)=2y\) not change the distribution?
A proposed rescaling changes the statistic but leaves both the natural parameter and log-partition function unchanged. What should be checked to decide whether the formula is still equivalent?