One-Parameter Exponential-Family Factorization

Why Factor a Probability Law?

A response model is often first written in terms of a parameter that has a direct interpretation, such as a rate or a shape. Exponential-family form rewrites the same law to expose how the parameter interacts with the observed response. The central task here is algebraic: isolate a function of the observation multiplied by a parameter-dependent coefficient, and collect the remaining parameter-only term.

The factorization is useful because it gives a common language for different probability models. It also requires care: the original parameter need not itself be the natural parameter, and a density whose support changes with the parameter may not admit the usual form.

Definition and Components

Definition: A one-parameter family has a (one-parameter) exponential-family representation if, with respect to a fixed dominating measure and on a common support, its density or mass function can be written as

f(y;\theta)=h(y)\exp\{\eta(\theta)T(y)-A(\eta(\theta))\}.

Here \(h(y)\) is the parameter-free base measure or carrier term, \(T(y)\) is the statistic, \(\eta(\theta)\) is the natural parameter, and \(A\) is the log-partition function. The symbol \(\theta\) denotes the original parameter; \(\eta(\theta)\) is a function of that parameter, and need not equal it.

For a discrete response, the dominating measure is usually counting measure and \(f\) is a mass function. For a continuous response, it is usually Lebesgue measure and \(f\) is a density. In either case, the factorization is an equality of the corresponding probability-law expressions on their support. Values outside a common support are assigned probability zero.

A practical route is to take the logarithm of a positive density or mass function. If the result can be arranged as a parameter-free term in \(y\), plus one product of a parameter-only coefficient and a fixed function of \(y\), plus a parameter-only term, exponentiating gives the desired structure. The fixed function of \(y\) is \(T(y)\); the coefficient is \(\eta(\theta)\).

Worked Example: A Three-Point Discrete Law

Let \(Y\) take values \(0,1,2\), and let \(\theta>0\). Suppose the probabilities are

P(Y=0)=\frac{2}{2+3\theta+\theta^2},\qquad P(Y=1)=\frac{3\theta}{2+3\theta+\theta^2},\qquad P(Y=2)=\frac{\theta^2}{2+3\theta+\theta^2}.

The probabilities sum to one because their numerators sum to the denominator. Set \(\eta=\log\theta\), so \(\theta=e^\eta\). For each possible response, write its numerator as a fixed weight times a power of \(\theta\):

f(y;\theta)= \frac{h(y)\theta^{T(y)}}{2+3\theta+\theta^2} =h(y)\exp\left\{\eta T(y)-\log(2+3e^\eta+e^{2\eta})\right\},

where \(h(0)=2,\ h(1)=3,\ h(2)=1\), and \(T(y)=y\). Thus \(\eta(\theta)=\log\theta\), and \(A(\eta)=\log(2+3e^\eta+e^{2\eta})\). The \(y\)-dependent weights belong in \(h\); the denominator, which depends only on the parameter, belongs in \(A\).

Worked Example: A Continuous Power Density

For \(\theta>-1\), consider the density \(f(y;\theta)=(\theta+1)y^\theta\) for \(0<y<1\), and zero elsewhere. First check that it is a density:

\int_0^1(\theta+1)y^\theta\,dy =(\theta+1)\left[\frac{y^{\theta+1}}{\theta+1}\right]_0^1=1.

Since \(y^\theta=\exp\{\theta\log y\}\) and \(\theta+1=\exp\{\log(\theta+1)\}\), this becomes

f(y;\theta)= \exp\{\theta\log y+\log(\theta+1)\} =1\cdot\exp\{\eta T(y)-A(\eta)\},

with \(\eta(\theta)=\theta\), \(T(y)=\log y\), \(h(y)=1\) on \((0,1)\), and \(A(\eta)=-\log(\eta+1)\) for \(\eta>-1\). The base measure here is understood on the common support \((0,1)\); the density is zero outside it. Notice that the statistic need not be the response itself: here it is the logarithm of the response.

Worked Example: A Density on a Bounded Interval

For real \(\theta\ne0\), define \(f(y;\theta)=\theta e^{\theta y}/(e^\theta-1)\) on \(0<y<1\). At \(\theta=0\), define \(f(y;0)=1\) on that interval. For nonzero \(\theta\), the density is positive, including when \(\theta<0\), because \(\theta\) and \(e^\theta-1\) have the same sign. Its integral is

\int_0^1\frac{\theta e^{\theta y}}{e^\theta-1}\,dy =\frac{\theta}{e^\theta-1}\left[\frac{e^{\theta y}}{\theta}\right]_0^1=1.

For \(\theta\ne0\), take \(\eta(\theta)=\theta\), \(T(y)=y\), and \(h(y)=1\) on \((0,1)\). Then

f(y;\theta) =\exp\left\{\eta y-\log\left(\frac{e^\eta-1}{\eta}\right)\right\}.

Thus \(A(\eta)=\log((e^\eta-1)/\eta)\) when \(\eta\ne0\). At \(\eta=0\), the ratio tends to one, so define \(A(0)=0\); the same factorization then gives the uniform density \(1\). This example shows that the log-partition function may need a separately defined value at a parameter where a displayed ratio has a removable singularity.

A Log-Ratio Criterion

The examples suggest a useful test. Comparing each law to a fixed reference parameter removes terms that do not change with the parameter. Under common-support and positivity conditions, the resulting log-density ratios must all use the same function of the response.

Theorem: Let \(\{f(y;\theta):\theta\in\Theta\}\) be densities or mass functions with respect to the same dominating measure, positive almost everywhere on a common support. Fix \(\theta_0\in\Theta\). Suppose there is one measurable function \(T(y)\) such that, for every \(\theta\),

\log f(y;\theta)-\log f(y;\theta_0) =b(\theta)T(y)+c(\theta)

almost everywhere, where \(b(\theta)\) and \(c(\theta)\) are scalars. If \(c(\theta)\) has the same value whenever \(b(\theta)\) has the same value, then the family has the factorization in the definition, with \(\eta(\theta)=b(\theta)\), \(h(y)=f(y;\theta_0)\), and \(A(b(\theta))=-c(\theta)\).

Proof. Exponentiate the assumed log-ratio identity to obtain \(f(y;\theta)=f(y;\theta_0)\exp\{b(\theta)T(y)+c(\theta)\}\). By the condition on \(c\), the value \(-c(\theta)\) can be defined as a function \(A\) of \(b(\theta)\); it is therefore well-defined even if distinct parameter values give the same \(b\). Substituting \(h(y)=f(y;\theta_0)\) and \(\eta(\theta)=b(\theta)\) yields \(f(y;\theta)=h(y)\exp\{\eta(\theta)T(y)-A(\eta(\theta))\}\). At \(\theta=\theta_0\), the log ratio is zero, so the representation can use \(b(\theta_0)=c(\theta_0)=0\). This proves the stated factorization. \(\square\)

The condition is also necessary for a factorization: subtracting the log factorization at \(\theta_0\) from the one at \(\theta\) gives a coefficient times \(T(y)\), plus a term that does not depend on \(y\). Thus the criterion is an exact way to check the one-statistic structure, not just a convenient algebraic trick.

Representation Choices and Support

The components are generally not unique. For example, suppose a representation uses \(T\) and \(\eta\), and choose a nonzero constant \(a\) and a constant \(b\). Define \(T^*(y)=aT(y)+b\) and \(\eta^*(\theta)=\eta(\theta)/a\). Then

\eta(\theta)T(y)-A(\eta(\theta)) =\eta^*(\theta)T^*(y) -\left[A(\eta(\theta))+b\eta^*(\theta)\right].

So the same law has another representation, with the adjusted log-partition term in brackets. The factorization identifies a useful structure, but it does not by itself make a particular scaling or offset for \(T\) mandatory.

A separate issue is support. The usual form uses a parameter-free base measure on a common support. If the set of possible observations changes with \(\theta\), its indicator generally depends on \(\theta\), so it cannot simply be placed inside \(h(y)\). For example, a uniform density on \((0,\theta)\) has a support indicator that changes with \(\theta\). One must not call that indicator a parameter-free carrier term. Some families can be represented in other ways or with broader conventions, but the common-support requirement in the definition must be checked rather than assumed.

Common Mistakes

Using the original parameter automatically as the natural parameter. In the three-point example, the natural parameter is \(\log\theta\), not \(\theta\). Identify the coefficient of \(T(y)\) after rewriting the law.
Putting every term involving the response into \(T\). The coefficient of \(T(y)\) must depend only on the parameter, while the base measure must not depend on it. Check the dependence of each factor separately.
Ignoring the support indicator. A factorization on a fixed support is not established if the possible values of \(y\) change with \(\theta\).
Calling the representation unique. Rescaling or shifting \(T\) changes the corresponding natural-parameter and log-partition terms while leaving the probability law unchanged.

The practical payoff is a disciplined way to inspect a response model: isolate the observation-dependent statistic, identify the parameter scale that multiplies it, and keep the carrier separate from the parameter-only term. These distinctions matter because later calculations use the components differently; a mistaken allocation can obscure the structure or produce an invalid factorization.

Check Your Understanding

For each question, focus on how the terms depend on the response and the parameter.

In the three-point example, what are \(h(y)\), \(T(y)\), and \(\eta(\theta)\), and why is the natural parameter not \(\theta\)?
For the power density, why is \(T(y)=\log y\) rather than \(y\)?
State the log-ratio condition that allows a family to be written using one fixed statistic.
Why must the log-ratio criterion check that \(c(\theta)\) is determined by \(b(\theta)\)?
What prevents a parameter-dependent support indicator from being included in a parameter-free base measure?