What the Natural Parameter and Cumulant Function Do
An exponential-family probability model separates the contribution of an observation from the contribution of its parameter. The natural parameter controls how strongly the statistic of an observation is weighted. The cumulant function supplies the normalizing factor that makes all probabilities sum to one, or makes a density integrate to one. The base measure specifies the reference scale and can also carry fixed weights or support restrictions.
The focus here is the one-parameter, unit-dispersion form. This is enough to see why the parameter and cumulant function must be considered together. A familiar generalized linear model form may include a separate dispersion factor; the relationship between those conventions is noted below.
The reference measure matters. For a discrete response, \(\nu\) is commonly counting measure, so normalization means summing over the support. For a continuous response, \(\nu\) is commonly Lebesgue measure, so normalization means integrating over the support. The factor \(h(y)\) is interpreted relative to that choice. It need not be a probability distribution on its own, and it may be zero outside the support.
In a common generalized linear model notation, the density or mass function is written \(h(y,\phi)\exp\{[y\theta-b(\theta)]/a(\phi)\}\). For fixed dispersion \(\phi\), \(\theta\) is often called the canonical or natural parameter and \(b(\theta)\) the cumulant function. Alternatively, the exponent can be written as \(y\eta-A(\eta)\), where \(\eta=\theta/a(\phi)\) and \(A(\eta)=b(\theta)/a(\phi)\), provided \(\phi\) is fixed and the reparameterization is valid. The naming convention depends on how the scale factor is separated; the normalizing requirement does not change.
Normalization Determines the Cumulant Function
For the unit-dispersion form, sum or integrate the unnormalized expression \(h(y)e^{\eta T(y)}\) over the support. The resulting quantity must be the reciprocal of the factor \(e^{-A(\eta)}\) in the probability model. Thus the cumulant function is not an arbitrary extra term: it is determined by the base measure, statistic, and value of the natural parameter.
When the integral is finite and positive, taking logarithms gives the normalization formula. In a discrete model, replace the integral with a sum. If the integral or sum is infinite for a proposed value of \(\eta\), no finite value of \(A(\eta)\) can normalize the model at that value.
Determine whether normalization requires a sum or an integral, and where the base measure is nonzero.
Write the exponent in the form \(\eta T(y)-A(\eta)\), identifying which part depends on the observation and which part is constant in \(y\).
Its logarithm gives \(A(\eta)\) wherever the sum or integral is finite and positive.
Include precisely the values of \(\eta\) for which normalization is possible; do not assume every real value is allowed.
This argument is also a short justification of the formula: integrating the proposed mass or density removes the dependence on the observation, leaving \(e^{-A(\eta)}\) times the normalizing sum or integral. Setting that product equal to one forces \(A(\eta)\) to be the logarithm of that sum or integral. The formula applies only where the sum or integral is finite and positive.
Worked Examples
Worked Example: A Finite-Support Model With Unequal Base Weights
A hypothetical discrete response \(Y\) takes values \(0,1,2\). Its base-measure weights are \(h(0)=1\), \(h(1)=2\), and \(h(2)=1\), with counting measure and \(T(y)=y\). Find the cumulant function and parameter space. Then, at \(\eta=\log 2\), calculate all three probabilities and check that they sum to one.
Method. Sum \(h(y)e^{\eta y}\) over the three supported values. Because the support is finite, this sum is finite for every real \(\eta\).
The unnormalized weights are \(1\), \(2e^\eta\), and \(e^{2\eta}\). Therefore the normalizing sum is \(1+2e^\eta+e^{2\eta}=(1+e^\eta)^2\), and \(A(\eta)=\log[(1+e^\eta)^2]=2\log(1+e^\eta)\). The parameter space is all real numbers.
At \(\eta=\log 2\), \(e^\eta=2\). The three unnormalized weights are \(1\), \(2(2)=4\), and \(2^2=4\). Their sum is \(1+4+4=9\), so \(A(\log 2)=\log 9\).
Dividing each weight by 9 gives \(P(Y=0)=1/9\), \(P(Y=1)=4/9\), and \(P(Y=2)=4/9\). The check is \(1/9+4/9+4/9=9/9=1\). A common mistake is to use weights \(1,e^\eta,e^{2\eta}\), omitting the base weight of 2 at \(y=1\); that would define a different model.
Worked Example: A Countable Support Restricts the Natural Parameter
Let \(Y\) take values \(0,1,2,\ldots\), with counting measure, \(h(y)=1\), and \(T(y)=y\). The proposed model is \(f_\eta(y)=\exp\{\eta y-A(\eta)\}\). Determine the parameter space and cumulant function. At \(\eta=\log(1/3)\), calculate \(A(\eta)\), \(P(Y=0)\), and \(P(Y=1)\).
Method. The normalizing sum is the geometric series \(\sum_{y=0}^{\infty}e^{\eta y}\). A geometric series with ratio \(r\) converges exactly when \(|r|<1\). Here \(r=e^\eta>0\), so convergence requires \(e^\eta<1\), or \(\eta<0\).
For \(\eta<0\), the sum is \(1/(1-e^\eta)\). Hence \(A(\eta)=\log[1/(1-e^\eta)]=-\log(1-e^\eta)\). At \(\eta=\log(1/3)\), \(e^\eta=1/3\), so the sum is \(1/(1-1/3)=1/(2/3)=3/2\), and \(A(\eta)=\log(3/2)\).
The probability at zero is \(e^{-A(\eta)}=(3/2)^{-1}=2/3\). The probability at one is \(e^{\eta-A(\eta)}=(1/3)(2/3)=2/9\). As a check, the first two probabilities sum to \(2/3+2/9=8/9\), which is less than one because the model also assigns positive probability to \(Y=2,3,\ldots\).
The restriction is essential: at \(\eta=0\), every term of the normalizing sum is 1, so the sum diverges. For \(\eta>0\), its terms grow and it also diverges. The common mistake is to use the finite-support conclusion from the preceding example and claim that every real \(\eta\) is allowed.
Worked Example: A Continuous Model and Its Base Measure
A hypothetical positive response \(Y\) has Lebesgue measure on \(y>0\), base factor \(h(y)=1\) on that support, and statistic \(T(y)=y\). For \(f_\eta(y)=\exp\{\eta y-A(\eta)\}\), find the parameter space and cumulant function. Then write the normalized density and calculate \(P(Y>1)\) at \(\eta=-2\).
Method. Integrate the unnormalized factor over the support: \(\int_0^\infty e^{\eta y}\,dy\). This integral is finite only if \(\eta<0\). For such \(\eta\), its antiderivative gives \(\lim_{b\to\infty}(e^{\eta b}-1)/\eta=-1/\eta\), since \(e^{\eta b}\to0\).
It follows that \(A(\eta)=\log(-1/\eta)=-\log(-\eta)\) for \(\eta<0\). At \(\eta=-2\), \(A(-2)=-\log 2\), so \(e^{-A(-2)}=2\). The normalized density is therefore \(f_{-2}(y)=2e^{-2y}\) for \(y>0\), and zero otherwise.
The requested probability is \(P(Y>1)=\int_1^\infty 2e^{-2y}\,dy\). An antiderivative is \(-e^{-2y}\), so the integral is \(\lim_{b\to\infty}[-e^{-2y}]_1^b=0-(-e^{-2})=e^{-2}\approx0.1353\). The density check is \(\int_0^\infty 2e^{-2y}\,dy=1\). The common mistake is to integrate over all real numbers even though the base measure is zero for \(y\leq0\); the support is part of the model.
Interpreting the Base Measure and Parameter Space
The base-measure factor is not merely a decorative constant. It can encode which outcomes are possible and how those outcomes are weighted before the natural parameter tilts them. In the finite-support example, the factor 2 at \(y=1\) changed the relative probability of that outcome. In the continuous example, the restriction to \(y>0\) determined the limits of integration and therefore the normalizing factor.
A distribution can sometimes be represented in more than one algebraically equivalent way by moving a parameter-independent factor between terms. For example, a factor depending only on \(y\) can be written as part of \(h(y)\), or its logarithm can appear in the exponent. Such rearrangements are valid only if the resulting expression and reference measure still define the same probabilities or density. The natural parameter is the coefficient of the chosen statistic in the chosen representation; changing the statistic or representation can change the displayed parameter.
Common Exam Traps
- Ignoring the support. Integrating over the whole real line when the density is defined only for positive values changes the normalizer and the model.
- Forgetting base weights. A nonconstant \(h(y)\) changes the unnormalized weights; it cannot be dropped unless the model is being changed or the factor is correctly accounted for elsewhere.
- Assuming the parameter space is all real numbers. That is true for the finite-support example, but not for the countable or continuous examples. Test convergence at the boundary as well as in the interior.
- Confusing the natural parameter with the response mean. The natural parameter is the coefficient in the exponent. It is not generally the mean of the response.
- Using the wrong scale convention. When a dispersion factor is present, check whether the chosen notation calls \(\theta\) or \(\theta/a(\phi)\) the natural parameter. The full exponent and normalizer must agree.
Check Your Understanding
For each model, identify the requested normalizer, parameter restriction, or probability. Answers assume counting measure for discrete models and Lebesgue measure for continuous models.
- A response has support \(\{0,1,2\}\), base weights \(1,3,2\), and statistic \(T(y)=y\). Give \(A(\eta)\), its parameter space, and the three probabilities at \(\eta=0\). (Answer: \(A(\eta)=\log(1+3e^\eta+2e^{2\eta})\); all real \(\eta\); probabilities \(1/6, 3/6, 2/6\).)
- A response has support \(0,1,2,\ldots\), base factor 1, and \(T(y)=y\). At \(\eta=\log(1/4)\), give \(A(\eta)\), \(P(Y=0)\), and \(P(Y=1)\). (Answer: \(A(\eta)=\log(4/3)\); \(P(Y=0)=3/4\); \(P(Y=1)=3/16\).)
- A continuous response has support \(y>0\), base factor 1, and density proportional to \(e^{-3y}\). Identify \(\eta\), \(A(\eta)\), and \(P(Y>1)\). (Answer: \(\eta=-3\); \(A(-3)=-\log 3\); \(P(Y>1)=e^{-3}\approx0.0498\).)
- A discrete response has support \(\{0,1,2\}\), base weights \(2,1,1\), and \(\eta=\log 2\). Find the normalizing sum, cumulant value, and probability of \(Y=2\). (Answer: normalizing sum \(8\); \(A(\eta)=\log 8\); \(P(Y=2)=1/2\).)
- For support \(0,1,2,\ldots\) with base factor 1 and \(T(y)=y\), is \(\eta=0\) permitted? Explain using the normalizing sum. (Answer: No; the sum is \(\sum_{y=0}^{\infty}1\), which diverges, so no finite cumulant function can normalize it.)