Why Separate the Parameter From the Data?
An exponential-family density organizes the relationship between a model parameter and an observed response. Its key feature is that the parameter-dependent part of the log density contains a product: a natural parameter multiplied by a sufficient statistic. Identifying that product helps reveal which aspect of the data informs the parameter, and it provides a useful common language for distributions that may look quite different in their original forms.
The natural parameter is not necessarily the parameter originally used to describe the model. For example, a Bernoulli model is commonly specified by a success probability \(p\), but its natural parameter is the log odds of success. The sufficient statistic is a function of the observed response, not a parameter. The two quantities have different roles even though they appear together in the same term.
The Natural Parameter and Sufficient Statistic
For a scalar response \(Y\), a one-parameter exponential-family density or mass function can be written in canonical form as
Here \(h(y)\) is a factor that does not depend on \(\eta\), \(T(y)\) is the sufficient statistic, \(\eta\) is the natural parameter, and \(A(\eta)\) is the log-partition function. The log density is
The natural parameter is therefore the coefficient of \(T(y)\) in the log density. It is often a transformation of a more familiar parameter, such as a probability or rate. The statistic \(T(y)\) is a function of the data; it does not change when the parameter changes.
In the one-observation canonical form above, \(T(Y)\) is sufficient for \(\eta\): the density factors into \(h(y)\), which is free of \(\eta\), and \(\exp\{\eta T(y)-A(\eta)\}\), whose data dependence is only through \(T(y)\). For \(n\) independent observations \(Y_1,\ldots,Y_n\) sharing the same natural parameter,
The parameter-dependent part of this joint density uses the data only through \(\sum_{i=1}^n T(Y_i)\). That sum is sufficient for \(\eta\). This statement concerns the specified model and its parameter; it does not say that the sum retains every feature of the data that could matter for other questions.
Worked Example: Bernoulli Response
Worked Example: Bernoulli Response
Suppose a single hypothetical response \(Y\) records whether a participant completes a task: \(Y=1\) for completion and \(Y=0\) otherwise. Assume \(Y\sim\operatorname{Bernoulli}(p)\), with \(0<p<1\). The probability mass function is \(f(y\mid p)=p^y(1-p)^{1-y}\) for \(y\in\{0,1\}\).
Taking logarithms and collecting the term involving the observation gives
Thus the natural parameter is \(\eta=\log\{p/(1-p)\}\), the log odds of completion, and the sufficient statistic is \(T(y)=y\). To put the remaining parameter term in canonical form, solve \(p=e^\eta/(1+e^\eta)\). Then \(\log(1-p)=-\log(1+e^\eta)\), so \(A(\eta)=\log(1+e^\eta)\), with \(h(y)=1\).
For a concrete check, take \(p=0.6\) and observe \(y=1\). Then \(\eta=\log(0.6/0.4)=\log(1.5)\approx0.4055\), \(T(y)=1\), and \(A(\eta)=\log(2.5)\approx0.9163\). The canonical log mass is \(0.4055-\log(2.5)\approx-0.5108\), so the mass is approximately \(e^{-0.5108}=0.6000\), as expected. The exact expression is \(e^{\log(1.5)-\log(2.5)}=0.6\); the rounded intermediate values are only for display.
Why the Sample Sum Is Sufficient
The factorization for independent observations gives a short justification for the sample result. In the Bernoulli case, each \(T(Y_i)=Y_i\), so the data enter the parameter-dependent part of the joint likelihood only through the number of completions, \(\sum_i Y_i\).
For independent Bernoulli responses with common probability \(p\), multiply their individual mass functions: \(L(p)=\prod_i p^{y_i}(1-p)^{1-y_i}\).
Let \(s=\sum_i y_i\). Then \(L(p)=p^s(1-p)^{n-s}\), which depends on the observed responses through \(s\) only.
The likelihood factors as a function of \(s\) and \(p\), with no other data-dependent factor. Therefore \(s=\sum_i Y_i\) is sufficient for \(p\).
More generally, the product form of independent exponential-family densities shows the same result whenever the observations share \(\eta\): the sum of the individual sufficient statistics is sufficient. Independence and a common natural parameter are assumptions here. If observations are dependent or have different parameters, this particular argument may not apply as written.
Worked Example: A Poisson Count
Worked Example: A Poisson Count
Suppose a hypothetical monitoring device records \(Y=3\) events during a fixed time interval. Assume a Poisson model with rate \(\lambda=2.5\) events per interval. For a nonnegative integer \(y\), the mass is \(f(y\mid\lambda)=e^{-\lambda}\lambda^y/y!\).
Rewrite the mass by putting the observation in the exponent:
The natural parameter is \(\eta=\log\lambda\), and the sufficient statistic is \(T(y)=y\). In canonical form, \(A(\eta)=e^\eta\), because \(\lambda=e^\eta\), and \(h(y)=1/y!\). For \(y=3\), the log mass is
Using the unrounded log mass, the probability is \(e^{-1.542887\ldots}\approx0.213763\), or \(0.2138\) to four decimal places. This agrees with the direct calculation \(e^{-2.5}(2.5)^3/6\). In particular, exponentiating the displayed rounded value \(-1.543\) would give approximately \(0.2137\); the four-decimal result above comes from the unrounded expression.
Worked Example: The Total of Independent Poisson Counts
Worked Example: The Total of Independent Poisson Counts
Suppose four hypothetical, independent intervals produce event counts \(1,0,2,3\), and assume each count has the same Poisson rate \(\lambda\) events per interval. The sample total is \(s=1+0+2+3=6\). The joint likelihood is
The only part involving the unknown rate depends on the observations through \(s=6\). The denominator, \(12\), depends on the particular observed counts but not on \(\lambda\). The factorization criterion therefore identifies \(\sum_iY_i\) as sufficient for the common rate \(\lambda\).
For example, the count sequence \(0,0,0,6\) also has total 6. Its likelihood and that of \(1,0,2,3\) are not numerically equal: their data-only denominators differ. But their likelihood ratio does not depend on \(\lambda\), since the factors \(e^{-4\lambda}\lambda^6\) cancel. Thus, for inference about this common rate under the stated model, the two data sets provide the same parameter-dependent likelihood information through their total.
How to Check an Exponential-Family Identification
A practical way to identify the natural parameter and sufficient statistic is to take the logarithm of the density or mass, then collect terms by whether they depend on the parameter or the data. The target is a term of the form \(\eta T(y)\), plus a parameter-only term and a data-only term. For a scalar natural parameter, the coefficient of the data statistic is the natural parameter; for a vector parameter, the corresponding term is an inner product.
The decomposition should be checked rather than guessed. In particular:
- Check parameter dependence. The factor \(h(y)\) must not depend on the parameter. A term involving both \(y\) and the parameter cannot be placed in \(h(y)\).
- Check data dependence. The statistic \(T(y)\) must be a function of the observation. A quantity involving the unknown parameter is not the sufficient statistic.
- Check the whole log density. Terms left over after identifying \(\eta T(y)\) must be accounted for by \(A(\eta)\) or \(\log h(y)\). Omitting \(A(\eta)\) generally gives an expression that does not normalize to a density or mass function.
- Check the model assumptions for a sample result. The sum \(\sum_i T(Y_i)\) follows from the stated independent, common-parameter likelihood; it should not be carried over automatically to a different dependence structure.
Natural-parameter form is a useful alternative to writing a distribution in terms of its original parameter. It makes the data summary visible in the likelihood and gives a common structure for probability models. However, a sufficient statistic is sufficient only relative to a particular model and parameter. It does not establish that the model is appropriate, guarantee a good estimate, or retain all information relevant to every possible scientific question.
Common Pitfalls and Equivalent Forms
A frequent error is to call the original parameter the natural parameter simply because it is the parameter named in the distribution. In the Bernoulli model, \(p\) is the success probability, but \(\log\{p/(1-p)\}\) is the coefficient of \(y\) in the log mass. In the Poisson model, \(\lambda\) is the rate, but \(\log\lambda\) is the coefficient of \(y\). The coefficient in the log density, not the original parameter’s name, determines the natural parameter.
Another error is to confuse the individual statistic with its sample aggregate. For one Bernoulli or Poisson observation, \(T(y)=y\). For independent observations sharing a parameter, the likelihood uses \(\sum_iY_i\). Both statements are correct at their respective levels.
The particular algebraic representation is not always unique. For example, replacing \(T(y)\) by \(cT(y)\) and \(\eta\) by \(\eta/c\), for a nonzero constant \(c\), leaves their product unchanged; the parameter-only term must still be expressed consistently. Thus, identify the convention used in a given representation before comparing symbols across formulas. In the Bernoulli and Poisson forms developed here, the standard choice is \(T(y)=y\).
Check Your Understanding
Answer each question, then compare with the solutions below.
- For a Bernoulli response with success probability \(p\), identify the natural parameter and the one-observation sufficient statistic.
- For a Poisson count with rate \(\lambda\), identify \(T(y)\) and the natural parameter in the canonical form.
- Why is \(\sum_iY_i\) sufficient for a common Bernoulli probability in an independent sample?
- In an independent Poisson sample with a common rate, why can two data sets with the same total have different likelihoods but the same parameter-dependent likelihood information?
Solutions. 1. The log mass contains \(y\log\{p/(1-p)\}\), so \(\eta=\log\{p/(1-p)\}\) and \(T(y)=y\). 2. The mass is \(y!^{-1}\exp\{y\log\lambda-\lambda\}\), so \(T(y)=y\) and \(\eta=\log\lambda\). 3. The joint likelihood is \(p^s(1-p)^{n-s}\), where \(s=\sum_iY_i\); its dependence on the observed data is only through \(s\), so the factorization criterion gives sufficiency. 4. The common parameter appears through \(e^{-n\lambda}\lambda^s\), determined by sample size \(n\) and total \(s\); data-only factorial terms may differ but cancel from the likelihood ratio as a function of \(\lambda\).