Why Write a Response Distribution in a Common Form?
A generalized linear model starts with a response distribution, but familiar distributions can look quite different when written using their usual parameters. A common representation makes their structure easier to compare: it separates the part that depends on the observed response from the part that normalizes the distribution, and it displays any dispersion parameter. This tutorial focuses on identifying those parts, not on deriving model means or variances from them.
The earlier material on the conditional mean and linear predictor introduced the response and the linear predictor. Here, \(\theta\) denotes the natural parameter in the exponential-family representation. It is not automatically the same quantity as the linear predictor, which is often denoted by \(\eta\). A link function or model specification determines any relationship between them.
The formula describes a density with respect to a continuous response or a mass function for a discrete response. The exponential function guarantees nonnegative values, but it does not by itself guarantee that those values integrate or sum to one. The log-partition function is the term that supplies the required normalization for each permitted value of \(\theta\).
The representation is useful when the support of \(Y\) is fixed as \(\theta\) varies, the normalizing function is finite on the parameter values under consideration, and the remaining term \(c(y,\phi)\) is free of \(\theta\). These are structural requirements, not optional labels. If a proposed decomposition leaves a \(\theta\)-dependent term inside \(c\), or omits a parameter-dependent normalizer, it is not the displayed representation.
Reading the Components
The response is the random variable whose observed value is \(y\). The statistic \(T(y)\) is the function of that response multiplied by \(\theta\) in the exponent. In the common GLM form it is simply \(y\); in other families it can be a different function. For independent observations with a common parameter, sums of these response statistics appear in the likelihood. This is why \(T(y)\) is often called the sufficient-statistic component of the representation.
The natural parameter \(\theta\) controls how the response statistic is weighted. The log-partition function \(A(\theta)\) is a function of that parameter alone and ensures normalization, together with the other terms in the representation. The dispersion contribution is displayed through \(a(\phi)\) and any additional dependence on \(\phi\) in \(c(y,\phi)\). It is important not to assume that every family has an estimable dispersion parameter: some common response models have a fixed scale, while others include one.
There can be more than one algebraically equivalent way to allocate constants between \(A\) and \(c\), but a decomposition must preserve the full density or mass function. For a chosen convention, \(A\) is the log normalizer associated with the natural parameter, while \(c\) must not depend on that parameter. A quick check is to substitute the proposed components back into the exponent and recover the original log density exactly.
Worked Example: A Finite Discrete Response
Suppose a hypothetical response \(Y\) takes values \(0,1,2\), with probability mass function \(P(Y=y;\theta)=\exp\{y\theta-A(\theta)\}\) on those three values. Identify the components and evaluate the probabilities at \(\theta=0.4\).
For the probabilities to sum to one, the normalizing constant must satisfy $$ \exp\{A(\theta)\} = \sum_{y=0}^{2}\exp(y\theta) = 1+\exp(\theta)+\exp(2\theta). $$ Therefore $$ A(\theta)=\log\{1+\exp(\theta)+\exp(2\theta)\}. $$ The response statistic is \(T(y)=y\), the natural parameter is \(\theta\), and the log-partition function is the \(A(\theta)\) above. There is no dispersion term here, so \(a(\phi)=1\) by convention, and \(c(y,\phi)=0\).
At \(\theta=0.4\), the normalizing sum is $$ 1+\exp(0.4)+\exp(0.8) \approx 1+1.4918247+2.2255409 =4.7173656. $$ Thus \(A(0.4)=\log(4.7173656)\approx1.5513\). Using the unrounded normalizing sum, the three probabilities are $$ P(Y=0)\approx\frac{1}{4.7173656}=0.2120,\qquad P(Y=1)\approx\frac{1.4918247}{4.7173656}=0.3162,\qquad P(Y=2)\approx\frac{2.2255409}{4.7173656}=0.4718. $$ They sum to \(1.0000\) at the displayed precision. This check confirms that the log-partition function normalizes the mass function.
Worked Identifications in Familiar Models
The next examples use familiar distributions only to practice reading the representation. The task in each case is to identify the components and verify the algebra, rather than to derive additional properties of the model.
Worked Example: Rewriting a Bernoulli Mass Function
Let \(Y\) be Bernoulli with success probability \(p\), where \(0<p<1\). Its mass function is \(f(y;p)=p^y(1-p)^{1-y}\) for \(y\in\{0,1\}\). Rewrite it in exponential-family form.
Take logarithms and collect the terms involving \(y\): $$ \log f(y;p) =y\log(p)+(1-y)\log(1-p) =y\log\left(\frac{p}{1-p}\right)+\log(1-p). $$ Set \(\theta=\log\{p/(1-p)\}\). Then \(p=\exp(\theta)/(1+\exp(\theta))\), so \(\log(1-p)=-\log(1+\exp(\theta))\). Consequently, $$ f(y;p) =\exp\left\{y\theta-\log(1+\exp(\theta))\right\}. $$ The components are \(T(y)=y\), natural parameter \(\theta=\log\{p/(1-p)\}\), log-partition function \(A(\theta)=\log(1+\exp(\theta))\), \(a(\phi)=1\), and \(c(y,\phi)=0\).
As a numerical check, take \(p=0.7\). Then \(\theta=\log(7/3)\approx0.8473\), and \(A(\theta)=\log(10/3)\approx1.2040\). For \(y=1\), the representation gives \(\exp(\theta-A)=\exp\{\log(7/3)-\log(10/3)\}=0.7\). For \(y=0\), it gives \(\exp(-A)=3/10=0.3\). These are the original success and failure probabilities.
Worked Example: Identifying Dispersion in a Normal Density
Suppose \(Y\) has a normal distribution with mean \(\mu\) and known variance \(\sigma^2\). Its density is $$ f(y;\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(y-\mu)^2}{2\sigma^2}\right\}. $$ Expand the square in the exponent: $$ -\frac{(y-\mu)^2}{2\sigma^2} = \frac{y\mu-\mu^2/2}{\sigma^2} -\frac{y^2}{2\sigma^2}. $$ The density therefore has the required form with \(T(y)=y\), \(\theta=\mu\), \(A(\theta)=\theta^2/2\), \(a(\phi)=\phi\) where \(\phi=\sigma^2\), and $$ c(y,\phi)=-\frac{y^2}{2\phi}-\frac{1}{2}\log(2\pi\phi). $$ The known variance is the dispersion value in this representation. Notice that the term \(-y^2/(2\phi)\) belongs in \(c\): it depends on the response and dispersion, but not on the natural parameter \(\theta=\mu\).
For a numerical check, set \(\mu=2\), \(\sigma^2=4\), and evaluate the density at \(y=3\). The natural-parameter part is \((3\cdot2-2^2/2)/4=1\). The remaining term is \(-9/8-\tfrac12\log(8\pi)\approx-2.7371\). Thus the log density is about \(-1.7371\), and the density is \(\exp(-1.7371)\approx0.1760\). Direct substitution into the normal density gives the same value, approximately \(0.1760\).
Why the Decomposition Works
The factorization also explains why the response statistic is central to likelihood calculations. For independent observations \(Y_1,\ldots,Y_n\) from the same representation, their joint density is the product of the individual densities. Taking logarithms gives $$ \log L(\theta,\phi) = \frac{\theta\sum_{i=1}^n T(y_i)-nA(\theta)}{a(\phi)} +\sum_{i=1}^n c(y_i,\phi). $$ For fixed \(\phi\), the dependence on \(\theta\) through the observed data occurs through \(\sum_iT(y_i)\). The remaining factor does not involve \(\theta\). By the likelihood factorization criterion, this sum is sufficient for \(\theta\) under the stated independent-sampling model. For a single observation, the corresponding statistic is \(T(Y)\). This justification relies on the product likelihood and the absence of \(\theta\) from \(c\); it is not justified merely by calling a term a statistic.
The derivation is a useful algebraic check: multiply the log density by the number of observations, sum its response-dependent terms, and collect the natural-parameter terms. If the result contains extra dependence on the observations and \(\theta\) that cannot be expressed through \(\sum_iT(y_i)\), the proposed decomposition should be checked again.
Checking a Proposed Representation
When converting a model, use a deliberate sequence rather than guessing the components from their names.
Include its support and every normalizing constant. Take its logarithm so products and powers become sums.
Identify a function \(T(y)\) multiplied by a parameter \(\theta\). Terms independent of \(\theta\) do not belong in that product.
Identify the term \(A(\theta)\) and the dispersion scaling \(a(\phi)\), if present. Confirm that the normalizer is finite over the parameter values being used.
The remainder must be free of \(\theta\). Substitute all components into the representation and recover the original log density, then check normalization when practical.
There are two useful ways to describe the same response model. Its ordinary parameterization, such as a probability \(p\) or a mean \(\mu\), is often easiest to interpret directly. The exponential-family parameterization reorganizes the same density around \(\theta\), \(T(y)\), and \(A(\theta)\), which makes likelihood structure visible and supports the GLM framework. The choice is not between two different probability models when the reparameterization is valid; it is between two ways of expressing the same model.
This representation has limits. A particular density may fail to fit this one-parameter form, may require more than one natural parameter, or may have support that changes with a parameter. A model with several natural parameters needs a corresponding vector-valued generalization, rather than forcing all parameter dependence into one scalar \(\theta\). Also, identifying a valid exponential-family form does not by itself establish that a chosen link, linear predictor, or mixed-effects structure is appropriate.
Common Mistakes and Diagnostics
- Confusing the linear predictor with the natural parameter. They may be related in a particular model, but the relationship must be specified; their symbols and roles are not automatically interchangeable.
- Putting a parameter-dependent term in \(c(y,\phi)\). The remainder term must not depend on \(\theta\). Re-expand the log density and check the dependence explicitly.
- Dropping the normalizing term. An exponent containing the response term alone may produce values that do not sum or integrate to one. Check the total probability or density normalization.
- Ignoring dispersion in the exponent. When \(a(\phi)\neq1\), it scales both the response-statistic and log-partition terms. Omitting it changes the distribution.
- Rounding too early. Use the unrounded normalizer in probability calculations and round only the final displayed values. Otherwise probabilities can be inconsistent with the stated density.
Warning signs include probabilities that fail to sum to one, a purported log-partition function that still depends on \(y\), or a remainder term that changes with \(\theta\). These indicate an algebraic error, an incomplete normalizer, or a representation that needs a broader form.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the representation \(f(y;\theta,\phi)=\exp\{[T(y)\theta-A(\theta)]/a(\phi)+c(y,\phi)\}\), which term must be free of \(\theta\), and why?
- A model has \(T(y)=y^2\) in its natural-parameter term. Is that automatically invalid because the usual scalar-response GLM form has \(T(y)=y\)? Explain.
- For the Bernoulli form in the worked example, give \(T(y)\), \(\theta\), and \(A(\theta)\).
- For independent observations from the same one-parameter representation with fixed \(\phi\), identify the data summary through which the likelihood depends on \(\theta\).
- What numerical check was used in the finite discrete example to assess the log-partition function?
Solutions. (1) The remainder \(c(y,\phi)\) must be free of \(\theta\); otherwise the displayed separation of natural-parameter and response-only terms is not valid. (2) No. \(T(y)=y^2\) can be a valid response statistic in a broader exponential-family representation; \(T(y)=y\) is the usual scalar-response GLM form, not a universal requirement. (3) \(T(y)=y\), \(\theta=\log\{p/(1-p)\}\), and \(A(\theta)=\log(1+\exp(\theta))\). (4) The likelihood depends on \(\theta\) through \(\sum_iT(y_i)\), as shown by the joint log likelihood. (5) The three probabilities were calculated using the same unrounded normalizing sum and shown to add to \(1.0000\) after rounding.