Why Independent Product Families Have a Block Structure
When several random variables are independent, their joint density or mass function is a product of their component densities or mass functions. If each component already has an exponential-family representation, that product can be written as a single exponential-family expression. Its statistic is formed by placing the component statistics in separate coordinate blocks, and its log-partition function is the sum of the component log-partition functions.
The block structure matters: it records which statistic and natural parameter belong to each variable. It also makes clear that a product family with independently varying parameters has a Cartesian-product natural parameter space. A model in which the component parameters are linked may use the same joint formula, but its realized natural parameters occupy only a restricted subset of that space.
The dimension of the concatenated statistic is \(p_1+\cdots+p_m\). The joint carrier is the product of the component carriers, and the pairing \(\eta^{\mathsf T}T\) is the sum of the component pairings. These statements are consequences of independence and the product-measure construction, not assumptions that the component variables have identical distributions.
Joint Canonical Representation and Natural Parameter Space
For each component, let \(\mu_i\) be a parameter-independent carrier measure on its sample space \(\mathcal{X}_i\). Assume the representation above holds with respect to \(\mu_i\), with \(h_i\) nonnegative and measurable. Define its natural parameter space by the finiteness of its normalizing integral: $$ \mathcal{H}_i= \left\{\eta_i\in\mathbb{R}^{p_i}: 0<\int_{\mathcal{X}_i}h_i(x_i)\exp\{\eta_i^{\mathsf T}T_i(x_i)\}\,\mu_i(dx_i)<\infty \right\}. $$ On this space, \(A_i(\eta_i)\) is the logarithm of that integral. The parameter-independent carrier assumption is important: parameter-dependent support requires separate care, as in nonregular families.
Proof. Independence gives the joint density as the product of the component densities. Substituting their canonical forms gives $$ \begin{aligned} f(x_1,\ldots,x_m;\eta) &=\prod_{i=1}^m h_i(x_i) \exp\{\eta_i^{\mathsf T}T_i(x_i)-A_i(\eta_i)\}\\ &=\left\{\prod_{i=1}^m h_i(x_i)\right\} \exp\left\{\sum_{i=1}^m\eta_i^{\mathsf T}T_i(x_i)-\sum_{i=1}^m A_i(\eta_i)\right\}. \end{aligned} $$ By the definitions of the concatenated vectors, the first sum in the exponent is \(\eta^{\mathsf T}T(x)\). Thus the displayed joint expression has the stated canonical form and log-partition function.
To identify the natural parameter space, integrate its unnormalized kernel. The integrand is nonnegative and measurable, so Tonelli's theorem gives $$ \begin{aligned} &\int_{\mathcal{X}_1\times\cdots\times\mathcal{X}_m} \prod_{i=1}^m h_i(x_i)\exp\{\eta_i^{\mathsf T}T_i(x_i)\} \,(\mu_1\otimes\cdots\otimes\mu_m)(d x)\\ &\qquad = \prod_{i=1}^m \int_{\mathcal{X}_i}h_i(x_i)\exp\{\eta_i^{\mathsf T}T_i(x_i)\}\,\mu_i(dx_i), \end{aligned} $$ where the equality also holds if one or more factors are infinite. Since each factor is positive, the product is positive and finite exactly when every factor is positive and finite. This is precisely the condition \(\eta_i\in\mathcal{H}_i\) for every \(i\). Taking the logarithm of the product of the component normalizing integrals gives \(\sum_i A_i(\eta_i)\), as claimed. \(\square\)
The theorem describes the full product family, in which each component natural parameter may vary independently throughout its own natural parameter space. A particular model may impose a parameterization \(\eta_i=\eta_i(\theta)\). Its realized joint natural parameters are then only the vectors \((\eta_1(\theta),\ldots,\eta_m(\theta))\) allowed by that parameterization. The joint canonical expression remains valid on this subset; the subset need not be the entire Cartesian product.
Derivation. Substitute \(\eta_i(\theta)\) for each component natural parameter in the product density. Its exponent contains the terms \(\eta_i(\theta)^{\mathsf T}T_i(x_i)-A_i(\eta_i(\theta))\). Collecting the normalizing terms gives \(-\sum_i A_i(\eta_i(\theta))\); collecting the statistic terms gives the pairing with the concatenated parameter vector. By definition, the vectors attained as \(\theta\) varies form the stated image. This derivation identifies the restricted parameter set without changing the full product-family natural parameter space. \(\square\)
Worked Representations
Worked Example: Independent Bernoulli Variables
Let \(X_1,\ldots,X_3\) be independent Bernoulli variables with success probabilities \(p_i\in(0,1)\), which need not be equal. With counting measure on \(\{0,1\}\), the mass function of \(X_i\) is \(p_i^{x_i}(1-p_i)^{1-x_i}\). Define \(\eta_i=\log\{p_i/(1-p_i)\}\). Solving for \(p_i\) gives \(p_i=e^{\eta_i}/(1+e^{\eta_i})\), so $$ p_i^{x_i}(1-p_i)^{1-x_i} =\exp\left\{x_i\eta_i-\log(1+e^{\eta_i})\right\}. $$ Thus \(T_i(x_i)=x_i\) and \(A_i(\eta_i)=\log(1+e^{\eta_i})\).
Independence now gives the joint mass function $$ f(x_1,x_2,x_3;\eta) =\exp\left\{\eta_1x_1+\eta_2x_2+\eta_3x_3 -\sum_{i=1}^3\log(1+e^{\eta_i})\right\}. $$ The concatenated statistic is \((x_1,x_2,x_3)^{\mathsf T}\). For one Bernoulli variable, the normalizing sum is \(1+e^{\eta_i}\), finite and positive for every real \(\eta_i\); hence \(\mathcal{H}_i=\mathbb{R}\). The full natural parameter space is \(\mathbb{R}^3\). If one additionally requires all three success probabilities to be equal, then all three natural parameters must be equal: this is a restricted model, not the full product family.
Worked Example: A Bernoulli Variable and a Poisson Variable
Let \(B\sim\operatorname{Bernoulli}(p)\) and \(N\sim\operatorname{Poisson}(\lambda)\) be independent, where \(0<p<1\) and \(\lambda>0\). The Bernoulli representation has \(T_1(B)=B\), \(\eta_1=\log\{p/(1-p)\}\), and \(A_1(\eta_1)=\log(1+e^{\eta_1})\). For the Poisson variable, using counting measure on the nonnegative integers, $$ P(N=n)=\frac{\lambda^n e^{-\lambda}}{n!} =\frac{1}{n!}\exp\{n\log\lambda-\lambda\}. $$ Thus \(h_2(n)=1/n!\), \(T_2(N)=N\), \(\eta_2=\log\lambda\), and \(A_2(\eta_2)=e^{\eta_2}\).
The joint representation is therefore $$ P(B=b,N=n) =\frac{1}{n!} \exp\left\{\eta_1b+\eta_2n -\log(1+e^{\eta_1})-e^{\eta_2}\right\}. $$ For the Poisson block, the normalizing sum is $$ \sum_{n=0}^{\infty}\frac{e^{\eta_2 n}}{n!} =\exp(e^{\eta_2}), $$ which is finite and positive for every real \(\eta_2\). Both component natural parameter spaces are \(\mathbb{R}\), so the joint natural parameter space is \(\mathbb{R}^2\). The carrier \(1/n!\) belongs to the Poisson coordinate; it does not alter the concatenated statistic or the additivity of the log-partition function.
Worked Example: Two Independent Normals With Unknown Means and Variances
Let \(X_1\sim N(\mu_1,\sigma_1^2)\) and \(X_2\sim N(\mu_2,\sigma_2^2)\) independently, with \(\sigma_i^2>0\). For a single Normal variable, expand the exponent of its density: $$ -\frac{(x-\mu)^2}{2\sigma^2} =-\frac{x^2}{2\sigma^2}+\frac{\mu x}{\sigma^2}-\frac{\mu^2}{2\sigma^2}. $$ Using Lebesgue measure and carrier \(h(x)=1\), take \(T(x)=(x,x^2)^{\mathsf T}\), \(\eta_1=\mu/\sigma^2\), and \(\eta_2=-1/(2\sigma^2)\). The normalizing integral is $$ \int_{\mathbb{R}}e^{\eta_1x+\eta_2x^2}\,dx =\sqrt{\frac{\pi}{-\eta_2}}\exp\left\{-\frac{\eta_1^2}{4\eta_2}\right\}, \qquad \eta_2<0. $$ This follows by completing the square; the integral is finite exactly when \(\eta_2<0\). Therefore $$ A(\eta_1,\eta_2) =-\frac{\eta_1^2}{4\eta_2}+\frac{1}{2}\log\left(\frac{\pi}{-\eta_2}\right). $$ Substituting \(\eta_1=\mu/\sigma^2\) and \(\eta_2=-1/(2\sigma^2)\) yields \(A=\mu^2/(2\sigma^2)+\tfrac12\log(2\pi\sigma^2)\), the normalizing term in the usual Normal density.
For the independent pair, the statistic and natural parameter are $$ T(x_1,x_2)=(x_1,x_1^2,x_2,x_2^2)^{\mathsf T}, \qquad \eta=(\eta_{11},\eta_{12},\eta_{21},\eta_{22})^{\mathsf T}, $$ and the log-partition function is $$ A(\eta)=A(\eta_{11},\eta_{12})+A(\eta_{21},\eta_{22}). $$ The full natural parameter space is \(\{(\eta_{11},\eta_{12},\eta_{21},\eta_{22}):\eta_{12}<0,\ \eta_{22}<0\}\). Each inequality is necessary to make the corresponding Gaussian integral finite. This example also shows why a block may contain more than one statistic coordinate.
Common Mistake: Confusing the Product Family With a Restricted Model
A frequent error is to call the natural parameter space of a joint model “the Cartesian product” even when the parameters are linked, or, conversely, to conclude that a linked model has no joint canonical representation. The correction is to keep two objects distinct: the full product family's natural parameter space, determined by component integrability, and the set of natural parameter vectors actually attained under the model's parameterization.
For example, the three Bernoulli variables above have full natural parameter space \(\mathbb{R}^3\). Under the additional equal-probability restriction, the realized vectors have the form \((\eta,\eta,\eta)\), not every point of \(\mathbb{R}^3\). The joint density still has the concatenated statistic \((x_1,x_2,x_3)\) and the sum of the three log-partition terms, evaluated at that restricted vector.
Another mistake is to multiply marginal densities when the variables are dependent. The product derivation begins with independence; without it, the joint density need not equal the product of the component densities, and the separate log-partition functions need not combine in this way. A factorization of a joint law must be justified, not inferred from the fact that each marginal belongs to an exponential family.
Check Your Understanding
Use the component canonical forms and the independence assumption in each question.
- If three independent components have statistic dimensions \(2\), \(1\), and \(4\), what is the dimension of the concatenated statistic?
- Why does Tonelli's theorem apply when deriving the joint normalizing integral from component integrals?
- Two independent Poisson variables have natural parameters \(\eta_1\) and \(\eta_2\). State their concatenated statistic and joint log-partition function.
- For two independent Normal variables with unknown variances, which inequalities define the full natural parameter space in the quadratic-statistic coordinates?
- Explain the difference between the full natural parameter space and the realized natural parameter set when two component parameters are constrained to be equal.