Exponential Tilting and Distributional Change

Reweighting a Baseline Distribution

A parameter change in an exponential family does more than change a label attached to a distribution: it reweights the possible observations. Values of the sufficient statistic that are larger in a chosen direction receive more weight when the natural parameter moves in that direction; values in the opposite direction receive less. The normalizing constant adjusts all the weights so that they define a probability distribution.

This change of distribution is called exponential tilting. It gives a direct way to compare two parameter values, calculate probabilities under the changed distribution, and express a new expectation using the baseline distribution.

The finiteness of \(Z_0(\delta)\) is essential: without it the weights cannot be normalized to have total probability one. Since the exponential weight is strictly positive wherever \(T(X)\) is finite, a well-defined tilt preserves the baseline distribution's support. In a discrete model it changes the probabilities of supported points; in a continuous model it changes the density without introducing new supported values.

Natural-Parameter Changes and Likelihood Ratios

Consider a canonical exponential family with density or mass function, relative to a common dominating measure, \(f_\eta(x)=h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}\). Here \(\eta\) is the natural parameter and \(A(\eta)\) is the log-normalizing function. Suppose both \(\eta_0\) and \(\eta_1=\eta_0+\delta\) are in the natural parameter space. The two distributions then share the same support, and their density ratio is obtained directly by division.

This is the likelihood ratio for one observation at \(x\), comparing \(\eta_1\) with \(\eta_0\). It is also the exponential-tilt weight. In particular, the baseline expectation of the unnormalized weight is \(E_{\eta_0}[\exp\{\delta^\mathsf{T}T(X)\}]=\exp\{A(\eta_1)-A(\eta_0)\}\). Thus the change in the log-normalizing function supplies exactly the constant needed to make the tilted density integrate or sum to one.

Proof. By the density formula, for every supported \(x\),

Integrating both sides with respect to the common dominating measure shows that \(1=\exp\{-[A(\eta_1)-A(\eta_0)]\}E_{\eta_0}[\exp\{\delta^\mathsf{T}T(X)\}]\). Therefore the expectation is \(\exp\{A(\eta_1)-A(\eta_0)\}\), which proves the normalizing-constant formula and the tilt representation. Multiplying the density identity by \(g(x)\) and integrating gives the stated expectation formula. The assumed absolute integrability makes this integral well-defined. \(\square\)

For \(n\) independent observations \(X_1,\ldots,X_n\), the likelihood ratio is the product of the individual ratios. Equivalently, its logarithm is \(\delta^\mathsf{T}\sum_{i=1}^n T(X_i)-n[A(\eta_1)-A(\eta_0)]\). Thus tilting acts on the accumulated sufficient statistic, while the normalizing adjustment is multiplied by the sample size.

Worked Discrete and Continuous Tilts

Why Shifted Expectations Work

The expectation identity is useful when an expectation under the changed distribution is harder to calculate directly than an expectation under the baseline. It says to multiply the quantity of interest by the likelihood ratio and average under the baseline. The normalization is not optional: using only the unnormalized weight gives a weighted total, not an expectation under a probability distribution.

A useful consistency check is that setting \(g(X)=1\) in the expectation identity gives \(E_{\eta_1}[1]=1\). The numerator then equals the normalizing constant. For an indicator \(g(X)=1_{\{X\in A\}}\), the same identity gives the probability of an event under the tilted distribution: \(P_{\eta_1}(X\in A)=E_{\eta_0}[1_{\{X\in A\}}\exp\{\delta^\mathsf{T}T(X)\}]/Z_{\eta_0}(\delta)\).

Proof. The density of \(P_\delta\) relative to \(P_0\) is \(\exp\{\delta^\mathsf{T}T(X)\}/Z_0(\delta)\). Tilting this density by \(\gamma\) multiplies it by \(\exp\{\gamma^\mathsf{T}T(X)\}\) and divides by \(E_\delta[\exp\{\gamma^\mathsf{T}T(X)\}]\). By the definition of \(P_\delta\), this denominator is \(Z_0(\delta+\gamma)/Z_0(\delta)\), which is finite by the assumption that the further tilt is well-defined. The resulting density relative to \(P_0\) is therefore \(\exp\{(\delta+\gamma)^\mathsf{T}T(X)\}/Z_0(\delta+\gamma)\), as claimed. \(\square\)

Composition gives a practical check on calculations: shifting from \(\eta_0\) to \(\eta_0+\delta\), and then by \(\gamma\), must give the same distribution as shifting directly by \(\delta+\gamma\). It also emphasizes that the tilt parameter is a change in natural coordinates, not generally an additive change in an ordinary probability or mean.

Common Pitfall

A frequent error is to multiply baseline probabilities or densities by \(\exp\{\delta^\mathsf{T}T(x)\}\) and stop. Those quantities generally do not sum or integrate to one. For the Bernoulli example, the unnormalized masses would be \(7/10\) and \(6/10\), whose total is \(13/10\), not \(1\). Dividing both by \(13/10\) gives \(7/13\) and \(6/13\), the valid tilted probabilities.

Another error is to interpret \(\delta\) as a direct change in the probability of success. In the Bernoulli example, \(\delta=\log 2\) multiplies the odds by \(2\); it does not add \(2\) to the probability. Always identify the statistic being weighted, calculate the normalizing constant, and then form the ratio.