The Exponential Distribution as a Gamma Special Case

From the Gamma Family to the Exponential Distribution

The exponential distribution is often introduced as a model for a positive waiting time, while the gamma distribution is introduced as a broader family of positive-valued distributions. Their connection is exact: the exponential distribution is the gamma distribution with shape one. This identity also gives a direct route to the exponential distribution's canonical representation, provided the rate and the sign of the canonical parameter are tracked carefully.

We will use the gamma parameterization in which the second parameter is a rate. That choice makes the relationship especially transparent: the exponential rate is the gamma rate when the gamma shape is one. The canonical parameter will turn out to be the negative of this rate.

Definition: A random variable $X$ has an exponential distribution with rate $\lambda>0$ if its density is $$ f(x;\lambda)=\lambda e^{-\lambda x}\mathbf{1}_{(0,\infty)}(x). $$ Here $\mathbf{1}_{(0,\infty)}(x)$ is one when $x>0$ and zero otherwise. A gamma distribution with shape $\alpha>0$ and rate $\beta>0$ has density $$ f(x;\alpha,\beta)=\frac{\beta^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}\mathbf{1}_{(0,\infty)}(x). $$

The rate parameter has units reciprocal to the units of $X$. For example, if $X$ is measured in hours, a rate is measured in inverse hours. A scale parameter, when used instead, is the reciprocal of the rate. Keeping these parameterizations distinct will be important in the examples.

Shape One Gives the Exponential Density

Set the gamma shape to $\alpha=1$ and its rate to $\beta=\lambda$. The gamma density becomes $$ f(x;1,\lambda)=\frac{\lambda^1}{\Gamma(1)}x^{1-1}e^{-\lambda x}\mathbf{1}_{(0,\infty)}(x). $$ For $x>0$, $x^0=1$, and $\Gamma(1)=1$. Therefore $$ f(x;1,\lambda)=\lambda e^{-\lambda x}\mathbf{1}_{(0,\infty)}(x), $$ which is precisely the exponential density with rate $\lambda$. The shape-one restriction removes the power $x^{\alpha-1}$ from the gamma density; it does not change the positive support or the rate term in the exponential.

Worked Example: Checking the Gamma Identity

Take a gamma distribution with shape $1$ and rate $3$. Its density, for $x>0$, is $$ f(x;1,3)=\frac{3^1}{\Gamma(1)}x^0e^{-3x}=3e^{-3x}. $$ For $x\leq 0$, the density is zero. This is the exponential density with rate $\lambda=3$.

For a numerical check at $x=1/2$, the gamma expression gives $$ f(1/2;1,3)=\frac{3}{1}(1/2)^0e^{-3/2}=3e^{-3/2}. $$ The exponential expression gives $$ f(1/2;3)=3e^{-3(1/2)}=3e^{-3/2}. $$ The two parameterizations give the same density value, as they must.

The equality is an equality of distributions, not merely an agreement at a particular observation. Both densities have the same value at every $x>0$, and both are zero at every $x\leq 0$. The shape parameter is fixed at one in this subfamily; the remaining gamma parameter is the exponential rate.

Canonical Exponential-Family Form

In a one-parameter canonical exponential-family representation, a density is written as a parameter-free carrier times an exponential involving a natural parameter and a statistic. For this distribution, the statistic is the observation itself. Define the natural parameter by $\eta=-\lambda$. Then $\lambda=-\eta$, and the restriction $\lambda>0$ becomes $\eta<0$.

Definition: The exponential density in canonical form is $$ f(x;\eta)=h(x)\exp\{\eta x-A(\eta)\},\qquad \eta<0, $$ where $h(x)=\mathbf{1}_{(0,\infty)}(x)$ is the carrier, $x$ is the canonical statistic, and $$ A(\eta)=-\log(-\eta). $$ The natural parameter is $\eta=-\lambda$, so the exponential rate is $\lambda=-\eta$.

To check the expression, substitute $\eta=-\lambda$ and $A(\eta)=-\log(\lambda)$ into the exponent. For $x>0$, $$ \exp\{\eta x-A(\eta)\} =\exp\{-\lambda x+\log(\lambda)\} =\lambda e^{-\lambda x}. $$ Multiplication by $h(x)$ sets the density to zero outside the positive support. Thus the canonical expression reproduces the exponential density, including its support.

Why the Natural Parameter Must Be Negative

The log-partition function is determined by the integral that normalizes the exponential term. Since the carrier is one on $x>0$, the relevant integral is $\int_0^\infty e^{\eta x}\,dx$. Its finiteness, and hence the natural-parameter domain, can be checked directly.

Result: The integral $\int_0^\infty e^{\eta x}\,dx$ is finite exactly when $\eta<0$. On that domain it equals $-1/\eta$, so its logarithm is the log-partition function $A(\eta)=-\log(-\eta)$.

Proof. If $\eta<0$, then for any $b>0$, $$ \int_0^b e^{\eta x}\,dx=\frac{e^{\eta b}-1}{\eta}. $$ As $b$ tends to infinity, $e^{\eta b}$ tends to zero. The improper integral therefore converges to $$ \int_0^\infty e^{\eta x}\,dx=\frac{-1}{\eta}=-\frac{1}{\eta}. $$ This value is positive because $\eta<0$. Its logarithm is $$ \log\left(-\frac{1}{\eta}\right)=-\log(-\eta). $$ If $\eta=0$, the integral is $\int_0^\infty 1\,dx$, which diverges. If $\eta>0$, then $e^{\eta x}\geq 1$ for $x\geq 0$, so the integral also diverges. Hence the finite natural-parameter domain is exactly $(-\infty,0)$, and the stated log-partition function follows. $\square$

This calculation also verifies normalization without relying only on the gamma identity. For $\eta<0$, $$ \int_{-\infty}^{\infty}h(x)\exp\{\eta x-A(\eta)\}\,dx =e^{-A(\eta)}\int_0^\infty e^{\eta x}\,dx =(-\eta)\left(-\frac{1}{\eta}\right)=1. $$ The integral is one, while for $\eta\geq 0$ no finite choice of $A(\eta)$ can normalize this carrier and exponential term.

Worked Example: Rate Three in Canonical Form

Let the exponential rate be $\lambda=3$. The natural parameter is $$ \eta=-\lambda=-3, $$ which lies in the required domain $\eta<0$. The log-partition function is $$ A(-3)=-\log(-(-3))=-\log(3). $$ For $x>0$, the canonical density is therefore $$ \exp\{\eta x-A(\eta)\} =\exp\{-3x+\log(3)\}=3e^{-3x}. $$ For $x\leq 0$, the carrier $h(x)$ is zero. The result is the exponential density with rate $3$, written with its canonical parameter.

Rate, Scale, and Canonical Parameter

The gamma and canonical forms use different symbols for the same one-parameter family. In the gamma representation with shape one, the gamma rate $\beta$ equals the exponential rate $\lambda$. In canonical form, that positive rate becomes a negative natural parameter. If the distribution is instead described by a positive scale $\theta$, then $\theta=1/\lambda$, so the canonical parameter is $\eta=-1/\theta$.

Parameter relationship: For the shape-one gamma representation and the exponential distribution, $$ \alpha=1,\qquad \beta=\lambda,\qquad \eta=-\lambda=-\beta. $$ If $\theta$ denotes the scale, then $$ \theta=\frac{1}{\lambda},\qquad \eta=-\frac{1}{\theta}. $$ Thus $\lambda=-\eta$ and $\theta=-1/\eta$, with $\lambda,\theta>0$ and $\eta<0$.

Worked Example: Converting a Scale to Canonical Form

Suppose the exponential scale is $\theta=3/2$. Its rate is the reciprocal, $$ \lambda=\frac{1}{\theta}=\frac{1}{3/2}=\frac{2}{3}. $$ The canonical parameter and log-partition function are $$ \eta=-\lambda=-\frac{2}{3}, \qquad A(\eta)=-\log\left(\frac{2}{3}\right)=\log\left(\frac{3}{2}\right). $$ At $x=3$, the scale-rate density expression gives $$ \lambda e^{-\lambda x}=\frac{2}{3}e^{-(2/3)3}=\frac{2}{3}e^{-2}. $$ The canonical expression gives $$ \exp\{\eta x-A(\eta)\} =\exp\left\{-2-\log\left(\frac{3}{2}\right)\right\} =\frac{2}{3}e^{-2}. $$ They agree. The carrier is one at $x=3$, so this is also the full density value there.

Common Parameterization Mistakes

The most common error is to treat the rate as the canonical parameter without changing its sign. The density contains $-\lambda x$, so the coefficient of the canonical statistic $x$ is $\eta=-\lambda$, not $+\lambda$. A second error is to substitute a scale where a rate is required: the rate is the reciprocal of the scale. For scale $\theta$, the coefficient is $-x/\theta$, giving $\eta=-1/\theta$, not $-\theta$.

A related mistake is to omit the log-partition term after identifying the coefficient of $x$. The term $-A(\eta)$ supplies the normalizing factor. Here it equals $\log(-\eta)$, so the canonical exponential term is $e^{\eta x+\log(-\eta)}=(-\eta)e^{\eta x}$, which is the required rate-times-exponential form on $x>0$.

Why This Representation Matters

Recognizing the exponential distribution as a shape-one gamma distribution lets one move cleanly between a familiar positive-response density and the gamma family’s parameterization. The canonical form makes the rate parameter’s role explicit: it is the negative natural parameter, and the normalizer is finite only for negative natural parameters. These identities are useful whenever the distribution is written in gamma notation in one place and exponential-family notation in another.

The support remains $x>0$ throughout, and the carrier does not depend on the parameter. Those details help keep the algebra honest: the rate belongs in the exponential term and its normalizing factor, while the support is enforced by the carrier. For this special case, every part of the representation can be checked directly from the density and its integral.

Check Your Understanding

Use the density, gamma, and canonical forms to answer the following.

Write the gamma density with shape $1$ and rate $5$, including its support, and identify the corresponding exponential rate.
For an exponential distribution with rate $\lambda=1/4$, give its natural parameter and log-partition function.
If the scale is $\theta=2$, find the rate and canonical parameter. Explain why the canonical parameter is negative.
Evaluate $\int_0^\infty e^{\eta x}\,dx$ for $\eta=-2$, and state the resulting log-partition value.
What goes wrong with the normalizing integral if the proposed natural parameter is $\eta=0$?