Sufficient Statistics for Gamma Samples

Why the Gamma Parameterization Matters

For gamma observations, the useful sufficient summary depends on which parameter is unknown. With shape fixed and scale unknown, the likelihood depends on the data through their sum. With scale fixed and shape unknown, it depends on the sum of their logarithms. The distinction comes directly from the different places these parameters appear in the gamma density.

We use the shape-scale parameterization throughout. In particular, the scale is not the rate: the exponential term in the density is $\exp(-x/\theta)$, so the rate would be $1/\theta$. Keeping this convention explicit prevents a common source of incorrect likelihoods.

Definition: A gamma random variable with shape $\alpha>0$ and scale $\theta>0$ has density $$ f(x\mid \alpha,\theta)=\frac{x^{\alpha-1}\exp(-x/\theta)}{\Gamma(\alpha)\theta^\alpha},\qquad x>0, $$ and density zero otherwise. Here $\Gamma(\alpha)=\int_0^\infty u^{\alpha-1}\exp(-u)\,du$. For independent observations $X_1,\ldots,X_n$ with this distribution, the joint density is the product of their individual densities on the fixed support $(0,\infty)^n$.

A statistic $T(X_1,\ldots,X_n)$ is sufficient for a parameter if, conditional on $T$, the distribution of the observed sample does not depend on that parameter. We use the likelihood factorization criterion: for a dominated model with support not depending on the parameter, a statistic is sufficient if the joint density can be written as $g_\eta(T(x))h(x)$, where all parameter dependence is in $g_\eta$, and $h$ does not depend on the parameter. The common support and the parameter-free factor are important parts of this criterion.

Fixed Shape, Unknown Scale

Suppose $X_1,\ldots,X_n$ are independent gamma observations with known shape $a>0$ and unknown scale $\theta>0$. Write the observed sample as $x_1,\ldots,x_n$, all positive. Independence gives the likelihood

L(\theta;x_1,\ldots,x_n) =\prod_{i=1}^n\frac{x_i^{a-1}\exp(-x_i/\theta)}{\Gamma(a)\theta^a} =\frac{1}{\Gamma(a)^n}\theta^{-na} \exp\!\left(-\frac{\sum_{i=1}^n x_i}{\theta}\right) \prod_{i=1}^n x_i^{a-1}.

The parameter-dependent terms involve the data only through $S=\sum_{i=1}^n x_i$. The remaining product depends on the data and the known shape, but not on $\theta$.

Theorem: For independent gamma observations with fixed known shape $a>0$ and unknown scale $\theta>0$, the sample sum $S=\sum_{i=1}^n X_i$ is sufficient for $\theta$.

Proof. On the common support $x_i>0$, factor the joint density as

f_\theta(x_1,\ldots,x_n) = \underbrace{\frac{\theta^{-na}}{\Gamma(a)^n} \exp\!\left(-\frac{S}{\theta}\right)}_{g_\theta(S)} \underbrace{\prod_{i=1}^n x_i^{a-1}}_{h(x_1,\ldots,x_n)}.

The first factor depends on the sample only through $S$, and the second factor does not depend on $\theta$. The support $(0,\infty)^n$ is the same for every $\theta>0$. The likelihood factorization criterion therefore gives sufficiency of $S$ for $\theta$. $\square$

Worked Example: Fixed Shape and Two Samples with the Same Sum

Let $n=4$, fix the shape at $a=2$, and leave the scale $\theta$ unknown. Compare the samples $(1,2,3,4)$ and $(1,1,4,4)$. Both have sum $S=10$. For shape 2, $\Gamma(2)=1$, and the likelihood is

L(\theta;x)=\theta^{-8}\exp(-S/\theta)\prod_{i=1}^4x_i.

For the first sample, the product is $1\cdot2\cdot3\cdot4=24$, so its likelihood is $24\theta^{-8}\exp(-10/\theta)$. For the second, the product is $1\cdot1\cdot4\cdot4=16$, giving $16\theta^{-8}\exp(-10/\theta)$. Their likelihood ratio is $24/16=3/2$, independent of $\theta$. Thus the differing products change only a parameter-free factor; the scale-dependent part is determined by the common sum.

Fixed Scale, Unknown Shape

Now suppose the scale is known to be $b>0$, while the shape $\alpha>0$ is unknown. The likelihood is

L(\alpha;x_1,\ldots,x_n) =\prod_{i=1}^n\frac{x_i^{\alpha-1}\exp(-x_i/b)}{\Gamma(\alpha)b^\alpha} =\frac{b^{-n\alpha}}{\Gamma(\alpha)^n} \exp\!\left((\alpha-1)\sum_{i=1}^n\log x_i\right) \exp\!\left(-\frac{\sum_{i=1}^n x_i}{b}\right).

Here the data-dependent quantity inside the shape-dependent exponential is $T=\sum_{i=1}^n\log x_i$. The factor involving $\sum x_i$ does not depend on $\alpha$, since $b$ is fixed. The sample sum is therefore not the summary identified by this factorization for the unknown shape.

Theorem: For independent gamma observations with fixed known scale $b>0$ and unknown shape $\alpha>0$, the statistic $T=\sum_{i=1}^n\log X_i$ is sufficient for $\alpha$. Equivalently, the product $\prod_{i=1}^n X_i$ is sufficient, since it is $\exp(T)$.

Proof. Regroup the joint density as

f_\alpha(x_1,\ldots,x_n) = \underbrace{\frac{b^{-n\alpha}}{\Gamma(\alpha)^n} \exp\!\left((\alpha-1)T\right)}_{g_\alpha(T)} \underbrace{\exp\!\left(-\frac{\sum_{i=1}^n x_i}{b}\right)}_{h(x_1,\ldots,x_n)}.

The factor $g_\alpha$ depends on the observations only through $T$; $h$ has no dependence on $\alpha$. The support is the fixed set $(0,\infty)^n$ for all $\alpha>0$. The factorization criterion proves that $T$ is sufficient. Because the exponential map is one-to-one, $T$ and $\exp(T)=\prod_i X_i$ determine one another, so the product is also sufficient. $\square$

Worked Example: Computing the Shape Summary

Suppose $n=3$, the scale is known to be $b=2$, and the observations are $1,2,8$. The relevant summary for the unknown shape is

T=\log 1+\log 2+\log 8=\log 16.

Substitution into the likelihood gives

L(\alpha) =\frac{2^{-3\alpha}}{\Gamma(\alpha)^3} \exp\!\left((\alpha-1)\log 16\right) \exp(-11/2),

because the observations sum to $1+2+8=11$. The last factor, $\exp(-11/2)$, is constant with respect to $\alpha$. The shape-dependent data term is $\log 16$, or equivalently the product $16$. In particular, the sum $11$ appears only in a factor that does not involve the unknown shape.

Reading the Factorization Correctly

A reliable way to identify the sufficient statistic is to write the full likelihood before deciding what to retain. Then group all terms involving the unknown parameter and inspect which function of the data appears in those terms. The two cases can be compared directly:

Known quantity	Unknown quantity	Sufficient summary from the factorization
Shape $a$	Scale $\theta$	$\sum_i X_i$
Scale $b$	Shape $\alpha$	$\sum_i\log X_i$, equivalently $\prod_i X_i$

Worked Example: One Sample, Two Different Summaries

Take observations $x_1=1/2,\ x_2=1,\ x_3=2$. Their sum is $1/2+1+2=7/2$, while their product is $(1/2)\cdot1\cdot2=1$, so their sum of logarithms is $\log(1/2)+\log 1+\log 2=0$.

If the shape is fixed at $a=3$ and the scale is unknown, the scale-dependent likelihood factor is $\theta^{-9}\exp(-(7/2)/\theta)$. The relevant summary is $7/2$. If instead the scale is fixed at $b=2$ and the shape is unknown, the shape-dependent likelihood factor is $2^{-3\alpha}\Gamma(\alpha)^{-3}\exp(0)$. The data-dependent shape summary is $0$, or product $1$. The same observations produce different sufficient summaries because the unknown parameter has changed.

Common Mistakes and Why Sufficiency Matters

Using the wrong gamma convention. With the shape-scale density used here, the exponential term is $\exp(-x/\theta)$. Replacing it with $\exp(-\theta x)$ changes the parameterization and the likelihood.
Always choosing the sample sum. The sum is the summary for unknown scale with fixed shape. With fixed scale and unknown shape, the parameter-dependent power of each $x_i$ produces $\sum_i\log x_i$.
Keeping a parameter-free term as if it changed sufficiency. For fixed shape, $\prod_i x_i^{a-1}$ is part of the likelihood, but it does not depend on the unknown scale. For fixed scale, $\exp(-\sum_i x_i/b)$ does not depend on the unknown shape. Such terms belong in $h(x)$.
Confusing sufficient with minimal sufficient. The factorizations establish sufficiency. They do not, by themselves, establish that no smaller sufficient summary exists; that is a separate question.

Sufficiency identifies which features of the sample carry the likelihood's dependence on the parameter in these models. It also provides a practical check on calculations: if the shape is fixed, the scale-dependent part must involve the sample sum; if the scale is fixed, the shape-dependent part must involve the sum of logs. A mismatch usually signals an algebra or parameterization error.

Check Your Understanding

For each question, start from the gamma density with shape-scale parameterization and identify which terms depend on the unknown parameter.

For independent gamma observations with known shape $a$ and unknown scale $\theta$, write the likelihood and identify its sufficient statistic.
With known scale $b$ and unknown shape $\alpha$, why does $\sum_i X_i$ occur only in a parameter-free factor?
For observations $2,3,6$, compute both the sample sum and the sum of logarithms. Which is the sufficient summary when shape is fixed, and which when scale is fixed?
Explain why the product of observations and the sum of their logarithms contain equivalent information for the fixed-scale, unknown-shape model.
What role does the common support $(0,\infty)^n$ play when applying the likelihood factorization criterion?

Known quantity	Unknown quantity	Sufficient summary from the factorization
Shape \(a\)	Scale \(\theta\)	\(\sum_i X_i\)
Scale \(b\)	Shape \(\alpha\)	\(\sum_i\log X_i\), equivalently \(\prod_i X_i\)