Sufficiency Under Reparameterization

Why Reparameterization Matters

A statistical model can often be described using different parameter labels. A Bernoulli probability \(p\), for example, can be replaced by its odds or its log-odds. The numerical parameter changes, but the probability distributions available to the model do not. A statistic that retains all the information in the data about the model should not stop being sufficient merely because we changed labels.

The key condition is that the transformation be one-to-one. If two distinct parameter values are mapped to the same new value, the new parameter may fail to identify a unique distribution. In that case, the question is not just whether a sufficient statistic is preserved: first we must ask whether the proposed transformed parameter describes a well-defined model at all.

Definition: Let \(X\) have a distribution \(P_\theta\) in a model \(\{P_\theta:\theta\in\Theta\}\), and let \(T=T(X)\) be a statistic. The statistic \(T\) is sufficient for the model if the conditional distribution of \(X\) given \(T\) does not depend on \(\theta\). Equivalently, once \(T\) is observed, the remaining details of \(X\) have a conditional distribution that is the same for every parameter value. For a dominated model, the likelihood factorization criterion gives a corresponding density-based characterization.

This definition treats sufficiency as a property of the family of distributions, not as a property of a particular symbol used for the parameter. We will assume the sample space is a standard Borel space, so the conditional distributions in this definition exist. In the likelihood calculations below, we use a common dominating measure and factor the density into a parameter-dependent function of \(T(x)\) and a parameter-free function of the data.

One-to-One Transformations Preserve Sufficiency

Suppose \(\eta=g(\theta)\), where \(g\) is a bijection from \(\Theta\) onto a new parameter space \(H\). The distribution indexed by \(\eta\) is defined by \(Q_\eta=P_{g^{-1}(\eta)}\). This is simply the original family with its labels changed: every original distribution appears exactly once in the new indexing.

Theorem: If \(g:\Theta\to H\) is a bijection, a statistic \(T\) is sufficient for \(\{P_\theta:\theta\in\Theta\}\) if and only if it is sufficient for \(\{Q_\eta:\eta\in H\}\), where \(Q_\eta=P_{g^{-1}(\eta)}\).

Proof. For any \(\eta\in H\), the distribution \(Q_\eta\) is exactly \(P_{g^{-1}(\eta)}\). If the conditional distribution of \(X\) given \(T\) is the same for every \(\theta\), it is therefore the same for every \(\eta\). This proves sufficiency under the new labels. Conversely, for any \(\theta\in\Theta\), set \(\eta=g(\theta)\). The conditional distribution under \(P_\theta\) is the one under \(Q_\eta\); if it is independent of \(\eta\), it is independent of \(\theta\). Thus sufficiency holds in both parameterizations. \(\square\)

The same argument is visible in the likelihood factorization criterion. If a density factors as \(f_\theta(x)=a_\theta(T(x))h(x)\), then the reparameterized density is \(q_\eta(x)=a_{g^{-1}(\eta)}(T(x))h(x)\), which is still a factorization through \(T\). The converse follows by substituting \(\eta=g(\theta)\) into a factorization in the new parameter. This algebra is useful in examples, but the conditional-distribution argument makes clear why the result does not depend on a particular choice of density.

Worked Example: Bernoulli Data in Log-Odds Form

Let \(X_1,\ldots,X_n\) be independent Bernoulli observations with success probability \(p\), where \(0<p<1\). The total number of successes is \(S=\sum_{i=1}^n X_i\). Replace \(p\) by the log-odds \(\eta=\log\{p/(1-p)\}\).

Check the transformation. Its inverse is \(p=e^\eta/(1+e^\eta)\), defined for every real \(\eta\). Thus the map from \(p\in(0,1)\) to \(\eta\in\mathbb R\) is one-to-one and onto.

Write the likelihood. For observed values \(x_1,\ldots,x_n\), each in \(\{0,1\}\), independence gives

L(p;x)=\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} =p^{s}(1-p)^{n-s},\qquad s=\sum_{i=1}^n x_i.

Substitute the new parameter. Since \(p=e^\eta/(1+e^\eta)\) and \(1-p=1/(1+e^\eta)\),

L(\eta;x) =\left(\frac{e^\eta}{1+e^\eta}\right)^s \left(\frac{1}{1+e^\eta}\right)^{n-s} =\frac{e^{\eta s}}{(1+e^\eta)^n}.

The parameter-dependent part involves the data only through \(s\). The remaining factor is \(1\), which has no parameter dependence. Hence the likelihood factors through \(S\), so \(S\) is sufficient in the log-odds parameterization, just as it is in the probability parameterization.

When a Many-to-One Map Is Valid

Now let \(g:\Theta\to H\) be many-to-one. The value \(\eta=g(\theta)\) groups together all parameter values in the same fiber, the set \(\{\theta:g(\theta)=\eta\}\). To define a distribution \(Q_\eta\) using \(\eta\) alone, every value of \(\theta\) in that fiber must give the same distribution.

Theorem: A family \(Q_\eta=P_\theta\), with \(\eta=g(\theta)\), is well-defined as a model indexed only by \(\eta\) if and only if \(P_\theta=P_{\theta'}\) whenever \(g(\theta)=g(\theta')\). When this condition holds, the transformed family contains exactly the same distributions as the original family. Consequently, a statistic is sufficient for one family if and only if it is sufficient for the other.

Proof. First suppose \(Q_\eta\) is well-defined. If \(g(\theta)=g(\theta')=\eta\), both \(P_\theta\) and \(P_{\theta'}\) must equal the single distribution \(Q_\eta\), so \(P_\theta=P_{\theta'}\). For the converse, suppose distributions agree whenever parameter values share the same image. For each \(\eta\) in the image of \(g\), choose any \(\theta\) with \(g(\theta)=\eta\) and set \(Q_\eta=P_\theta\). The assumption ensures that this choice does not affect \(Q_\eta\), so the new family is well-defined. Every \(Q_\eta\) is an original distribution, and every original distribution equals \(Q_{g(\theta)}\); thus the two families contain the same distributions. Sufficiency depends on this family of distributions, so it is unchanged. \(\square\)

This result separates a genuine collapse of redundant labels from loss of information about which distribution applies. In the valid case, distinct parameter values were only different labels for the same distribution. In the invalid case, the proposed new parameter assigns one label to multiple distributions.

Worked Example: A Valid Many-to-One Map for a Normal Model

Suppose \(X_1,\ldots,X_n\) are independent with \(X_i\sim N(\theta^2,\sigma^2)\), where \(\theta\in\mathbb R\) and the variance \(\sigma^2>0\) is known. Let \(\eta=\theta^2\), so \(\eta\geq 0\).

The map is many-to-one because \(\theta\) and \(-\theta\) have the same square. But both values give the same normal distribution, \(N(\theta^2,\sigma^2)\). Thus \(\eta\) does index a well-defined family.

To identify a sufficient statistic, write the joint density in terms of \(\eta\):

\begin{aligned} f_\eta(x) &=(2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\eta)^2\right\}\\ &=(2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{\sum_{i=1}^n x_i^2}{2\sigma^2}\right\} \exp\left\{\frac{\eta\sum_{i=1}^n x_i}{\sigma^2} -\frac{n\eta^2}{2\sigma^2}\right\}. \end{aligned}

The first exponential involving the observations depends on the data but not on \(\eta\). The other parameter-dependent factor uses the data only through \(\sum_i X_i\). Therefore the likelihood factors through \(\sum_i X_i\), which is sufficient for this model. Because the \(\theta\)-indexed and \(\eta\)-indexed models contain the same distributions, this statistic is sufficient under either labeling.

Worked Example: A Many-to-One Map That Does Not Define a New Parameterization

Let \(\theta\in\{-1,1\}\), and let \(X_1,\ldots,X_n\) be independent Bernoulli observations with success probability \(p_\theta=(2+\theta)/4\). Thus \(p_{-1}=1/4\) and \(p_1=3/4\). Consider \(\eta=\theta^2\).

Both values of \(\theta\) map to \(\eta=1\), but the distributions are not the same. In particular, \(P_{\theta=-1}(X_1=1)=1/4\), whereas \(P_{\theta=1}(X_1=1)=3/4\). A model indexed only by \(\eta=1\) would need to assign one distribution to this value, but there are two different distributions to assign. Therefore \(\eta\) alone does not parameterize this model.

For the original model, the likelihood is

L(\theta;x)=p_\theta^s(1-p_\theta)^{n-s},\qquad s=\sum_{i=1}^n x_i.

It depends on the observations only through \(s\), so the likelihood factorization criterion shows that \(S=\sum_i X_i\) is sufficient for the original model. But it would be incorrect to call this a reparameterization by \(\eta=\theta^2\): the transformed label does not determine the distribution. Additional information that distinguishes the two values of \(\theta\) would be needed to specify the original model.

Common Mistake and Practical Check

A common mistake is to argue, “The transformed likelihood can be written using \(\eta=g(\theta)\), so sufficiency is automatically preserved,” even when \(g\) is many-to-one. This skips the necessary check that one \(\eta\) corresponds to one distribution. In the Bernoulli example, the same \(\eta=1\) corresponds to success probabilities \(1/4\) and \(3/4\), so there is no single likelihood indexed by \(\eta\) alone.

For a one-to-one transformation, verify the inverse and substitute it into the likelihood or use the conditional-distribution definition. For a many-to-one map, compare distributions within each fiber before proceeding. If all distributions agree within each fiber, the map removes redundant labels and sufficiency is unchanged. If they do not agree, it is not a valid reparameterization of that model.

Identify the map.
Specify the original parameter space, the transformed parameter, and the image of the transformation.

Check injectivity.
If the map is bijective onto its image, the new parameter is a relabeling and sufficiency is preserved in both directions.

Check distributions within fibers.
If the map is many-to-one, verify that all parameter values with the same transformed value give the same distribution.

Apply the factorization criterion if useful.
After the model is properly indexed, check whether the likelihood depends on the data through the proposed statistic.

Sufficiency concerns how the distributions vary across the model. A bijective transformation changes only the coordinate used to label those distributions. A valid many-to-one transformation merges labels that were already distributionally indistinguishable. Neither operation changes the model's sufficient statistics; an invalid collapse, however, does not define the claimed transformed model.

Check Your Understanding

Answer each question using the distinction between parameter labels and probability distributions.

Why does a bijection between parameter spaces preserve sufficiency in both directions?
For Bernoulli observations with \(0<p<1\), explain why the log-odds is a one-to-one parameterization of \(p\).
What condition must hold among parameter values in the same fiber of a many-to-one map for the transformed model to be well-defined?
In the normal model \(X_i\sim N(\theta^2,\sigma^2)\), why is \(\eta=\theta^2\) a valid transformed parameter despite not being one-to-one?
In the two-point Bernoulli example, why does \(\eta=\theta^2\) fail to index a unique distribution?

Sufficiency Under Reparameterization

What You'll Learn