Reparameterization Invariance of Maximum Likelihood

Why Reparameterization Should Not Change the Estimate

A statistical model can describe the same probability distributions using different coordinates. A Gaussian covariance matrix, for example, can be expressed directly as a covariance matrix or through its inverse, the precision matrix. These coordinates may make different calculations convenient, but they do not change which probability distribution best fits the observed data.

The maximum-likelihood principle formalizes this idea. When a change of coordinates is one-to-one and covers the same parameter space, maximizing the likelihood in the new coordinates must identify the same model point. The numerical estimate changes its representation, not the distribution it represents.

Definition: Let a model have parameter $\theta\in\Theta$ and likelihood $L(\theta)$ for observed data. A reparameterization is a map $\eta=g(\theta)$ that gives coordinates $\eta\in H$ for the same model. When $g:\Theta\to H$ is a bijection, define the likelihood in the new coordinates by $$ L_{\eta}(\eta)=L\bigl(g^{-1}(\eta)\bigr),\qquad \eta\in H. $$ An MLE is a parameter value at which the likelihood attains its global maximum. If more than one value attains the maximum, the set of all such values is the MLE set.

For the result below, the key condition is that the reparameterization is a bijection between the original and new parameter spaces. A smooth map with a smooth inverse (a diffeomorphism) is a common, convenient case. Smoothness is useful for relating derivatives, but it is not needed to establish the basic invariance of a global maximum.

The Invariance Result

Theorem: Suppose $g:\Theta\to H$ is a bijection and $L_{\eta}(\eta)=L(g^{-1}(\eta))$. Then the MLE set in the new coordinates is exactly the image under $g$ of the MLE set in the original coordinates: $$ \operatorname*{arg\,max}_{\eta\in H} L_{\eta}(\eta) = g\!\left(\operatorname*{arg\,max}_{\theta\in\Theta}L(\theta)\right). $$ In particular, if a unique MLE $\hat\theta$ exists, the unique new-coordinate MLE is $\hat\eta=g(\hat\theta)$.

Proof. Let $A=\operatorname*{arg\,max}_{\theta\in\Theta}L(\theta)$. First take any $\theta^*\in A$, and set $\eta^*=g(\theta^*)$. For every $\eta\in H$, bijectivity gives a $\theta=g^{-1}(\eta)\in\Theta$. Since $\theta^*$ maximizes $L$, $$ L_{\eta}(\eta^*)=L(\theta^*)\geq L(\theta)=L_{\eta}(\eta). $$ Thus $\eta^*$ maximizes $L_{\eta}$, so $g(A)$ is contained in the new MLE set.

Conversely, let $\eta^*$ maximize $L_{\eta}$ and put $\theta^*=g^{-1}(\eta^*)$. For every $\theta\in\Theta$, $g(\theta)\in H$, and therefore $$ L(\theta^*)=L_{\eta}(\eta^*)\geq L_{\eta}(g(\theta))=L(\theta). $$ Hence $\theta^*\in A$, which shows that every new-coordinate maximizer belongs to $g(A)$. The two sets are equal. If $A$ contains one point, its image also contains one point because $g$ is one-to-one. $\square$

This argument is about global maximizers and compares the same likelihood values at corresponding model points. If the maximum is not attained, neither coordinate system creates an MLE: bijectivity preserves the supremum, but does not turn a supremum into an attained maximum.

A Scalar Example: Variance and Log-Variance

Worked Example: Reparameterizing a Gaussian Variance

Let $X_1,\ldots,X_4$ be independent $N(0,v)$ observations, where the mean is known to be zero and $v>0$ is the variance. Suppose the observed values are $1,-1,2,-2$. Find the MLE of $v$, then re-express it as the MLE of $\tau=\log v$.

The sum of squares is $$ \sum_{i=1}^4 x_i^2=1+1+4+4=10. $$ Ignoring terms that do not depend on $v$, the log-likelihood is $$ \ell(v)=-\frac{4}{2}\log v-\frac{10}{2v} =-2\log v-\frac{5}{v}. $$ Its derivative is $$ \frac{d\ell}{dv}=-\frac{2}{v}+\frac{5}{v^2} =\frac{5-2v}{v^2}. $$ The derivative is positive for $0<v<5/2$ and negative for $v>5/2$, so the global maximum occurs at $\hat v=5/2$.

The map $g(v)=\log v$ is a bijection from $(0,\infty)$ to $\mathbb R$, with inverse $v=e^\tau$. The new log-likelihood is the original likelihood composed with this inverse: $$ \ell_{\tau}(\tau)=\ell(e^\tau)=-2\tau-5e^{-\tau}. $$ Its derivative is $-2+5e^{-\tau}$, which vanishes at $\hat\tau=\log(5/2)$. Its second derivative is $-5e^{-\tau}<0$, confirming a maximum. As predicted, $\hat\tau=\log\hat v$; the fitted variance remains $e^{\hat\tau}=5/2$.

Covariance and Precision Coordinates

For a multivariate Gaussian model, covariance and precision provide a direct application. A covariance matrix $\Sigma$ is symmetric positive definite, and its precision matrix is $\Omega=\Sigma^{-1}$, also symmetric positive definite. Matrix inversion is a smooth bijection on this space, and applying it twice returns the original matrix.

Theorem: Let $X_1,\ldots,X_n$ be independent $N_p(\mu,\Sigma)$ observations, with $\mu$ known. Define $$ S=\frac{1}{n}\sum_{i=1}^n (X_i-\mu)(X_i-\mu)^{\mathsf T}. $$ If $S$ is positive definite, the MLEs over the positive-definite covariance and precision spaces are $$ \hat\Sigma=S,\qquad \hat\Omega=S^{-1}. $$

Proof. Up to a term constant in $\Sigma$, the Gaussian log-likelihood is $$ \ell(\Sigma)=-\frac{n}{2}\log|\Sigma|-\frac{n}{2}\operatorname{tr}(\Sigma^{-1}S). $$ Set $\Omega=\Sigma^{-1}$. Since $\log|\Sigma|=-\log|\Omega|$, the log-likelihood in precision coordinates is $$ \ell_{\Omega}(\Omega)=\frac{n}{2}\log|\Omega|-\frac{n}{2}\operatorname{tr}(\Omega S). $$ To find its global maximum, form the positive-definite matrix $B=S^{1/2}\Omega S^{1/2}$. Then $$ \log|\Omega|=\log|B|-\log|S|,\qquad \operatorname{tr}(\Omega S)=\operatorname{tr}(B). $$ If $\lambda_1,\ldots,\lambda_p$ are the positive eigenvalues of $B$, the part of the log-likelihood depending on $B$ is proportional to $$ \log|B|-\operatorname{tr}(B)=\sum_{j=1}^{p}(\log\lambda_j-\lambda_j). $$ For every $\lambda>0$, the function $\log\lambda-\lambda$ has derivative $1/\lambda-1$, which is positive below $1$ and negative above $1$. It is therefore uniquely maximized at $\lambda=1$. The sum is uniquely maximized when every eigenvalue equals $1$, which for a symmetric positive-definite $B$ means $B=I$. Thus $S^{1/2}\Omega S^{1/2}=I$, so $\hat\Omega=S^{-1}$, and consequently $\hat\Sigma=S$. $\square$

Worked Example: Transforming a Covariance MLE to Precision

Suppose the known mean is zero and the four observed vectors are $(1,0)^{\mathsf T},(-1,0)^{\mathsf T},(0,2)^{\mathsf T},(0,-2)^{\mathsf T}$. Calculate the covariance and precision MLEs.

The outer products are $$ \begin{pmatrix}1&0\\0&0\end{pmatrix},\quad \begin{pmatrix}1&0\\0&0\end{pmatrix},\quad \begin{pmatrix}0&0\\0&4\end{pmatrix},\quad \begin{pmatrix}0&0\\0&4\end{pmatrix}. $$ Their sum, divided by $n=4$, gives $$ S=\begin{pmatrix}1/2&0\\0&2\end{pmatrix}. $$ Both diagonal entries are positive, so $S$ is positive definite and the covariance MLE exists. The theorem gives $$ \hat\Sigma=\begin{pmatrix}1/2&0\\0&2\end{pmatrix}, \qquad \hat\Omega=\hat\Sigma^{-1} =\begin{pmatrix}2&0\\0&1/2\end{pmatrix}. $$ In particular, the precision estimate is the inverse of the covariance estimate, not a separate maximizer representing a different fitted Gaussian distribution.

Standard Deviations and Correlation

For a bivariate Gaussian model, another useful coordinate system consists of the marginal standard deviations $s_1,s_2$ and correlation $\rho$. The correspondence with covariance coordinates is $$ \Sigma= \begin{pmatrix} s_1^2 & \rho s_1s_2\\ \rho s_1s_2 & s_2^2 \end{pmatrix}, \qquad s_1>0,\quad s_2>0,\quad -1<\rho<1. $$ The inverse map is $s_1=\sqrt{\Sigma_{11}}$, $s_2=\sqrt{\Sigma_{22}}$, and $\rho=\Sigma_{12}/(s_1s_2)$. Positive definiteness implies these coordinates are in the stated ranges, so the correspondence is a smooth bijection.

Worked Example: Reading Standard Deviations and Correlation From an MLE

With known mean zero, consider the four observed vectors $(1,1)^{\mathsf T},(-1,-1)^{\mathsf T},(2,-1)^{\mathsf T},(-2,1)^{\mathsf T}$. Find the MLE in covariance coordinates and then in standard-deviation and correlation coordinates.

The first diagonal entry of $S$ is $(1+1+4+4)/4=5/2$. The second is $(1+1+1+1)/4=1$. The off-diagonal entry is $(1+1-2-2)/4=-1/2$. Hence $$ S=\begin{pmatrix}5/2&-1/2\\-1/2&1\end{pmatrix}, \qquad |S|=\frac{5}{2}-\frac{1}{4}=\frac{9}{4}>0. $$ Its first diagonal entry and determinant are positive, so this symmetric matrix is positive definite. The covariance MLE is therefore $\hat\Sigma=S$. Transforming its entries gives $$ \hat s_1=\sqrt{5/2},\qquad \hat s_2=1,\qquad \hat\rho=\frac{-1/2}{\sqrt{5/2}\,(1)}=-\frac{1}{\sqrt{10}}. $$ The transformed values satisfy $\hat s_1>0$, $\hat s_2>0$, and $|\hat\rho|<1$. Substitution into the covariance-coordinate formula recovers $S$, verifying that the two estimates represent the same fitted covariance matrix.

Derivatives and a Common Pitfall

When the reparameterization is a diffeomorphism and the log-likelihood is differentiable, the score equations also transform consistently. Write $\theta=g^{-1}(\eta)$, and let $Dg^{-1}(\eta)$ denote the derivative matrix of the inverse map. The chain rule gives $$ \nabla_{\eta}\ell_{\eta}(\eta) = Dg^{-1}(\eta)^{\mathsf T}\nabla_{\theta}\ell(\theta). $$ The derivative matrix is nonsingular for a diffeomorphism. Thus a point has zero score in one coordinate system exactly when its corresponding point has zero score in the other. This observation concerns stationary points; the global-maximizer result above does not depend on differentiability.

A common mistake is to multiply the likelihood by the Jacobian of the parameter transformation. That Jacobian is used when transforming a density for a random variable, or a prior density in a Bayesian calculation. Here the likelihood is the data likelihood as a function of the parameter, so the new likelihood is just $L(g^{-1}(\eta))$.

Worked Example: Why a Parameter Jacobian Gives the Wrong MLE

Return to the variance example, where the correct likelihood as a function of $v$ is proportional to $v^{-2}\exp(-5/v)$. Under $\tau=\log v$, the correct likelihood is proportional to $e^{-2\tau}\exp(-5e^{-\tau})$, which is the original likelihood evaluated at $v=e^\tau$. Its maximum corresponds to $v=5/2$.

If one incorrectly multiplies by the transformation Jacobian $dv/d\tau=e^\tau=v$, the expression becomes proportional to $v^{-1}\exp(-5/v)$. Its log, apart from a constant, is $-\log v-5/v$, whose derivative is $-1/v+5/v^2$. Setting this to zero gives $v=5$, not the MLE $5/2$. The extra factor changed the function being maximized; it treated the likelihood as though it were a density over parameter values.

Bijectivity matters as well. If a map omits some values of the original parameter space, maximizing over its image can impose a restriction and change the estimate. If it is many-to-one, a new-coordinate likelihood is well-defined only when the original likelihood assigns the same value to every parameter with the same new coordinate. Checking the parameter space and the meaning of the transformed likelihood is therefore part of using invariance correctly.

Key takeaway: Under a bijective reparameterization, compose the original likelihood with the inverse map; do not add a Jacobian. The MLE set maps exactly through the reparameterization, so covariance, precision, and valid standard-deviation/correlation coordinates describe the same maximizing model.

Check Your Understanding

Use the definitions and results above to answer the following questions.

What condition on a reparameterization guarantees that the MLE sets correspond exactly, even if the maximum is not unique?
For $v>0$ and $\tau=\log v$, express the MLE of $\tau$ in terms of the MLE of $v$.
Why does a positive-definite $S$ matter in the covariance-to-precision calculation?
For a bivariate covariance matrix, how is the correlation recovered from its entries?
Why is a parameter-transformation Jacobian not included in the reparameterized likelihood?