Tutorials › Econometrics (Graduate) › Parameterizing Nonlinear Likelihoods

Home  /  Tutorials  /  Nonlinear Likelihood Estimation

Nonlinear Likelihood Estimation · Tutorial 434 of 952

Parameterizing Nonlinear Likelihoods

Learn to choose and check parameter transformations that enforce a model’s constraints without changing its likelihood maximization problem.

Intermediate 10 min read

What You'll Learn

  • Distinguish a model’s original parameter space from the space of its transformed parameters.
  • Use an exponential transformation to represent a strictly positive scale parameter.
  • Use the logit transformation to represent probabilities in the open unit interval.
  • Translate likelihoods into transformed parameters while preserving their values.
  • Identify when a transformation excludes boundary values and may prevent a finite transformed estimate.
  • Check whether a proposed parameterization is one-to-one and covers the intended parameter space.

Why Parameterize a Likelihood?

Likelihood models often involve parameters that cannot take arbitrary real values. A scale may need to be strictly positive, while a probability must be between zero and one, inclusive. A likelihood written directly in these parameters has a constrained domain. One way to handle the restrictions is to express the original parameters as functions of new, often unrestricted, parameters.

A transformation can make a restriction automatic, but it also changes the domain on which the likelihood is written. The key questions are therefore not only whether the transformed parameter satisfies the restriction, but also whether the transformation represents every value allowed by the original model. That distinction matters for likelihood maximization, especially when a maximum lies at a boundary.

Definition: Suppose a model is indexed by a parameter \(\theta\) in a parameter space \(\Theta\). A reparameterization writes \(\theta=g(\eta)\), where \(\eta\) belongs to a new space \(H\). The likelihood in the new parameter is \(L_\eta(\eta)=L_\theta(g(\eta))\). The transformed parameterization represents the original model exactly when the image of \(g\) is all of \(\Theta\). If \(g\) is also one-to-one, each original parameter value has exactly one transformed representation.

For a nondegenerate model, a standard deviation parameter \(\sigma\) is strictly positive, so the exponential transformation \(\sigma=\exp(\eta)\) is convenient. In general, a standard deviation is nonnegative: a constant random variable can have standard deviation zero. A probability is in the closed interval from zero to one, inclusive; a logit transformation represents only the open interval between them for finite transformed values.

What a Reparameterization Changes

A likelihood is a function of the parameter for the observed data. Substituting \(\theta=g(\eta)\) changes the argument of that function; it does not change the observed data or multiply the likelihood by a Jacobian. A Jacobian is relevant when transforming a probability density with respect to a variable, not simply when expressing the same likelihood using a different parameter.

If \(g\) is one-to-one and maps \(H\) onto \(\Theta\), maximizing the likelihood over \(H\) gives the same maximum likelihood value as maximizing over \(\Theta\). The maximizing values correspond through \(\theta=g(\eta)\). The reason is that the two maximizations evaluate the same likelihood values over exactly the same set of model parameter values.

$$ \sup_{\eta\in H} L_\eta(\eta) = \sup_{\eta\in H} L_\theta(g(\eta)) = \sup_{\theta\in\Theta} L_\theta(\theta). $$

If the transformation is not onto, the transformed model covers only part of the original parameter space. Its maximum can then differ from the original maximum, or the original maximizing value may be approached but never attained at a finite transformed parameter. This is a domain issue, not a numerical optimization issue.

Worked Example: A Positive Normal Scale

Worked Example: A Positive Normal Scale

Suppose three hypothetical observations \(x_1=1\), \(x_2=-1\), and \(x_3=2\) are independent and identically distributed as \(N(0,\sigma^2)\), with the mean fixed at zero and the nondegenerate scale parameter restricted to \(\sigma>0\). The normal model and independence assumption give the likelihood

$$ L_\sigma(\sigma) = (2\pi\sigma^2)^{-3/2} \exp\left(-\frac{1^2+(-1)^2+2^2}{2\sigma^2}\right), \qquad \sigma>0. $$

The sum of squared observations is \(1+1+4=6\). Set \(\sigma=\exp(\eta)\), where \(\eta\) can be any real number. Since the exponential maps the real line one-to-one onto the strictly positive numbers, this transformation represents every allowed scale value. Substitution gives

$$ L_\eta(\eta) = (2\pi)^{-3/2}\exp(-3\eta) \exp\left(-3\exp(-2\eta)\right), \qquad -\infty<\eta<\infty. $$

For example, at \(\sigma=\sqrt{2}\), the transformed value is \(\eta=\log(\sqrt{2})\approx0.3466\). The log likelihood in the original parameter is

$$ \ell_\sigma(\sqrt{2}) = -\frac{3}{2}\log(2\pi)-3\log(\sqrt{2})-\frac{6}{2(\sqrt{2})^2} \approx -5.297. $$

Using \(\eta\) gives the same value:

$$ \ell_\eta(0.3466) = -\frac{3}{2}\log(2\pi)-3(0.3466)-3\exp(-2(0.3466)) \approx -5.297. $$

The transformation enforces a positive scale while preserving the likelihood values. It does not assert that every standard deviation in every model must be positive: here positivity follows from the specified nondegenerate normal model.

Probabilities and the Logit Transformation

For a probability parameter \(p\), the logit transformation uses an unrestricted real parameter \(\gamma\):

$$ p=\frac{\exp(\gamma)}{1+\exp(\gamma)}, \qquad \gamma=\log\left(\frac{p}{1-p}\right). $$

Every finite real \(\gamma\) yields \(0<p<1\), and every probability strictly between zero and one has a unique finite \(\gamma\). The inverse expression is defined only for interior probabilities. Thus the logit parameterization is a one-to-one representation of the open interval, not of the closed interval \([0,1]\).

Worked Example: A Binomial Probability in Logit Form

Suppose a hypothetical sequence of ten independent Bernoulli trials has seven successes and three failures, with a common success probability \(p\). For the observed success count \(y=7\), the binomial likelihood is

$$ L_p(p)=\binom{10}{7}p^7(1-p)^3, \qquad 0\leq p\leq 1. $$

The binomial model assumes independent trials with the same success probability. Substituting the logit parameterization gives

$$ L_\gamma(\gamma) = \binom{10}{7} \left(\frac{\exp(\gamma)}{1+\exp(\gamma)}\right)^7 \left(\frac{1}{1+\exp(\gamma)}\right)^3 = \binom{10}{7}\frac{\exp(7\gamma)}{(1+\exp(\gamma))^{10}}. $$

At \(\gamma=0\), \(p=1/2\), so

$$ L_\gamma(0)=\binom{10}{7}(0.5)^{10} =120(0.0009765625) =0.1172. $$

At \(\gamma=\log(7/3)\approx0.8473\), \(p=0.7\). The likelihood is

$$ L_\gamma(\log(7/3)) = 120(0.7)^7(0.3)^3 \approx 0.2668. $$

For this data, the likelihood is maximized at \(p=0.7\). In the original parameter, the derivative of the log likelihood, omitting the constant \(\log\binom{10}{7}\), is \(7/p-3/(1-p)\). This is positive for \(p<0.7\), zero at \(p=0.7\), and negative for \(p>0.7\), so the maximum is attained there. The corresponding transformed estimate is the finite value \(\gamma=\log(7/3)\). Both parameterizations produce the same likelihood at corresponding values.

Boundary Values Need Separate Attention

The probability constraint \(0\leq p\leq1\) includes its endpoints. A finite logit parameter cannot produce either endpoint, so a model whose likelihood is maximized at \(p=0\) or \(p=1\) has no finite maximizing logit value. The transformed likelihood may have a supremum as \(\gamma\) tends to infinity or negative infinity, without attaining that supremum at any finite \(\gamma\).

Worked Example: All Observations Are Successes

Consider five hypothetical independent Bernoulli trials with a common success probability \(p\), and suppose all five outcomes are successes. The likelihood is \(L_p(p)=p^5\) for \(0\leq p\leq1\). This function increases over the allowed interval and attains its maximum, \(1\), at \(p=1\). Thus the original parameterization has a maximum likelihood estimate at the boundary.

Under the logit transformation, every finite \(\gamma\) gives \(p=\exp(\gamma)/(1+\exp(\gamma))<1\). For example, \(\gamma=\log(9)\) gives \(p=0.9\) and likelihood \(0.9^5=0.59049\). As \(\gamma\) increases without bound, \(p\) approaches one and the likelihood approaches one, but no finite \(\gamma\) attains it. The transformed likelihood therefore has a supremum but no finite maximizer, even though the original likelihood has a maximizer.

The discrepancy does not mean the likelihood changed. The logit transformation covers only \(0<p<1\), while the original parameter space includes \(p=1\). To represent the boundary estimate, retain the original parameterization or explicitly account for the boundary; do not report a finite logit estimate as if it attained the maximum.

Choosing and Checking a Parameterization

A useful parameterization should enforce the intended restrictions while preserving the model values of interest. Before using one, check its domain, image, and whether distinct transformed values can represent the same original parameter. The exponential and logit transformations are one-to-one on their stated domains, but not every convenient formula has these properties.

1
State the original parameter space.
Write the allowed values explicitly, including whether boundary values such as zero or one are included.
2
Specify the transformation and its domain.
For example, \(\sigma=\exp(\eta)\) uses \(\eta\in\mathbb{R}\), while \(p=\exp(\gamma)/(1+\exp(\gamma))\) uses \(\gamma\in\mathbb{R}\).
3
Check the image and uniqueness.
Confirm which original values are represented and whether the transformation assigns each represented value a unique transformed value.
4
Substitute into the likelihood.
Form \(L_\eta(\eta)=L_\theta(g(\eta))\), preserving all likelihood terms and not adding a Jacobian for a parameter substitution.
5
Check the maximum, including boundaries.
If the original maximum may occur at a value outside the transformation’s image, the transformed parameter may have no finite maximizer.

Some restrictions involve several parameters at once. For instance, a model might impose that one probability equals a specified function of another parameter. A reparameterization should then encode the restriction as an equality by construction, and its image must still be checked against the intended constrained parameter space. Merely writing down a formula that satisfies the restriction does not show that all allowed parameter combinations are represented.

Common Mistake

A common error is to assume that a transformation automatically preserves every feature of the original parameter space. For example, one might say, “The logit keeps probabilities between zero and one, so maximizing over a finite logit parameter is equivalent to maximizing over all probabilities.” The correction is that the logit maps finite real values only to the open interval \(0<p<1\). If the original model allows \(p=0\) or \(p=1\), the transformed space omits those values; a boundary maximum can consequently become an unattained supremum.

Another error is to multiply a likelihood by the derivative of the parameter transformation. The likelihood is evaluated at the transformed parameter value, \(L_\eta(\eta)=L_\theta(g(\eta))\); a derivative factor is not part of that substitution. Always distinguish re-expressing a likelihood from transforming a density with respect to a variable.

Key takeaway: A transformation is a way to express a likelihood on a new parameter domain. Check whether it enforces the restrictions, whether it covers the full intended parameter space, and whether boundary estimates remain attainable. A one-to-one transformation onto the original space preserves the likelihood maximization problem.

Check Your Understanding

Use the parameter spaces and likelihoods stated in each question.

  1. What set of scale values is represented by \(\sigma=\exp(\eta)\) when \(\eta\) ranges over all real numbers? Does this include zero?
  2. Does the finite logit parameter \(\gamma\) represent probabilities \(p=0\) and \(p=1\)? Explain.
  3. Why is a Jacobian not multiplied into a likelihood merely because the model parameter has been re-expressed using \(\theta=g(\eta)\)?
  4. For five independent Bernoulli trials that are all failures, where is the maximum of \(L(p)=(1-p)^5\) on \(0\leq p\leq1\)? What happens to the logit parameterization at that maximum?
  5. What two properties should be checked to determine whether a one-to-one reparameterization represents the full original parameter space?

Previous

Next

Back to Tutorials
Clinical Biostats