A Worked Curved Normal Likelihood

Introduction

A normal model usually assigns separate parameters to its mean and variance. Here those quantities are tied together: the mean is \(\mu\), the variance is \(\mu^2\), and \(\mu\) must be positive. This restriction makes the model a useful concrete example of a curved exponential family. Its likelihood is simple enough to analyze directly, but the constraint matters: the estimate must maximize the likelihood along the curve of allowed parameter values, not over an unconstrained two-parameter normal family.

Assume \(X_1,\ldots,X_n\) are independent and identically distributed with \(X_i\sim N(\mu,\mu^2)\), where \(\mu>0\). Write \(S_1=\sum_{i=1}^n X_i\) and \(S_2=\sum_{i=1}^n X_i^2\). We will derive the natural-parameter curve, the score and information, and the likelihood estimate.

The Curved Natural-Parameter Representation

The density of one observation is

The sufficient-statistic terms are \(x\) and \(x^2\). In the two-dimensional normal natural-parameter representation, their coefficients are \(\eta_1=\mu/\sigma^2\) and \(\eta_2=-1/(2\sigma^2)\). Substituting \(\sigma^2=\mu^2\) gives the following restriction.

Along this curve, the log normalizing term is \(A(\eta_1,\eta_2)=-\log\eta_1+\frac12+\frac12\log(2\pi)\). Indeed, substituting \(\eta_1=1/\mu\) gives \(A=\log\mu+\frac12+\frac12\log(2\pi)\), exactly the constant term in the log density. The curve is nonlinear, so differentiation with respect to \(\mu\) must account for how both natural parameters change.

Log-Likelihood and Score

For the independent sample, the log-likelihood, up to no omitted parameter-dependent terms, is

Differentiating each parameter-dependent term gives the score \(U(\mu)=\ell'(\mu)\) and its derivative:

The score equation can be written \(S_2-S_1\mu-n\mu^2=0\). In particular, its terms involve both the sum and the sum of squares; setting the score to zero does not amount to matching the sample mean to the model mean.

Fisher Information

For this one-parameter model, the Fisher information is the expectation of the squared score, or equivalently the negative expectation of the second derivative when the expectation is finite and the usual score identity conditions hold. Here both calculations are direct.

To verify the result, write one observation as \(X=\mu(1+Z)\), where \(Z\sim N(0,1)\). Its score is \(u(\mu)=-1/\mu+X^2/\mu^3-X/\mu^2=(Z^2+Z-1)/\mu\). Since \(E[Z]=E[Z^3]=0\), \(E[Z^2]=1\), and \(E[Z^4]=3\),

The observations are independent, so the variance of the sum of their scores is the sum of their variances; each score has mean zero. Hence \(I_n(\mu)=3n/\mu^2\). The same answer follows by expectation of the negative second derivative: \(E[S_1]=n\mu\) and \(E[S_2]=nE[X^2]=2n\mu^2\), so

Likelihood Estimate and Its Uniqueness

Suppose \(S_2>0\), as occurs with probability one for a sample from this continuous model. The score equation is equivalent, for \(\mu>0\), to \(n\mu^2+S_1\mu-S_2=0\). This quadratic has exactly one positive root, since its two roots have product \(-S_2/n<0\). The positive root is

Proof. Multiplying the score by \(\mu^3>0\) shows that its sign is the sign of \(q(\mu)=S_2-S_1\mu-n\mu^2\). The equation \(q(\mu)=0\) has exactly one positive root: the quadratic \(n\mu^2+S_1\mu-S_2\) has roots of opposite signs because their product is \(-S_2/n<0\). Since \(q(0)=S_2>0\) and \(q(\mu)\) is negative for sufficiently large \(\mu\), the score is positive before that positive root and negative after it. Thus the log-likelihood increases up to the root and decreases thereafter, making it the unique global maximum on \((0,\infty)\). Solving the quadratic gives the displayed estimate. \(\square\)

If \(S_2=0\), every observation is zero. Then the log-likelihood contains \(-n\log\mu-n/2\) plus a constant and tends to \(+\infty\) as \(\mu\) decreases to zero. There is no finite maximum likelihood estimate in the parameter space. This boundary case is possible for an arbitrary recorded data set, although it has probability zero under the continuous model.

Common Pitfall and Why the Constraint Matters

A common mistake is to maximize the normal likelihood as if the mean and variance were unrelated, then substitute the restriction \(\sigma^2=\mu^2\) afterward. That procedure need not give the maximum on the restricted family: the unrestricted estimates optimize over a larger parameter space and may not lie on the allowed curve. The correct approach is to substitute the constraint into the likelihood first and maximize the resulting one-parameter function, as above.

A second mistake is to treat observed information as though it were always the same as Fisher information. In the worked sample, the observed information at \(2\) is \(3/8\); for that same \(n=2,\mu=2\), the expected information is \(3n/\mu^2=3/2\). The discrepancy is not an arithmetic error: one is a sample-dependent curvature, while the other is its expectation under the model. This example makes the distinction particularly visible.