Why the Parameter Space Matters
A maximum likelihood estimate depends not only on the observed data and the likelihood function, but also on which parameter values are allowed. The unrestricted estimate uses the full parameter space of the model. A constrained estimate maximizes the same likelihood after the parameter space has been limited, for example by an equality such as \(\lambda_1=\lambda_2\) or an inequality such as \(p\leq 0.4\). This distinction is central to likelihood ratio methods: before comparing likelihoods, one must find the maximum in each relevant parameter space.
A constrained estimate may lie in the interior of the permitted set or on its boundary. A boundary point is a maximum only if it is included in the set. If the likelihood approaches its largest value as a parameter approaches an excluded endpoint, the likelihood has a supremum there but no maximum. Thus, solving a score equation alone does not settle every constrained optimization problem.
For a fixed observed data set, maximizing the likelihood is equivalent to maximizing its logarithm whenever the likelihood is positive, because the logarithm is strictly increasing. Log-likelihoods are usually easier to differentiate and compare. A data-only factor that does not depend on the parameter can be omitted during optimization, but it must be treated consistently if actual likelihood values are compared.
A Reliable Optimization Procedure
The practical method is to identify the feasible parameter set first, then search it for all possible maxima. In a one-dimensional closed interval, this means checking interior stationary points and included endpoints. For a multidimensional restriction, a maximum may occur inside the feasible set, on a boundary, or where several boundaries meet; the likelihood must be optimized over each relevant part of the set. A stationary point is only a candidate until it is shown to be a maximum and compared with other candidates.
State the likelihood and the parameter values allowed by the model. Then express any added equality or inequality restriction precisely.
Optimize over the full parameter space, checking parameter-space endpoints when a score equation does not give an admissible maximizer.
Use substitution for a convenient equality restriction, or find interior and boundary candidates for an inequality restriction.
Check that each proposed value is permitted and that it gives at least as large a likelihood as every other feasible candidate. An excluded endpoint can yield only a supremum.
Why an Inequality Can Put the Estimate on a Boundary
For a one-dimensional parameter restricted to a closed interval, a differentiable log-likelihood can be increasing across the entire feasible interval. In that case, its maximum occurs at the right endpoint if that endpoint is included. If the endpoint is excluded, the same monotonicity does not produce a maximum: the likelihood may only approach its supremum. The corresponding statement holds for a decreasing log-likelihood and the left endpoint.
The Bernoulli model gives a direct illustration. Suppose \(X\) is the number of successes in \(n\) independent trials with success probability \(p\), where \(0<p<1\). For an observed count \(x\), the log-likelihood, apart from a parameter-free term, is \(x\log p+(n-x)\log(1-p)\). Its derivative is positive when \(p<x/n\) and negative when \(p>x/n\), so the unrestricted maximum is at \(x/n\) when that value is in the parameter space. Under a restriction \(p\leq c\), if \(x/n>c\), the likelihood increases throughout the permitted interval and the constrained maximum is at \(c\), provided \(c\) is included.
Worked Example: A Bernoulli Estimate at an Included Boundary
Model and method. A hypothetical quality audit samples 10 independent components. Let each component have probability \(p\) of failing inspection, and suppose 7 fail. The model is binomial with \(n=10\), observed count \(x=7\), and parameter space \(0\leq p\leq1\). Find the unrestricted MLE and the constrained MLE under \(p\leq0.4\). The method is to differentiate the log-likelihood and then optimize over the restricted interval.
Ignoring the binomial coefficient, the likelihood and log-likelihood are
For \(0<p<1\), the score is
The denominator is positive on this interval. The score is zero at \(p=7/10=0.7\), positive below \(0.7\), and negative above it. Therefore the unrestricted MLE is \(\widehat p=0.7\).
The constrained interval is \(0\leq p\leq0.4\). Since every interior value in this interval is below \(0.7\), the score is positive throughout its interior. Thus the likelihood increases up to \(p=0.4\), and the endpoint is permitted. The constrained MLE is \(\widehat p_0=0.4\).
Check and trap avoided. The unrestricted estimate \(0.7\) is not feasible under the restriction, so it cannot be the constrained answer. The constrained value \(0.4\) is included and has a larger likelihood than any smaller permitted value. The common mistake is to report the unrestricted estimate without checking the constraint.
Equality Restrictions and Substitution
An equality restriction reduces the set of possible parameter values. A convenient first step is often to substitute the restriction into the likelihood, leaving a likelihood in fewer free parameters. The resulting maximum must still be checked against any other restrictions in the model. Alternatively, for a differentiable likelihood and regular equality constraints, one may use Lagrange multipliers to find candidates; those equations alone do not establish that a candidate is a global maximum.
Worked Example: Two Poisson Rates Restricted to Be Equal
Model and method. In a hypothetical monitoring study, event counts in two independent observation periods are \(Y_1=6\) over exposure \(e_1=2\) and \(Y_2=4\) over exposure \(e_2=1\). Assume \(Y_j\sim\operatorname{Poisson}(e_j\lambda_j)\), independently, with \(\lambda_1>0\) and \(\lambda_2>0\). Find the unrestricted rate estimates and the constrained estimate under the equality restriction \(\lambda_1=\lambda_2=\lambda\). Use the score equations, then substitute the equality into the joint likelihood.
For a Poisson count \(y\) with exposure \(e\), the likelihood as a function of its rate is proportional to \(\lambda^y e^{-e\lambda}\). The two unrestricted log-likelihood contributions are \(6\log\lambda_1-2\lambda_1\) and \(4\log\lambda_2-\lambda_2\), apart from constants. Their score equations are
Solving gives \(\widehat{\lambda}_1=6/2=3\) and \(\widehat{\lambda}_2=4/1=4\).
Under the equality restriction, the joint log-likelihood, apart from constants, becomes
Its derivative is \(10/\lambda-3\). Setting it to zero gives \(\widehat{\lambda}=10/3\). The second derivative is \(-10/\lambda^2<0\) for \(\lambda>0\), so this stationary point is a maximum. The restricted estimate is therefore \(\widehat{\lambda}_0=10/3\), approximately \(3.333\).
Check and trap avoided. The restricted rate is the total count divided by total exposure: \((6+4)/(2+1)=10/3\). It is not the unweighted average \((3+4)/2=3.5\), because the observation periods have different exposures. The unrestricted estimates differ, while the equality restriction requires one common rate.
A Normal Mean Restriction with Unknown Variance
When a restriction applies to one parameter while other parameters remain unknown, those nuisance parameters must also be re-estimated under the restriction. In particular, a constrained estimate of a normal mean does not generally leave the variance estimate unchanged. The variance that maximizes the likelihood depends on the selected mean.
Worked Example: Constrained Normal Mean and Re-estimated Variance
Model and method. Three independent observations from \(N(\mu,\sigma^2)\), with unknown \(\mu\in\mathbb{R}\) and \(\sigma^2>0\), are \(2,3,5\). Find the unrestricted MLEs and the constrained MLEs under \(\mu\leq2\). Use the normal likelihood, first optimizing the mean and variance without restriction, then minimizing the residual sum of squares over the permitted means.
The sample mean is
For independent normal observations, the unrestricted MLE of \(\mu\) is \(\bar{x}=10/3\). The sum of squared deviations from this mean is
The unrestricted variance MLE is the residual sum of squares divided by \(n=3\), so \(\widehat{\sigma}^2=(14/3)/3=14/9\).
Under \(\mu\leq2\), the closest permitted mean to the sample mean \(10/3\) is \(\mu=2\). To verify this choice and re-estimate the variance, the residual sum of squares at a permitted \(\mu\) can be written as
Since \(\bar{x}=10/3>2\), the term \((\mu-\bar{x})^2\) is minimized over \(\mu\leq2\) at the included endpoint \(\mu=2\). The residual sum of squares there is \((2-2)^2+(3-2)^2+(5-2)^2=0+1+9=10\). For a fixed mean, the normal likelihood is maximized over \(\sigma^2>0\) at the residual sum of squares divided by \(n\). Hence the constrained estimates are \(\widehat{\mu}_0=2\) and \(\widehat{\sigma}_0^2=10/3\).
Check and trap avoided. The observed-data mean is \(10/3\), not \(2\); \(2\) is the constrained mean estimate because of the inequality. The variance must then be recomputed around that constrained mean. Reusing the unrestricted variance \(14/9\) would fail to maximize the likelihood under the restriction.
Existence, Boundaries, and Common Errors
A constrained MLE need not exist for every parameter space. For example, if a likelihood increases as a parameter approaches an excluded endpoint, no allowed value attains the limiting likelihood. This differs from the Bernoulli example above, where \(p=0.4\) was included. For a continuous likelihood on a closed and bounded feasible set, continuity guarantees that a maximum is attained. When the feasible set is not compact, or when the likelihood is not continuous on it, existence must be checked rather than assumed.
For complicated restrictions, the fact that the constrained estimate lies on a boundary does not mean one can simply replace an unrestricted estimate by a nearby boundary value. One must maximize the likelihood over the entire feasible set, including any boundary portions, and compare candidate likelihood values. Likewise, an equality-constrained estimate is not necessarily an arithmetic average of unrestricted estimates; the likelihood and the information in the observations determine the result.
- Ignoring feasibility: An unrestricted MLE outside the permitted set is not a constrained MLE.
- Forgetting nuisance parameters: Re-optimize every unrestricted parameter that remains free after applying the constraint.
- Assuming an excluded endpoint is a maximum: An excluded endpoint can give a supremum without an attained estimate.
- Stopping at a score equation: Check admissibility, boundaries, and whether the candidate gives a global maximum.
Check Your Understanding
For each problem, identify the permitted parameter set before finding or assessing the estimate.
- A hypothetical assay has 12 independent Bernoulli outcomes, of which 3 are positive. Find the unrestricted MLE of \(p\) and the constrained MLE under \(p\geq0.4\). (Answer: Unrestricted \(\widehat p=0.25\); constrained \(\widehat p_0=0.4\).)
- A count is modeled as \(Y\sim\operatorname{Poisson}(\lambda)\), with observed value \(Y=8\) and restriction \(0<\lambda\leq4\). Find the unrestricted and constrained MLEs. (Answer: Unrestricted \(\widehat\lambda=8\); constrained \(\widehat\lambda_0=4\).)
- Independent observations \(1,2,5\) come from \(N(\mu,1)\), with known variance \(1\). Find the unrestricted mean MLE and the constrained mean MLE under \(\mu\geq3\). (Answer: Unrestricted \(\widehat\mu=8/3\); constrained \(\widehat\mu_0=3\).)
- Independent Poisson counts are \(Y_1=9\) over exposure \(e_1=3\) and \(Y_2=6\) over exposure \(e_2=1\). Find the two unrestricted rate MLEs and the MLE under \(\lambda_1=\lambda_2\). (Answer: Unrestricted estimates \(3\) and \(6\); common constrained estimate \(15/4=3.75\).)
- Four Bernoulli trials all result in success. The parameter space is \(0<p<1\), with the additional restriction \(p<0.8\). Does a constrained MLE exist? State the supremum of the likelihood. (Answer: No constrained MLE exists; the likelihood has supremum \(0.8^4=0.4096\), approached as \(p\) increases to \(0.8\).)