Constrained Maximum Likelihood And Reparameterization

Why Constrain a Likelihood?

Likelihood maximization is often presented as an optimization over all possible parameter values. In applications, however, theory or the definition of a model may require parameters to satisfy restrictions: probabilities must lie between zero and one, variances must be positive, or two parameters may be required to sum to a known total. An estimate that ignores a genuine restriction may be impossible to interpret or may not belong to the model being fitted.

A constraint can be built into the parameterization, or it can be handled directly as part of the optimization problem. The first approach can simplify computation, but it must represent the entire feasible set. The second approach makes the role of active restrictions explicit through the Karush–Kuhn–Tucker (KKT) conditions.

The likelihood and its regularity conditions still matter: the data must be represented by the specified likelihood, and the log-likelihood and constraint functions must be sufficiently differentiable for the derivatives used below to exist. KKT conditions are local optimization conditions; they do not by themselves guarantee a global maximum. When the log-likelihood is concave and the feasible set is convex, a point satisfying the KKT conditions is a global maximum. A numerical optimizer's convergence report is not a substitute for checking feasibility and optimality.

Equality Constraints Through Reparameterization

Suppose an equality restriction can be represented by \(\theta=r(\phi)\), where every allowed \(\phi\) maps to a feasible \(\theta\), and every feasible \(\theta\) of interest is represented. Then the constrained problem becomes an unconstrained problem in \(\phi\):

The chain rule gives the transformed score. If \(J_r(\phi)\) is the Jacobian matrix of \(r\), then

A transformed optimum therefore requires the original score to have no component in a feasible direction. It need not be zero in every direction in the unrestricted parameter space: some directions are ruled out by the equality restriction. This is why simply solving the unrestricted score equations and checking the constraint afterward generally does not produce the constrained estimate.

Reparameterization requires care. A transformation that only covers part of the feasible set can silently exclude the maximum. A transformation should also be checked for identifiability: different values of \(\phi\) should not represent the same \(\theta\) in a way that creates unnecessary ambiguity. For example, exponentiating a parameter enforces strict positivity, but cannot attain zero for any finite transformed parameter. If zero is an allowed boundary value and could maximize the likelihood, that transformation does not cover the full feasible set.

Inequality Constraints and KKT Conditions

For differentiable constraints \(g_j(\theta)\geq0\), write the Lagrangian for maximization as \(\mathcal{L}(\theta,\lambda)=\ell(\theta)+\sum_j\lambda_j g_j(\theta)\), with multipliers \(\lambda_j\). Under a suitable constraint qualification, such as linearly independent gradients of the active constraints, a local optimum must satisfy the KKT conditions:

Complementary slackness separates the two cases. If a constraint is inactive, \(g_j(\hat{\theta})>0\), so its multiplier must be zero: it has no local effect on the optimum. If its multiplier is positive, the constraint must be active. Intuitively, at an active boundary the likelihood's gradient can point toward a better value only in a direction that is infeasible. The multiplier measures the local value of relaxing that restriction, in the units associated with the constraint.

Why the KKT Conditions Identify a Constrained Optimum

The stationarity condition follows from considering feasible small changes in the parameter. For clarity, consider one smooth active constraint \(g(\theta)\geq0\), with \(g(\hat{\theta})=0\) and \(\nabla g(\hat{\theta})\neq0\). A direction \(d\) pointing into the feasible region must satisfy \(\nabla g(\hat{\theta})^\mathsf{T}d\geq0\) to first order. At a local maximum, the directional derivative of the log-likelihood cannot be positive in such a direction: \(\nabla\ell(\hat{\theta})^\mathsf{T}d\leq0\).

In particular, directions tangent to the constraint satisfy \(\nabla g(\hat{\theta})^\mathsf{T}d=0\). Both \(d\) and \(-d\) are tangent, so the directional derivative must be zero in every tangent direction. Consequently, \(\nabla\ell(\hat{\theta})\) is proportional to \(\nabla g(\hat{\theta})\). The sign restriction from feasible inward directions makes that proportionality consistent with \(\nabla\ell(\hat{\theta})+\lambda\nabla g(\hat{\theta})=0\) for \(\lambda\geq0\). For multiple regular constraints, the corresponding argument gives a nonnegative combination of the active constraint gradients. Inactive constraints have zero multipliers, yielding complementary slackness.

These conditions are necessary under regularity conditions, not automatically sufficient. In practice, check the constraints, the multipliers, and the curvature of the objective in feasible directions. If the log-likelihood is concave and the feasible set is convex, the KKT conditions are sufficient for a global maximum; without those properties, they may describe only a local candidate.

What Constraints Change About Inference

A restriction affects uncertainty as well as the point estimate. For a linear equality restriction \(A\theta=c\), suppose the restriction is part of the model and the likelihood has regular curvature. Let \(I\) denote the information matrix for the unrestricted parameter vector, evaluated at the constrained estimate, and assume \(I\) is nonsingular and \(A\) has full row rank. The large-sample covariance of the constrained estimator, to first order under the restriction, is

The subtracted term removes variation in directions forbidden by the equality restrictions. In the two-mean example, the restriction \(\mu_1+\mu_2=10\) removes variation in the sum: estimation uncertainty remains along the line \(\mu_1+\mu_2=10\), not perpendicular to it. Inference should therefore use the constrained covariance or a valid reduced-parameter formulation, rather than treating the two estimates as unrestricted.

An active inequality can likewise place the estimate on the boundary of the feasible set. Standard symmetric Wald intervals based on an unrestricted, unconstrained approximation may then be inappropriate: they can suggest parameter values outside the model, and the usual interior normal approximation need not apply at a boundary. The inferential method must reflect the constraint and whether it is genuinely part of the model. Boundary-specific distributional results require additional care; an active constraint is not, by itself, a reason to report an ordinary unconstrained standard error.

Common Mistakes

• Estimating first and imposing the restriction afterward. Replacing an infeasible unrestricted estimate with an arbitrary feasible value need not maximize the likelihood over the feasible set. Optimize subject to the restriction.
• Using the wrong multiplier sign. For a maximization problem written \(g(\theta)\geq0\), use \(\mathcal{L}=\ell+\lambda g\) with \(\lambda\geq0\). Reversing the inequality changes the multiplier convention.
• Assuming every constraint is active. An inactive constraint has a zero multiplier. Check its value at the solution rather than imposing equality automatically.
• Treating KKT stationarity as proof of a global maximum. Check curvature and the feasible set, or otherwise establish that the candidate is a maximum rather than a minimum or saddle point.
• Using an interior standard error at a boundary without checking its validity. Constraints can change both feasible directions and the sampling distribution; inference must respect the restricted parameter space.