Regularity Audit for MLE Asymptotics

Why Audit Regularity Before Using MLE Asymptotics?

A normal approximation for a maximum likelihood estimator is not guaranteed merely because an estimator is defined by maximizing a likelihood. Its usual justification depends on structural features of the model: the parameter must be identifiable, the true value must be interior, the likelihood must be sufficiently smooth, and the information must be nonsingular. Global maximization also matters: a local calculation around the truth does not establish that the estimator actually selects that local maximum.

A regularity audit makes these requirements explicit before an asymptotic approximation is used. We apply the same questions to binomial, exponential, and endpoint-uniform models. The first two illustrate familiar fixed-support cases; the uniform endpoint model shows how parameter-dependent support can invalidate the usual smooth, square-root asymptotic analysis.

A Structured Regularity Audit

Definition: A regularity audit for an MLE checks, at a specified true parameter value \(\theta_0\), whether (1) the model is identifiable, (2) the support is suitably independent of the parameter, (3) \(\theta_0\) is interior to the parameter space, (4) the likelihood is sufficiently differentiable under suitable domination conditions, (5) Fisher information is finite and nonsingular, and (6) a global maximizer exists and is selected in a way compatible with the local likelihood analysis.

These are diagnostic checks, not a claim that a short checklist alone proves asymptotic normality. In particular, a theorem may impose additional assumptions, such as consistency, control of likelihoods away from the truth, or uniform convergence of derivatives. The audit identifies common failures and checks that the model has the basic structure required by the usual theory.

Identifiability. Different parameter values should give different distributions: if \(F_{\theta_1}=F_{\theta_2}\), then \(\theta_1=\theta_2\).
Support. The set on which the density or mass function is positive should not change with the parameter, apart from sets of probability zero, for the usual fixed-support argument.
Interiority. The true value \(\theta_0\) should lie in the interior of the parameter space, so small perturbations in both directions are allowed.
Differentiability. The log-likelihood should have the derivatives needed for local expansions. Differentiating an integral or expectation also requires conditions that justify interchanging differentiation and integration.
Information. Fisher information should be finite and strictly positive in a scalar model (or positive definite in a multivariate model).
Global maximization. A local score equation is not enough: check whether a global maximum exists, whether it is unique, and whether a boundary or distant mode can compete.

When Differentiation Gives the Information Identity

The following result explains one role of the support and differentiability checks. It states sufficient conditions for the score to have mean zero and for the two usual definitions of Fisher information to agree. These identities should not be used automatically in a model where differentiation under the integral has not been justified.

Theorem: Suppose a scalar-parameter model has density or mass function \(f_\theta(x)\) with respect to a measure \(\mu\), and the support is common in a neighborhood of \(\theta_0\). Suppose \(f_\theta(x)>0\) there on that support. Assume that, near \(\theta_0\), the first two derivatives with respect to \(\theta\) exist and differentiation can be interchanged with integration for both \(f_\theta\) and its first derivative. Assume also that the score and second log-derivative are integrable. With \(u_\theta(x)=\partial \log f_\theta(x)/\partial\theta\), these conditions imply \(E_{\theta_0}[u_{\theta_0}(X)]=0\) and \(E_{\theta_0}[u_{\theta_0}(X)^2]=-E_{\theta_0}[\partial^2\log f_{\theta_0}(X)/\partial\theta^2]\).

Proof. Normalization gives \(\int f_\theta(x)\,d\mu(x)=1\). Differentiate this equality at \(\theta_0\), using the assumed interchange:

0=\int \left.\frac{\partial f_\theta(x)}{\partial\theta}\right|_{\theta=\theta_0}d\mu(x) =\int f_{\theta_0}(x)u_{\theta_0}(x)d\mu(x) =E_{\theta_0}[u_{\theta_0}(X)].

For the second identity, differentiate the normalization equality twice. Since \(\partial^2 f_\theta/\partial\theta^2 =f_\theta\{(\partial\log f_\theta/\partial\theta)^2+\partial^2\log f_\theta/\partial\theta^2\}\), the second derivative of the integral is

0=\int f_{\theta_0}(x) \left\{u_{\theta_0}(x)^2+ \left.\frac{\partial^2\log f_\theta(x)}{\partial\theta^2}\right|_{\theta=\theta_0} \right\}d\mu(x).

Rearranging yields the stated information identity. The common-support and interchange assumptions are essential: a moving support can contribute boundary effects that this calculation does not capture. \(\square\)

Worked Example: Binomial Counts With a Fixed Number of Trials

Worked Example: Auditing a Binomial Likelihood

Suppose \(X_1,\ldots,X_5\) are independent \(\operatorname{Bin}(10,p)\) counts, with parameter space \(0<p<1\). The observed counts are \(6,7,5,8,6\). Audit the model and find its likelihood maximizer.

Identifiability and support. The mean of each count is \(10p\), so its distribution determines \(p\); the model is identifiable. Each count takes values in \(\{0,1,\ldots,10\}\), independent of \(p\), so the support is fixed.

Interiority and smoothness. For a true value \(0<p_0<1\), the parameter is interior. Omitting factors that do not depend on \(p\), the log-likelihood is

\ell(p)=32\log p+18\log(1-p)+C, \qquad \ell'(p)=\frac{32}{p}-\frac{18}{1-p}, \qquad \ell''(p)=-\frac{32}{p^2}-\frac{18}{(1-p)^2}.

The total number of successes is \(6+7+5+8+6=32\), and the total number of trials is \(50\). Setting the score to zero gives \(32(1-p)=18p\), so \(\hat p=32/50=0.64\). The second derivative is negative throughout \(0<p<1\), so this is the unique maximizer for these data.

Information. For one \(\operatorname{Bin}(10,p)\) observation, the score is \(X/p-(10-X)/(1-p)\). Since \(E[X]=10p\) and \(\operatorname{Var}(X)=10p(1-p)\), its variance, and hence its Fisher information, is \(10/\{p(1-p)\}\). It is finite and positive for \(0<p<1\).

Existence caveat. If all 50 trials are successes, the likelihood is proportional to \(p^{50}\), whose supremum over the open space \(0<p<1\) is approached as \(p\) tends to 1 but is not attained. Thus a global MLE need not exist on the open parameter space for every possible sample. If the parameter space is instead \([0,1]\), the maximum exists at the boundary; the usual interior asymptotic argument still concerns a true value \(0<p_0<1\).

Worked Example: Exponential Rates

Worked Example: Auditing an Exponential Likelihood

Let \(X_1,\ldots,X_4\) be independent exponential observations with rate \(\lambda>0\), so \(f_\lambda(x)=\lambda e^{-\lambda x}\) for \(x\geq0\). The observed values are \(0.4,0.8,1.2,1.6\).

Identifiability and support. The mean is \(1/\lambda\), so distinct positive rates give distinct distributions. The support \([0,\infty)\) is fixed.

Interiority and maximization. The true rate \(\lambda_0>0\) is interior. The sample sum is \(S=0.4+0.8+1.2+1.6=4\), and the log-likelihood is

\ell(\lambda)=4\log\lambda-4\lambda, \qquad \ell'(\lambda)=\frac{4}{\lambda}-4, \qquad \ell''(\lambda)=-\frac{4}{\lambda^2}.

The score is zero at \(\hat\lambda=1\), and the second derivative is negative for every \(\lambda>0\). Also, \(\ell(\lambda)\) tends to \(-\infty\) as \(\lambda\) tends to zero or infinity. Therefore \(\hat\lambda=1\) is the unique global maximizer.

Information. For one observation, the score is \(1/\lambda-X\), with mean zero and variance \(1/\lambda^2\), because an exponential observation has variance \(1/\lambda^2\). Thus the information per observation is \(1/\lambda^2\), finite and positive for every allowed rate. The regularity checks support the usual likelihood analysis at any fixed \(\lambda_0>0\).

Worked Example: A Uniform Endpoint Parameter

Worked Example: Auditing a Parameter-Dependent Support

Let \(X_1,\ldots,X_4\) be independent \(\operatorname{Uniform}[0,\theta]\) observations, with \(\theta>0\). Use the density version \(f_\theta(x)=1/\theta\) for \(0\leq x\leq\theta\), and zero otherwise. The observed values are \(0.3,0.7,1.1,1.4\).

The model is identifiable because \(E[X]=\theta/2\). However, the support \([0,\theta]\) depends on the parameter. For each \(\theta_0>0\), the true value is interior to the parameter space \((0,\infty)\), but support dependence already prevents the standard fixed-support audit from passing.

Let \(M=\max_i X_i=1.4\). Under the stated density version, the likelihood is

L(\theta)= \begin{cases} \theta^{-4}, & \theta\geq 1.4,\\ 0, & 0<\theta<1.4. \end{cases}

On the allowed range \(\theta\geq1.4\), \(\theta^{-4}\) is decreasing, because its derivative is \(-4\theta^{-5}<0\). The likelihood is positive at \(\theta=1.4\), so its global maximum is attained there: \(\hat\theta=M=1.4\). The value at the endpoint matters. If instead the density were specified on \(0\leq x<\theta\), an observation equal to \(M\) would have likelihood zero at \(\theta=M\); the likelihood would have supremum \(M^{-4}\) as \(\theta\) decreases to \(M\) from above, but no maximizer. The closed-support density version used here avoids that nonattainment.

Although \(\log f_\theta(x)=-\log\theta\) for observations strictly inside the support, the support itself moves with \(\theta\). Treating \(-1/\theta\) as an ordinary score and applying the fixed-support information identity would give a nonzero expected score, which signals that the required interchange argument fails. The usual regular likelihood approximation is therefore not justified by the fixed-support conditions.

Why the Endpoint Model Has a Different Rate

The support failure is not merely formal. It changes the behavior of the maximum. Under the true value \(\theta\), the sample maximum \(M_n=\max(X_1,\ldots,X_n)\) has, for \(0\leq x\leq n\),

P_\theta\left(\frac{n(\theta-M_n)}{\theta}>x\right) =P_\theta\left(M_n<\theta\left(1-\frac{x}{n}\right)\right) =\left(1-\frac{x}{n}\right)^n.

The last equality follows because each observation independently falls below \(\theta(1-x/n)\) with probability \(1-x/n\). For fixed \(x\geq0\), the right-hand side tends to \(e^{-x}\). Hence \(n(\theta-M_n)/\theta\) converges in distribution to an exponential random variable with rate 1. The error has order \(1/n\), not the usual \(1/\sqrt n\) scale. The endpoint model is therefore a concrete warning against assuming a normal MLE approximation from a smooth-looking log-likelihood alone.

Common Mistakes in a Regularity Audit

Checking only the score equation. A root of \(\ell'(\theta)=0\) may be a local minimum, one of several modes, or not a global maximizer. Check curvature and likelihood behavior at the parameter-space boundaries and elsewhere.
Confusing an interior truth with a regular model. The uniform parameter \(\theta_0>0\) is interior in \((0,\infty)\), but its support still depends on \(\theta\).
Ignoring the parameter space's endpoints. In the binomial model, all successes give no maximizer on \(0<p<1\), even though the likelihood has a finite supremum. A closed parameter space changes the existence answer.
Using a density formula without checking its endpoint convention. For the endpoint-uniform model, the likelihood at \(\theta=M_n\) depends on whether the density includes the endpoint. With support \([0,\theta]\), the maximum is attained; with support \([0,\theta)\), it is not.
Interchanging differentiation and integration without conditions. A formal derivative of the log-density does not establish that the score has mean zero, especially when the support moves.

The practical value of the audit is that it separates routine calculations from assumptions that need special attention. Binomial and exponential models pass the basic fixed-support checks at interior parameter values, though global existence can still depend on the sample and parameter-space convention. The endpoint-uniform model fails the fixed-support check, and its maximum has a different convergence scale. These distinctions determine whether standard MLE asymptotics are appropriate or whether a model-specific analysis is needed.

Check Your Understanding

Use the audit questions to answer each item.

For independent \(\operatorname{Bin}(m,p)\) observations, which basic regularity checks hold when \(0<p_0<1\), and what sample can prevent an MLE from existing on the open parameter space?
For the exponential rate model, explain why the support is fixed and calculate the Fisher information per observation.
For \(X_i\sim\operatorname{Uniform}[0,\theta]\), why is the MLE the sample maximum under the closed-support density version?
What is wrong with applying the fixed-support score identity to the endpoint-uniform model without checking differentiation under the integral?
What limiting distribution does \(n(\theta-M_n)/\theta\) have, and what does that imply about the estimator's rate?