Identifiability and Separation of Likelihood Maxima

Why Identifiability Matters

A likelihood maximum is useful for inference only if its location represents a distinguishable model. If two different coefficient vectors produce the same response distributions at every observed covariate value, the data cannot tell those vectors apart. The likelihood is constant across those equivalent coefficient vectors; if a maximum is attained, every such vector is also a maximizer, though a maximum need not be attained.

Two related questions are important. First, does the population likelihood have a unique maximizer? Second, is its value strictly lower, by a positive amount, at parameters outside any fixed neighborhood of that maximizer? The second property is a form of separation of the population maximum. It is stronger than uniqueness and requires additional conditions.

Definition: A parameter $\beta$ is identifiable in a model if distinct values of $\beta$ imply distinct probability distributions for the observable data. In a GLM, the coefficient vector determines conditional distributions through the linear predictor $X^\mathsf{T}\beta$, the inverse link, and the response distribution. If distinct coefficient vectors give the same conditional distributions at the covariate values under consideration, those coefficients are not identifiable.

For a random-design formulation, let $X$ have covariate distribution $H$, and suppose the conditional distribution of $Y$ given $X=x$ is $f_\beta(y\mid x)$. Write $\beta_0$ for the true coefficient vector. The population log-likelihood criterion is

Q(\beta)=E_{\beta_0}\big[\log f_\beta(Y\mid X)\big].

Assume this expectation is well-defined for the parameters being compared. A fixed-design version uses an average over the design rows in place of the expectation over $H$. The uniqueness conclusion below applies under the corresponding assumptions when the design matrix has full column rank; separation still requires a separate condition, such as compactness and continuity.

The Likelihood Comparison and Its Equality Case

The key comparison is between the true conditional distribution and the distribution proposed by a candidate coefficient. For probability densities or mass functions $p$ and $q$ with respect to the same dominating measure, their Kullback–Leibler divergence is

D(p\|q)=\int_{\{p>0\}}p\log\!\left(\frac{p}{q}\right),

with $D(p\|q)=+\infty$ if $q=0$ on a set of positive $p$-probability. The integral is understood in its usual extended sense. This divergence is nonnegative, and equals zero exactly when $p=q$ almost everywhere.

Result: If $P$ is the true distribution and $Q$ is any candidate distribution, then $D(P\|Q)\geq 0$, with equality if and only if $P=Q$ as distributions.

To verify nonnegativity without assuming that the candidate distribution has no probability outside the support of the true distribution, use $-\log u\geq 1-u$ for $u>0$. On $\{p>0,q>0\}$, this gives $p\log(p/q)\geq p-q$. If $q=0$ on a set with positive $p$-probability, the divergence is infinite and the claim follows. Otherwise, integrating the inequality over $\{p>0\}$ gives

D(p\|q)\geq \int_{\{p>0\}}(p-q) =1-\int_{\{p>0\}}q \geq 0.

The last inequality holds because $q$ has total integral one. For the equality case, if $D(p\|q)=0$, the displayed lower bound implies $\int_{\{p>0\}}q=1$, so $q$ has no mass outside the support of $p$. Also, equality in $-\log u\geq 1-u$ requires $u=1$; hence $q=p$ almost everywhere on $\{p>0\}$. Conversely, if $p=q$ almost everywhere, the divergence is zero. This proves the result. $\square$

Apply the result conditionally at each $x$. When the expected log-likelihood comparison is well-defined, the difference between the true and candidate population criteria is

Q(\beta_0)-Q(\beta) = E_X\!\left[ D\!\left(f_{\beta_0}(\,\cdot\mid X)\,\middle\|\,f_\beta(\,\cdot\mid X)\right) \right] \geq 0.

Thus the true parameter maximizes the population criterion. Equality holds precisely when the candidate conditional distribution agrees with the true conditional distribution for $H$-almost every $x$. A unique maximum follows if that agreement is possible only when $\beta=\beta_0$. The equality condition matters: nonnegativity alone does not establish uniqueness.

How Full Rank Gives Uniqueness in a GLM

For a GLM with a scalar response mean, write $\mu_\beta(x)=g^{-1}(x^\mathsf{T}\beta)$, where $g$ is the link function. The following result gives useful sufficient conditions for a unique population maximum.

Theorem: Suppose the model is correctly specified at $\beta_0$, and the conditional log-likelihood comparison above is well-defined. Assume:

The link $g$ is one-to-one on the relevant mean parameter space.
At fixed dispersion, the conditional response distribution is identifiable by its mean: different means imply different conditional distributions.
$E[XX^\mathsf{T}]$ exists and is positive definite.

Then $Q(\beta_0)>Q(\beta)$ for every $\beta\ne\beta_0$ in the parameter space. In particular, $\beta_0$ is the unique population likelihood maximizer.

Proof. Take any $\beta\ne\beta_0$ and put $v=\beta-\beta_0$, so $v\ne0$. Positive definiteness gives

E[(X^\mathsf{T}v)^2]=v^\mathsf{T}E[XX^\mathsf{T}]v>0.

Consequently $P(X^\mathsf{T}v\ne0)>0$; otherwise the nonnegative random variable $(X^\mathsf{T}v)^2$ would be zero almost surely and its expectation would be zero. On this positive-probability set, $X^\mathsf{T}\beta\ne X^\mathsf{T}\beta_0$. The one-to-one link implies $\mu_\beta(X)\ne\mu_{\beta_0}(X)$, and identifiability of the conditional response distribution by its mean implies that the two conditional distributions differ there. Their conditional KL divergence is therefore strictly positive on a set of positive $H$-probability and is nonnegative elsewhere. Its expectation is strictly positive. The likelihood comparison gives $Q(\beta_0)-Q(\beta)>0$, as claimed. $\square$

For a fixed design matrix with rows $x_i^\mathsf{T}$, the analogous rank condition is that the matrix has full column rank. Equivalently, there is no nonzero $v$ for which $x_i^\mathsf{T}v=0$ for every row. If each row receives positive weight in the population criterion, a difference in linear predictors for at least one row yields a strict likelihood comparison under the same link and distribution assumptions.

Worked Example: A Full-Rank Poisson Regression

Let $X$ equal $0$ or $1$, each with probability $1/2$. Given $X=x$, suppose $Y$ has a Poisson distribution with mean $\mu_\beta(x)=\exp(\beta_0+\beta_1x)$, and the true coefficient is $\beta_0=(\log 2,\log 3)^\mathsf{T}$. The true conditional means are $2$ at $x=0$ and $6$ at $x=1$.

The two design rows are $(1,0)$ and $(1,1)$, which are linearly independent. Indeed, if $a(1,0)^\mathsf{T}+b(1,1)^\mathsf{T}=0$, the second coordinate gives $b=0$, and then the first gives $a=0$. The log link is one-to-one, and a Poisson distribution is identified by its mean. The theorem therefore gives a unique population maximum at the true coefficient.

For a direct likelihood check, take the candidate $\beta=(\log 3,\log 2)^\mathsf{T}$. It predicts means $3$ at $x=0$ and $6$ at $x=1$. For Poisson distributions with means $\mu_0$ and $\mu$, the divergence is

D\{\operatorname{Pois}(\mu_0)\|\operatorname{Pois}(\mu)\} =\mu_0\log\!\left(\frac{\mu_0}{\mu}\right)+\mu-\mu_0.

At $x=0$, this is $2\log(2/3)+1$, which is about $0.1891$; at $x=1$, it is zero. The population likelihood gap is therefore

Q(\beta_0)-Q(\beta) =\tfrac12\{2\log(2/3)+1\}+\tfrac12(0) \approx 0.0945>0.

The nonzero gap confirms that this candidate does not maximize the population criterion.

When the Design Is Rank-Deficient

If the design has a nonzero null direction, changing coefficients in that direction leaves every linear predictor unchanged. The mean parameters and conditional distributions are then unchanged as well. No likelihood calculation can distinguish coefficient vectors that generate identical distributions.

Worked Example: Aliased Predictors in a Poisson GLM

Suppose a Poisson log-link model has predictors $z_1$ and $z_2$, but the observed design satisfies $z_1=z_2$ for every unit. Its linear predictor is $\eta=\beta_0+\beta_1z_1+\beta_2z_2$. Since the two predictors agree, this becomes $\eta=\beta_0+(\beta_1+\beta_2)z_1$: the likelihood depends on the two slope coefficients only through their sum.

For example, the vectors $(0,1,2)^\mathsf{T}$ and $(0,2,1)^\mathsf{T}$ both give slope sum $3$. At any row with $z_1=z_2$, both predict $\eta=3z_1$, so they produce the same Poisson mean $\exp(3z_1)$ and exactly the same conditional distribution. If one of these vectors is a maximizer, the other is also a maximizer. More generally, any change $(0,t,-t)^\mathsf{T}$ leaves all fitted distributions unchanged. The coefficients are not identifiable, although their sum may be.

In matrix terms, the predictor columns are identical, so the design matrix is not full rank. The failure is structural: reporting one optimizer’s coefficient values does not make the separate slopes estimable from this likelihood.

Uniqueness Versus Separation by a Positive Gap

Uniqueness says that every fixed alternative $\beta\ne\beta_0$ has a lower population criterion. Separation asks for more: for every $\varepsilon>0$, all parameters at least $\varepsilon$ away from $\beta_0$ must have criterion values bounded below $Q(\beta_0)$ by a positive amount.

Definition: A unique maximizer $\beta_0$ is separated on a parameter set $\Theta$ if, for every $\varepsilon>0$ for which the set is nonempty, $$ \sup_{\{\beta\in\Theta:\ \|\beta-\beta_0\|\geq\varepsilon\}}Q(\beta)<Q(\beta_0). $$

A standard sufficient condition is compactness. If $\Theta$ is compact, $Q$ is continuous on $\Theta$, and $\beta_0$ is its unique maximizer, then the maximum criterion value on the closed set $\{\beta\in\Theta:\|\beta-\beta_0\|\geq\varepsilon\}$ is attained. Every point in that set differs from $\beta_0$, so its criterion value is strictly smaller than $Q(\beta_0)$. The attained maximum is therefore strictly smaller too, giving the required positive gap.

On an unbounded parameter space, uniqueness and continuity alone do not establish this result: a sequence of parameters can move arbitrarily far away while its criterion values approach the maximum. A positive gap then needs some further condition, such as a direct separation assumption or suitable control of the criterion at large parameter values. Full rank and identifiable means establish uniqueness under the theorem’s assumptions; by themselves, they should not be mistaken for a proof of global separation.

Worked Example: Logistic Regression With Identifiable Coefficients

Let $X$ take the two values $(1,0)^\mathsf{T}$ and $(1,1)^\mathsf{T}$ with equal probability. Use the logistic link, so $P_\beta(Y=1\mid X=x)=\operatorname{expit}(x^\mathsf{T}\beta)$, where $\operatorname{expit}(u)=1/(1+\exp(-u))$. At the true coefficient $\beta_0=(0,\log 3)^\mathsf{T}$, the success probabilities are $p_0=1/2$ and $p_1=3/4$.

The design rows are linearly independent, and the logistic mean is one-to-one in the linear predictor. Bernoulli distributions are identified by their success probabilities, so the population maximum is unique. To see a numerical likelihood comparison, consider the candidate $\beta=(0,0)^\mathsf{T}$, which predicts probability $1/2$ at both design points.

For Bernoulli distributions with success probabilities $p$ and $q$, the divergence is $p\log(p/q)+(1-p)\log\{(1-p)/(1-q)\}$. At $x=0$, the divergence is zero. At $x=1$, it is

\frac34\log\!\left(\frac{3/4}{1/2}\right) +\frac14\log\!\left(\frac{1/4}{1/2}\right) =\frac34\log(3/2)+\frac14\log(1/2) \approx 0.1308.

Averaging over the two covariate values gives $Q(\beta_0)-Q(\beta)\approx 0.0654>0$. This candidate therefore has strictly lower population log-likelihood.

Common Mistakes

Equating a unique maximum with a separated maximum. A strict inequality at each fixed alternative does not, without extra conditions, provide one positive gap uniformly over all distant alternatives. Compactness and continuity are one sufficient route to that stronger property.
Assuming a full-rank matrix settles every identifiability question. Full rank distinguishes linear predictors. A one-to-one link and a response distribution identified by its mean are also needed to transfer that distinction to conditional distributions.
Treating an optimizer’s output as proof of identifiable coefficients. With aliased predictors, software may return a particular coefficient vector, but other vectors can generate exactly the same fitted distributions.
Ignoring support when comparing distributions. The KL inequality must account for probability the candidate distribution places outside the true distribution’s support; integrating only over the true support gives $1-\int_{\{p>0\}}q$, not automatically zero.

Key takeaway: The population likelihood compares candidate conditional distributions with the true ones through KL divergence. Full-rank design, an injective mean map, and identifiable response distributions make the true coefficient the unique maximizer. A positive gap away from it is a further property that requires its own justification.

Check Your Understanding

Answer each question using the distinction between coefficients, fitted means, and conditional distributions.

Why does $E[XX^\mathsf{T}]$ being positive definite rule out a nonzero coefficient change that leaves $X^\mathsf{T}\beta$ unchanged almost surely?
In an aliased-predictor model, what can remain identifiable even when the individual coefficients are not?
What is the equality condition in the KL comparison of the true and candidate conditional distributions?
Why are compactness and continuity useful for proving a positive likelihood gap away from a unique maximum?
Does full rank alone guarantee identifiable coefficients in every model? State the additional features used in the theorem.