Why Likelihood Ascent Needs a Careful Argument
EM treats the random effects as latent data. At each iteration, it averages the complete-data log-likelihood under the conditional distribution of those effects given the observed data and the current parameter value. The M-step then seeks a parameter value that increases this conditional average. The important result is that, under suitable conditions, such an increase guarantees that the observed-data likelihood does not decrease. The result is a bound, not an assertion that the two increases are equal.
The conditions matter. In particular, comparing two values that are both negative infinity does not establish that their difference is nonnegative. A valid ascent argument requires a finite, well-defined difference between the Q-functions, as well as likelihoods for which the logarithms and conditional distributions are defined.
Likelihoods and the Current Conditional Distribution
Let \(y\) denote the observed data, \(b\) the latent random effects, and \(\theta\) the population parameter vector. Write \(p(y,b\mid\theta)\) for the complete-data density (or mass function), with respect to a common dominating measure. The observed-data likelihood integrates out the latent effects:
At the current iterate \(\theta^{(t)}\), the conditional density of the effects is
where this ratio is defined on the support of the current complete-data density. The EM auxiliary function is the conditional expectation of the complete-data log-likelihood:
The integration distribution in this definition stays fixed at the current iterate while \(\theta\) varies. An exact EM update chooses \(\theta^{(t+1)}\) to maximize this function over the parameter space. For the ascent property, it is enough that the update increases \(Q\); a global maximum is not required.
For the standard argument, assume \(0<L(\theta^{(t)})<\infty\), \(0<L(\theta^{(t+1)})<\infty\), and that the conditional expectation of the log-density ratio below is finite and well-defined. In particular, require both Q values to be finite and require \(Q(\theta^{(t+1)}\mid\theta^{(t)})-Q(\theta^{(t)}\mid\theta^{(t)})\) to equal that expectation. The candidate complete-data density must be positive \(q_t\)-almost everywhere so that the log ratio is finite almost everywhere. These conditions are sufficient and make the comparison meaningful; other technical conditions can be used if they guarantee the same finite expectations and inequalities.
The Likelihood-Ascent Bound
Define the complete-data density ratio \(r(b)=p(y,b\mid\theta^{(t+1)})/p(y,b\mid\theta^{(t)})\) wherever the current density is positive. Under the conditions just stated, \(r(b)>0\) for \(q_t\)-almost every \(b\). Also, its conditional expectation is finite, because the candidate likelihood is finite.
The inequality allows the candidate density to assign mass outside the current support: that mass contributes to its likelihood but not to the displayed integral. Since the logarithm is increasing, the inequality implies \(\log E_{q_t}[r(b)]\leq\log\{L(\theta^{(t+1)})/L(\theta^{(t)})\}\). Jensen’s inequality for the concave logarithm gives the other side of the bound:
The final equality follows by writing the log ratio as \(\log p(y,b\mid\theta^{(t+1)})-\log p(y,b\mid\theta^{(t)})\) and integrating both terms under the same density \(q_t\). Thus the likelihood increase is at least as large, on the log scale, as the Q increase. If Q does not decrease, then the log likelihood does not decrease; exponentiating gives \(L(\theta^{(t+1)})\geq L(\theta^{(t)})\).
Jensen’s inequality is an equality when \(r(b)\) is constant \(q_t\)-almost everywhere. In general the bound can be strict. Consequently, the Q increase is a sufficient condition for likelihood ascent, not a numerical prediction of the exact likelihood increase.
Worked Example: A Q Increase Gives a Likelihood Increase
Consider a hypothetical model with a binary latent effect \(b\in\{0,1\}\) and binary observation \(y\in\{0,1\}\). The following entries are joint probabilities for the displayed value \(y=1\); the remaining probability in each model is assigned to \(y=0\).
| Parameter value | \(p(y=1,b=0\mid\theta)\) | \(p(y=1,b=1\mid\theta)\) | \(L(\theta)\) |
|---|---|---|---|
| Current | 0.4 | 0.1 | 0.5 |
| Candidate | 0.6 | 0.2 | 0.8 |
The current conditional probabilities are \(q_t(0)=0.4/0.5=0.8\) and \(q_t(1)=0.1/0.5=0.2\). The complete-data density ratios are \(0.6/0.4=1.5\) and \(0.2/0.1=2\). Therefore the Q difference is
This is positive, so the candidate satisfies the Q-increase condition. The actual log-likelihood increase is \(\log(0.8/0.5)=\log(1.6)=0.4700\), which is at least \(0.4630\). Hence the likelihood rises from \(0.5\) to \(0.8\). All probabilities are positive and finite, and the latent support is the same at both parameter values, so the expectations and ratios used in the argument are well-defined.
Why the Bound Can Be Strict
The gap between the two sides of the bound reflects variation in the complete-data density ratio across latent effects. When that ratio is constant under the current conditional distribution, observing the latent effect would not change the ratio between candidate and current complete-data densities on the relevant support. Otherwise, averaging the ratio and taking its logarithm need not give the same result as averaging its logarithm.
Worked Example: Equality When the Ratio Is Constant
Suppose a hypothetical current model assigns joint probabilities \(0.3\) and \(0.2\) to \((y=1,b=0)\) and \((y=1,b=1)\), respectively. Its observed likelihood is \(0.5\), so \(q_t(0)=0.6\) and \(q_t(1)=0.4\). Suppose the candidate assigns probabilities \(0.36\) and \(0.24\) to those same outcomes, with the rest of its probability assigned to \(y=0\).
Both complete-data ratios are \(1.2\). Thus \(Q(\theta^{(t+1)}\mid\theta^{(t)})-Q(\theta^{(t)}\mid\theta^{(t)})=0.6\log(1.2)+0.4\log(1.2)=\log(1.2)=0.1823\). The observed likelihood changes from \(0.5\) to \(0.36+0.24=0.6\), giving \(\log(0.6/0.5)=\log(1.2)=0.1823\). The bound is exact because the ratio is constant over the current conditional distribution’s support. The probabilities are valid and positive, so both Q values are finite and the equality calculation applies.
Conditions, Failure Modes, and Diagnostics
In a nonlinear mixed-effects model, the latent vector \(b\) may contain several subject-specific effects, and \(p(y,b\mid\theta)\) combines the observation model with the random-effects distribution. The ascent proof itself does not require a Gaussian random-effects distribution, a particular pharmacokinetic model, or a closed-form E-step. It does require that the stated conditional expectations and likelihoods exist and that the update actually meets the Q-increase condition.
- Undefined or infinite Q values. If either Q value is infinite, their comparison may not define a finite difference. In particular, \(-\infty\geq-\infty\) in an extended-real convention cannot be used as evidence of ascent.
- Candidate density zero under the current conditional distribution. Then the log ratio can be \(-\infty\), and the finite-difference argument does not apply as stated. Check the support and integrability conditions rather than silently subtracting infinities.
- Approximate E- or M-steps. Numerical integration error can make the computed Q difference have the wrong sign; an optimizer that stops early may also fail to increase Q. The exact theorem guarantees ascent only when the true conditional Q does not decrease and the finiteness conditions hold.
- Likelihood arithmetic errors. Underflow, inaccurate integration, or inconsistent evaluations can make successive computed likelihoods appear to decrease. Check integration accuracy and evaluate log-likelihoods stably, especially when likelihood contributions are very small.
A useful diagnostic is to monitor both the conditional Q change and the marginal log-likelihood change using sufficiently accurate calculations. If a reliable Q increase is accompanied by a genuine likelihood decrease, at least one premise or computation is wrong. A decrease in Q, on the other hand, removes this guarantee but does not by itself prove that the observed likelihood fell.
Worked Example: Why Two Negative-Infinite Q Values Are Not an Ascent Condition
Consider the hypothetical density \(f_t(x)=1/[x(\log x)^2]\) for \(x\geq e\). It integrates to one: using \(u=\log x\), its integral is \(\int_1^\infty u^{-2}\,du=1\). Suppose that, for the observed outcome, the current complete-data density is \(f_t(x)\), while a candidate assigns density \(f_c(x)=\tfrac12 f_t(x)\); the candidate assigns the remaining probability to another observed outcome.
Under the current model, the conditional density of \(x\) given the observed outcome is \(f_t(x)\), and the current observed likelihood contribution is \(1\). Under the candidate, that contribution is \(1/2\), so the log-likelihood change is \(\log(1/2)=-0.6931\): the likelihood decreases.
However, the current expected log density is negative infinity. Indeed, under \(u=\log x\), \(u\) has density \(1/u^2\) on \(u\geq1\), and \(\log f_t(x)=-u-2\log u\) has expectation \(-\infty\). The candidate log density differs by the finite constant \(\log(1/2)\), so its expected log density is also negative infinity. Comparing these two extended-real Q values does not establish a nonnegative Q difference: the subtraction \((-\infty)-(-\infty)\) is undefined. This example violates the finite-Q and finite-difference conditions, so it does not contradict the ascent bound.
How EM Ascent Relates to Other Updates
An exact EM update maximizes \(Q(\theta\mid\theta^{(t)})\), while a generalized EM update need only find a candidate that does not decrease it. Both satisfy the ascent argument when the required conditions hold. A direct optimizer of the observed-data likelihood instead targets \(L(\theta)\) itself and does not rely on the Q bound, although it must evaluate or approximate the marginal likelihood. These approaches have different computational requirements; the ascent property alone does not say which will be faster or more accurate in a given model.
Nor does monotone likelihood ascent guarantee convergence to a unique or global maximum. It establishes that each valid update does not lower the likelihood, not that every sequence of updates reaches the best possible parameter value. A flat likelihood, boundary behavior, or poor numerical approximations can still complicate fitting.
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions below.
- What distribution is used to calculate \(Q(\theta\mid\theta^{(t)})\), and which part of that definition changes as \(\theta\) varies?
- If a valid update increases Q by \(0.12\), what can be concluded about the log-likelihood change?
- Can a Q decrease prove that the observed likelihood decreased? Explain.
- Why is a comparison of two Q values both equal to negative infinity insufficient to establish likelihood ascent?
Solutions. (1) The expectation uses the current conditional distribution \(p(b\mid y,\theta^{(t)})\); the candidate parameter in the complete-data log density varies. (2) The log-likelihood change is at least \(0.12\), so the likelihood increases by a factor of at least \(\exp(0.12)\), approximately \(1.128\), under the theorem’s conditions. (3) No. The ascent bound gives a sufficient condition; a negative Q change provides no conclusion about the sign of the likelihood change. (4) Subtracting two negative infinities is undefined, so there is no well-defined finite Q difference to place in the bound.