Averaging Sufficient Statistics During SAEM

Why Stabilize Sufficient Statistics?

In SAEM, each latent-effect simulation produces a noisy realization of the complete-data sufficient statistics. The M-step uses the current statistic estimate to update the population parameters. If each new simulation replaced the previous statistic estimate outright, the parameter updates could fluctuate sharply from one iteration to the next. Stochastic approximation instead blends the new statistic with the accumulated estimate. This reduces the influence of any one draw, especially when the gain becomes small late in the run.

The object being stabilized matters. SAEM is built around accumulating sufficient statistics and then applying the M-step defined by the complete-data model. Averaging parameter vectors after separate M-steps is a different operation. When the M-step is nonlinear, the two operations need not agree, and a parameter average may not correspond to any valid sufficient-statistic update.

Definition: Let $Z_k$ denote the simulated latent effects at iteration $k$, and let $T(y,Z_k)$ be the complete-data sufficient-statistic vector for observed data $y$. A stochastic-approximation update is $$ S_k=S_{k-1}+\gamma_k\{T(y,Z_k)-S_{k-1}\} =(1-\gamma_k)S_{k-1}+\gamma_k T(y,Z_k), $$ where $S_k$ is the running statistic estimate and $0<\gamma_k\leq 1$ is the gain. The parameter update is $\theta_k=M(S_k)$, where $M$ is the M-step mapping implied by the complete-data model.

In the usual SAEM interpretation, $Z_k$ is drawn from, or generated by a transition kernel targeting, the latent-effects distribution conditional on the data and the current parameter iterate. Thus the statistic being estimated may change as $\theta_k$ changes. The update is intended to track the corresponding conditional expectation, not to average arbitrary quantities. For the standard stochastic-approximation convergence argument, the model and statistic representation must be appropriate, the latent-effect simulation must adequately target the conditional distribution, and the gain schedule must satisfy suitable conditions. A common schedule has $\sum_k\gamma_k=\infty$ and $\sum_k\gamma_k^2<\infty$ after any initial adaptation phase. These conditions alone do not compensate for a poorly mixing simulation or a misspecified model.

What the Gain Does to the Statistic

The update is a weighted average. A gain of 1 replaces the old statistic with the new draw. A gain between 0 and 1 gives the new draw weight $\gamma_k$ and retains weight $1-\gamma_k$ on the previous estimate. Repeated updates therefore smooth iteration-to-iteration noise. With a decreasing gain, early draws can move the estimate substantially, while later draws have progressively less effect.

Worked Example: A Decreasing Gain Smooths Noisy Draws

Suppose one component of the sufficient-statistic vector has hypothetical draw values $T_1=12$, $T_2=8$, $T_3=10$, and $T_4=9$. Start at $S_0=8$, and use the gains $\gamma_1=0.5$, $\gamma_2=0.25$, $\gamma_3=0.25$, and $\gamma_4=0.25$. This scalar component has the same units as its underlying statistic.

The successive updates are:

\begin{aligned} S_1&=8+0.5(12-8)=10,\\ S_2&=10+0.25(8-10)=9.5,\\ S_3&=9.5+0.25(10-9.5)=9.625,\\ S_4&=9.625+0.25(9-9.625)=9.46875. \end{aligned}

The last simulated value is 9, but the updated statistic is 9.46875 rather than 9: earlier information still contributes. The calculation shows smoothing, not proof that the statistic has converged. Whether the sequence is tracking its intended conditional expectation depends on the latent-effect simulation, the changing parameter values, and the gain schedule.

For a fixed target and independent draws with finite variance, averaging $m$ draws reduces the variance of their mean to the single-draw variance divided by $m$. The intuition is that positive and negative deviations tend to offset. SAEM draws are often dependent, however, because latent effects may be generated by Markov chain transitions. Dependence reduces the benefit of averaging: the variance is governed by both the marginal draw variability and the serial correlation. A small gain can damp visible changes while still retaining bias or slow drift if the simulation has not adequately explored the conditional distribution.

Average Statistics First, Then Apply the M-Step

The sufficient-statistic representation connects the latent-data calculation to the M-step. Suppose the complete-data log-likelihood can be expressed as a parameter-dependent function of $T(y,Z)$, plus terms not involving the parameters. Then its conditional expectation depends on the latent effects through the conditional expectation of $T(y,Z)$. SAEM approximates that statistic and applies the model’s M-step mapping to it. This is the reason the statistic scale—not an arbitrary parameter scale—is central to the update.

Worked Example: Gaussian Random-Effect Statistics

Consider a hypothetical model with $m=2$ random intercepts $b_1,b_2$, independently distributed as $N(\mu,\omega^2)$. For this Gaussian model, the relevant sufficient statistics are $T_1=b_1+b_2$ and $T_2=b_1^2+b_2^2$. Given statistic values, the M-step is

\hat{\mu}=\frac{T_1}{m}, \qquad \hat{\omega}^2=\frac{T_2}{m}-\hat{\mu}^{\,2}.

For illustration, take three hypothetical latent-effect draws: $(0,2)$, $(2,2)$, and $(4,0)$. Their statistic pairs $(T_1,T_2)$ are $(2,4)$, $(4,8)$, and $(4,16)$. The average statistics are

\bar{T}_1=\frac{2+4+4}{3}=\frac{10}{3}, \qquad \bar{T}_2=\frac{4+8+16}{3}=\frac{28}{3}.

Applying the M-step to those averaged statistics gives

\hat{\mu}=\frac{10/3}{2}=\frac{5}{3}, \qquad \hat{\omega}^2=\frac{28/3}{2}-\left(\frac{5}{3}\right)^2 =\frac{14}{3}-\frac{25}{9} =\frac{17}{9}\approx 1.889.

The mean is in the units of $b_i$, and the variance is in squared units. This is an illustration of applying the M-step to averaged statistics; the three draws are not being claimed as a converged SAEM estimate. Notice that a finite collection of averaged statistics must also be compatible with the M-step’s parameter constraints. In this example the resulting variance is positive.

A useful justification for the distinction from parameter averaging comes directly from the variance formula. For a draw of two random effects, the M-step variance is the average of their squares minus the square of their mean. If one first computes this variance separately for each draw and then averages, the result generally differs from applying the M-step to the averaged statistics. Specifically, for $m$ effects,

\frac{\overline{T_2}}{m} -\left(\frac{\overline{T_1}}{m}\right)^2 = \overline{\left(\frac{T_2}{m} -\left(\frac{T_1}{m}\right)^2\right)} + \overline{\left(\frac{T_1}{m}\right)^2} -\left(\overline{\frac{T_1}{m}}\right)^2.

The final two terms on the right are the variance, across draws, of the draw-specific means. Thus the variance from averaged statistics is the average draw-specific variance plus the variation in draw-specific means. Unless those means are constant, the two procedures differ. This is one specific instance of a general fact: applying a nonlinear function after averaging is not generally equivalent to averaging after applying the function.

When Further Averaging Helps—and When It Misleads

The stochastic-approximation recursion is itself a way to stabilize sufficient statistics. Other approaches, such as averaging several statistic evaluations into a batch before updating, can reduce Monte Carlo noise when the draws are sufficiently informative. Such averaging should preserve the statistic vector required by the M-step. It should also be interpreted in light of the simulation dependence and the possibility that the conditional expectation changes as the parameter iterate moves.

Worked Example: A Moving Average Can Lag Behind a Changing Target

Suppose a statistic is near 0 for three iterations, and then its target shifts to 10. For simplicity, let the observed statistic values be $0,0,0,10,10$. A trailing three-value average is 0 after the first three values, then

\frac{0+0+10}{3}=3.333 \qquad\text{and}\qquad \frac{0+10+10}{3}=6.667

after the fourth and fifth values, respectively. The averages reduce abrupt variation but lag the new level of 10. This is a hypothetical illustration of finite-window smoothing, not a prescription for a window length. If the target is still changing because the parameter estimates are moving, an apparently smooth statistic can be a stale statistic rather than a stable estimate of the current target.

Averaging parameter iterates is not automatically wrong, but it needs a separate justification. For example, an averaging scheme may be established for a particular stochastic approximation problem with specified assumptions and a defined inferential target. That is not the same as replacing SAEM’s statistic update with an unmotivated average of parameter vectors. Even if each parameter vector satisfies constraints, their average may not be the M-step result for the averaged sufficient statistics. Conversely, averaging on a constrained parameter scale can produce results that do not preserve the model’s intended covariance or parameterization.

Approach	What is averaged?	Main consideration
SAEM stochastic approximation	New sufficient statistics with the running statistic estimate	Uses the model’s statistic-to-parameter M-step; gain and latent-effect simulation still matter.
Batch or window averaging	Several statistic evaluations, or a selected sequence of statistics	Can reduce visible noise but may retain dependence or introduce lag when the target moves.
Parameter-iterate averaging	Parameter vectors after M-steps	Not generally equivalent to averaging sufficient statistics; requires its own rationale and constraint checks.

Diagnostics and Practical Limits

A smooth trace is not sufficient evidence that the statistics have stabilized correctly. Examine statistic traces alongside parameter traces and latent-chain behavior. If statistics continue to drift, parameter changes remain substantial, or different chains explore different regions, smoothing may conceal rather than resolve the problem. A highly persistent trace can reflect a small gain, strong simulation autocorrelation, a slowly changing target, or genuine instability; those explanations require different responses.

Check that every statistic component has the expected scale and that the M-step output is valid. For example, a covariance update must satisfy its model’s constraints; a smooth update does not guarantee this. Large abrupt jumps can arise from poor initial values, unstable simulation, or a statistic near a boundary. Conversely, a near-constant trace can arise because the gain has become very small, even when the chain has not adequately explored the conditional distribution. Compare independent runs or inspect simulation diagnostics where appropriate, and do not treat smoothing as a repair for poor mixing, model misspecification, or an unsuitable sufficient-statistic representation.

Underweighting new information: a very small gain can make adaptation to a changed target extremely slow.
Overreacting to individual draws: a gain that remains large can leave substantial late-iteration fluctuation.
Confusing smoothness with convergence: a running average can look stable while the underlying draws remain poorly representative.
Averaging the wrong object: post hoc averages of parameter estimates do not automatically implement the SAEM M-step.

Key takeaway: In SAEM, stabilize the sufficient statistics that define the complete-data M-step. The gain controls how much each simulated statistic contributes. Additional averaging can be useful only when its target, dependence, and effect on the M-step are understood; parameter averaging is not a substitute without a separate justification.

Check Your Understanding

Try each question, then compare with the solutions below.

If $S_{k-1}=6$, $T(y,Z_k)=14$, and $\gamma_k=0.25$, what is $S_k$?
Why might a three-value moving average respond slowly after the target statistic changes?
For two random effects, explain why averaging draw-specific variance estimates need not equal the variance from averaged sufficient statistics.
What is one reason a smooth sufficient-statistic trace might fail to show that SAEM is behaving well?

Solutions. (1) $S_k=6+0.25(14-6)=8$, in the same units as the statistic. (2) The average retains earlier values; for example, the mean of $0,0,10$ is $10/3$, below the new target of 10. (3) The M-step variance subtracts the square of the mean. The square of an average is generally not the average of squares; the difference includes the across-draw variance of the draw-specific means. (4) A small gain can make the trace look nearly constant even if the latent-effect simulation mixes poorly, so smoothness should be checked alongside simulation and parameter diagnostics.