Burn-In and the SAEM Gain Transition

Why Separate SAEM Into Two Gain Phases?

In SAEM, each iteration uses a simulated latent-effect draw to update estimates of the conditional sufficient statistics, then uses those statistics to update the population parameters. Early in a run, the current statistics may be poor starting values. A larger gain lets a new simulated draw move the statistics quickly; a smaller gain later reduces the influence of any one draw and helps stabilize the recursion.

A burn-in phase in this context is a finite initial part of the SAEM recursion that uses relatively large gains before a later stabilization phase uses a decreasing gain schedule. It is a design choice, not a guarantee that the algorithm has reached a good estimate by the time the phase ends. In particular, it does not replace checking whether the latent-effect simulation is exploring the relevant conditional distribution.

Definition: Let $S_k$ be the running estimate of the sufficient-statistic vector at iteration $k$, and let $s(Z_k)$ be the statistic evaluated at the simulated latent-effect state $Z_k$. The SAEM stochastic-approximation update is $$ S_k=S_{k-1}+\gamma_k\{s(Z_k)-S_{k-1}\} =(1-\gamma_k)S_{k-1}+\gamma_k s(Z_k), $$ where $0<\gamma_k\leq 1$ is the gain. The population-parameter update is written $\theta_k=M(S_k)$, where $M$ denotes the M-step mapping. A gain transition changes the gains used in this recursion; it does not reset $S_k$.

When $\gamma_k=1$, the update sets $S_k=s(Z_k)$: the new draw entirely replaces the previous statistic estimate. For $0<\gamma_k<1$, the update combines the old estimate and the new draw. Thus, a large gain can speed adaptation but also leaves the statistics and parameter estimates more sensitive to simulation noise.

Choosing a Transition and a Stabilization Schedule

A simple staged design uses a finite high-gain period followed by a decreasing sequence. For example, let $K$ be the last iteration of the initial phase, set $\gamma_k=1$ for $k\leq K$, and use the following tail:

\gamma_k=\frac{1}{k-K+1}, \qquad k>K.

This is one illustrative schedule, not a universally optimal choice. The relevant stochastic-approximation conditions commonly include $\sum_k \gamma_k=\infty$ and $\sum_k\gamma_k^2<\infty$, alongside conditions on the model, the M-step, and the simulation process. For the displayed tail, writing $j=k-K$ gives $\gamma_{K+j}=1/(j+1)$. Its sum diverges, while its squared sum converges. A finite prefix with $\gamma_k=1$ adds only finitely many terms to either sum, so it does not change these tail properties.

This argument justifies the compatibility of a finite high-gain prefix with that particular asymptotic gain requirement. It does not, by itself, prove that the SAEM estimates converge: the other assumptions needed for convergence still matter. Nor does it justify keeping $\gamma_k=1$ indefinitely. With a gain fixed at one, the statistic estimate continues to be replaced by individual simulated statistics rather than increasingly averaging them.

The transition time should be chosen to balance two risks. Switching too early can make the later, smaller gains slow to correct initial adaptation. Switching too late keeps the recursion unnecessarily sensitive to individual draws and delays stabilization. There is no universal iteration number that separates these phases. A defensible choice uses the behavior of relevant sufficient statistics and parameter estimates across iterations, considered alongside the quality of the conditional simulation.

Worked Example: Calculating Updates Across the Transition

Suppose a scalar sufficient statistic is initialized at $S_0=4$. In a hypothetical run, the simulated statistic values for the first six iterations are $5, 7, 6, 8, 5,$ and $7$. Use $\gamma_k=1$ for iterations 1 through 3, then $\gamma_k=1/(k-3+1)$ for later iterations. Calculate the statistic estimates.

For the first three iterations, a gain of one replaces the old value with the new simulated statistic:

S_1=5,\qquad S_2=7,\qquad S_3=6.

At iteration 4, $\gamma_4=1/(4-3+1)=1/2$, so $S_4=(1-1/2)(6)+(1/2)(8)=7$. At iteration 5, $\gamma_5=1/3$, giving $S_5=(2/3)(7)+(1/3)(5)=19/3\approx6.333$. At iteration 6, $\gamma_6=1/4$, giving $S_6=(3/4)(19/3)+(1/4)(7)=6.5$.

The gain transition did not reset the statistic: the first smaller-gain update starts from $S_3=6$. The later values are less dominated by a single new draw than the values in the gain-one phase. This calculation illustrates the mechanics only; six updates cannot establish that a real run has stabilized.

Assessing Whether the Switch Is Timely

A transition assessment should ask whether the high-gain phase has done enough to move the recursion away from its initial transient, and whether the post-transition behavior is consistent with gradual stabilization. Inspect more than one relevant quantity: a parameter trace can look flat while an underlying sufficient statistic continues to drift, or one parameter may be stable while another is not. If the latent-effect simulation is poorly mixing, a smooth trace can also be misleading because it may reflect limited exploration rather than a well-estimated conditional expectation.

A practical assessment can use a few complementary checks:

Look for sustained movement before the proposed switch. Large, directional changes in key statistics or parameters across several iterations suggest the run is still adapting. A single change is not enough to diagnose a trend.
Inspect behavior after the switch. Smaller gains should reduce the effect of individual draws, but estimates may still move. Persistent directional drift after the transition is evidence to investigate, not proof by itself that the gain schedule is wrong.
Check simulation quality. Consider whether latent-effect chains explore adequately, using the chain-mixing concepts relevant to the model. A gain change cannot fix a chain that repeatedly fails to explore important regions.
Compare several quantities and, where feasible, repeated runs. Agreement across statistics and parameters is more informative than apparent stability in a single trace. Different initial values or simulation seeds can reveal sensitivity that one run conceals.

These checks support a reasoned choice; they do not supply a formal test for the correct transition iteration. Do not choose a switch merely because one short segment appears flat. Conversely, do not insist that every parameter be constant before switching: stochastic updates remain noisy, and the purpose of the decreasing-gain phase is to reduce that noise while allowing continued adaptation.

Worked Example: Comparing Two Candidate Switches

Consider a hypothetical parameter trace with values $2.0, 3.4, 4.6, 5.1, 5.3, 5.2, 5.25,$ and $5.22$ at iterations 1 through 8. A proposed early switch is after iteration 3; a later switch is after iteration 6. For a simple descriptive check, calculate the range in each three-iteration block: maximum minus minimum.

For iterations 1–3, the range is $4.6-2.0=2.6$, showing substantial movement. For iterations 4–6, it is $5.3-5.1=0.2$. For iterations 6–8, it is $5.25-5.2=0.05$. The trace therefore shows much smaller variation near iterations 6–8 than at the beginning.

On this parameter alone, switching after iteration 3 appears early: the trace continues to rise from $4.6$ to $5.3$ over the next two iterations. Extending the high-gain phase to iteration 6 may be reasonable, since the values from iterations 4–6 vary less. But the small range in iterations 6–8 does not prove that the later switch is optimal; it covers only three values and may reflect simulation noise or a temporary plateau. Before choosing, inspect other parameters and statistics and assess the latent-effect simulation. This is a descriptive decision aid, not a hypothesis test or a universal cutoff rule.

Burn-In Is Not MCMC Burn-In

The phrase “burn-in” is also used for discarding early states of a Markov chain before using later states to represent a target distribution. That is distinct from the SAEM gain burn-in defined here. In SAEM, the gain controls how strongly a simulated statistic changes $S_k$; a high-gain phase does not automatically mean that early chain states have been discarded. Likewise, reducing the SAEM gain does not establish that a chain has mixed adequately.

Worked Example: Separating Gain Choice From Chain Quality

Suppose a hypothetical SAEM run uses $\gamma_k=1$ for its first 20 iterations, while the latent-effect simulation at each iteration remains near the same narrow region of its conditional distribution. The statistics change as each new draw replaces the old one, but the draws do not explore sufficiently.

Changing to smaller gains may make the statistic trace look smoother, because each draw has less influence. It does not make the simulated draws explore regions they have not visited. The appropriate conclusion is that there are two separate issues: the gain schedule controls the stochastic-approximation update, while chain mixing concerns the adequacy of the latent-effect simulation. Both must be considered; a smooth post-transition trace alone cannot establish that the SAEM estimates are reliable.

Alternatives, Limitations, and Common Mistakes

One alternative is to use a decreasing gain from the first iteration. This avoids a separate high-gain phase and may work well when the initialization is already suitable. A staged schedule instead gives the early recursion more capacity to adapt before the gain becomes small. A third option—keeping a large gain for a very long time—can make adaptation responsive, but leaves persistent sensitivity to individual draws and is not a substitute for a decreasing tail when the usual stochastic-approximation conditions are required.

The staged approach has limitations. It adds a transition choice, and traces provide evidence rather than certainty about when adaptation is adequate. A transition selected after inspecting a run can also complicate theoretical guarantees: the standard gain-sequence argument above directly covers a fixed finite prefix followed by the specified tail, but data-dependent switching is not automatically covered by that argument. If a data-dependent rule is used, its interaction with the convergence conditions must be justified rather than assumed.

Common mistake: “The trace was nearly flat for a few iterations, so the transition is proven to be correctly timed and the algorithm has converged.” This reasoning confuses a short-term visual pattern with a convergence guarantee. A correction is to treat trace stability as one diagnostic, inspect multiple statistics and parameters, check simulation quality, and retain the distinction between a reasonable transition decision and proof of convergence.

Key takeaway: A finite high-gain phase can speed early adaptation, while a decreasing tail reduces the influence of individual simulated draws. Choose the transition using several sources of evidence, and remember that neither a gain schedule nor a smooth trace can compensate for inadequate latent-effect simulation.

Check Your Understanding

Answer each question, then compare with the solutions below.

If $\gamma_k=1$, what does the SAEM statistic update become?
For a finite high-gain prefix followed by $\gamma_{K+j}=1/(j+1)$, why do the finite initial gains not change whether the tail meets the usual sum and squared-sum conditions?
A parameter trace looks smooth after the gain is reduced, but the latent-effect chain explores only a narrow region. What can and cannot be concluded?
Give one reason a transition after only a few iterations could be too early and one reason a very long high-gain phase could be undesirable.

Solutions. (1) $S_k=s(Z_k)$: the new simulated statistic replaces the previous estimate. (2) A finite prefix contributes only finite amounts to both sums, so the tail determines whether the sums diverge or converge. (3) The update may be less sensitive to individual draws, but the smoothness does not establish adequate exploration or reliable estimates; chain quality must be assessed separately. (4) An early switch may slow correction of ongoing initial drift; a long high-gain phase keeps the statistic highly sensitive to individual simulated draws and delays stabilization.