Checking a Canonical Model in Simulation

Why Check a Canonical Model by Simulation?

A canonical joint distribution gives precise predictions about the behavior of cumulative score statistics across analysis looks. Simulation provides a practical way to check whether code intended to represent that distribution reproduces its key features. The check is especially useful when several information levels are involved: an implementation can produce plausible values at each look while getting the dependence between looks wrong.

The aim here is to check a specified canonical model, not to establish that a real study satisfies its assumptions. A simulation can reveal coding errors or discrepancies from the model’s mathematical predictions. It cannot, by itself, show that the model is appropriate for data from a particular trial.

The information levels need not be integer-valued. Their spacing determines how much new information each increment contributes. Write \(t_0=0\), so the increment between looks \(k-1\) and \(k\) has information length \(t_k-t_{k-1}\).

Generating a Canonical Score Path

A convenient simulation method starts with independent standard normal random variables \(\varepsilon_1,\ldots,\varepsilon_K\), each with mean zero and variance one. For each look, the cumulative score numerator includes the drift accumulated to that information level and the noise from all increments up to that look. Dividing by the square root of the information level gives the standardized statistic.

This construction is repeated independently for each simulated path. The same increment \(\varepsilon_j\) contributes to every later cumulative statistic, because later looks retain information accumulated at earlier looks. That reuse is essential: it creates the dependence between the cumulative statistics.

Why These Are the Canonical Predictions

The mean follows directly from the construction. Each \(\varepsilon_j\) has mean zero, so the only expected contribution to the numerator at look \(k\) is \(\theta t_k\). Dividing by \(\sqrt{t_k}\) gives \(E(Z_k)=\theta\sqrt{t_k}\).

For the variance and correlation, define the accumulated noise \(W_k=\sum_{j=1}^{k}\sqrt{t_j-t_{j-1}}\,\varepsilon_j\). Independent increments have zero covariance with one another. Therefore the variance of \(W_k\) is the sum of the increment variances:

Since \(Z_k\) is the sum of a fixed drift and \(W_k/\sqrt{t_k}\), its variance is \(t_k/t_k=1\). For \(i<j\), the noise \(W_i\) is part of \(W_j\); the additional increments from \(i+1\) through \(j\) are independent of \(W_i\). Thus \(\operatorname{Cov}(W_i,W_j)=\operatorname{Var}(W_i)=t_i\). Standardizing both statistics gives:

Because both variances are one, this covariance is also their correlation. The drift terms are fixed constants and do not affect covariance. This derivation also explains why the correlation is determined by information levels rather than by the value of \(\theta\).

A Practical Simulation Check

For a simulation with \(M\) independent paths, calculate the empirical mean and sample variance of the \(M\) values at each look. Calculate the sample correlation for each pair of looks. Compare those summaries with their theoretical targets, rather than expecting a finite simulation to reproduce them exactly.

Worked Examples

Interpreting the Check and Avoiding a Common Error

Empirical means, variances, and correlations are useful summaries, but they are not exact acceptance criteria. Their values vary from run to run. Larger \(M\) generally reduces that Monte Carlo variation, while a mismatch that remains substantial across repeated runs merits investigation. Inspecting the information levels, increment variances, and reuse of earlier increments can help locate an implementation error.

A common mistake is to generate a separate independent standard normal value for each cumulative statistic. That procedure gives variance one at each look under the null, so a check of marginal variances alone may appear successful. But it gives zero correlation between distinct looks, rather than the canonical positive correlation \(\sqrt{t_i/t_j}\). Matching each marginal distribution is not sufficient to match the joint distribution.

There are two sound implementation approaches. Independent normal increments make the source of dependence explicit and are convenient for generating paths. Alternatively, one can generate a multivariate normal vector with the canonical mean vector and covariance matrix. That approach is mathematically equivalent when the covariance matrix is correctly specified, but may make it less obvious how earlier increments contribute to later looks. In either case, checking only means and variances misses dependence errors.

The check has limits. Agreement of these summaries is not a proof that every feature of the simulated joint distribution is correct, and a canonical-model simulation does not validate the assumptions that justify using that model for a real trial. The present check is best treated as an implementation verification step: it asks whether the code reproduces the stated model’s basic moment and dependence structure.