Generating Trial Paths for Operating-Characteristic Simulations

Why Simulate Complete Trial Paths?

An operating-characteristic simulation needs more than a plausible statistic at each planned analysis. It needs a plausible sequence of statistics from the same hypothetical trial. When interim analyses use accumulating data, a later statistic includes information already used at earlier looks. The statistics are therefore dependent. Simulating each look separately, or drawing independent standard normal values for each look, loses that dependence and produces trial paths that do not represent repeated analyses of accumulating evidence.

This tutorial develops a standard canonical model for generating those paths. It shows how to specify the information at each look, how the null and alternative parameters determine the path distribution, and how to generate a path using independent increments. The aim is to produce the joint statistics correctly; decisions about stopping are applied afterward.

Definition: A trial path is the ordered vector of test statistics $(Z_1,\ldots,Z_K)$ that would be observed at the $K$ planned interim and final analyses in one hypothetical trial, if the study’s accumulating data were available at every look. A path-generation method preserves the joint structure when it generates this vector from the specified joint distribution, not merely the correct marginal distribution at each look.

The Canonical Joint Model

Suppose the planned analyses occur at increasing information levels $0<I_1<\cdots<I_K$. Information measures the precision available for the treatment-effect estimate; it need not equal the number of participants. The canonical model assumes that the cumulative score-like quantity $Y_k$ at look $k$ has mean $\theta I_k$ and variance $I_k$. Here $\theta$ is a fixed parameter under the scenario being simulated. Under the null, $\theta=0$; under a specified alternative, $\theta$ is the alternative value on this canonical scale.

The model’s key structural assumption is that the increments $Y_k-Y_{k-1}$ from non-overlapping information intervals are independent, with $Y_0=I_0=0$. Each increment has mean $\theta(I_k-I_{k-1})$ and variance $I_k-I_{k-1}$. The standardized statistic is $Z_k=Y_k/\sqrt{I_k}$. Thus, statistics at successive looks are not independent, even though the increments used to construct them are.

Canonical path model: For $k=1,\ldots,K$, let $\Delta I_k=I_k-I_{k-1}$, with $I_0=0$. Generate independent $X_k\sim N(0,1)$, then set $$ Y_k=Y_{k-1}+\theta\Delta I_k+\sqrt{\Delta I_k}\,X_k, \qquad Z_k=\frac{Y_k}{\sqrt{I_k}}. $$ Equivalently, $Z_1,\ldots,Z_K$ are jointly normal, with $$ E(Z_k)=\theta\sqrt{I_k}, \qquad \operatorname{Cov}(Z_i,Z_j)=\sqrt{\frac{I_i}{I_j}}\quad (i\leq j). $$

For planning, it is often convenient to work with information fractions $t_k=I_k/I_{\max}$, where $I_{\max}=I_K$. If the final-analysis mean under the alternative is $\delta=\theta\sqrt{I_{\max}}$, then $E(Z_k)=\delta\sqrt{t_k}$. The covariance is $\sqrt{t_i/t_j}$ for $i\leq j$. This parameterization makes the effect scenario and timing of looks easy to specify. It does not change the underlying joint model.

Why the Statistics Are Correlated

The dependence follows directly from data accumulation. For $i\leq j$, $Y_i$ is part of $Y_j$. The later score contains the earlier score plus new information. Under the independent-increment model, the new increment after look $i$ contributes no covariance with $Y_i$, so $\operatorname{Cov}(Y_i,Y_j)=\operatorname{Var}(Y_i)=I_i$. Standardizing each cumulative score gives the correlation structure below.

\operatorname{Cov}(Z_i,Z_j) = \operatorname{Cov}\left(\frac{Y_i}{\sqrt{I_i}},\frac{Y_j}{\sqrt{I_j}}\right) = \frac{I_i}{\sqrt{I_i I_j}} = \sqrt{\frac{I_i}{I_j}},\qquad i\leq j.

In particular, the correlation between two interim statistics depends on their relative information, not directly on the treatment effect. Under the null, their means are zero; under an alternative, their means shift according to information, while the canonical covariance remains the same. This is a model-based property: if the analysis statistics or data-generating mechanism do not satisfy the canonical assumptions, this covariance formula may not be appropriate.

Worked Example: Calculating the Joint Structure

Consider three planned looks at information fractions $t_1=0.25$, $t_2=0.60$, and $t_3=1.00$. Under the canonical model, calculate the correlations and the means if the final-analysis mean under the alternative is $\delta=2.40$.

For the first and second looks, the correlation is $$ \operatorname{Corr}(Z_1,Z_2)=\sqrt{\frac{0.25}{0.60}}=0.6455. $$ For the first and final looks it is $\sqrt{0.25/1.00}=0.5000$; for the second and final looks it is $\sqrt{0.60/1.00}=0.7746$. The full correlation matrix is therefore $$ \begin{pmatrix} 1 & 0.6455 & 0.5000\\ 0.6455 & 1 & 0.7746\\ 0.5000 & 0.7746 & 1 \end{pmatrix}. $$

The mean at each look is $\delta\sqrt{t_k}$. The three means are $2.40\sqrt{0.25}=1.200$, $2.40\sqrt{0.60}=1.859$, and $2.40\sqrt{1.00}=2.400$. These calculations specify the means and correlations of the path; they do not specify any stopping decision.

Generating a Path from Independent Increments

A direct way to simulate is to generate one independent standard normal value for each information interval, then accumulate the resulting increments. This approach makes the source of the correlation explicit. It also ensures that the same simulated trial has a coherent cumulative statistic at every look.

Specify information and the scenario.
Record increasing information levels $I_1,\ldots,I_K$, and choose $\theta$ for the null or alternative scenario.

Generate independent increments.
Draw independent standard normal values $X_1,\ldots,X_K$. For interval $k$, form $\theta\Delta I_k+\sqrt{\Delta I_k}X_k$.

Accumulate and standardize.
Add each increment to the previous cumulative score to obtain $Y_k$, then calculate $Z_k=Y_k/\sqrt{I_k}$.

Retain the whole vector.
Store $(Z_1,\ldots,Z_K)$ as one simulated path, along with the scenario and information schedule used to generate it.

Worked Example: Building One Alternative Path

Use information fractions $0.25,0.60,1.00$, final-analysis mean $\delta=1.50$, and independent normal draws $X_1=0.40$, $X_2=-1.00$, and $X_3=0.50$. On the information-fraction scale, the parameter per unit of information is $\theta=\delta=1.50$, and the interval lengths are $0.25,0.35,0.40$.

The first increment is $$ \Delta Y_1=1.50(0.25)+\sqrt{0.25}(0.40)=0.375+0.200=0.575. $$ Thus $Y_1=0.575$, and $Z_1=0.575/\sqrt{0.25}=1.150$.

At the second interval, $$ \Delta Y_2=1.50(0.35)+\sqrt{0.35}(-1.00) =0.525-0.5916=-0.0666. $$ So $Y_2=0.575-0.0666=0.5084$, and $Z_2=0.5084/\sqrt{0.60}=0.6563$.

At the final interval, $$ \Delta Y_3=1.50(0.40)+\sqrt{0.40}(0.50) =0.600+0.3162=0.9162. $$ Therefore $Y_3=0.5084+0.9162=1.4246$, and $Z_3=1.4246/\sqrt{1.00}=1.425$. The generated path is approximately $(1.150,0.6563,1.425)$. Its middle statistic can be lower than the first: new data can move the cumulative estimate in either direction, even under a positive alternative.

Checks on a Generated Path

A single path is random, so it is not expected to match the theoretical means or correlations exactly. Checks are most informative across many generated paths. Before interpreting simulated operating characteristics, verify that the simulation setup and generated quantities correspond to the intended model.

Check the information schedule. Information must be positive and strictly increasing. Zero-length or negative increments make the stated increment model invalid.
Check marginal means and variances across paths. At look $k$, simulated $Z_k$ values should have mean near $\theta\sqrt{I_k}$ and variance near 1, subject to ordinary simulation variation.
Check pairwise dependence. Across paths, the sample correlation of $Z_i$ and $Z_j$ should be near $\sqrt{I_i/I_j}$ for $i<j$. A near-zero correlation at successive looks is a warning that independent statistics may have been generated by mistake.
Check the construction. If increments are generated independently, confirm that the same cumulative $Y_k$ is used to form each corresponding $Z_k$, and that each statistic is divided by $\sqrt{I_k}$.

Worked Example: Detecting Independent-Look Generation

Suppose two analyses are planned at information fractions $0.40$ and $1.00$. The canonical correlation is $$ \operatorname{Corr}(Z_1,Z_2)=\sqrt{\frac{0.40}{1.00}}=0.6325. $$ A programmer instead generates $Z_1$ and $Z_2$ as independent standard normal values under the null. Each marginal distribution is correct: both have mean 0 and variance 1. But their correlation is 0, rather than 0.6325. The joint path is therefore wrong even though checks of each look separately would pass.

The correction is to generate an initial score increment for information $0.40$, then an independent additional increment for information $0.60$. The first statistic uses the initial cumulative score; the second uses the sum of both increments and is standardized by $\sqrt{1.00}$. This construction produces the required positive dependence.

Alternative Generation Methods and Their Limits

The independent-increment method is usually a transparent choice for canonical paths. An equivalent method is to draw the full vector from its multivariate normal distribution, using the mean vector and covariance matrix derived above. For instance, a Cholesky factorization can transform a vector of independent standard normal draws into a vector with the target covariance. This method can be convenient when the joint distribution is already specified, but it requires constructing the covariance matrix correctly and can obscure the cumulative-data interpretation. The two approaches should give the same joint distribution under the canonical model.

A third possibility is to simulate the trial’s underlying data and calculate each planned statistic from the data available at that look. This can be more suitable when the canonical normal approximation is poor or when the analysis statistic’s behavior depends on features not represented by the canonical model. It requires a defensible data-generating model and correctly reproducing the planned analysis at each look; it is not automatically more reliable merely because it simulates individual observations.

The canonical approach assumes a fixed parameter within the scenario, increasing information, independent increments, and approximately normal standardized statistics with the stated means and variances. If information is random, the effect changes over time, observations are not independent in the required way, or the statistic has a materially non-normal distribution, the simple formula may be inadequate. In those cases, the path generator must reflect the actual analysis and data-generating assumptions rather than impose canonical correlations by default. A path generator describes the joint behavior of statistics; it does not itself establish that the assumptions are scientifically or operationally plausible.

Common Pitfall

A common mistake is to argue that every look has the correct standard normal distribution under $H_0$, so independently generating a standard normal value at each look must be sufficient. That reasoning checks only the marginal distributions. Repeated analyses use overlapping accumulated information, so the joint distribution matters too. Under the canonical model, two looks at information fractions $0.40$ and $1.00$ have correlation $0.6325$, not zero. Generate independent increments, accumulate them, and then standardize; do not generate independent cumulative statistics.

Key takeaway: A valid canonical trial path is generated from independent information increments and then accumulated into correlated interim statistics. Specify the information schedule and scenario parameter first, and check both marginal behavior and dependence across looks.

Check Your Understanding

Answer each question, then compare with the solutions below.

For information fractions $0.36$ and $1.00$, what is the canonical correlation between the two interim statistics?
With final-analysis mean $\delta=2.00$, what is the expected statistic at information fraction $0.25$?
Why can separately generated standard normal statistics have correct marginal distributions but still form invalid trial paths?
Name one assumption of the canonical increment model that could fail and explain the consequence for path generation.

Solutions. (1) The correlation is $\sqrt{0.36/1.00}=0.6000$. (2) The expected statistic is $\delta\sqrt{t}=2.00\sqrt{0.25}=1.000$. (3) Correct marginal distributions do not ensure the correct joint dependence; independent draws have correlation zero, while the canonical correlation here depends on the information fractions. (4) For example, if standardized statistics are materially non-normal, normal increments may not reproduce their joint behavior. The generator would need to reflect a suitable data-generating and analysis model instead.