Comparing Realized and Nominal Spending

Why Compare Spending With Realized Error?

An alpha-spending function specifies how much of the one-sided Type I error budget is intended to have been spent by each information time. A set of efficacy boundaries, however, determines what actually happens under the null: the probability of crossing at or before each look. These quantities should agree in a correctly calibrated design, apart from numerical approximation and rounding. Comparing them is therefore an audit of the design, not just a restatement of its nominal plan.

The distinction matters because a boundary's marginal tail probability is not generally the probability of first crossing at that look. Some paths that exceed a later boundary have already crossed an earlier one. Counting those paths again overstates the probability of a new crossing. The comparison must use the joint null distribution of the sequential test statistics and the rule that testing stops at the first efficacy crossing.

Nominal Spending and Realized Crossing Probability

Consider a one-sided efficacy design with $K$ planned looks at increasing information fractions $0<t_1<\cdots<t_K=1$. Let $Z_i$ be the standardized test statistic at look $i$, and let $c_i$ be its upper efficacy boundary. Under $H_0$, assume the canonical joint-normal model: $Z_i\sim N(0,1)$, and for $i<j$, $\operatorname{Corr}(Z_i,Z_j)=\sqrt{t_i/t_j}$. Equivalently, the standardized process has independent increments when expressed on the information scale. The observations and information times are assumed to follow the design used for calibration; the boundary rule is to stop for efficacy at the first look with $Z_i\ge c_i$. This tutorial considers no futility stopping rule.

Definition: The nominal cumulative spending target at look $i$ is $A(t_i)$, where $A$ is the prespecified alpha-spending function. The realized cumulative null crossing probability is $$ \widetilde{\alpha}_i=P_{H_0}\left(\bigcup_{j=1}^{i}\{Z_j\ge c_j\}\right). $$ The realized first-crossing probability at look $i$ is $$ q_i=P_{H_0}(Z_1<c_1,\ldots,Z_{i-1}<c_{i-1},Z_i\ge c_i). $$ These disjoint first-crossing events give $\widetilde{\alpha}_i=\sum_{j=1}^{i}q_j$. A calibrated design targets $\widetilde{\alpha}_i=A(t_i)$ at every planned look.

The incremental nominal spending between looks $i-1$ and $i$ is $A(t_i)-A(t_{i-1})$, with $A(t_0)=0$. Calibration chooses boundaries so that $q_i$ equals this increment. It is the first-crossing probability, not the marginal tail probability $P_{H_0}(Z_i\ge c_i)$, that corresponds to incremental spending.

Calculating Realized Spending

Under the canonical model, the conditional distribution of the next statistic is $Z_i\mid Z_{i-1}=x\sim N(\rho_i x,1-\rho_i^2)$, where $\rho_i=\sqrt{t_{i-1}/t_i}$. The Markov property of this process allows the probability of surviving earlier looks to be carried forward recursively. Define $g_i(z)$ as the joint density of $Z_i=z$ and no efficacy crossing through look $i$. Then $g_1(z)=\phi(z)$ for $z<c_1$, and $g_1(z)=0$ otherwise. For $i\ge2$,

g_i(z)=\mathbf{1}(z<c_i)\int_{-\infty}^{c_{i-1}}g_{i-1}(x) \frac{1}{\sqrt{1-\rho_i^2}} \phi\!\left(\frac{z-\rho_i x}{\sqrt{1-\rho_i^2}}\right)\,dx .

Here $\phi$ is the standard normal density and $\mathbf{1}(\cdot)$ is the indicator function. The first-crossing probability at look $i$ is the mass that transitions from a surviving path at look $i-1$ to $Z_i\ge c_i$:

q_i=\int_{-\infty}^{c_{i-1}}g_{i-1}(x) \left[1-\Phi\!\left(\frac{c_i-\rho_i x}{\sqrt{1-\rho_i^2}}\right)\right]\,dx , \qquad \widetilde{\alpha}_i=\widetilde{\alpha}_{i-1}+q_i .

For the first look, $q_1=1-\Phi(c_1)$. These equations show why calculating each marginal tail separately is insufficient: the recursion restricts the integration to paths that have not already crossed.

Worked Example: A Marginal Endpoint Tail Is Not the Sequential Error

Suppose a hypothetical two-look design has $t_1=0.5$, $t_2=1$, $c_1=2.576$, and $c_2=1.960$. Under the canonical null model, the correlation is $\rho=\sqrt{0.5}=0.7071$. The first-look crossing probability is approximately $1-\Phi(2.576)=0.0050$. At the second look, the marginal tail is $1-\Phi(1.960)=0.0250$, but this includes paths that already crossed at look 1.

The joint probability of crossing both boundaries is $P(Z_1\ge2.576,Z_2\ge1.960)$. Conditional on $Z_1=x$, $Z_2$ has mean $0.7071x$ and standard deviation $0.7071$. Thus the joint upper-tail probability is

\int_{2.576}^{\infty}\phi(x) \left[1-\Phi\!\left(\frac{1.960-0.7071x}{0.7071}\right)\right]dx \approx 0.0027 .

Subtract that overlap from the marginal second-look tail to obtain the probability of first crossing at look 2: $q_2\approx0.0250-0.0027=0.0223$. Therefore the realized cumulative crossing probability is $\widetilde{\alpha}_2\approx0.0050+0.0223=0.0273$, not $0.0250$. These boundaries do not realize a nominal cumulative target of $0.0250$ at the final look, even though the second boundary has a marginal one-sided tail of $0.0250$.

To calibrate the second boundary for a cumulative target of $0.0250$, keep $c_1=2.576$ and solve $q_2=0.0250-0.0050=0.0200$ using the first-crossing integral. The resulting boundary is higher than $1.960$. Using that root in the same integral makes the realized total $0.0050+0.0200=0.0250$, up to numerical tolerance.

Auditing a Calibrated Design

A practical audit compares the cumulative targets and realized probabilities at every look, not only at the final look. The boundary calibration should use the same information fractions, null model, sidedness, stopping rule, and numerical conventions as the audit. Otherwise a mismatch may reflect different inputs rather than an error in the boundary calculation.

Worked Example: Checking a Three-Look Calibration

Consider a hypothetical three-look design with nominal cumulative targets $A(t_1)=0.0025$, $A(t_2)=0.0100$, and $A(t_3)=0.0250$. Suppose the boundaries were calibrated recursively under the canonical null model. A numerical integration audit reports realized cumulative probabilities $0.00250$, $0.01000$, and $0.02500$, respectively, to the displayed precision.

The implied first-crossing probabilities are obtained by differencing successive cumulative values: $q_1=0.00250$, $q_2=0.01000-0.00250=0.00750$, and $q_3=0.02500-0.01000=0.01500$. The corresponding nominal increments are $0.0025$, $0.0075$, and $0.0150$. Thus each realized increment agrees with its spending allocation, and the cumulative comparison also agrees at all three looks.

This check is stronger than verifying only that the final value is $0.0250$. In principle, a calculation can have the right final total but allocate crossing probability incorrectly across earlier looks. A look-by-look comparison detects that problem.

Worked Example: Quantifying a Small Calibration Discrepancy

Suppose an audit of another hypothetical design gives the following outputs. The values are cumulative null crossing probabilities, computed from the boundary vector and rounded to five decimal places.

Look	Nominal target	Realized probability	Realized minus nominal
1	0.00500	0.00500	0.00000
2	0.01500	0.01508	0.00008
3	0.02500	0.02511	0.00011

At look 2 the realized cumulative probability exceeds its target by $0.01508-0.01500=0.00008$, or $0.008$ percentage points. At look 3 the excess is $0.02511-0.02500=0.00011$, or $0.011$ percentage points. The final realized probability is therefore $0.02511$, slightly above the nominal $0.02500$.

Whether this is meaningful depends on the integration error, root-finding tolerance, and rounding used in the calibration. If the numerical uncertainty is, for example, $0.00020$, these differences are smaller than that uncertainty and do not establish a substantive mismatch. If the calculation is accurate to $0.00001$, the excess is larger than the stated numerical uncertainty and warrants investigation or tighter recalibration. A small difference should not be declared harmless without reference to the accuracy of the computation.

Interpreting Diagnostics and Limitations

The primary diagnostic is the signed difference $d_i=\widetilde{\alpha}_i-A(t_i)$ at each look. Positive values indicate more null crossing probability has accumulated than intended by that look; negative values indicate less. Also inspect the incremental differences, since they identify which look's first-crossing allocation is inconsistent. A discrepancy that grows with each look can indicate accumulated numerical error or a boundary implementation issue. A discrepancy that appears abruptly at one look focuses attention on that look's boundary, information time, or integration step.

A useful comparison is with marginal tails. The marginal quantity $1-\Phi(c_i)$ answers whether the statistic at look $i$, considered alone, exceeds its boundary. It does not account for earlier stopping. Direct multivariate-normal probability calculations can provide an independent check, while the recursive integral explicitly incorporates survival through prior looks and is convenient for sequential calibration. Both require the same joint model and event definition. Numerical integration and root finding have tolerances; report enough digits to assess the mismatch, but do not mistake displayed rounding for exact equality.

This audit is conditional on the design assumptions. If the actual information times differ from those used to compute the boundaries, the joint correlations and crossing probabilities may differ. Misspecified dependence, noncanonical statistics, an altered stopping rule, or unaccounted looks can likewise invalidate the comparison. A discrepancy alone does not identify which assumption or computation failed; it is evidence to inspect inputs and implementation. The audit is also not a statement about the probability that $H_0$ is true. It evaluates a repeated-sampling error probability under $H_0$.

Common Mistake

Common mistake: “The last boundary has a one-sided tail probability of 0.025, so the design's cumulative Type I error is 0.025.” This ignores paths that cross earlier and would be counted again by the marginal endpoint tail. The correction is to calculate each look's first-crossing probability under the joint null distribution and sum those disjoint probabilities. In the two-look example, the endpoint marginal tail is 0.0250, but the realized cumulative probability is about 0.0273 because the first-look crossing paths overlap the endpoint tail.

Key takeaway: Nominal spending is the target; realized spending is the probability implied by the boundaries and the sequential null distribution. A calibrated design should match the target look by look, within justified numerical tolerance.

Check Your Understanding

Answer each question, then compare with the solutions.

At two looks, why is $P(Z_2\ge c_2)$ not generally equal to the incremental Type I error spent at look 2?
A design has realized cumulative probabilities $0.004$ at look 1 and $0.018$ at look 2. What is its first-crossing probability at look 2?
A final nominal target is $0.025$, while the realized final crossing probability is $0.0247$. State the discrepancy and one quantity you would check before deciding whether it is meaningful.
What assumptions are needed to use the conditional-normal recursion for realized crossing probabilities?

Solutions. (1) The marginal tail includes paths that may already have crossed at look 1; the incremental error is the probability of crossing at look 2 after surviving earlier looks. (2) $q_2=0.018-0.004=0.014$. (3) The realized probability is $0.0003$ below the nominal target; compare that difference with the numerical integration and calibration tolerances. (4) The canonical joint-normal null model with the specified information times and dependence, the stated efficacy boundaries, and the first-crossing stopping rule must apply.