A Worked Monthly Monitoring Example

Why a Monthly Monitoring Example Needs an Explicit Model

Suppose a hypothetical trial team checks a significance test every month and plans to stop if a result is significant. The trial may appear to have a single significance level, but repeated opportunities to reject \(H_0\) change the probability of at least one false rejection. To calculate that probability, we must specify how the monthly tests relate to one another.

This tutorial uses a deliberately simple model: each month’s test is based on new data independent of the data used in every other month. That assumption makes the monthly rejection events independent. It is not generally a good description of tests that repeatedly analyze accumulating trial data, where looks share observations. The earlier material on dependence between cumulative interim tests explains why that distinction matters.

Model and assumptions: There are \(K\) scheduled monthly tests. Under the global null hypothesis, \(H_0\) is true at every look. At look \(j\), the test is valid at its prespecified level \(\alpha_j\), so its probability of rejecting \(H_0\) is \(\alpha_j\). For the calculations in this tutorial, the monthly rejection events are assumed independent. The schedule and thresholds are fixed in advance, and the analysis stops at the first rejection.

A valid level-\(\alpha_j\) test has a null rejection probability no greater than \(\alpha_j\); when we calculate exact values below, we assume it equals \(\alpha_j\). For example, a continuous test statistic with a correctly calibrated critical value can attain the stated level. If a test is conservative, the exact overall probability may be smaller than the calculations using equality.

Monthly Rejections and the First-Rejection Event

Let \(R_j\) denote the event that the test at month \(j\) rejects \(H_0\). The first-rejection event at month \(j\) is different: it requires no rejection in any earlier month and a rejection at month \(j\). In event notation, it is \(R_j\) together with non-rejection at months \(1,\ldots,j-1\).

Definition: The overall null rejection probability, also called the family-wise Type I error probability for this monitoring rule, is the probability of at least one rejection across the \(K\) scheduled looks when \(H_0\) is true. Under the independent-looks model, with exact monthly null rejection probabilities \(\alpha_j\), it is

P\left(\bigcup_{j=1}^{K} R_j\mid H_0\right) =1-\prod_{j=1}^{K}(1-\alpha_j).

The complement of “at least one rejection” is “no rejection at any month.” Under independence, the probability of no rejection at month \(j\) is \(1-\alpha_j\), and the probability of no rejection at every month is the product of those probabilities. Subtracting that product from 1 gives the overall probability of at least one rejection.

The same reasoning gives the probability that the first rejection occurs at month \(j\): all \(j-1\) earlier tests must not reject, and the month-\(j\) test must reject. Independence therefore gives

P(\text{first rejection at month }j\mid H_0) =\left\{\prod_{i=1}^{j-1}(1-\alpha_i)\right\}\alpha_j.

These first-rejection events are mutually exclusive: a single monitoring sequence cannot have its first rejection at two different months. Their probabilities sum to the overall null rejection probability. If the trial stops at the first rejection, later tests are not performed after that event; the formula describes the planned sequence of opportunities and the stopping rule, not additional observed tests after stopping.

Worked Monthly Monitoring Examples

Worked Example: Tracing the First Rejection in a Hypothetical Trial

A hypothetical trial prespecifies monthly two-sided tests at level \(\alpha=0.05\), stopping at the first significant result. For illustration, suppose the reported monthly p-values for the first five months are \(0.120\), \(0.041\), \(0.080\), \(0.009\), and \(0.200\). Identify the first rejection and its implication for the monitoring sequence.

At month 1, \(0.120>0.05\), so the test does not reject \(H_0\). At month 2, \(0.041<0.05\), so the test rejects \(H_0\). Thus the first-rejection event occurs at month 2, and the prespecified rule stops monitoring then. The later displayed values are included only to make the sequence concrete; under the stopping rule, months 3–5 would not actually be tested after the month-2 rejection.

Under the independent-looks null model, the probability of a first rejection at month 2 is

$$ (1-0.05)(0.05)=0.95(0.05)=0.0475. $$

So, if \(H_0\) is true, the probability that the first rejection is specifically at month 2 is 4.75%. The observed p-value sequence identifies where this hypothetical sequence first rejects; it does not by itself establish that the null hypothesis is false.

Worked Example: Overall Null Rejection Probability Across 12 Months

Consider a hypothetical 12-month monitoring plan with an independent, valid level-\(0.05\) test each month. Assume each test has null rejection probability exactly \(0.05\). Calculate the probability of at least one false rejection during the year.

For one month, the probability of no rejection under \(H_0\) is \(1-0.05=0.95\). Under the independence assumption, the probability of no rejection at all 12 months is \(0.95^{12}\). Therefore,

P(\text{at least one rejection}\mid H_0) =1-(0.95)^{12} =1-0.5404 =0.4596.

The overall null rejection probability is approximately \(0.4596\), or 46.0%. It is not 5%, even though each individual test uses a 5% level. This result is about the chance of at least one false rejection under the stipulated model; it is not the probability that \(H_0\) is true after a significant result.

Worked Example: Monthly Thresholds That Vary

Suppose a hypothetical 12-month plan uses level \(0.01\) in each of the first three months and level \(0.05\) in each of the remaining nine months. Assume independent monthly rejection events with null rejection probabilities equal to the specified levels. Find the overall null rejection probability and the probability that the first rejection occurs in month 4.

The probability of no rejection during the first three months is \((1-0.01)^3=0.99^3\). The probability of no rejection during the last nine months is \(0.95^9\). Thus,

P(\text{at least one rejection}\mid H_0) =1-(0.99)^3(0.95)^9 \approx 1-(0.9703)(0.6302) \approx 0.3885.

The overall null rejection probability is about 38.85%. For the first rejection to occur in month 4, there must be no rejection during months 1–3, followed by a rejection at month 4:

P(\text{first rejection at month 4}\mid H_0) =(0.99)^3(0.05) =0.04851.

Thus, the probability of a first rejection specifically in month 4 is about 4.851%. The changing thresholds alter both the chance of surviving earlier months without rejection and the chance of rejection at the month in question.

Why the Independence Assumption Matters

The product calculation is exact only when the monthly rejection events are independent and their null rejection probabilities are as specified. Monthly calendar spacing does not create independence by itself. If every monthly analysis includes the same participants’ data collected so far, the test statistics share information, and the rejection events are generally dependent. In that setting, multiplying the monthly non-rejection probabilities as if the tests were independent is not justified.

The assumption can be reasonable in a designed illustration where each month enrolls a separate, non-overlapping cohort and each test uses only that month’s data, provided the cohorts and their outcomes are independent under the model. It can also be a useful simplifying assumption for learning how repeated opportunities affect error. But it should not be silently substituted for a dependence model in an actual cumulative trial.

One alternative is to use a conservative per-look threshold based on the Bonferroni inequality. For \(K\) tests, assigning each test level \(\alpha/K\) ensures that the probability of at least one rejection is at most \(\alpha\), without requiring independence. This bound does not generally give the exact overall probability, and it may be conservative. A single final test at level \(\alpha\) avoids repeated testing but does not provide the same opportunity to stop early. These approaches answer different design needs; none makes an independent-looks calculation valid for dependent cumulative tests.

Diagnostics, Limitations, and a Common Mistake

Before using the independent formula, check the design and analysis rather than relying on the number of months alone. Ask whether the monthly data sets overlap, whether outcomes or participants can induce dependence across months, and whether every test has the stated null rejection probability. Also verify that the monthly schedule and thresholds were fixed, and that the stated stopping rule matches the calculation. A mismatch in any of these details can make the numerical answer misleading.

Overlapping data: Reusing accumulating observations is a warning that monthly rejections may be dependent.
Different or conservative tests: A nominal level does not guarantee an exact rejection probability equal to that level; discrete or otherwise conservative tests may reject less often.
Changed schedule or thresholds: Adding unplanned looks or changing thresholds means the original probability calculation no longer describes the actual rule.
Stopping and reporting: If the rule stops at the first rejection, do not describe later hypothetical results as tests that were actually carried out.

Common mistake: “Each month is a 5% test, so the chance of a false positive over the year is 5%.” This confuses a single-look error rate with the probability of at least one rejection across repeated opportunities. Under 12 independent tests at level \(0.05\), the latter is \(1-0.95^{12}\approx 0.4596\), not 0.05. A second mistake is using that independent formula for cumulative tests just because the checks happen monthly. The correction is to establish the relationship between the tests first; the formula here depends on independent rejection events.

Key takeaway: Under a fixed schedule of independent tests with exact null rejection probabilities \(\alpha_1,\ldots,\alpha_K\), the chance of at least one null rejection is \(1-\prod_{j=1}^{K}(1-\alpha_j)\). The probability that the first rejection is at month \(j\) is \(\alpha_j\prod_{i=1}^{j-1}(1-\alpha_i)\). These formulas do not automatically apply to repeated analyses of cumulative data.

Check Your Understanding

Answer each question using the independent-looks model where it is specified. Solutions follow each question.

A hypothetical study has four independent monthly tests, each with null rejection probability \(0.05\). What is the probability of at least one rejection? Solution: \(1-(0.95)^4=1-0.8145=0.1855\), or about 18.55%.
For the four-test plan in Question 1, what is the probability that the first rejection occurs in month 3? Solution: The first two tests must not reject and month 3 must reject, so \((0.95)^2(0.05)=0.045125\), or about 4.513%.
Three independent tests have levels \(0.02\), \(0.02\), and \(0.10\). Calculate the overall null rejection probability. Solution: \(1-(0.98)(0.98)(0.90)=1-0.86436=0.13564\), or about 13.56%.
A trial tests the same accumulating data every month. Is the independent-looks product formula automatically justified? Solution: No. The analyses share observations, so the rejection events are generally dependent. The independent formula requires an additional assumption that has not been established by the monthly schedule.