Checking a Gray-Test Power Simulation

Why a Power Estimate Needs Checking

A simulation-based power calculation repeatedly generates hypothetical studies under a specified alternative, applies the planned Gray test, and estimates power by the fraction of tests that reject. That fraction is useful only if the simulated studies represent the design and the test behaves appropriately. A coding mistake, unexpected event mix, excessive censoring, or Monte Carlo noise can all produce a convincing-looking but misleading estimate.

The earlier material on Simulation-Based Sample-Size Planning describes the basic planning strategy. Here the emphasis is on validation: check the test under a null scenario, inspect the simulated outcomes under both null and alternative scenarios, and quantify the random error in the estimated rejection fraction. The Gray test compares cumulative incidence functions for a specified cause while accounting for competing events; the checks below do not replace that test or its assumptions.

Check Calibration Under the Null

Before trusting estimated power, simulate studies under a null hypothesis that satisfies the test’s null condition: the target-cause cumulative incidence functions are equal across the groups being compared. For each replicate, generate the full competing-risks data, apply the planned Gray test to the planned target cause and groups, and record whether its p-value is below the prespecified significance level \(\alpha\). The empirical rejection rate estimates the test’s type I error under that particular null-generating scenario.

A valid check must reproduce the analysis as it will be used: the sample sizes, allocation, follow-up, censoring, event definitions, strata if applicable, test implementation, and rejection threshold. The simulation mechanism should also satisfy the assumptions needed for the planned analysis. In particular, right censoring should be independent of the event process under the assumed model, at least conditional on the variables represented in the analysis. If censoring depends on unmodelled prognosis, a high or low rejection rate may reflect a mismatch between the data-generating process and the test, not merely a test-programming defect.

The nominal level is a long-run rate, not a promise that one finite simulation will reject exactly \(\alpha B\) times. Compare the observed rate with its Monte Carlo uncertainty. Also examine more than one plausible null scenario when important features vary—for example, if the competing-event process or censoring differs between groups. Equal target-cause cumulative incidence is the null being checked; it does not require every other aspect of the groups’ event processes to be identical.

Audit Events and Censoring

The rejection fraction is not enough. For each simulated scenario, summarize the numbers and proportions of target-cause events, competing events, and censored observations, separately by group. Check that these quantities are plausible under the design inputs and consistent with the simulation code. When the data-generating model provides known cumulative incidence probabilities at the study horizon, compare the generated outcomes with those probabilities across many replicates, allowing for sampling variation.

A single replicate’s fraction with a target event before censoring is not generally the target-cause cumulative incidence. Participants censored before the horizon have unknown later event status; the observed event fraction therefore depends on censoring as well as the event process. Use the simulation model’s known cumulative incidence as a reference when available, and use averages across replicates to diagnose systematic discrepancies. For censoring, distinguish administrative censoring at the planned end of follow-up from loss to follow-up, and check censoring times as well as final status counts. Similar overall censoring percentages can conceal very different timing patterns or group-specific censoring.

There is no universal event proportion that guarantees adequate power. The relevant question is whether the generated numbers agree with the assumed design and whether the study produces enough information for the planned comparison. A large competing-event burden can reduce opportunities to observe the target event. Very few target events, heavy early censoring, or pronounced discrepancies between groups and their inputs warrant investigation before interpreting power.

Quantify Monte Carlo Uncertainty

For independent replicates under a fixed scenario, each \(R_b\) is a Bernoulli random variable with rejection probability \(q\). Thus \(\operatorname{Var}(R_b)=q(1-q)\). Since \(\hat{q}\) is the average of \(B\) independent indicators,

Replacing the unknown \(q\) by \(\hat{q}\) gives the Monte Carlo standard error \(SE_{\mathrm{MC}}=\sqrt{\hat{q}(1-\hat{q})/B}\). This measures simulation noise in the estimated rejection probability; it is not the standard error of the Gray test statistic within a study. For a rough large-\(B\) interval, use \(\hat{q}\pm1.96SE_{\mathrm{MC}}\). A Wilson or exact binomial interval is preferable when the number of replicates is small or the rejection rate is near zero or one.

The same calculation applies to an estimated power, replacing \(q\) with the alternative-scenario rejection probability. Choose \(B\) to achieve useful precision, rather than relying on a customary replicate count. Because \(q(1-q)\leq0.25\), a conservative approximate requirement for Monte Carlo standard error no greater than \(d\) is \(B\geq0.25/d^2\). For example, targeting \(d=0.01\) gives \(B\geq2{,}500\) as a conservative planning value; a tighter interval-width target may require more replicates.

A Practical Validation Sequence

A disciplined check separates problems in the simulation from uncertainty in the final estimate. The following sequence is useful before reporting a power result.

Simulation errors can also appear as impossible statuses, event times after administrative follow-up, or a mismatch between intended and actual group sizes. Do not silently drop failed test runs: report how many failed, determine why, and correct the process or define a principled handling rule before estimating power. A large number of failures can make the completed-run rejection fraction unrepresentative.

How This Check Differs From Other Design Approaches

An approximate sample-size calculation is faster and can clarify how event rates, allocation, and effect size affect a design. It relies on its approximation and input assumptions. Simulation can represent specified competing-event and censoring mechanisms more directly, but it is only as credible as its code and data-generating assumptions. It also adds Monte Carlo error, which must be reported or made small enough for the decision at hand.

Likewise, comparing the empirical null rejection rate with \(\alpha\) is a diagnostic, not a substitute for checking the model. A rate near \(\alpha\) does not establish that the simulation has the right event mix, nor does it validate an alternative scenario. Conversely, a modest difference from \(\alpha\) may be ordinary Monte Carlo variation. Interpret rejection rates together with outcome audits, test-failure counts, and uncertainty intervals.

Common Pitfall

Wrong reasoning: “The null simulation rejected in 6% of 1,000 runs instead of 5%, so the test is invalid and the power estimate is unusable.” The estimate is \(0.060\), and its Monte Carlo standard error is \(\sqrt{0.06(0.94)/1{,}000}\approx0.00751\). A rough 95% interval is approximately \(0.0453\) to \(0.0747\), which includes \(0.05\). The observed difference alone is not strong evidence of a calibration failure.

Correction: Assess the discrepancy relative to simulation uncertainty, then check the null construction, event and censoring patterns, code, and test runs. Increase \(B\) if more precise calibration evidence is needed. Do not treat a nonsignificant calibration discrepancy as proof of validity, and do not treat every finite-simulation difference from \(\alpha\) as a defect.