Why a Simulated P-Value Is Not Exact
A permutation test compares an observed statistic with the values it could take under a specified null model. If every possible null rearrangement is evaluated, the resulting tail proportion is an exact permutation p-value for that permutation distribution. When only a random sample of rearrangements is evaluated, the tail proportion is estimated: a different run of the simulation could produce a different count and therefore a different p-value.
This tutorial focuses on that distinction. It explains how to count simulated null statistics as extreme as the observed statistic, why the plus-one correction is used, and how to quantify the simulation variability. The correction does not remove Monte Carlo error. It prevents a finite simulation from reporting a p-value of zero and gives a natural rank-based calculation that includes the observed statistic.
For a right-sided test, “at least as extreme” usually means \(T_j^* \ge T_{\text{obs}}\); for a left-sided test it means \(T_j^* \le T_{\text{obs}}\). For a two-sided test, the rule must reflect the chosen measure of extremeness. For example, if departures in either direction count equally, one may count statistics satisfying \(|T_j^*| \ge |T_{\text{obs}}|\). State the rule clearly and use it consistently.
The null distribution and the permutation scheme must be appropriate for the test. In particular, the simulated arrangements must follow the null model being tested, as established in permutation tests. Increasing the number of simulations reduces Monte Carlo variability, but it cannot make an invalid permutation scheme valid.
Calculating the Plus-One P-Value
Suppose the null hypothesis is that the effect is zero, and a positive difference in means supports the alternative. The simulated null statistics are compared with the observed difference: simulations at least as large as the observed difference count toward the right tail. The numerator includes one additional extreme result, representing the observed statistic in the rank calculation, and the denominator includes that observed result along with the \(B\) simulated results.
Worked Example: A Right-Sided Test
In a hypothetical two-group study, the observed difference in sample means is \(T_{\text{obs}}=2.4\) points, with positive values favoring the alternative. A simulation generates \(B=999\) null statistics. Of these, \(b=23\) are at least \(2.4\) points.
The right-tail rule is \(T_j^* \ge 2.4\). Applying the plus-one formula gives $$ \hat{p}_{\text{MC}}=\frac{b+1}{B+1} =\frac{23+1}{999+1} =\frac{24}{1000} =0.024. $$
The estimated p-value is 0.024. In context, if the null model and permutation scheme are appropriate, a statistic at least as favorable to the positive-effect alternative as the observed 2.4-point difference is uncommon under that null model. This is not the probability that the null hypothesis is true.
Without the correction, the simulated tail proportion would be \(23/999 \approx 0.0230\). That is a reasonable estimate of the tail probability from the sampled rearrangements, but it is not the plus-one p-value. If none of the simulated statistics were as extreme as the observation, \(b/ B\) would be zero. The plus-one calculation would instead give \(1/(B+1)\), reflecting that the observed statistic itself is part of the rank comparison.
Why the Plus-One Formula Is Natural
The formula can be understood as a rank calculation. Imagine putting the observed statistic together with the \(B\) simulated null statistics. For a right-sided test, a more extreme statistic has a larger value. If the observed value is included among the values ranked, then the number of values at least as large as it is \(b+1\), and the total number of values is \(B+1\). Their proportion is \((b+1)/(B+1)\). For a left-sided test, rank in the lower tail instead; the same formula applies with \(b\) counting simulated values no greater than the observation.
Under the null, when the observed outcome and the simulated null outcomes are exchangeable under the permutation scheme, this rank comparison gives a valid finite-simulation test when the p-value is compared with the chosen significance level. Ties are included in the tail count, which can make the test conservative. This rank argument explains why the observed statistic is included; it does not imply that the simulation has enumerated every possible arrangement or that Monte Carlo error has vanished.
When every possible arrangement is enumerated, no random simulation count is involved. The exact p-value is calculated from the full permutation distribution, with the tail rule and treatment of ties stated. A Monte Carlo p-value is used when only a random subset of arrangements is generated. It is an estimate based on that subset, with the plus-one adjustment described above.
Monte Carlo Standard Error
The count \(b\) varies from one simulation run to another. To quantify this variability, treat each simulated statistic as producing an indicator: it is 1 if it falls in the chosen tail and 0 otherwise. If the null tail probability is \(q\), each indicator has variance \(q(1-q)\). For \(B\) independent simulations, the variance of the simulated tail proportion is \(q(1-q)/B\). Replacing the unknown \(q\) by an estimate gives the Monte Carlo standard error (MCSE).
This formula is a plug-in approximation for the simulation error in the estimated tail probability. It is often adequate when the simulation count is reasonably large and the estimated probability is not extremely close to zero or one. The MCSE is measured in probability units, not in the units of the original test statistic. It describes variation from using a finite random simulation, not uncertainty in the study's data or uncertainty about whether the null hypothesis is true.
Worked Example: Quantifying Simulation Error
Use the right-sided test above, where \(B=999\), \(b=23\), and the plus-one estimate is \(0.024\). The approximate Monte Carlo standard error is $$ SE_{\text{MC}} \approx \sqrt{\frac{0.024(1-0.024)}{999}} = \sqrt{\frac{0.023424}{999}} \approx \sqrt{0.00002345} \approx 0.00484. $$
A rough range extending two MCSEs in either direction is $$ 0.024 \pm 2(0.00484) =0.024\pm 0.00968, $$ or approximately \(0.0143\) to \(0.0337\). This is a rough way to show simulation variability, not a confidence interval for the scientific effect and not a guarantee that the exact permutation p-value lies in that range. It indicates that a new run with 999 simulations could produce a noticeably different estimate.
The estimate 0.024 is well below 0.05, and even the rough upper end is below 0.05. The simulation result therefore appears separated from that threshold, although the MCSE still describes nontrivial variation. When an estimate is close to a decision threshold, it is especially important to consider simulation error.
Two-Sided P-Values and the Tail Rule
For a two-sided test, the p-value must represent outcomes at least as inconsistent with the null as the observed outcome in either direction. There is not one universal numerical rule that fits every statistic and null distribution. For a statistic centered at zero with symmetric null values, counting simulated statistics whose absolute values are at least the observed absolute value is a clear rule. The choice of tail rule should be made as part of defining the test, not selected after seeing which rule gives a smaller p-value.
Worked Example: A Two-Sided Tail Count
In a hypothetical analysis, the observed statistic is \(T_{\text{obs}}=-1.8\). The two-sided rule is to count null statistics with absolute value at least \(1.8\). Among \(B=1999\) simulated statistics, \(b=91\) satisfy \(|T_j^*|\ge 1.8\).
The plus-one estimate is $$ \hat{p}_{\text{MC}} =\frac{91+1}{1999+1} =\frac{92}{2000} =0.046. $$
The approximate MCSE is $$ SE_{\text{MC}} \approx \sqrt{\frac{0.046(1-0.046)}{1999}} = \sqrt{\frac{0.043884}{1999}} \approx \sqrt{0.00002195} \approx 0.00469. $$
The estimated two-sided p-value is 0.046, with an MCSE of about 0.00469. Because 0.046 is close to 0.05, the simulation result alone does not locate the exact tail probability precisely relative to that threshold. More simulations can reduce the Monte Carlo error. They do not change the observed statistic or the null model.
Choosing the Number of Simulations
The MCSE formula also helps plan a simulation. If the tail probability is expected to be near \(q\) and a target MCSE is \(\delta\), solve approximately $$ \delta \approx \sqrt{\frac{q(1-q)}{B}} \quad\Longrightarrow\quad B \approx \frac{q(1-q)}{\delta^2}. $$ The value of \(q\) is usually not known in advance, so planning requires a plausible guess. A preliminary run can also help, but the number of simulations should then be increased deliberately if more precision is needed. Since the standard error decreases approximately as \(1/\sqrt{B}\), reducing it by half takes about four times as many simulations.
Worked Example: Planning for a Smaller MCSE
Suppose a p-value near \(0.030\) is anticipated, and the goal is an MCSE no greater than \(0.002\). Using \(q=0.030\) as the planning value gives $$ B \approx \frac{0.030(1-0.030)}{0.002^2} = \frac{0.0291}{0.000004} =7275. $$
Thus, at least about 7,275 simulations are indicated by this approximation; choosing \(B=8000\) provides a modest buffer. If a run with 8,000 simulations yields \(b=239\), the corrected p-value is $$ \hat{p}_{\text{MC}} =\frac{239+1}{8000+1} =\frac{240}{8001} \approx 0.0300. $$
Its estimated MCSE is approximately $$ \sqrt{\frac{0.0300(1-0.0300)}{8000}} \approx \sqrt{0.000003638} \approx 0.00191. $$ This is just below the target of 0.002. The calculation is a planning approximation: the actual MCSE depends on the tail probability, which is not known exactly.
Specify what counts as at least as extreme as the observed statistic, including the rule for a two-sided test.
Use the permutation scheme that corresponds to the null hypothesis, and record the number \(B\) of simulations.
Count \(b\) simulated statistics in the specified tail, then calculate \((b+1)/(B+1)\).
Estimate the MCSE and decide whether more simulations are needed, especially if the p-value is near a threshold relevant to the analysis.
Common Mistakes and Interpretation
A simulation p-value is evidence about how unusual the observed statistic is under the null model and permutation scheme. It is not the probability that the null is true, and it is not an effect-size measure. A small p-value does not by itself establish that an effect is important or that the observed association is causal.
- Reporting zero after no simulated exceedances. If \(b=0\), the plus-one p-value is \(1/(B+1)\), not zero. The simulation has limited resolution; it has not established that the tail probability is zero.
- Using the wrong tail. Counting only statistics greater than the observation answers a right-sided question. It does not automatically provide a two-sided p-value. Define the alternative and tail rule first.
- Treating the MCSE as data uncertainty. The MCSE describes the run-to-run variation caused by a finite number of simulated statistics. It does not quantify sampling uncertainty in the original data.
- Stopping because the estimate looks favorable. A result near a threshold may shift across it with more simulations. Report the simulation count and consider the MCSE rather than presenting the estimate as exact.
- Assuming more simulations fix a poor null scheme. A very large \(B\) can make a simulation estimate precise for the wrong distribution. The null model and permutation scheme still need to be appropriate.
Check Your Understanding
Try these questions before reviewing the examples.
- In a right-sided test, \(B=499\) and 14 simulated statistics are at least as large as the observed statistic. Calculate the plus-one p-value.
- Why does the plus-one correction avoid reporting a p-value of zero when no simulated statistic is as extreme as the observation?
- For a two-sided test using absolute values, what should count toward the tail if \(T_{\text{obs}}=2.1\)?
- What does the Monte Carlo standard error describe, and what kind of uncertainty does it not describe?
- If a p-value estimate is close to a significance threshold, how can increasing \(B\) help, and what can it not repair?