Why Results Near a Cutoff Need Careful Interpretation
A significance threshold turns a p-value into a rule-based decision. That can be useful: when a testing plan specifies a significance level in advance, the analyst can apply the same rule consistently. But a result close to the cutoff can tempt us to tell a much stronger story than the numbers support. A p-value just below the threshold does not suddenly become compelling evidence, and one just above it does not suddenly become uninformative.
The key is to keep two things separate: the p-value, which summarizes how unusual the observed result is under the null model, and the decision rule, which compares that value with a chosen threshold. Close p-values can lead to different decisions because a rule has a boundary. Their evidential summaries, however, remain close in numerical value.
A p-value is calculated under the null model: it is the probability, assuming \(H_0\) and the test’s conditions hold, of obtaining a test statistic at least as extreme as the one observed, according to the test’s definition of extremeness. The significance level is a cutoff selected for the decision rule; it is not a special property of the observed data. For example, choosing \(\alpha=0.05\) does not make 0.05 a natural dividing line between two kinds of evidence.
A Decision Boundary Is Not an Evidence Cliff
Suppose a test has \(\alpha=0.05\). A p-value of 0.049 is just below the cutoff, while 0.051 is just above it. Applying the rule gives different decisions. Yet the values differ by only 0.002. The decision changes abruptly at the boundary because the rule is defined that way; the numerical evidence summarized by the p-value does not undergo an equally abrupt change.
This distinction can be made explicit. Write the decision indicator \(D(p)\) as 1 when the result leads to rejection and 0 otherwise. The rule is a step function: it changes from 1 to 0 at the threshold. But if two p-values differ only slightly, their numerical distance remains slight, regardless of whether they fall on opposite sides of the step.
For example, with \(\alpha=0.05\), compare \(p_1=0.0499\) and \(p_2=0.0501\). Their difference is
The rule assigns different outcomes, but the p-values are separated by just 0.0002. This does not mean that p-values should be treated as a complete or linear measure of evidence. It means that a cutoff-based decision should not be mistaken for a sudden change in the underlying evidential information.
Worked Example: Two Results on Opposite Sides of 0.05
Imagine two hypothetical analyses of the same research question, using the same test and the same prespecified significance level, \(\alpha=0.05\). The first reports \(p=0.047\); the second reports \(p=0.053\). Compare the decisions and describe what the small difference does, and does not, justify saying.
First result. Compare the p-value with the threshold:
The first result meets the stated rule for rejecting \(H_0\).
Second result. Make the same comparison:
The second result does not meet the rule for rejecting \(H_0\). The difference between the reported p-values is
It is accurate to say that the decisions differ under the prespecified rule. It is not accurate to describe the first result as conclusive and the second as showing no evidence solely because they fall on opposite sides of 0.05. The p-values are close, so their numerical summaries are close as well. These comparisons do not establish that the analyses have identical evidence in every respect; they show why crossing the threshold alone is not a sound basis for describing a dramatic evidential difference.
Rounding Can Hide the Comparison
A threshold decision depends on the unrounded p-value. If a displayed value has been rounded, it may not contain enough information to determine whether the original value was above, below, or exactly equal to \(\alpha\). This is especially important when the reported value rounds to the threshold itself.
For instance, with \(\alpha=0.05\), both \(p=0.0496\) and \(p=0.0504\) round to 0.05 to two decimal places. But the first satisfies \(p\leq\alpha\), while the second satisfies \(p>\alpha\). A report that gives only “\(p=0.05\)” cannot distinguish those cases. If the exact value is available, compare it with the prespecified cutoff before rounding. If it is not available, do not infer a decision that the reported precision cannot support.
Worked Example: A Rounded P-Value at the Cutoff
A hypothetical test uses \(\alpha=0.05\). The output gives an unrounded p-value of 0.0504, but a summary table displays p-values to two decimal places. Find the displayed value, make the decision using the unrounded value, and explain what the displayed value alone would tell a reader.
Rounding. To two decimal places, 0.0504 is 0.05: the third decimal place is 0, so the second decimal place does not change.
Decision using the available unrounded result. Compare the original value with the threshold:
Thus, this result does not meet the stated rule for rejecting \(H_0\). The displayed “0.05” alone is insufficient to recover that comparison: it could represent an unrounded value on either side of 0.05, depending on how it was rounded. Reporting more digits, or stating the comparison based on the unrounded value, prevents this ambiguity.
The reverse situation also matters. If an exact p-value equals the threshold, the rule stated above assigns it to the rejection side because it uses \(p\leq\alpha\). But a displayed value that appears equal to the threshold after rounding does not establish that the exact p-value equals it. Treat equality as a mathematical comparison of the available value, not as something implied by a rounded display.
Worked Example: Reporting a Result Just Below a Stricter Cutoff
Suppose a hypothetical analysis uses a prespecified \(\alpha=0.01\) and reports \(p=0.0098\). A second result for the same question reports \(p=0.0102\). Compare both results with the chosen threshold, and consider how rounding to two decimal places would affect the report.
For the first result,
It meets the rule for rejecting \(H_0\). For the second result,
It does not meet that rule. The p-values differ by
Both values round to 0.01 to two decimal places. Therefore, a report that gives only two decimal places would obscure their position relative to the threshold. The decision should be based on the more precise values, with enough precision reported to make the comparison clear. The differing decisions follow from the selected \(\alpha=0.01\); they do not establish a sudden change in evidence between the two results.
A Practical Way to Interpret a Near-Threshold Result
When a p-value is close to \(\alpha\), a careful interpretation can be built in three steps. First, state the p-value and the significance level at sufficient precision. Second, apply the comparison rule exactly as specified. Third, describe the result without turning the decision boundary into a claim that the evidence changed sharply. The distinction is useful whether the p-value falls just below or just above the cutoff.
Use the \(\alpha\) chosen for the test; do not move it after seeing the p-value to obtain a preferred decision.
Check whether \(p\leq\alpha\) or \(p>\alpha\). If only a rounded value is available and it does not resolve the comparison, say so rather than guessing.
Report which side of the rule the result falls on, while noting that a small numerical distance from the cutoff does not imply a dramatic evidential difference.
For example, “The p-value was 0.049, just below the prespecified 0.05 threshold, so the result met the rule for rejecting \(H_0\)” reports the comparison directly. It is more measured than calling the result a decisive discovery merely because 0.049 is below 0.05. Likewise, “The p-value was 0.051, just above 0.05, so the result did not meet the rejection rule” is more precise than claiming that the data provide no evidence.
Common Mistake: Treating 0.05 as a Natural Divide
A common mistake is to reason that \(p=0.049\) demonstrates an effect, while \(p=0.051\) demonstrates its absence. The correction is to separate what the rule says from what the p-value says. Under a prespecified \(\alpha=0.05\), the first value falls on the rejection side and the second falls on the other side. The small numerical difference does not, by itself, justify describing one result as strong evidence and the other as none.
Another mistake is to make the decision from a rounded number. If a displayed p-value is 0.05 to two decimal places, do not assume the exact value equals 0.05 or that it lies on either side. Use the unrounded value if available; otherwise report that the displayed precision does not determine the comparison. Preserving this distinction makes the decision reproducible and the interpretation appropriately cautious.
Check Your Understanding
For each question, distinguish the comparison rule from what can be inferred about the evidence.
- With \(\alpha=0.05\), a test reports \(p=0.048\). Which side of the decision rule does it fall on?
- Two otherwise comparable analyses report \(p=0.049\) and \(p=0.052\). Why is it misleading to describe them as showing radically different evidence solely because one is below 0.05?
- A report gives \(p=0.05\) to two decimal places, with \(\alpha=0.05\). Can the displayed number alone establish whether the unrounded p-value is above or below the threshold? Explain.
- With \(\alpha=0.01\), compare \(p=0.0100\) and \(p=0.0101\) using the stated rule \(p\leq\alpha\). What is the decision for each?
- Why should an analyst compare the unrounded p-value with the prespecified threshold before reporting a rounded value?