Significance Thresholds and Decision Rules

From a P-Value to a Decision

A p-value summarizes how extreme the observed test result is under the null model, using the test’s specified definition of extremeness. A decision rule answers a different question: given a cutoff chosen in advance, does this p-value meet the rule for declaring statistical significance? Keeping these roles separate prevents a numerical summary of evidence from being mistaken for the decision policy applied to it.

The cutoff is called the significance level and is denoted by \(\alpha\). It must be specified before examining the result. Common choices include \(0.05\) and \(0.01\), but no one value is appropriate for every setting. Once \(\alpha\) and the test are fixed, the p-value is compared with that cutoff.

The equality case is included: when the p-value is exactly \(\alpha\), reject \(H_0\) under this convention. The rule uses the p-value to make a classification, but the p-value itself is not the classification. It remains a numerical measure defined relative to the null model and the chosen alternative; \(\alpha\) is the prespecified cutoff against which that measure is assessed.

What the Significance Level Controls

A test uses a null model to assess how unusual the observed result would be if \(H_0\) were true. Before using the decision rule, assume the hypotheses, test statistic, and definition of extremeness have been specified, and that the test’s assumptions make its p-value valid under the null model. A p-value is valid at level \(\alpha\) when, for every parameter value allowed by \(H_0\), the chance of obtaining a p-value at most \(\alpha\) is no greater than \(\alpha\).

This gives the significance level its repeated-sampling interpretation: if the null hypothesis is true and the same valid testing procedure is repeated under its stated conditions, the probability of rejecting \(H_0\) is at most \(\alpha\). For a continuous test with exact calibration, that probability is often exactly \(\alpha\); for some discrete tests it can be smaller. The level is therefore a bound on the test procedure’s long-run probability of a false rejection when \(H_0\) is true. It is not the probability that \(H_0\) is true after observing the data.

Why the Rule Has This Form

The p-value is defined from the null distribution as the probability, assuming \(H_0\), of a result at least as extreme as the observed result, according to the test’s specified extremeness rule. Small p-values correspond to outcomes the null model assigns relatively little probability. A threshold rule rejects for the p-values in the range from zero through \(\alpha\).

For a valid p-value, the defining calibration property is \(P(p\text{-value}\leq\alpha\mid H_0)\leq\alpha\), with the condition understood for each parameter value in a composite null hypothesis. Thus, using the rule “reject when the p-value is at most \(\alpha\)” limits the probability of rejection under the null to the chosen level. In an exactly calibrated continuous test, the p-value is uniformly distributed on \([0,1]\) under the null, so the probability that it falls in \([0,\alpha]\) is the length of that interval:

This argument explains both the cutoff and the importance of calibration: the rule has the stated error-rate control only when the p-value comes from a valid test under its assumptions. It does not say that every p-value has a uniform distribution in every setting, or that the null hypothesis has probability \(\alpha\) of being true.

Applying the Decision Rule

In the examples below, the p-values are supplied as results from hypothetical tests. In each case, suppose the null and alternative hypotheses and a valid test were specified before the data were examined, and the test’s assumptions are satisfied. The task here is to apply the decision rule, not to calculate the p-value from raw data.

Why Prespecification Matters

The significance level determines how demanding the rejection rule will be. A smaller \(\alpha\) creates a smaller range of p-values that lead to rejection. As the derivation shows, that choice also sets a tighter upper bound on the probability of rejecting a true null hypothesis, provided the test is valid. Choosing the level before seeing the result makes the decision rule transparent and prevents the cutoff from being adjusted simply to obtain a desired classification.

A decision rule is useful because it makes the criterion explicit. Two people who agree on the hypotheses, valid test, p-value, and prespecified \(\alpha\) will apply the same comparison and reach the same classification. But the label “statistically significant” is only the output of that rule. It is not a measure of the size of an effect, a guarantee of a reproducible result, or proof that a scientific claim is true.

Common Mistakes

• Treating the p-value as the cutoff. Saying “the p-value is \(0.05\)” does not state the decision rule unless the chosen \(\alpha\) is also known. Compare the p-value with the prespecified significance level.
• Reversing the inequality. A smaller p-value meets a fixed cutoff more readily. The conventional rule is \(p\leq\alpha\), not \(p\geq\alpha\).
• Excluding equality. Under the stated convention, a p-value equal to \(\alpha\) leads to rejection because the rule uses “less than or equal to.”
• Choosing \(\alpha\) after seeing the result. Adjusting the level to change the classification undermines the planned error-rate interpretation. Prespecify the level.
• Reading the decision as certainty. Rejection is a rule-based conclusion, not proof that \(H_0\) is false. The significance level is not a probability that the null is true.