Statistical Significance Versus Practical Importance

Two Different Questions

A test can provide strong evidence that an effect differs from zero without showing that the effect is large enough to matter. Conversely, an estimated effect can look important in its real-world setting while the data leave substantial uncertainty about whether an effect is present. These are not contradictions: statistical significance and practical importance answer different questions.

Statistical significance describes a result relative to a specified null model and a chosen significance level. Practical importance asks whether the estimated effect is large enough to matter for the people or decisions involved. The first is assessed using a test statistic and p-value; the second requires understanding the outcome’s units, consequences, costs, and relevant benchmarks.

For example, suppose a study estimates how much a new service changes the average time customers wait. The estimated difference is expressed in minutes, while its p-value summarizes how incompatible the observed test statistic is with a null model of no difference. A small p-value does not tell us whether the estimated change is one second or one hour. To judge the change’s practical importance, we need the estimate in minutes and a reasoned account of what size of reduction would make a meaningful difference.

Effect Estimates and the Null Model

Let \(\delta\) denote the population difference of interest, defined as the mean outcome in a treatment group minus the mean outcome in a comparison group. The sample estimate, \(\hat{\delta}\), is the observed difference between the corresponding sample means. A common two-sided test asks whether the data are inconsistent with \(H_0:\delta=0\), against \(H_a:\delta\ne 0\).

For the examples below, suppose the study design and measurements support the stated comparison, and the reported standard errors and p-values come from an appropriate analysis. Suppose also that the test statistic has the stated standard normal reference distribution under \(H_0\). These are assumptions for the illustrative calculations, not facts guaranteed by a small p-value. The examples use two-sided tests, so departures in either direction count as evidence against the null.

With an estimated difference \(\hat{\delta}\) and its standard error \(SE\), the standardized test statistic in these examples is

Under the null model, the two-sided p-value is the probability of a standard normal statistic at least as far from zero as the observed \(z\):

The calculation expresses the estimate in standard-error units. The result is used to assess evidence against the null model; practical interpretation still requires returning to \(\hat{\delta}\), its original units, and the setting.

A Short Justification: Why a P-Value Does Not Measure Importance

The distinction follows from what the test calculation uses. Under the stated normal reference model, the two-sided p-value is determined by the absolute standardized statistic, \(|z|=|\hat{\delta}|/SE\) when \(H_0:\delta=0\). The practical question, by contrast, concerns the magnitude of \(\hat{\delta}\) in its original units compared with a meaningful benchmark.

These quantities answer different questions: the standardized statistic locates the observed result relative to a null reference distribution, while the estimate states the observed difference in interpretable units. Dividing by a standard error does not convert the estimate into a measure of clinical, financial, or social value. The p-value therefore cannot supply the missing context or choose a practical benchmark for us.

How to Assess Practical Importance

A practical benchmark should be grounded in the outcome and the decision, not selected after seeing a p-value. Depending on the setting, it might be a minimum change patients would notice, a reduction large enough to justify a cost, or a difference that alters a policy decision. There may be no single uncontested benchmark; if so, explain the values and trade-offs behind the judgment rather than presenting it as a purely mathematical fact.

It is also important to examine the direction and possible consequences of an effect. A change can be large but undesirable. A modest average improvement might still matter if it affects many people or has almost no cost; the same improvement might not justify a burdensome or expensive intervention. The effect’s units and context belong in the interpretation, not just in the data table.

Reporting Both Conclusions

A useful report separates the evidence from the judgment about importance. State the estimated effect with its direction and units, report the p-value and the significance decision when relevant, then compare the estimate with a justified practical benchmark. If the estimate is uncertain, do not turn failure to reject into proof of no effect; if it is statistically significant, do not let that label substitute for evaluating its size.

For instance, a careful summary might say: “The new scheduling system was associated with an estimated 0.8-minute reduction in mean waiting time; the two-sided test was statistically significant (\(p=0.00137\)). The estimated reduction was below the 2-minute threshold specified as worthwhile for this service.” This phrasing communicates the direction, magnitude, evidence, and practical judgment without treating them as interchangeable.

Common Mistake

Wrong reasoning: “The p-value is below 0.05, so the effect must be important.” This confuses evidence against a null model with the size and consequences of the estimated effect.

Correction: A small p-value supports a conclusion about how unusual the test statistic would be under the null model, subject to the analysis assumptions. It does not set a minimum meaningful effect. Report the estimate in context and judge it against a relevant benchmark. The reverse mistake is also possible: a nonsignificant result does not prove that the effect is practically negligible, especially when the estimate itself is substantial but uncertain.