Understanding P-Values in Clinical Research

Introduction

Few concepts are used more often—and misunderstood more often—in clinical research than the p-value. A p-value appears in statistical analysis plans, clinical study reports, journal articles, tables, and regulatory discussions, but its correct interpretation requires more than asking whether it is "less than 0.05."

A p-value measures how compatible the observed data are with a specified null hypothesis, under the assumptions of the statistical test. It is not the probability that the null hypothesis is true, and it is not the probability that the observed finding occurred "by chance."

What Is a P-Value?

Suppose a randomized clinical trial compares an experimental treatment with control on a prespecified endpoint. Let the null hypothesis be that there is no treatment difference in the population.

In words, this is the probability—assuming the null hypothesis and the assumptions of the test are correct—of obtaining a result at least as extreme as the one observed.

The exact calculation depends on the statistical test. A two-sample t-test, chi-square test, logistic regression, Cox model, mixed model, and other procedures use different test statistics and reference distributions.

The Null and Alternative Hypotheses

A p-value cannot be interpreted without knowing what hypothesis is being tested. For a simple treatment comparison, one might define:

where \(\theta\) represents the treatment effect defined by the analysis. For a one-sided test, the alternative may instead specify a direction:

The choice between a one-sided and two-sided hypothesis test should be determined before examining the results and should follow the prespecified statistical analysis plan.

How a P-Value Is Calculated

Consider a treatment effect estimate \(\hat{\theta}\) with standard error \(SE(\hat{\theta})\). A common standardized test statistic is:

where \(\theta_0\) is the value specified under the null hypothesis. If the null hypothesis is no treatment difference, \(\theta_0\) is often zero.

For a two-sided large-sample normal test:

where \(\Phi\) is the standard normal cumulative distribution function. The more extreme the test statistic, the smaller the p-value tends to be.

A Simple Numerical Example

Suppose a trial estimates a treatment difference of 5 units with a standard error of 2 units. Testing \(H_0:\theta=0\):

For a two-sided normal approximation, the corresponding p-value is approximately:

This means that, if the null hypothesis and the assumptions of the test were true, a result at least this extreme would occur with probability about 1.24% under the specified reference distribution. It does not mean that there is a 1.24% probability that the null hypothesis is true.

What a P-Value Does Mean

A large p-value indicates that the data are not sufficiently inconsistent with the null hypothesis to reject it at the chosen significance level. It does not establish that the null hypothesis is true.

What a P-Value Does Not Mean

Statistical Significance

A commonly used significance level is:

If the p-value is less than the prespecified \(\alpha\), the result is conventionally described as statistically significant at that level.

The wording "reject" and "fail to reject" is preferable to "accept" and "prove." A nonsignificant result may reflect a genuinely small effect, substantial variability, a limited sample size, or insufficient information.

The Meaning of p = 0.05

The boundary of 0.05 is a convention rather than a universal scientific law. A p-value of 0.049 and a p-value of 0.051 should not be treated as fundamentally different scientific discoveries.

Clinical interpretation should consider the estimated treatment effect, confidence interval, endpoint importance, multiplicity, study design, and totality of evidence.

Small P-Values and Large P-Values

These p-values do not, by themselves, tell us whether the treatment effect is large, small, beneficial, harmful, or clinically meaningful.

P-Value Versus Effect Size

The estimated effect is often more clinically informative than the p-value alone. Suppose two studies both produce \(p=0.01\), but their treatment effects are very different.

The identical p-values do not imply identical clinical importance. The appropriate interpretation depends on the endpoint scale and what constitutes a meaningful difference.

P-Value Versus Confidence Interval

A confidence interval describes a range of effect estimates compatible with the data under the interval's assumptions. It provides information about both magnitude and precision.

A p-value may answer whether the data provide evidence against a particular null value. The confidence interval additionally shows how large or small plausible effects may be.

A P-Value Does Not Measure Effect Size

A very small p-value can occur with a small effect when the study is large and precise. Conversely, a clinically meaningful effect can have a relatively large p-value when the study is small or highly variable.

Sample Size Has a Major Influence

As sample size increases, standard errors often decrease. Holding an estimated effect approximately constant, a smaller standard error produces a more extreme test statistic:

Therefore, very large studies can produce small p-values for effects that are clinically trivial. Small studies have the opposite problem: even a potentially important effect may not reach conventional statistical significance because the estimate is imprecise.

Clinical Significance Versus Statistical Significance

Clinical significance asks a different question: Is the magnitude of the effect important to patients, clinicians, or decision makers?

For example, a very large trial could identify a statistically significant difference in systolic blood pressure of 0.5 mmHg:

The result may be statistically persuasive but clinically unimportant, depending on the context and endpoint.

Failure to Reject Is Not Proof of No Effect

Suppose a trial estimates:

The appropriate conclusion is that the prespecified test did not provide sufficient evidence to reject the null hypothesis at the chosen significance level. It would be incorrect to conclude that the treatment has exactly zero effect.

Power and P-Values

Statistical power is the probability of rejecting the null hypothesis under a specified alternative, assuming the model and design are correct.

A nonsignificant p-value in an underpowered study therefore does not necessarily provide strong evidence of no meaningful treatment effect.

Type I and Type II Errors

The significance level \(\alpha\) controls the Type I error rate under the specified testing framework. Power is related to the probability of avoiding a Type II error.

One-Sided Versus Two-Sided P-Values

For a two-sided test, departures from the null in either direction can contribute to the p-value. A one-sided test focuses on a prespecified direction.

The direction must be established before looking at the observed treatment effect. Choosing a one-sided test after seeing the data can invalidate the intended Type I error control.

P-Values in Randomized Clinical Trials

In a randomized trial, a p-value for a treatment comparison is generally intended to quantify the evidence against the null hypothesis specified for that endpoint and analysis. For a continuous endpoint, an analysis might estimate:

The interpretation must follow the exact estimand and model specified in the protocol and statistical analysis plan.

P-Values in Logistic Regression

A hypothesis test may assess \(H_0:\beta_j=0\). The associated p-value provides evidence concerning that coefficient under the specified model. The odds ratio and confidence interval should also be examined.

P-Values in Cox Regression

A p-value testing \(H_0:\beta_T=0\) assesses evidence against the null hazard ratio of 1. The hazard ratio and its confidence interval remain essential for interpretation.

P-Values in Repeated-Measures Analyses

Clinical trial data are often longitudinal. Mixed models, repeated-measures models, and other approaches may produce p-values for treatment effects, visit effects, treatment-by-visit interactions, contrasts, or other parameters.

Multiplicity and Multiple P-Values

Testing many hypotheses creates an increased opportunity for false-positive findings if each hypothesis is evaluated independently at the same nominal significance level.

For 20 independent tests at \(\alpha=0.05\):

This illustrates why unplanned multiplicity can substantially change interpretation. Clinical trial procedures may instead use prespecified hierarchical, graphical, gatekeeping, or other multiplicity-control strategies.

Primary, Secondary, and Exploratory Endpoints

The evidentiary role of a p-value depends on whether the endpoint is primary, key secondary, secondary, or exploratory and on the prespecified testing strategy. A p-value from an exploratory analysis should not automatically be interpreted with the same confirmatory strength as a p-value from a prespecified primary hypothesis.

P-Hacking and Analysis Flexibility

When investigators repeatedly analyze data, change endpoints, select favorable subgroups, or try many models until a small p-value appears, the nominal p-value may no longer provide the claimed Type I error control.

This is why clinical trials emphasize prospective specification of endpoints, analysis populations, statistical models, contrasts, multiplicity procedures, missing-data methods, and interim-analysis rules.

Subgroup P-Values

Subgroup analyses can be clinically valuable, but small subgroups often have limited precision. A significant result in one subgroup and a nonsignificant result in another do not automatically demonstrate that treatment effects differ.

Interim Analyses

Repeated looks at accumulating clinical trial data can affect Type I error if unplanned repeated testing is performed. Formal interim analyses therefore require an appropriate prespecified statistical framework. Group-sequential methods and alpha-spending approaches can define decision boundaries while maintaining overall error control.

Adjusted Versus Unadjusted P-Values

An unadjusted p-value typically reflects a single specified hypothesis test without adjustment for a family of related hypotheses. An adjusted p-value or adjusted testing procedure accounts for a prespecified multiplicity structure.

Missing Data and P-Values

The statistical model and missing-data strategy can influence both the treatment-effect estimate and its p-value. For longitudinal clinical trial endpoints, approaches may include likelihood-based models, multiple imputation, tipping-point analyses, pattern-mixture approaches, or other sensitivity analyses depending on the estimand and assumptions.

A p-value cannot rescue an analysis whose handling of missing data is inappropriate for the scientific question.

Assumptions Matter

A p-value is conditional on the assumptions underlying the statistical method. Examples include assumptions concerning independence, distributional form, variance structure, proportional hazards, model specification, and measurement. If the model is inappropriate, the nominal p-value may not have its intended statistical properties.

The Clinical Trial Interpretation Workflow

Why "p < 0.05" Is Not the Whole Story

Trial A provides highly precise evidence for a small effect. Trial B provides imprecise evidence around a potentially much larger effect. Looking only at whether \(p<0.05\) would obscure this important distinction.

Equivalence and Non-Inferiority Are Different

A conventional superiority test asking whether two treatments differ is not the same as an equivalence or non-inferiority analysis. A nonsignificant superiority test does not establish equivalence.

Bayesian Probability Is Different

A p-value is a frequentist quantity. A Bayesian posterior probability can instead express probability statements about a parameter conditional on the observed data and a prior distribution.

This is conceptually different from:

Software Output: Read the Entire Result

Statistical software can make p-values appear deceptively simple. A clinical statistician should inspect the full output rather than copying a single number.

• Parameter estimate
• Standard error
• Test statistic
• Degrees of freedom
• P-value
• Confidence interval
• Model-specific diagnostics

Example SAS Interpretation

proc ttest data=trial;
  class TRT;
  var CHANGE;
run;

The resulting p-value tests the treatment comparison defined by the procedure and data structure. Verify treatment coding, analysis population, endpoint derivation, assumptions, and whether the test matches the prespecified SAP.

Example R Interpretation

fit <- lm(CHANGE ~ TRT + BASELINE, data = analysis)

summary(fit)

If the treatment coefficient has a small p-value, the result indicates evidence against the specified null for that coefficient under the model. It does not mean that the model is automatically appropriate or that the treatment effect is clinically important.

Reporting P-Values in a Clinical Study Report

A good CSR presentation should make the statistical context clear. A typical result might report the treatment effect, confidence interval, and p-value together:

The exact wording should reflect the estimand, model, multiplicity procedure, and prespecified reporting conventions.

Common Mistakes in Clinical Research

1. Interpreting the p-value as the probability that the null is true. It is not.
2. Calling p > 0.05 "proof of no effect." Failure to reject is not proof of equality.
3. Ignoring the effect estimate. Statistical significance does not measure clinical magnitude.
4. Ignoring the confidence interval. Precision and plausible effect sizes matter.
5. Testing many endpoints without addressing multiplicity. A collection of nominal p-values can exaggerate false-positive evidence.
6. Choosing the analysis after seeing the results. This can invalidate the intended inferential interpretation.
7. Assuming significance means clinical importance. Clinical relevance requires context.
8. Assuming nonsignificance means equivalence. Equivalence requires a different hypothesis and prespecified margin.
9. Comparing subgroup significance rather than treatment-effect differences. An interaction test is generally needed.
10. Reporting software output without understanding the tested hypothesis. Every p-value must be tied to a defined parameter or contrast.

Practical P-Value Interpretation Checklist

Quick Reference

The Most Important Concept

The p-value is best understood as a measure of how unusual the observed data are under a specified null hypothesis and statistical model.

It is not a probability that the null hypothesis is true. It is not a measure of clinical importance. It is not a substitute for an effect estimate or confidence interval. And a p-value greater than 0.05 does not prove that there is no treatment effect.

In clinical research, the most informative interpretation combines the prespecified hypothesis, treatment-effect estimate, confidence interval, p-value, multiplicity considerations, study design, statistical assumptions, clinical importance, safety, and totality of evidence.

References

Wasserstein, R.L., Lazar, N.A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.
Greenland, S., Senn, S.J., Rothman, K.J., et al. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31, 337–350.
Sterne, J.A.C., Smith, G.D. (2001). Sifting the evidence—what's wrong with significance tests? BMJ, 322, 226–231.
ICH. E9 Statistical Principles for Clinical Trials. International Council for Harmonisation.