Why a P-Value Depends on More Than the Test Statistic
A p-value describes how unusual a test statistic would be if the null hypothesis and the test’s reference model were both correct. That reference model includes assumptions about how observations were collected and what the measurements represent. If those assumptions fail, the calculation may still produce a number, but that number may not have the stated probability interpretation.
This matters because a small p-value can look precise while being based on the wrong uncertainty or even the wrong quantity. For example, treating measurements from the same person as independent can make the apparent sample size too large. A systematic measurement offset can make the data appear inconsistent with a null value even when the underlying quantity matches it.
To make the assumptions concrete, consider a one-sample z test for a population mean \(\mu\), with null value \(\mu_0\) and known population standard deviation \(\sigma\). If the observations are independent and normally distributed with mean \(\mu_0\) under the null, then \(\bar{x}\) has a normal distribution with mean \(\mu_0\) and standard deviation \(\sigma/\sqrt{n}\). The test statistic is
A p-value is then calculated from the null distribution of \(Z\), using the tail or tails specified by the alternative hypothesis. This procedure relies on several linked conditions: the null value must describe the quantity actually tested; the observations must have the assumed dependence structure; and the measurements and stated value of \(\sigma\) must be appropriate. Normality is one sufficient condition for the stated z reference distribution in this setup. Other procedures may use different conditions, but no procedure makes its assumptions optional.
Dependence Changes the Uncertainty
The usual standard error \(\sigma/\sqrt{n}\) for a sample mean relies on independence. When observations move together, the sample mean usually varies more from sample to sample than that formula predicts. If the standard error is understated, a test statistic can be too large in magnitude, producing a p-value that is too small.
Worked Example: Two Related Measurements in Each Cluster
In a hypothetical study, a researcher records a change in a laboratory value for two people in each of 10 clinics. There are \(n=20\) recorded changes. Suppose the marginal standard deviation of each change is known to be 10 units, and the two observations within each clinic have correlation \(0.8\). Different clinics are independent. Under the null hypothesis, the mean change is 0 units. The observed average change is 6 units. Compare the standard independent-observation calculation with one that uses the stated within-clinic dependence.
Assumptions for the comparison. Suppose the 10 clinic pairs follow the same bivariate normal model, with each measurement having standard deviation 10 units and within-pair correlation 0.8. Assume the pairs are independent of one another, and that these model quantities are known. This gives a specified null distribution for the sample mean. The independent-observation calculation does not match this model because observations in a clinic are correlated.
Calculation that incorrectly assumes independence. Using \(\sigma=10\) and \(n=20\), the standard error would be \(10/\sqrt{20}=2.236\) units. The resulting statistic is
For a two-sided test, the standard normal reference model gives \(P(|Z|\geq 2.683)\approx 0.0073\). That calculation treats all 20 measurements as independent, which contradicts the stated model.
Calculation using the dependence. The variance of a sum is the sum of the individual variances plus twice each pairwise covariance. Within each clinic, the covariance is \(0.8(10)(10)=80\) square units. There are 20 individual variances of \(100\) and 10 within-clinic covariance pairs. Thus
The dependence-aware statistic is \(z=(6-0)/3=2\), giving the two-sided p-value \(P(|Z|\geq 2)\approx 0.0455\). Under the specified model, this is the appropriate calculation. The difference between 0.0073 and 0.0455 is not a change in the observed average; it is a consequence of using different, and here only one correct, models for its sampling variation.
The general lesson is not that every correlation makes a p-value smaller or larger by a fixed amount. Rather, the null distribution must reflect the dependence actually present. Repeated observations on one person, members of one family, or patients treated at the same clinic may carry less independent information than the same number of unrelated observations. A count of recorded rows is not automatically a count of independent units.
Biased Measurements Can Test the Wrong Null
A test can model random variation accurately and still be misleading if the measurement process shifts every observation away from the quantity of interest. Suppose \(X_i\) is the true value for unit \(i\), while the recorded value is \(Y_i=X_i+b+\epsilon_i\). Here \(b\) is a systematic measurement offset and \(\epsilon_i\) is random measurement error. If the target is the mean of the true values, a null hypothesis about that mean does not generally imply that the recorded values have the same mean.
Worked Example: A Known Calibration Drift
Suppose a hypothetical instrument adds 2 units to every measurement. A researcher takes \(n=25\) independent measurements of a quantity whose true mean is being tested against \(\mu_0=0\) units. Assume the random measurement errors are normally distributed with known standard deviation 5 units, and that their realized average in this particular sample is 0 units. The observed average is therefore 2 units. What does a test that ignores the drift report, and what follows from the calibrated model?
Assumptions for the calibrated calculation. The offset is exactly 2 units for every measurement, the random errors are independent normal variables with mean 0 and known standard deviation 5 units, and the null concerns the true mean. Under that null, the expected recorded average is \(0+2=2\) units. The stated sample average of 2 units follows from the example’s assumption that the realized random errors average to 0; it is not guaranteed just because the true mean is zero.
Ignoring the offset. If the researcher compares the observed average directly with 0, the standard error is \(5/\sqrt{25}=1\) unit. The statistic is
A two-sided standard normal calculation gives \(P(|Z|\geq 2)\approx 0.0455\). But this reference calculation treats the expected recorded average under the null as 0. The measurement model says it is 2 units. Thus the calculation does not provide a valid test of the stated null about the true mean.
Using the known offset. The recorded average expected under the null is \(\mu_0+b=0+2=2\) units. The appropriate standardized difference is therefore
The two-sided p-value is 1 under this model. The observed average is exactly at its null expectation after calibration. A systematic offset can therefore create apparent evidence against the true-mean null when the test is incorrectly centered. If the question instead concerned whether the instrument’s recorded mean equals 0, that would be a different parameter and a different null hypothesis.
Calibration Uncertainty Also Belongs in the Model
Sometimes an instrument’s offset is not known exactly; it is estimated in a separate calibration process. Correcting the observed average by the estimated offset does not make the uncertainty in that estimate disappear. If the calibration estimate is independent of the study measurements, its variance contributes to the variance of the corrected average.
Worked Example: An Estimated Instrument Offset
In a hypothetical measurement study, \(n=25\) independent readings have observed average 1.8 units. The known standard deviation of individual random measurement errors is 2.5 units. A separate, independent calibration estimates the additive offset as 1.0 unit, with a normal error distribution and standard error 0.6 units. Test the null that the true mean is 0, using a two-sided normal reference calculation and these stated model assumptions.
Model and assumptions. Write the study average as \(\bar{Y}=\mu+b+\bar{\epsilon}\), where \(b\) is the true offset and the study errors have standard deviation 2.5 units. Let \(\hat{b}\) be the independent calibration estimate. Assume \(\hat{b}\) is unbiased for \(b\), its error is normally distributed with standard error 0.6 units, and it is independent of the study errors. The corrected estimate is \(\bar{Y}-\hat{b}\). Under the null, its expected value is 0 units.
The corrected estimate is \(1.8-1.0=0.8\) units. The study-measurement contribution to its standard error is \(2.5/\sqrt{25}=0.5\) units. Because the calibration error is independent, the two variance contributions add:
The statistic is \(z=0.8/0.781\approx 1.024\), with two-sided p-value \(P(|Z|\geq 1.024)\approx 0.306\). If the researcher incorrectly treated the estimated offset as exact, the standard error would be 0.5 units, giving \(z=0.8/0.5=1.6\) and a two-sided p-value of about 0.110. That smaller p-value ignores real uncertainty in the calibration. It is not the result of the stated model.
These examples separate two problems that are sometimes confused. A known offset changes the null expectation of the recorded measurements. An uncertain offset adds uncertainty to the corrected estimate. In either case, the test needs a model that matches how the measurement was produced.
A Short Justification: Why Dependence Alters the Standard Error
The independent-observation standard error follows from a variance calculation. If \(X_1,\ldots,X_n\) are independent with the same variance \(\sigma^2\), then the variance of their average is
If observations are dependent, the variance of the sum also includes covariance terms:
The familiar \(\sigma/\sqrt{n}\) standard error follows only when the covariance terms are zero. Positive covariances increase the variance of the average; negative covariances can decrease it. This is why an independence assumption is not a minor technical detail: it determines the reference distribution used to calculate the p-value.
Common Mistakes and a Practical Check
- Counting measurements as independent units. Twenty readings do not necessarily provide the same information as 20 independent observations. Check whether observations share people, places, equipment, or other sources of dependence.
- Assuming a large sample repairs dependence or bias. More observations can reduce random uncertainty under an appropriate model, but they do not automatically remove correlated errors or a systematic measurement offset.
- Testing the recorded value while making a claim about the true value. State which quantity the null describes. If recorded values are shifted or noisy, relate their distribution to the target quantity before calculating the p-value.
- Treating a correction as exact when it was estimated. If a calibration estimate has uncertainty, account for it in the sampling variation of the corrected estimate.
- Interpreting a calculated p-value as proof that assumptions hold. The p-value is conditional on the null model; it cannot validate the model that produced it.
Before relying on a p-value, describe the observational units and how they were selected, identify any repeated or clustered measurements, and explain how the measurement process relates recorded values to the target. Then ask whether the null distribution used by the test reflects those features. If it does not, the remedy is not to reinterpret the same p-value more forcefully; the analysis needs an appropriate model or better measurements.
Check Your Understanding
Use the model and assumptions in each question to explain what the p-value calculation would require.
- Why can treating two highly correlated measurements from the same clinic as independent make a standard error too small?
- An instrument adds a known 3 units to every recorded value. If the null concerns a true mean of 0, what is the expected recorded mean under that null?
- In the estimated-calibration example, why is the variance of the corrected estimate the sum of two variance contributions?
- Could increasing the number of measurements remove a fixed calibration offset? Explain why or why not.
- A test produces a small p-value under an independence model, but the observations were collected in strongly related pairs. What should be checked before interpreting that p-value?