Why a Large P-Value Does Not Decide a Term’s Role
A regression coefficient test answers a focused question: under the specified model, is the coefficient distinguishable from zero with the available data and assumptions? Model specification asks a different question: which terms are needed to represent the scientific question and the planned comparison? These questions are related, but they are not interchangeable.
If a term has a large p-value, the data may provide weak evidence that its coefficient differs from zero, conditional on the other included terms. That result alone does not show that the term is irrelevant to the research question, that it is not needed for an adjustment, or that removing it would leave the intended comparison unchanged. A term may therefore remain in a model for a stated scientific or design reason even when its individual test is weak.
What an Individual Coefficient Test Says
For independent observations, a common regression model for a quantitative outcome is
Here, \(Y_i\) is the outcome for observation \(i\), the \(X\) terms are predictors, the \(\beta\) values are population coefficients, and \(\varepsilon_i\) is the error. For the usual exact individual t test, assume the errors are independent, have mean zero and common variance \(\sigma^2\), and are normally distributed conditional on the predictors. The model’s mean structure must also be appropriate: the included terms should represent the conditional mean being modeled.
For a predictor \(X_j\), the individual two-sided test is \(H_0:\beta_j=0\) against \(H_a:\beta_j\ne 0\). Given an estimated coefficient \(\hat{\beta}_j\) and its standard error \(SE(\hat{\beta}_j)\), the test statistic is
Under the null hypothesis and the stated model assumptions, this statistic has a t distribution with the model’s residual degrees of freedom. The p-value is the probability, assuming \(H_0\) and the model, of a statistic at least as far from zero as the observed one. It measures evidence about that coefficient under that model. It does not directly test whether the term belongs in the specification.
Worked Examples: Retaining Terms for Their Stated Roles
Worked Example: Prespecified Baseline Adjustment
Hypothetical data. In a randomized study of 120 adults, the outcome is systolic blood pressure at follow-up, measured in mmHg. The prespecified model includes treatment group and baseline systolic blood pressure. The baseline measurement is included to adjust the treatment comparison for participants’ starting levels.
Suppose the fitted baseline coefficient is \(-0.12\) mmHg per 1 mmHg higher baseline pressure, with standard error \(0.09\) mmHg per mmHg. There are 117 residual degrees of freedom. The individual test for the baseline coefficient gives
For a two-sided test with 117 degrees of freedom, the p-value is approximately \(0.185\). At a significance level of \(0.05\), this is not strong evidence against a zero baseline coefficient in this fitted model. It does not show that baseline pressure should be removed: the model was specified to compare treatment groups adjusted for the baseline measurement. Removing that term would change the comparison being reported.
For this hypothetical example, suppose participants’ outcomes are independent, the relationship between baseline pressure and follow-up pressure is adequately represented by a straight-line term, and residual checks show no substantial pattern, unequal spread, or serious departure from normality. Those conditions support using the stated t test. The reason to retain baseline pressure is its prespecified adjustment role, not a claim that its coefficient is statistically significant.
Worked Example: A Prespecified Confounder
Hypothetical data. A cross-sectional analysis of 85 workers relates weekly exposure hours to a lung-function measure, in liters. Age, measured in years, was specified in advance as an adjustment term because it may be related to both exposure patterns and lung function. The fitted age coefficient is \(-0.08\) liters per year, with standard error \(0.07\) liters per year. The model has 82 residual degrees of freedom.
The age test statistic is
The two-sided p-value is approximately \(0.256\). This is weak evidence against a zero age coefficient conditional on exposure hours and the other model terms. It is not a test of whether age could affect the exposure–lung-function comparison if omitted. Age remains because its adjustment role was identified from the scientific rationale before interpreting this individual test.
Suppose each worker contributes one independent observation, the conditional mean is adequately described by the included terms, and residual diagnostics show no pronounced curvature, changing spread, or highly influential observation. These checks make the model-based test interpretable under the stated assumptions. Because the data are observational, retaining age does not by itself establish a causal effect of exposure; other sources of confounding may remain.
Worked Example: Keeping the Predictor That Defines the Question
Hypothetical data. A study of 48 classrooms asks whether instructional time is associated with a test score, measured in points. The prespecified model includes instructional time, class size, and a baseline score. Instructional time is measured in units of 10 minutes per week. Suppose its fitted coefficient is \(0.31\) test-score points per 10 additional minutes, with standard error \(0.24\) points per 10 minutes. With three predictors and an intercept, the residual degrees of freedom are \(48-4=44\).
The test statistic for the instructional-time coefficient is
The two-sided p-value with 44 degrees of freedom is approximately \(0.203\). The data do not provide strong evidence against a zero conditional coefficient at the \(0.05\) level. However, instructional time is the predictor named in the research question. Removing it would leave a model that no longer estimates the association the study set out to describe.
A 95% confidence interval helps show the uncertainty in the estimate. Using a t critical value of approximately \(2.015\), the interval is
Under the model assumptions, this interval procedure has 95% coverage in repeated samples. In context, the interval ranges from an estimated decrease of 0.174 points to an increase of 0.794 points per additional 10 minutes per week, conditional on class size and baseline score. The interval includes zero and is fairly wide, so the estimate is uncertain; it does not prove there is no association. Suppose classroom observations are independent, the conditional mean form is adequate, and residual checks reveal no substantial violations of the model assumptions. The predictor remains because it defines the question, while the weak test and wide interval describe how imprecisely its coefficient is estimated.
Why the Test Depends on the Model
An individual coefficient is interpreted with the other terms in the model held fixed. Consequently, its estimate and standard error are properties of the fitted specification as well as of the data. Changing the included terms can change the coefficient being estimated, its uncertainty, and the meaning of a comparison. A p-value computed for one specification is not a context-free verdict about a variable.
This is why a term’s role should be stated before relying on its individual test. For example, a term might be included to adjust a planned comparison, represent the principal predictor named in the research question, or preserve a prespecified definition of the estimand. If that rationale remains applicable, a large p-value alone is not a sufficient reason to remove it. Conversely, the fact that a coefficient has a small p-value does not by itself establish that a term is scientifically essential.
A useful justification has two parts: name the term’s role, then report what the data say about its coefficient. This keeps model specification separate from uncertainty assessment. It also avoids presenting a model chosen only because its included coefficients happened to pass a significance cutoff as though that model had been determined solely by the scientific question.
A Practical Decision Sequence
Explain whether the term represents the target predictor, an adjustment, or another part of the prespecified comparison.
Ask whether removing the term would change the question or the comparison the model is intended to describe.
Report the coefficient, its standard error or interval, and the p-value as evidence about that coefficient under the specified model.
Do not turn a weak coefficient test into a claim that the term is unnecessary, or a retained term into a claim that its coefficient is nonzero.
Common Mistake: Treating a Large P-Value as an Exclusion Rule
The mistaken reasoning is: “This term has \(p>0.05\), so it has no effect and should be deleted.” A p-value above the chosen threshold is not proof that the coefficient equals zero. It may reflect a small association, substantial uncertainty, limited information, or overlap with other predictors. More importantly here, even a weakly estimated coefficient can belong to a model because the term has a prespecified role.
The correction is to separate the specification decision from the coefficient inference. First decide whether the term is needed for the stated scientific question or comparison. Then report the coefficient test and interval as evidence, including their uncertainty. If a term has no stated role and is only a candidate for an optional extension, its retention is a separate model-development decision; an individual p-value alone still does not establish the full scientific case for including or excluding it.
Check Your Understanding
Answer each question by distinguishing the term’s role from the evidence about its coefficient.
- A prespecified adjustment term has a p-value of \(0.31\). What does this p-value say, and what does it not decide?
- Why can deleting a required adjustment term change the comparison represented by a regression model?
- A 95% confidence interval for a primary predictor’s coefficient includes zero. Does that show the predictor is irrelevant to the research question? Explain.
- In a regression model, what does “the coefficient is tested conditional on the other terms” mean?
- Give two distinct statements: one that justifies retaining a term, and one that accurately summarizes weak evidence about its coefficient.