Common Errors in Interpreting Covariate P Values

What a Covariate P-Value Answers

During covariate selection, a p-value can look like a compact verdict: a small value may be read as proof that a covariate matters, while a larger value may be read as proof that it does not. Neither conclusion follows from the p-value alone. The value summarizes how unusual the observed test statistic would be under a specified null hypothesis and the assumptions used to calculate its reference distribution. It does not directly measure effect size, clinical importance, causality, or the chance that a selected relationship will recur.

In a population PK model, a covariate test commonly compares a model that includes a specified covariate effect with a nested model that does not. For example, a null hypothesis might state that the exponent relating body weight to typical clearance is zero. The exact null, parameterization, test statistic, and reference distribution matter: a p-value is not meaningful without them.

Definition: For a specified test statistic \(T\), the p-value is the probability, assuming \(H_0\) and the test's other assumptions, of observing a value of \(T\) at least as incompatible with \(H_0\) as the value observed. “At least as incompatible” is determined by the test's tail rule. A p-value is not \(P(H_0\mid\text{data})\), nor is it the probability that a finding is true or will replicate.

For a likelihood-ratio test of a covariate effect, let \(OFV_0\) denote the objective function value for the null model and \(OFV_1\) the value for the nested model containing the effect. Under the null and standard regularity conditions, the likelihood-ratio statistic is approximately chi-square distributed, with degrees of freedom equal to the difference in the number of estimated parameters.

\Delta OFV = OFV_0 - OFV_1,\qquad p \approx P\left(\chi^2_{df} \geq \Delta OFV_{\text{observed}}\mid H_0\right)

This approximation relies on an adequately specified likelihood, appropriate optimization, nested models, and regularity conditions for the parameters being tested. A Wald test instead compares an estimate with its standard error and relies on an adequate normal approximation. These tests can disagree when estimates are imprecise, likelihoods are asymmetric, or the model is poorly behaved. A reported p-value should therefore be read as the result of a particular test under particular assumptions—not as a model-independent property of a covariate.

Significance Is Not Effect Size

Statistical significance describes the compatibility of the data with a null hypothesis under a test. Effect size describes the magnitude of the estimated relationship on a defined scale. Precision describes uncertainty in that estimate. These are related but distinct: with enough information, a small effect can produce a small p-value; with limited information, a potentially substantial effect can have a large p-value.

Worked Example: A Small Effect With a Small P-Value

Suppose a hypothetical population PK analysis uses the weight model \(CL_i=TVCL(WT_i/70)^\beta\), where \(CL_i\) is individual clearance, \(TVCL\) is typical clearance at 70 kg, and \(WT_i\) is weight in kg. The estimated weight exponent is \(\hat{\beta}=0.014\), and a likelihood-ratio test of \(H_0:\beta=0\) gives \(\Delta OFV=10.8\) with \(df=1\). Assume the nested models are correctly fit and the chi-square approximation is suitable.

The p-value is the upper-tail probability for a chi-square random variable with one degree of freedom:

p=P(\chi^2_1\geq10.8)\approx0.00102

This is evidence against the specified zero-exponent null under the test assumptions. It does not mean there is a 0.102% probability that the null is true. To see the separate scale of the estimate, compare predicted clearance at 140 kg with that at 70 kg:

\frac{CL(140)}{CL(70)} =\left(\frac{140}{70}\right)^{0.014} =2^{0.014} =\exp(0.014\ln 2) \approx1.00975

The fitted model therefore implies about a 0.975% higher typical clearance at 140 kg than at 70 kg. The small p-value does not turn that estimated contrast into a large one, nor does it establish whether that difference is clinically important. That requires a separate interpretation in the relevant clinical context.

The reverse mistake is also common: treating a p-value above a threshold as proof that there is no effect. A non-small p-value means the test did not provide sufficiently strong evidence against its null at the chosen threshold. It does not show that the effect is exactly zero. The estimate and its uncertainty are needed to see whether the data remain compatible with both negligible and substantial effects.

Worked Example: A Potentially Meaningful Estimate With a Larger P-Value

In a second hypothetical analysis, a power model estimates a covariate exponent as \(\hat{\beta}=0.30\) with \(SE=0.17\). Assume the estimator is approximately normal under the null, the standard error is valid, and this test was specified before examining the result. A two-sided Wald test of \(H_0:\beta=0\) uses:

z=\frac{\hat{\beta}-0}{SE} =\frac{0.30}{0.17} \approx1.765,\qquad p=2P(Z\geq1.765)\approx0.0776

At \(\alpha=0.05\), this test does not reject \(H_0\). That decision is not evidence that the exponent is zero. A normal-approximation 95% interval for the exponent is:

0.30\pm1.96(0.17) =0.30\pm0.3332 =(-0.0332,\ 0.6332)

For a doubling of the covariate under this power model, the fitted ratio is \(2^{0.30}\approx1.231\), or about 23.1% higher. Transforming the exponent interval gives \(2^{-0.0332}\approx0.977\) to \(2^{0.6332}\approx1.551\). This broad range includes a ratio near 1 as well as appreciably larger ratios. The correct conclusion is that this analysis leaves substantial uncertainty; it is not that the effect is absent. The interval is an uncertainty summary under the model and approximation, not a 95% probability statement about the fixed parameter.

Association Is Not Causation

A covariate p-value in an observational population PK analysis tests a model-based association, conditional on the model terms and assumptions used. It does not establish that changing the covariate would cause a change in a PK parameter. A covariate may be associated with the outcome because it is related to another determinant, because of selection into the study, because of measurement or model error, or because the fitted model does not represent the data-generating process adequately.

Worked Example: A Significant Renal Marker Association

Consider a hypothetical model for typical clearance that includes a renal marker \(R\):

TVCL_i=TVCL\left(\frac{R_i}{90}\right)^\beta

Suppose \(\hat{\beta}=0.25\), \(SE=0.08\), and the analysis reports a two-sided Wald test of \(H_0:\beta=0\). If the estimate is approximately normal and the standard error and model are valid, then:

z=\frac{0.25}{0.08}=3.125,\qquad p=2P(Z\geq3.125)\approx0.00178

This provides evidence of an association between the renal marker and typical clearance, conditional on the fitted model. For example, the fitted model predicts a ratio of \(2^{0.25}\approx1.189\) between marker values of 180 and 90 in the model's units. But the p-value does not show that experimentally increasing the marker would increase clearance by that amount. Disease severity, treatment, other measured or unmeasured factors, and the construction of the marker could contribute to the observed relationship. The causal interpretation requires design and assumptions beyond this significance test.

The distinction matters when covariates are used to describe patients, predict PK, or inform decisions. An association may be useful for prediction without being causal. Conversely, a plausible mechanism does not rescue a poorly estimated or misspecified association. State what the fitted model supports, and do not claim more than the design and assumptions justify.

A Small P-Value Does Not Guarantee Replication

A p-value is calculated for a particular analysis; it is not the probability that the same covariate will be selected in a new sample. Selection procedures can test many candidate effects, and some small p-values can arise by chance even when all the corresponding null hypotheses are true. Searching, changing the structural model, or selecting the most favorable result after examining the data can also make a nominal p-value overstate the strength of evidence.

Worked Example: Chance Findings Across Many Tests

Suppose 20 independent covariate tests are performed, all null hypotheses are true, and each test uses \(\alpha=0.05\). For any one test, the probability of not producing a false positive is \(1-0.05=0.95\). Under independence, the probability that none of the 20 tests is significant is \(0.95^{20}\). Thus the probability of at least one nominally significant result is:

P(\text{at least one }p<0.05) =1-0.95^{20} \approx1-0.3585 =0.6415

So even under these simplifying assumptions, there is about a 64.2% chance of at least one p-value below 0.05. Real covariate tests may be dependent, so this calculation is not a universal correction; it illustrates why the number and dependence of tests matter. A selected p-value alone cannot quantify the probability that the selected effect will recur. Earlier material on simulation-based search evaluation, bootstrap stability, and cross-validation addresses complementary aspects of selection and predictive performance; those assessments answer different questions from a single test.

A p-value also does not give the probability of a successful replication. Replication depends on factors such as the true effect, sample size, measurement quality, model choices, and how selection is conducted in the new data. A small p-value in one selected analysis may coexist with unstable selection across resamples or settings. Conversely, a covariate may have a plausible and reproducible predictive role even if a particular test is not significant in a limited data set.

Checks, Alternatives, and Common Pitfalls

Before interpreting a reported covariate p-value, check that the tested null is clear, the models or test statistic are appropriate, and the approximation used for the reference distribution is credible. In population PK data, sparse individual information, influential subjects, parameter correlations, boundary constraints, and model misspecification can affect estimation and uncertainty. For example, a standard chi-square likelihood-ratio reference can fail when regularity conditions do not hold, such as some tests involving a parameter on the boundary of its allowable range. A p-value from an invalid reference distribution is not repaired by reporting more decimal places.

The test itself is only one summary. A Wald test is convenient when the estimate is well approximated by a normal distribution and its standard error is reliable. A likelihood-ratio test compares fitted likelihoods and can behave differently in finite samples or with asymmetric uncertainty. Profile likelihood or simulation-based calibration may be more informative when the approximation is doubtful, but they also require appropriate model specification and computation. Reporting the estimate with its uncertainty, the exact test, the null hypothesis, and the analysis context is more informative than reporting a p-value alone.

Common errors include:

“The p-value is the probability the null is true.” It is calculated assuming the null, not by assigning a probability to the null.
“A smaller p-value means a larger covariate effect.” It can instead reflect greater precision or more information.
“A non-significant result proves no effect.” It may reflect limited information; inspect the estimate and uncertainty.
“A significant covariate causes the PK change.” The test establishes, at most, evidence for the model-based association under its assumptions.
“A selected significant result will replicate.” Replication is not guaranteed by a p-value and must be assessed using suitable new data or stability and validation analyses.

Key takeaway: Interpret a covariate p-value as evidence against a specified null under a specified model and test. Keep that evidence separate from the estimated effect size, its uncertainty, clinical interpretation, causal claims, and the likelihood of replication.

Check Your Understanding

Answer each question, then compare with the solutions below.

A likelihood-ratio test gives \(p=0.018\). Does this mean there is a 1.8% probability that the null hypothesis is true?
A covariate estimate is small but has a very small p-value. What does the p-value establish, and what does it not establish about the effect?
A covariate test has \(p=0.12\). Is it correct to conclude that the covariate has no effect? Explain.
Twenty independent tests are performed at \(\alpha=0.05\), with all null hypotheses true. What is the probability of at least one p-value below 0.05?
A renal marker has a significant conditional association with clearance in an observational model. What additional claim cannot be made from that p-value alone?

Solutions. (1) No. The p-value is the probability, assuming the null and test assumptions, of a test statistic at least as extreme as the observed one. It is not the probability that the null is true. (2) It provides evidence against the specified null under the test assumptions; it does not establish that the effect is large or clinically important. (3) No. The test did not reject its null at the chosen threshold, but the data may still be compatible with nonzero effects; examine the estimate and its uncertainty. (4) \(1-0.95^{20}\approx0.6415\), or about 64.2%, assuming independence and valid tests. (5) The p-value alone cannot establish that changing the marker would cause a change in clearance.