Multiple Testing During Covariate Screening

Why Repeated Covariate Tests Change the Interpretation

In a population pharmacokinetic analysis, a model may be tested against alternatives that add different candidate covariates to a parameter such as clearance. Each test can be individually well specified and use the same nominal significance level. But if many candidates are tested and any result below that level is treated as evidence for an effect, the chance of at least one false-positive result can be substantially higher than the significance level of a single test.

This matters because a covariate screen is often used to make decisions: which effects to investigate further, which terms to retain, and which findings to report as potentially important. A low p-value from such a search is evidence about a test considered in the context of that search. It is not automatically the same kind of evidence as a low p-value from one hypothesis specified before examining the data.

For covariate screening, the family might include the tests of age, body size, renal function, and genotype on clearance, if those tests are all searched and any positive result may be selected. The family should be defined by the scientific and decision context, not merely by which results are eventually reported. If the same data are used to test additional covariates, transformations, or parameter effects after seeing initial results, those choices can enlarge the effective search.

How the Probability Accumulates

First consider \(m\) tests with true null hypotheses, each having a valid per-test Type I error rate \(\alpha\). If the tests are independent, the probability that a particular test does not reject its null is \(1-\alpha\). Independence means the probability that none of the \(m\) tests rejects is the product of these probabilities. Therefore:

This is the FWER under the global null, the situation in which every tested covariate null hypothesis is true. For small \(\alpha\) and moderate \(m\), the risk can rise quickly. The expected number of false rejections under the global null is \(m\alpha\), because each test has probability \(\alpha\) of a false rejection and expectations add. That expected count is not the same quantity as the probability of at least one false rejection.

Covariate tests are often dependent. For example, body weight and body surface area may carry overlapping information, so their test statistics need not vary independently. In that case, \(1-(1-\alpha)^m\) is not generally the exact FWER. Dependence does not make the multiple-testing issue disappear; it means the exact probability depends on the joint distribution of the test results.

A general bound does not require independence. The probability of the union of events is no greater than the sum of their probabilities. Applying this to false rejections gives:

This bound can be loose, especially when tests are strongly dependent, but it shows that individual error rates alone do not specify the family-level risk. The calculation also relies on valid individual tests: if a test statistic is poorly calibrated, its actual Type I error rate may differ from the stated one.

Screening Is Not the Same as Confirmation

An exploratory screen is a search for candidates worth further attention. Its purpose may be to identify patterns, generate hypotheses, or prioritize follow-up analyses. In that setting, an unadjusted p-value can be reported as a result of an individual test, but it should be interpreted alongside the number and nature of the tests considered. Selecting the smallest p-value and presenting it as though that single test had been chosen in advance hides the search that produced it.

A confirmatory test answers a narrower question. The hypothesis, analysis, and decision rule are specified before the confirmatory data are examined. If a candidate covariate was selected using one data set and tested in an independent validation data set, then, under valid model and test assumptions, the validation p-value can be interpreted for the prespecified test in that validation analysis. The selection process in the first data set does not by itself invalidate an independent validation test.

Independence of data is not enough if the validation analysis quietly reopens the search. For example, testing six candidate covariates in the validation data and reporting only whichever has the smallest p-value recreates a multiple-testing problem. The confirmatory claim must correspond to the test that was actually prespecified, including its outcome, parameter, covariate coding, and analysis procedure.

The distinction is not that exploratory results are useless and confirmatory results are automatically correct. Exploration can be scientifically valuable, particularly when it identifies plausible covariates for further study. Confirmation offers a more focused test of a stated claim, but still depends on appropriate model specification, measurement quality, and a valid testing procedure.

Interpreting and Reporting a Covariate Screen

A useful report makes the search visible. State how many candidates were considered, which parameter each test addressed, whether the tests were planned or added after examining results, and what decision rule was used. If the search included multiple encodings or repeated analyses, describe those choices as well. Reporting only the selected covariate can make a broad exploratory search look like a single planned test.

A p-value below a nominal threshold is not the probability that the null hypothesis is true, and a p-value above it does not establish that the covariate has no effect. In a population PK model, the estimated effect size and its uncertainty remain important alongside the test result. A statistically notable screening result may be small in practical terms, unstable under alternative specifications, or difficult to distinguish from the effects of correlated covariates.

Multiplicity also does not explain every failure to replicate. A candidate can be selected by chance, but results can also differ because of sampling variation, different covariate distributions, measurement error, model assumptions, or differences between populations. Screening results should therefore motivate careful evaluation rather than an automatic claim that an association has been established.

Common Pitfalls and Limits

• Treating each 0.05 test as a 5% chance of any false finding. The 0.05 rate applies to one valid test under its null. For multiple tests, the chance of one or more false rejections can be larger.
• Assuming all tests are independent. Related covariates can produce dependent statistics. The independence formula is then not generally exact, even though multiplicity remains relevant.
• Defining the family only after seeing results. Excluding unreported candidates from the stated search conceals the process that led to the selected result.
• Calling a selected screening result confirmatory. A test chosen because it looked promising in the same data is still part of that exploratory search.
• Equating a false-positive risk calculation with a conclusion about a particular finding. FWER is a probability over repeated applications of a procedure under specified conditions. It does not tell whether a specific selected covariate effect is true.

There is no single probability calculation that captures every search. The independent-test formula requires independence and a global null; the union bound requires valid per-test error rates but can be conservative. When some null hypotheses are false, FWER concerns false rejections among the true nulls, and its value depends on the joint test behavior. The central practical point remains: the interpretation of a covariate result depends on how many related opportunities to find a result were considered and how selection was performed.