Multiplicity and Winner’s Curse in Covariate Searches

Why a Covariate Search Changes the Evidence

In an NLME analysis, a covariate search may examine several candidate relationships with one or more model parameters. Even if each individual test uses a conventional significance level, searching many candidates creates more opportunities to find an apparently strong result by chance. The search also affects effect estimates: when a candidate is retained because its estimated effect is unusually large, that estimate tends to overstate the underlying effect. These are related but distinct problems: multiplicity concerns false-positive decisions, while the winner’s curse concerns the magnitude of an effect selected from noisy estimates.

For a clear starting point, suppose each candidate covariate relationship has a null hypothesis that its population effect is zero, and its test produces a p-value. In the simplest calculations below, assume tests are valid at level \(\alpha\): under a true null, the probability of rejecting it is at most \(\alpha\). Wald tests in NLME models commonly rely on adequate model specification and an approximate normal distribution for the estimated effect relative to its standard error. These approximations may be unreliable with sparse data, weak identification, or boundary estimates; multiplicity calculations do not repair invalid individual tests.

The family should reflect the search that supports the claim. If a model-building process examines six covariates and highlights whichever appears most associated with clearance, the relevant family is not merely the single final test displayed in a table. It includes the candidate tests that could have led to that choice. Defining the family after seeing the results can make error control look stronger than it is.

How Multiple Tests Inflate False-Positive Risk

If \(m\) independent null hypotheses are all true and test \(j\) has actual type I error \(q_j\leq\alpha\), the probability of avoiding false positives in all \(m\) tests is \(\prod_{j=1}^m(1-q_j)\). Therefore,

Equality holds when each test’s actual type I error is exactly \(\alpha\).

This exact expression requires independence and the complete-null condition that all \(m\) hypotheses are true. Covariate tests are often correlated, for example when candidate covariates themselves are correlated, so the independence expression may not apply. A dependence-robust bound follows from the union bound: if each true null test has type I error no greater than \(\alpha\), then the probability of at least one false rejection is no greater than \(m\alpha\). The bound can be conservative, but it does not require independent tests.

A simple way to control FWER without assuming independence is the Bonferroni procedure. For \(m\) tests and a desired family-wise level \(\alpha\), compare each p-value with \(\alpha/m\). Equivalently, multiply each p-value by \(m\), capping the adjusted p-value at 1, and compare it with \(\alpha\). The union bound justifies the procedure: the chance of any false rejection is at most the sum of the individual error probabilities, which is at most \(m(\alpha/m)=\alpha\). Bonferroni control can be conservative, particularly when tests are strongly correlated.

Holm’s step-down procedure is another standard way to control FWER under arbitrary dependence. It orders the p-values from smallest to largest, compares the smallest with \(\alpha/m\), the next with \(\alpha/(m-1)\), and continues until a comparison fails; that p-value and all larger ones are not rejected. Holm is at least as powerful as ordinary Bonferroni while retaining strong FWER control. Neither adjustment makes a model or test valid if the underlying assumptions fail.

Why the Selected Effect Can Be Too Large

Multiplicity adjustment addresses rejection decisions, but a selected estimate can remain exaggerated. To see the mechanism, consider a simplified setting in which an estimated effect \(\hat{\beta}\) has a normal sampling distribution, known standard error \(s_\beta\), and true effect zero. Then \(Z=\hat{\beta}/s_\beta\) follows a standard normal distribution. If an analyst keeps a candidate only when its estimate is positive and its one-sided statistic exceeds a cutoff \(c\), the retained estimates necessarily have \(Z>c\). Their conditional average is greater than zero, despite the true effect being zero.

The conditional mean can be derived directly. Write the standard normal density as \(\phi(z)\) and its upper-tail probability as \(1-\Phi(c)\). Since \(\phi'(z)=-z\phi(z)\), p>

The equality follows because \(\int_c^\infty z\phi(z)\,dz=[-\phi(z)]_c^\infty=\phi(c)\). This result describes selection above a fixed positive threshold under the stated normal model. Selecting the largest estimate among several candidates can produce further upward selection, and the exact amount depends on the number and dependence of candidates and on the selection rule.

Practical Safeguards and Diagnostics

The most effective safeguard is to distinguish confirmatory questions from exploratory searches before interpreting results. Prespecify the covariates and hypotheses that form the primary family, the parameter scale on which the effects will be evaluated, and any multiplicity procedure. A limited, clinically justified primary family is easier to interpret than a broad search whose scope changes after results are seen. Prespecification does not guarantee a true effect; it makes the evidential claim and its error rate clearer.

For exploratory analyses, report the breadth of the search, not just the retained covariates. State how many candidates and tests were considered, how candidates were screened, what adjustment was used, and which analyses were data-driven. Present effect estimates with uncertainty and units, while making clear that ordinary confidence intervals and p-values generally do not account for selection unless the inferential procedure explicitly does so.

Stability checks provide useful information about whether a finding depends heavily on particular observations or modeling choices. Earlier material on bootstrap stability and cross-validation describes ways to assess reproducibility; here the key interpretation is that repeated selection across resamples or validation splits is supportive evidence of robustness, not a guarantee of an unbiased effect or formal error control. A finding that disappears under modest resampling, or depends on a few influential subjects or sparse subgroups, deserves especially cautious language. Likewise, residual stratification patterns may motivate investigation, but searching many plots and strata also creates opportunities for chance patterns.

There are several common approaches, with different goals:

The earlier material on shrinkage-based selection and penalized screening introduces methods that can temper extreme estimates during selection. Such methods do not make the search invisible: their tuning, candidate set, and assumptions still matter, and the resulting estimates should be interpreted according to the method used. No single adjustment or stability check can substitute for transparent reporting and independent confirmation when strong claims are required.

Common Pitfalls and Reporting

• Reporting only the winning p-value. A p-value of 0.04 after examining many candidates is not the same evidence as a p-value of 0.04 from one prespecified test. Report the search scope and adjustment or label the finding exploratory.
• Using the independence formula for correlated tests. \(1-(1-\alpha)^m\) is exact only under independence and the complete null. Use a justified dependence-aware calculation or a conservative procedure such as Bonferroni or Holm when appropriate.
• Treating adjusted significance as proof of a large effect. Multiplicity procedures address false-positive decisions, not clinical importance, model misspecification, or winner’s curse. Report the estimate, units, uncertainty, and selection context.
• Calling a stable selection confirmatory. Repeated selection may strengthen a robustness argument, but it does not by itself establish the population effect or remove bias from the search.

A cautious report might state: “Among the six candidate relationships examined, the covariate–clearance association had an estimated effect of 0.20 L/h per covariate standard deviation. Its unadjusted p-value was 0.0455, but it did not meet the Bonferroni family-wise threshold of 0.00833. The result is exploratory, and the selected effect may be upwardly biased.” This reports the evidence in context without interpreting a nominal result as confirmation.