Why Separate Selection From Confirmation?
When several treatments are compared, the treatment that looks most promising in an initial sample may look unusually good partly because of random variation. Testing that same treatment on the same observations can carry the initial selection advantage into the confirmatory test. An independent validation sample offers a clear design boundary: use one set of observations to select, then new observations to test the selected hypotheses.
Independence does not make every confirmatory analysis valid automatically. The validation endpoint, treatment contrast, direction of testing, analysis model, and multiplicity procedure still need to be specified. Nor does independence remove the need to account for testing more than one selected treatment. Its particular benefit is that, under suitable assumptions, selection based on the first sample does not distort the validation test’s null distribution.
In a clinical trial, this usually means enrolling distinct participants for validation after the selection decision is fixed, with treatment assignment and outcome measurement specified for that stage. Independence concerns the observations and information used in the test; it does not by itself ensure that the validation participants represent the same target population or that the outcome model is correct. Those requirements must be considered separately.
Designing the Validation Stage
A practical design identifies the candidate treatments, the selection rule, the maximum or planned number of treatments to validate, the confirmatory hypotheses, and the familywise significance level. The validation stage should then preserve a clean separation: selection is completed without access to validation outcomes, and confirmatory decisions are made using the validation outcomes and a prespecified multiplicity procedure.
State which treatment comparisons may be confirmed, which outcome and time point define each comparison, and whether the alternative is one-sided or two-sided.
Use the selection data to choose a set \(S\), and document that decision before validation outcomes are examined.
Collect new observations for the selected comparisons. Decide in advance how many treatments can be tested and how the significance level will be allocated.
Apply tests whose null distributions are valid for the validation design, then make confirmatory claims only according to the prespecified multiplicity rule.
If at most \(K\) candidate treatments could be tested, one conservative option is to allocate \(\alpha/K\) to each candidate, even when fewer are selected. If exactly \(m\) treatments are selected and tested, Bonferroni allocation at \(\alpha/m\) controls the familywise error rate for that selected family. These allocations do not require the validation test statistics to be independent of one another. In shared-control designs, for example, validation contrasts may be correlated; the union bound still applies.
Worked Example: Testing Two Selected Treatments
Suppose a hypothetical trial selects treatments A and B from five candidates using a pilot sample. A separate randomized validation sample compares each selected treatment with a shared control on a continuous endpoint. Assume independent participants, a common known standard deviation of 8 outcome units in each group, and a normal model for the group means. The prespecified confirmatory tests are one-sided superiority tests at familywise level \(\alpha=0.05\). Because two treatments are selected, allocate \(0.05/2=0.025\) to each test.
In the validation sample, each group has \(n=36\). The control mean is 1.2 units, the mean for A is 6.0, and the mean for B is 3.0. For either treatment-control contrast, the standard error is
For A, the estimated difference is \(6.0-1.2=4.8\) units, giving \(z=4.8/1.886=2.546\). The one-sided p-value is approximately 0.0055. Since \(0.0055<0.025\), A meets the prespecified confirmatory criterion. For B, the estimated difference is \(3.0-1.2=1.8\) units, so \(z=1.8/1.886=0.954\), with one-sided p-value approximately 0.170. B does not meet its criterion. These calculations rely on the stated normal model and known standard deviation; a trial using an estimated variance would need the corresponding prespecified test and reference distribution.
The selection-stage measurements do not enter these test statistics. The two validation contrasts share a control and are therefore not independent, but Bonferroni control does not depend on their mutual independence. The conclusion is limited to the validation population and the stated endpoint and analysis.
Why Conditional Validity Protects Error Control
The key argument conditions on the entire selection-stage data. Once \(D\) is fixed, the selected set \(S=s(D)\) is fixed too. If a true null treatment is selected, an independently collected validation sample still supplies a valid null test for that treatment, provided the model, randomization, and analysis assumptions hold. Thus selection can change which hypotheses are tested without changing the conditional null calibration of their validation p-values.
For a selected family of size \(m=|S|>0\), reject \(H_i\) only when \(p_i\leq\alpha/m\). Let \(I_0\) be the set of true null hypotheses. Conditional on \(D\), a false rejection requires at least one \(i\in I_0\cap S\) to cross its threshold. The union bound and conditional validity give
If \(S\) is empty, no confirmatory rejection is possible, so the conditional error probability is zero. Averaging the conditional bound over all possible selection-stage data gives unconditional familywise error control as well:
This argument explains what the earlier material on data-independent selection supplies: selection based on \(D\) is harmless to this conditional bound when the validation tests remain valid given \(D\). It does not say that any test applied after selection is automatically valid. The conditional validity premise is essential.
Worked Example: Checking the Endpoint in p-Value Validity
Suppose that, under a true null and conditional on every possible selection-stage dataset \(D\), a validation design A produces a p-value uniformly distributed on \([0,1]\). Then \(P(p\leq u\mid D)=u\) for every \(u\in[0,1]\). Design B instead always reports \(p=1\). For B,
Both designs satisfy conditional validity, because the probability is no greater than \(u\) throughout the full interval \([0,1]\). Design B is conservative but has no ability to reject at a threshold below 1. The endpoint matters: at \(u=1\), its probability is 1, not 0. This example distinguishes validity from exact uniformity; a valid p-value need not have a uniform null distribution.
Planning Sample Size and Checking the Design
An independent validation stage consumes participants who cannot also contribute selection-stage outcomes to the confirmatory test. Its sample size should therefore be planned for the adjusted threshold, not just for an unadjusted single comparison. The following calculation uses a normal approximation with a known standard deviation to make that consequence explicit.
Worked Example: Approximate Power for Validation
Assume a hypothetical validation design will test two selected treatments against control, with familywise \(\alpha=0.05\) and Bonferroni allocation \(0.025\) per one-sided test. Suppose the validation sample has 40 participants per group, the outcome standard deviation is 12 units, and the true treatment-control difference of interest is 8 units. Under independent normal observations and known standard deviation, the standard error is
The mean of the test statistic under this difference is \(8/2.683=2.981\). The one-sided critical value for level 0.025 is approximately 1.960. The approximate power is the probability that a standard normal variable shifted by 2.981 exceeds 1.960:
Thus the design has approximately 84.6% power for an 8-unit difference under the stated assumptions and allocation. This calculation does not guarantee that the selected treatments have that effect, and actual power may differ if the standard deviation, allocation, missingness, or outcome distribution differs from the assumptions.
Before launch, verify that participant identifiers do not overlap between stages, access controls prevent validation outcomes from influencing selection, and the selected family and testing plan are recorded before validation unblinding. During conduct, monitor recruitment, treatment allocation, outcome timing, and missingness. A broken firewall between stages or an unplanned change in the testing family can invalidate the intended conditional argument.
Independence is also not the same as comparability. If the validation stage occurs later, uses different sites, or recruits a systematically different population, its observations may be independent of the pilot but may estimate a different treatment effect. Specify the target population and estimand, and assess whether the design maintains them across stages. Prespecifying sample size helps operationally; if its value is adapted using selection data, the test must still be calibrated conditional on that choice.
Alternatives, Limitations, and Common Pitfalls
An independent validation sample is particularly attractive when a clean confirmatory interpretation is a priority and enough participants are available. It is not the only approach. A trial can instead test all candidate treatments under a multiplicity procedure, whether or not they appeared promising, or use a method specifically calibrated for inference after selection. Those approaches can make more use of existing observations, but their validity depends on their own assumptions and calculations. Independent validation provides a relatively transparent separation; it does not eliminate multiplicity, guarantee replication, or repair a misspecified outcome model.
| Approach | What it uses | Main design consideration |
|---|---|---|
| Independent validation | New observations for selected hypotheses | Protect the separation and adjust for the tested family |
| Test all candidates | Confirmatory observations for every candidate | Account for multiplicity across the full candidate family |
| Selection-adjusted inference | Data involved in selection, analyzed with a calibrated procedure | Use a method whose assumptions match the selection rule and data structure |
A frequent mistake is to say, “The selected treatment was tested on a fresh sample, so no multiplicity adjustment is needed.” Fresh data address the connection between selection and the validation test, but testing several selected hypotheses still creates a family of opportunities for false rejection. Another mistake is to call two samples independent merely because they have different labels. Shared participants, outcomes, or information used to choose the test can reintroduce dependence. The correction is to define the data boundary explicitly, check it operationally, and apply a multiplicity procedure to the validation family.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Four candidate treatments are available, and exactly two will be selected for validation. At familywise \(\alpha=0.05\), what Bonferroni threshold applies to each selected test?
- One selected treatment has validation p-value 0.018 and the other has p-value 0.041 under the design in Question 1. Which hypotheses are rejected by the stated procedure?
- Why does independence between the selection and validation samples not require the two validation treatment-control statistics to be independent for the Bonferroni argument?
- A p-value is always 1 under a true null. Does it satisfy conditional validity at \(u=1\), and what is \(P(p\leq u\mid D)\) for \(0\leq u<1\)?
- The validation sample is recruited at different sites from the pilot, with independent participants. Is that enough to establish that the validation estimate targets the same population effect? Explain.
Solutions. (1) The threshold is \(0.05/2=0.025\) per test. (2) Reject the first null because \(0.018<0.025\); do not reject the second because \(0.041>0.025\). (3) The union bound controls the probability of any false rejection using each test’s conditional validity, without assuming independence among test statistics. (4) Yes. For \(0\leq u<1\), the probability is 0; at \(u=1\), it is 1, which is no greater than \(u\). (5) No. Independent participants establish separation of observations, not equivalence of populations. Differences in sites or recruited participants may change the target population or effect, so comparability of the estimand and study conditions must also be considered.