Sample Splitting for Data-Adaptive Mean Tests

Why Separate Selection and Testing?

In a high-dimensional mean problem, the data may suggest which coordinates carry a signal, which threshold is promising, or which regularization level to use. But if the same observations both choose a procedure and assess its evidence, the resulting statistic has been selected to look favorable. Its null distribution is generally not the null distribution calculated as though the procedure had been fixed in advance.

Sample splitting addresses this issue by assigning distinct observations to selection and testing. The selection sample can determine a coordinate set or tuning choice. Conditional on that sample, the choice is fixed; an independent test sample can then be used for a test whose null calibration is valid for that fixed choice. Independence protects the calibration only when the test statistic’s null distribution is also correctly specified.

Definition: Let \(\boldsymbol{X}_1,\ldots,\boldsymbol{X}_n\) be independent observations with mean vector \(\boldsymbol{\mu}\in\mathbb{R}^p\). A sample split partitions the observations into a selection sample \(A\) and a test sample \(B\), of sizes \(n_A\) and \(n_B\). A data-adaptive choice, such as a coordinate set \(S(A)\) or tuning parameter \(\lambda(A)\), is made using only \(A\). A test statistic and its p-value are then computed using \(B\), with the choice from \(A\) held fixed.

For clarity, many exact calculations below use the model \(\boldsymbol{X}_i\sim N(\boldsymbol{\mu},I_p)\), independently, with the identity covariance known. Under the global null \(H_0:\boldsymbol{\mu}=\boldsymbol{0}\), each test-split coordinate statistic \(Z_j=\sqrt{n_B}\,\bar X_{B,j}\) has distribution \(N(0,1)\). Other models can be used, but they require an appropriate test-split calibration; independence by itself does not imply normality.

Conditional Validity After Selection

The basic guarantee is stated in terms of a p-value that is valid conditional on the selection data. A p-value \(P_j\) for testing \(H_{0,j}:\mu_j=0\) is conditionally super-uniform if, for every \(u\in[0,1]\), \(P(P_j\leq u\mid A)\leq u\) almost surely when \(H_{0,j}\) holds. Exact continuous p-values satisfy equality. This condition concerns the null distribution on the test split, not merely the fact that a selected set is fixed once \(A\) is observed.

Theorem: Suppose a test-split p-value \(P_j\) is conditionally super-uniform given \(A\) whenever \(H_{0,j}\) is true. Then it is also super-uniform without conditioning: \(P(P_j\leq u)\leq u\) for every \(u\in[0,1]\).

Proof. By the law of total probability, expressed as conditional expectation,

P(P_j\leq u) =E\!\left[P(P_j\leq u\mid A)\right] \leq E[u] =u.

The inequality holds because conditional validity gives \(P(P_j\leq u\mid A)\leq u\) almost surely under the null. Thus selection on \(A\) does not invalidate a test-split p-value whose validity holds conditional on \(A\). \(\square\)

For example, under independent \(N(\boldsymbol{\mu},I_p)\) observations, the two-sided coordinate p-value \(2\{1-\Phi(|Z_j|)\}\) is exactly uniform under \(H_{0,j}\), where \(\Phi\) is the standard normal distribution function. Conditional on \(A\), a selected coordinate \(j\) is fixed, and the independent test observations still give \(Z_j\sim N(0,1)\) under that coordinate null. With a different distribution or an unknown scale, a standard-normal p-value may not be valid; one must use a justified alternative, such as a valid studentized test under its assumptions or a test based on a specified null distribution.

Worked Example: Testing the Coordinate Selected by the Training Split

Suppose independent observations follow \(N(\boldsymbol{\mu},I_2)\), and the null is \(\boldsymbol{\mu}=\boldsymbol{0}\). Use \(n_A=20\) observations for selection and \(n_B=25\) for testing. Suppose the two training-sample coordinate means are \(0.10\) and \(-0.35\). Select the coordinate with the larger absolute training mean.

Since \(|-0.35|=0.35>|0.10|=0.10\), the selected coordinate is coordinate 2. Suppose the test-sample mean in that coordinate is \(\bar X_{B,2}=-0.40\). Its test statistic is

Z_2=\sqrt{25}(-0.40)=5(-0.40)=-2.

The two-sided p-value is \(2\{1-\Phi(2)\}\), approximately \(0.0455\). At level \(0.05\), reject the null for the selected coordinate. Under the global null, conditional on the training data, coordinate 2 is fixed and \(Z_2\sim N(0,1)\); hence this one-coordinate test has conditional type I error \(0.05\). The conclusion relies on the Gaussian model and known identity covariance, not on the split alone.

Testing Several Selected Coordinates

If the selection split returns several coordinates, separate valid p-values do not automatically justify testing each at level \(\alpha\) and declaring any rejection significant. A simple conditional family-wise error procedure is Bonferroni. Let \(S(A)\) be the selected set and \(m(A)=|S(A)|\). For each selected coordinate, reject when its valid test-split p-value is at most \(\alpha/m(A)\). Conditional on \(A\), this controls the probability of at least one false rejection among the selected true nulls.

Theorem: Suppose each selected true-null coordinate \(j\in S(A)\) has a conditionally super-uniform test-split p-value. If \(m(A)\geq1\), the rule that rejects \(H_{0,j}\) only when \(P_j\leq\alpha/m(A)\) has conditional family-wise error rate at most \(\alpha\), and therefore unconditional family-wise error rate at most \(\alpha\). If the selected set is empty, make no rejections.

Proof. Conditional on \(A\), let \(S_0(A)\) be the selected coordinates whose null hypotheses are true. The event of at least one false rejection is contained in the union of the events \(P_j\leq\alpha/m(A)\) over \(j\in S_0(A)\). The union bound and conditional super-uniformity give

P(\text{at least one false rejection}\mid A) \leq \sum_{j\in S_0(A)}P\!\left(P_j\leq\frac{\alpha}{m(A)}\mid A\right) \leq \frac{|S_0(A)|}{m(A)}\alpha \leq \alpha.

Taking expectations over \(A\) preserves the upper bound \(\alpha\), by the law of total probability. No independence among the test-split p-values is needed for this Bonferroni argument. \(\square\)

Worked Example: Bonferroni Calibration for a Selected Set

Suppose a training rule selects three coordinates, and their test-split p-values are \(0.008\), \(0.031\), and \(0.20\). Assume each is conditionally valid under its coordinate null; for instance, this holds under independent Gaussian observations with known identity covariance when the p-values are the two-sided normal p-values. For family-wise level \(\alpha=0.05\), the Bonferroni cutoff is

\frac{\alpha}{m}=\frac{0.05}{3}\approx0.0167.

The first p-value is below \(0.0167\), while \(0.031\) and \(0.20\) are above it. Thus only the first selected coordinate is rejected. Conditional on any training data that selected three coordinates, the probability of one or more false rejections is at most \(0.05\); the same bound holds unconditionally.

Choosing Thresholds on One Split

A threshold can also be selected using \(A\), then treated as fixed while a test statistic is calculated from \(B\). The null distribution used for calibration must reflect the selected coordinates and the statistic actually computed. In particular, a thresholded statistic need not have a chi-square distribution if its coordinates are dependent or if the covariance structure is not accounted for.

Worked Example: A Training-Selected Threshold and Quadratic Test

Assume independent \(N(\boldsymbol{\mu},I_4)\) observations and test the global null \(\boldsymbol{\mu}=\boldsymbol{0}\). Suppose the training-split statistics \(\sqrt{n_A}\bar X_{A,j}\) are \(2.4,1.7,0.4,-0.2\). Choose threshold \(1.5\) and select coordinates whose training statistics have absolute value greater than \(1.5\). The selected set is \(S=\{1,2\}\).

Suppose the test-split statistics for these coordinates are \(Z_1=2.0\) and \(Z_2=1.0\). With the selected set held fixed, use \(Q=Z_1^2+Z_2^2\). Under the stated Gaussian identity-covariance global null, conditional on the training data, \(Z_1,Z_2\) are independent standard normal variables. Therefore \(Q\sim\chi^2_2\), and the upper-tail p-value is

Q=2.0^2+1.0^2=5,\qquad P(\chi^2_2\geq5)=e^{-5/2}\approx0.0821.

This test does not reject at level \(0.05\). The calculation is valid because selection used only \(A\), and because the conditional null distribution of the test statistic is explicitly known. A threshold selected from the test-split statistics would require a different calibration that accounts for that selection.

Choosing Regularization on One Split

The same principle applies when \(A\) selects a regularization level. Once \(\lambda(A)\) is chosen, the test statistic on \(B\) must be calibrated using the null distribution corresponding to that fixed \(\lambda\). If the distribution depends on estimated nuisance quantities, those estimates and their effects on the null law must also be handled correctly. It is not enough to substitute a selected value into a reference distribution that was derived for another statistic.

Worked Example: Conditional Calibration of a Weighted Regularized Statistic

Assume independent \(N(\boldsymbol{\mu},I_2)\) observations and the global null \(\boldsymbol{\mu}=\boldsymbol{0}\). Let the training split choose \(\lambda\) from \(\{0,1\}\). For the test split define

T_\lambda=Z_1^2+\frac{1}{(1+\lambda)^2}Z_2^2, \qquad Z_j=\sqrt{n_B}\bar X_{B,j}.

Suppose the training rule chooses \(\lambda=1\), and the test-split statistics are \(Z_1=0.5\) and \(Z_2=2\). Then

T_1=(0.5)^2+\frac{1}{4}(2)^2=0.25+1=1.25.

Conditional on the training data, \(Z_1,Z_2\) are independent \(N(0,1)\). Thus the null reference distribution is \(U+V/4\), where \(U,V\) are independent \(\chi^2_1\) variables. The correctly calibrated p-value is

P(U+V/4\geq1.25) = \int_0^5 \left[2\left\{1-\Phi\!\left(\sqrt{1.25-v/4}\right)\right\}\right]f_{\chi^2_1}(v)\,dv +P(V\geq5).

Here \(f_{\chi^2_1}\) is the \(\chi^2_1\) density. For \(0\leq v\leq5\), the required lower-tail threshold for \(U\) is \(1.25-v/4\geq0\), giving the integrand shown; for \(v\geq5\), the event holds for every \(U\geq0\), giving the final term. This expression specifies the exact upper-tail calibration for the selected regularization level. It would be incorrect to compare \(T_1\) directly with a \(\chi^2_2\) reference distribution, since the two squared coordinates have different weights.

Conditions, Costs, and a Common Pitfall

A valid split-based procedure has three essential features: the selection rule uses only \(A\); the test-split observations are independent of \(A\) under the sampling design; and the p-value or critical value is valid for the test statistic conditional on the selection data. For a fixed split of independent observations, the second condition follows from independence of the observations. If the split itself is randomized independently of the observed values, conditioning on that split also preserves the argument.

Splitting has a cost: only \(n_B\) observations contribute to the test statistic, so testing can be less powerful than a procedure that uses all observations efficiently. The benefit is a transparent calibration despite data-adaptive choices. The method does not guarantee power, and it does not repair a misspecified null distribution.

Wrong reasoning: “The selected coordinate is fixed conditional on the training data, so its standard-normal p-value is valid.” Being fixed does not establish that its test statistic is standard normal.
Correction: First verify the test-split null law. Under independent Gaussian observations with known identity covariance, the standardized coordinate mean is standard normal; under other assumptions, use a calibration justified for those assumptions.
Wrong reasoning: “The observations are independent and have covariance \(I_p\), so the coordinate statistics are normal.” Independence and covariance specify neither a Gaussian distribution nor an exact normal law for a finite-sample mean.
Correction: State a distributional model or a valid approximation theorem with its conditions. Otherwise, do not claim exact normal p-values.
Wrong reasoning: “Bonferroni is unnecessary because the selected coordinates were fixed before testing.” A fixed family can still contain multiple tests; adjust for its size when controlling family-wise error.

Key takeaway: Sample splitting separates the act of choosing a mean-test procedure from the act of assessing it. The resulting protection is conditional: after selection, the test split must still have a correctly calibrated null distribution for the chosen coordinates, threshold, or regularization level.

Check Your Understanding

Use the assumptions stated in each question, and distinguish independence from distributional calibration.

Under independent \(N(\boldsymbol{\mu},I_p)\) observations with known covariance, why does choosing a coordinate using the training split preserve the conditional standard-normal null law on the test split?
Suppose selected-coordinate p-values are conditionally super-uniform. Show how the Bonferroni cutoff changes when the selected set has size \(m\).
Why does covariance \(I_p\) alone not justify an exact standard-normal p-value for a finite-sample test-split mean?
In the threshold example, what assumptions make the selected-coordinate quadratic statistic have a \(\chi^2_2\) null distribution conditional on selection?
For a weighted regularized statistic, why must its reference distribution use the selected weights rather than an unweighted \(\chi^2\) distribution?