Honest Inference by Sample Splitting

Why Separate Selection From Inference?

In a high-dimensional regression, a researcher may use the data to choose a manageable set of predictors and then estimate coefficients using that set. The difficulty is that if the same observations both determine which variables are retained and estimate their coefficients, the usual standard errors and intervals generally do not account for the selection step. A coefficient can look unusually large partly because the selection procedure favored variables that fit the observed outcomes well.

Sample splitting creates a simple separation. One portion of the observations is used to select predictors. A second, independent portion is used to estimate the coefficients and quantify uncertainty. Since the estimation observations did not determine which predictors were selected, conventional inference can be justified under explicit model and sampling assumptions. The separation addresses dependence created by reusing data; it does not guarantee that the selected regression is correctly specified.

Definition: Sample splitting for inference partitions an independent sample into a selection sample and an estimation sample. A data-dependent procedure uses only the selection sample to choose a set of regressors. The chosen regressors are then treated as fixed when an estimator and its uncertainty are calculated using only the estimation sample.

Model and Conditions for Valid Inference

Consider independent observations \((Y_i,X_i)\), where \(Y_i\) is an outcome and \(X_i\) is a vector of candidate regressors. Write the linear conditional-mean model as

Y_i = X_i^\top \beta + \varepsilon_i,\qquad E(\varepsilon_i\mid X_i)=0.

Here \(\beta\) is the population coefficient vector. In a high-dimensional setting, the number of candidate regressors can be large, while the number of nonzero coefficients is assumed to be relatively small. The selection procedure may be Lasso or another method, but this tutorial's inference step does not depend on a particular selection algorithm.

Randomly divide the observations into a selection sample \(D_1\), of size \(n_1\), and an estimation sample \(D_2\), of size \(n_2\), so \(n=n_1+n_2\). Let \(\widehat S\) be the set chosen using \(D_1\). The estimation sample contains the outcomes and regressors needed to fit OLS using only the columns indexed by \(\widehat S\). For a target coordinate \(j\in\widehat S\), a valid conventional interval requires more than the fact that the two samples are independent.

Independent observations and splitting: observations are independent and identically distributed, and the split is made without using outcomes to assign observations. Consequently, the estimation sample is independent of the selection sample.
Correct selected model: the selected regression must contain the regressors needed to represent the conditional mean. A sufficient condition in the sparse linear model is that the selected set contains the true nonzero support. Extra regressors with zero coefficients are permissible.
Full rank and adequate size: the estimation-sample design matrix for the selected regressors has full column rank, and \(n_2\) exceeds the number of fitted coefficients.
Classical finite-sample inference conditions: for the exact \(t\) interval described below, conditional on the estimation-sample regressors, the errors are independent and normally distributed with mean zero and a common variance \(\sigma^2\). We also require that this error model holds in the selected regression.

The support condition is important and usually cannot be certified merely by looking at the selection sample. If a relevant regressor is omitted, OLS in the estimation sample may estimate a different, generally biased quantity. Independent data can protect the inference calculation from selection-induced dependence, but cannot make an incorrect regression model correct.

OLS and a Conditional \(t\) Interval

Let \(k\) be the number of columns in the selected estimation-sample regression, counting an intercept if one is included. Let \(X_{2,\widehat S}\) be its \(n_2\)-by-\(k\) design matrix and \(Y_2\) its outcome vector. Provided the matrix has full column rank, the OLS estimator is

\widehat\beta_{\widehat S} = \left(X_{2,\widehat S}^{\top}X_{2,\widehat S}\right)^{-1} X_{2,\widehat S}^{\top}Y_2.

Write \(\widehat\beta_j\) for the estimate of the target coefficient, and let \(e_j\) denote the corresponding coordinate vector in the selected regression. Estimate the error variance by the residual sum of squares divided by its residual degrees of freedom:

\widehat\sigma^2 = \frac{\operatorname{RSS}}{n_2-k}, \qquad \operatorname{RSS} = \left(Y_2-X_{2,\widehat S}\widehat\beta_{\widehat S}\right)^\top \left(Y_2-X_{2,\widehat S}\widehat\beta_{\widehat S}\right).

The estimated standard error for the target coefficient is

\widehat{SE}(\widehat\beta_j) = \widehat\sigma \sqrt{ e_j^\top \left(X_{2,\widehat S}^{\top}X_{2,\widehat S}\right)^{-1} e_j }.

Theorem: Suppose the selected regression contains the true conditional-mean model, the estimation-sample design has full column rank, and, conditional on that design, the errors are independent \(N(0,\sigma^2)\). Conditional on the selection sample and the estimation-sample design, the usual OLS statistic for a true selected coefficient has a \(t\) distribution with \(n_2-k\) degrees of freedom. Thus a two-sided \(100(1-\alpha)\%\) interval is

\widehat\beta_j \ \pm\ t_{1-\alpha/2,\,n_2-k}\, \widehat{SE}(\widehat\beta_j).

Why the Conditional Reference Distribution Works

The selection step makes \(\widehat S\) random. But once we condition on the selection sample, the chosen set is fixed. Independence of the two samples then means that the estimation-sample errors have not been selected for favorable values. To see the resulting \(t\) distribution, condition also on the estimation-sample design and write \(X=X_{2,\widehat S}\). Under the stated model, OLS satisfies

\widehat\beta_{\widehat S}-\beta_{\widehat S} = (X^\top X)^{-1}X^\top\varepsilon, \qquad \widehat\beta_j-\beta_j \sim N\!\left(0,\sigma^2 e_j^\top(X^\top X)^{-1}e_j\right).

Normality gives the first distribution. For the variance estimate, let \(M_X=I-X(X^\top X)^{-1}X^\top\), the projection onto the residual space. Then \(\operatorname{RSS}=\varepsilon^\top M_X\varepsilon\). The matrix \(M_X\) is a symmetric idempotent projection with rank \(n_2-k\), so under the normal error assumption,

\frac{\operatorname{RSS}}{\sigma^2} \sim \chi^2_{n_2-k}.

The fitted-coefficient error depends on the projection of \(\varepsilon\) onto the columns of \(X\), whereas the residual depends on its projection onto the orthogonal residual space. Under normality these projections are independent. Therefore the standardized coefficient error divided by the square root of the independent residual variance estimate has a \(t_{n_2-k}\) distribution. This justifies the interval, conditional on the selection sample and the estimation design, whenever the selected model is correct. In particular, the interval's conditional coverage is \(1-\alpha\) under these assumptions; this is not a probability statement that a fixed parameter is random and lies in the realized interval.

Worked Example: A Coefficient Interval After Selection

Suppose 400 hypothetical independent observations are randomly divided into \(n_1=240\) selection observations and \(n_2=160\) estimation observations. The selection procedure retains three regressors, and the estimation regression also includes an intercept, so \(k=4\). Assume, for this example, that the selected variables include all nonzero regressors in the conditional-mean model, the estimation design has full rank, and the errors satisfy the conditional normal, homoskedastic model.

The estimation-sample OLS coefficient for a predictor measured in years is \(2.40\) outcome units per year. Its estimated standard error is \(0.50\) outcome units per year. The residual degrees of freedom are

$$ df=n_2-k=160-4=156. $$

For a 95% interval, \(t_{0.975,156}\) is approximately \(1.975\). Therefore

2.40\pm 1.975(0.50) = 2.40\pm 0.9875 = [1.4125,\ 3.3875].

Rounded to two decimal places, the interval is \([1.41,3.39]\) outcome units per year. Under the stated assumptions, the procedure has 95% conditional coverage for the population coefficient, given the selection sample and estimation design. In repeated applications of the full procedure under these assumptions, intervals constructed this way cover the fixed coefficient 95% of the time. The interval does not assert a 95% probability that the fixed coefficient lies between these particular endpoints.

The Precision Cost of Splitting

The direct cost of splitting is that only \(n_2\) observations estimate the coefficient. Under a stable design and with a modest number of fitted regressors, coefficient variance is approximately proportional to \(1/n_2\), and standard error is approximately proportional to \(1/\sqrt{n_2}\). If a fraction \(\rho\) of the full sample is assigned to selection, then \(n_2=(1-\rho)n\). Relative to using all \(n\) observations with the same correctly specified regression, the approximate standard-error inflation is

\frac{SE_{\mathrm{split}}}{SE_{\mathrm{all}}} \approx \sqrt{\frac{n}{n_2}} = \frac{1}{\sqrt{1-\rho}}.

This is an approximation: finite-sample degrees of freedom, changes in the selected model size, and variation in the design can also affect precision. But it makes the basic trade-off transparent. A larger selection sample may help the selection procedure, while leaving fewer observations for estimation and widening the interval.

Worked Example: Quantifying the Sample-Size Cost

Suppose a hypothetical study has \(n=1{,}000\) independent observations and reserves 30% for selection. The estimation sample then has \(n_2=700\) observations. Assume the design and error variance are otherwise comparable, and suppose the standard error using all 1,000 observations would be \(0.200\) coefficient units.

The approximate split-sample standard error is

SE_{\mathrm{split}} \approx 0.200\sqrt{\frac{1{,}000}{700}} = 0.200(1.195) = 0.239.

Thus the standard error is about 19.5% larger. The corresponding variance inflation is approximately

\frac{\operatorname{Var}_{\mathrm{split}}}{\operatorname{Var}_{\mathrm{all}}} \approx \frac{1{,}000}{700} = 1.429,

or about 42.9% larger. These figures quantify the cost of using only 700 observations for estimation; they do not measure any benefit the larger selection sample might provide for choosing regressors.

What Splitting Does—and Does Not—Guarantee

The guarantee is about the relationship between selection and estimation. Conditional on what happened in the selection sample, the independent estimation sample has not been used to choose a favorable model or coefficient. This makes the usual inference calculation appropriate when the selected model and error assumptions are also correct.

In practice, sample splitting has two notable limitations. First, a selection method can fail to retain an important predictor. If that predictor belongs in the conditional mean, the selected regression can suffer omitted-variable bias even though its estimation sample is independent. Second, the method discards some observations from coefficient estimation. The precision example shows that this loss can be material. Choosing the split fraction is therefore a design decision: there is no universal fraction that is best for all selection and estimation problems.

A single random split can also make results depend noticeably on which observations happen to be assigned to each portion. Repeating splits and combining estimates requires a procedure with its own justified uncertainty calculation; simply averaging intervals or standard errors does not automatically preserve nominal coverage. The basic single-split argument remains useful because its separation and assumptions are explicit.

Worked Example: Independence Does Not Fix an Omitted Regressor

Consider a hypothetical population model with two standardized regressors:

Y=\beta_1X_1+\beta_2X_2+\varepsilon, \qquad \beta_1=1,\quad \beta_2=2,

where \(\operatorname{Var}(X_1)=\operatorname{Var}(X_2)=1\), \(\operatorname{Corr}(X_1,X_2)=0.60\), and \(E(\varepsilon\mid X_1,X_2)=0\). Suppose the selection sample retains \(X_1\) but omits \(X_2\). In the independent estimation sample, regressing \(Y\) on \(X_1\) alone gives the population slope

\frac{\operatorname{Cov}(X_1,Y)}{\operatorname{Var}(X_1)} = \frac{\beta_1\operatorname{Var}(X_1) +\beta_2\operatorname{Cov}(X_1,X_2) +\operatorname{Cov}(X_1,\varepsilon)} {\operatorname{Var}(X_1)} = 1+2(0.60) = 2.20.

The final equality uses \(\operatorname{Cov}(X_1,\varepsilon)=0\) and \(\operatorname{Cov}(X_1,X_2)=0.60\). The target structural coefficient is \(\beta_1=1\), but the selected one-regressor regression targets a slope of \(2.20\). Having independent samples has not removed the omitted-variable bias. A conventional interval for the selected regression's slope cannot be interpreted as a valid interval for \(\beta_1\) on the basis of sample splitting alone; the selected-model correctness condition fails.

Common Mistake

Wrong reasoning: “The second sample is independent, so any standard error computed there is valid for the coefficient I wanted.”
Correction: Independence prevents the selection sample from inducing dependence in estimation-sample errors, but validity still requires a correctly specified selected regression, suitable design rank, and the stated error assumptions. If relevant variables are omitted, or if inference reuses observations that influenced selection, the usual interval argument does not apply.

The practical lesson is to report the split, selection rule, selected model, estimation sample size, and assumptions that support the uncertainty calculation. An interval is only as informative as the model and sampling conditions that justify it.

Check Your Understanding

Use the model and assumptions stated above when answering.

Why does conditioning on the selection sample make the chosen set of regressors fixed for estimation-sample inference?
An estimation sample has \(n_2=120\) observations and the selected regression has \(k=5\) coefficients, including the intercept. What are the residual degrees of freedom?
If 20% of observations are reserved for selection, what is the approximate standard-error inflation relative to using the full sample, assuming the same design and model size?
Why is the true-support inclusion condition relevant even though the estimation sample is independent of the selection sample?
State one reason the exact finite-sample \(t\) result may fail if the errors are not conditionally normal with a common variance.