Sample Splitting For Edgewise Confidence Intervals

Why Separate Discovery From Estimation?

A graph-learning procedure searches the data for edges that appear plausible. If the same observations are then used to estimate those edges and report ordinary confidence intervals, the reported estimates can be unusually large simply because they helped trigger selection. Sample splitting addresses this source of selection bias by assigning distinct roles to independent portions of the data: one portion discovers edges, and the other estimates precision-matrix entries.

Splitting does not, by itself, make an estimator accurate or normally distributed. The estimation sample still needs an inferential procedure with a valid sampling distribution. We first give a fully worked fixed-dimension Gaussian procedure, then state how the same separation principle applies when a valid high-dimensional estimator is available.

Definition: Let $X_1,\ldots,X_n$ be independent observations from a $p$-variate Gaussian distribution $N_p(\mu,\Sigma)$, where $\Sigma$ is positive definite and $\Theta=\Sigma^{-1}$ is the precision matrix. A sample-splitting edge procedure partitions the observations into a discovery sample and an independent inference sample. The discovery sample selects a set of edges $\widehat E$; the inference sample estimates $\Theta_{ij}$ for each $(i,j)\in\widehat E$ and constructs confidence intervals.

In a Gaussian graphical model, an undirected edge between vertices $i$ and $j$ corresponds to $\Theta_{ij}\ne 0$. Thus edgewise inference concerns off-diagonal entries of $\Theta$. An interval for $\Theta_{ij}$ that includes zero is compatible with no edge, but an interval excluding zero does not automatically establish simultaneous validity across all selected edges.

A Basic Split-Sample Procedure

Choose a split before examining the observations. Write $n_1$ for the discovery-sample size and $m=n-n_1$ for the inference-sample size. Apply any graph-selection rule to the discovery data, producing $\widehat E$. The rule might be a regularized likelihood fit or neighborhood selection. Then, without reusing the discovery observations for estimation, calculate inference-sample estimates and intervals for the selected entries.

For fixed $p$, a simple inference estimator is the inverse of the inference-sample covariance matrix. Let $\bar X$ be the mean of the $m$ inference observations and define

S=\frac{1}{m}\sum_{r=1}^{m}(X_r-\bar X)(X_r-\bar X)^{\mathsf T}, \qquad \widehat\Theta=S^{-1}.

The observations in this display are the inference observations only. If $m>p$, $S$ is invertible with probability one under a nonsingular Gaussian distribution. This estimator is a useful fixed-dimensional benchmark, not a general solution when $p$ is comparable to or larger than $m$.

Theorem: Suppose $p$ is fixed, $X_1,\ldots,X_m$ are independent $N_p(\mu,\Sigma)$ observations with positive-definite $\Sigma$, and $S$ is their centered sample covariance as defined above. For each fixed pair $i\ne j$, $$ \sqrt m\,(\widehat\Theta_{ij}-\Theta_{ij}) \xrightarrow{d} N\!\left(0,\Theta_{ii}\Theta_{jj}+\Theta_{ij}^{\,2}\right). $$ Consequently, if $\widehat v_{ij}=\widehat\Theta_{ii}\widehat\Theta_{jj}+\widehat\Theta_{ij}^{\,2}$, an asymptotic $100(1-\alpha)\%$ interval is $$ \widehat\Theta_{ij}\ \pm\ z_{1-\alpha/2}\sqrt{\widehat v_{ij}/m}, $$ where $z_{1-\alpha/2}$ is the $1-\alpha/2$ standard-normal quantile.

Proof. For fixed $p$, the multivariate central limit theorem for the entries of the sample covariance gives $\sqrt m\,\operatorname{vec}(S-\Sigma)\xrightarrow{d}N(0,V)$, where the covariance of the limiting entries indexed by $(a,b)$ and $(c,d)$ is $\Sigma_{ac}\Sigma_{bd}+\Sigma_{ad}\Sigma_{bc}$. Centering by the estimated mean rather than the true mean does not change this limit: the difference between the two covariance matrices is $(\bar X-\mu)(\bar X-\mu)^{\mathsf T}=O_p(m^{-1})$, which becomes $o_p(m^{-1/2})$ after multiplication by $\sqrt m$.

The map $g(A)=A^{-1}$ is differentiable at positive-definite $\Sigma$. Differentiating $A A^{-1}=I$ gives $dg=-\Theta(dA)\Theta$. Applying this derivative to the covariance limit yields the limiting covariance between entries $(i,j)$ and $(k,l)$ of $\sqrt m(\widehat\Theta-\Theta)$: $\Theta_{ik}\Theta_{jl}+\Theta_{il}\Theta_{jk}$. In particular, set $k=i$ and $l=j$. The variance for entry $(i,j)$ is then $\Theta_{ii}\Theta_{jj}+\Theta_{ij}^{2}$. The continuous mapping and Slutsky theorems justify replacing the unknown variance by its consistent plug-in estimate, giving the stated interval. $\square$

What Sample Splitting Guarantees

Let $D_1$ denote the discovery data and $D_2$ the inference data. Since the observations are independent and the split is fixed in advance, $D_1$ and $D_2$ are independent. The selected set $\widehat E$ is a function of $D_1$; an interval computed only from $D_2$ therefore does not inherit the direct selection bias caused by using the same noise to select and estimate an edge.

More precisely, suppose an inference procedure for a fixed edge $(i,j)$ produces an interval $C_{ij}(D_2)$ with asymptotic coverage $1-\alpha$. Conditional on any discovery outcome that selects $(i,j)$, the distribution of $D_2$ is unchanged. Hence the interval retains that marginal coverage, provided the procedure's coverage claim applies to the data-generating distribution. With fixed $p$, there are finitely many possible edges, so pointwise asymptotic coverage for each edge also gives coverage uniformly over this finite collection.

Important qualification: Sample splitting separates selection from estimation; it does not establish the inference estimator's validity. In high dimensions, an ordinary inverse sample covariance may not exist, and a regularized estimate may not have a normal sampling distribution. A high-dimensional procedure needs its own conditions and a justified standard error—for example, a valid debiased precision estimator applied to the inference sample. Independence from discovery cannot repair bias, failed sparsity conditions, or an invalid variance estimate.

There is a second possible procedure: use discovery data to choose a graph, then fit a Gaussian graphical model constrained to that graph on the inference sample. This approach needs care. If the selected graph omits a true edge, the constrained model is misspecified, and ordinary model-based intervals need not cover the true precision entries. A sufficient route to the usual fixed-dimensional justification is consistent graph selection, together with regularity conditions for the selected-model fit; then the probability of fitting the correct model tends to one. The unrestricted inverse-covariance procedure above avoids that particular omitted-edge model constraint when $p$ is fixed, though it does not solve the high-dimensional problem.

Worked Procedures and Calculations

Worked Example: Defining the Split and the Inferential Target

Suppose $n=800$ independent Gaussian observations are available on $p=5$ variables. A graph-selection rule is to be used to identify candidate edges, and the target is $\Theta_{24}$.

Choose $n_1=300$ observations for discovery and $m=500$ for inference. Run the graph-selection rule on only the first 300 observations and record whether $(2,4)$ belongs to $\widehat E$. If it is selected, estimate $\Theta_{24}$ using only the other 500 observations.

The split makes the selection decision independent of the inference observations. It does not guarantee that $(2,4)$ is a true edge or that the estimator is accurate; those claims require graph-selection and estimation assumptions, respectively. If $(2,4)$ is not selected, this procedure reports no interval for it.

Worked Example: A Two-Variable Edgewise Interval

Suppose the inference sample has $m=400$ observations on two variables, and its centered covariance matrix is $S=\begin{pmatrix}2&0.6\\0.6&1.5\end{pmatrix}$. Assume the discovery sample selected edge $(1,2)$.

The determinant is $2(1.5)-(0.6)^2=3-0.36=2.64$, so \[ \widehat\Theta=S^{-1} =\frac{1}{2.64}\begin{pmatrix}1.5&-0.6\\-0.6&2\end{pmatrix} =\begin{pmatrix}0.56818&-0.22727\\-0.22727&0.75758\end{pmatrix}. \]

The estimated variance numerator for the off-diagonal entry is \[ \widehat v_{12} =(0.56818)(0.75758)+(-0.22727)^2 \approx 0.43044+0.05165=0.48209. \] Thus the estimated standard error is $\sqrt{0.48209/400}\approx 0.03472$. Using $z_{0.975}\approx1.96$, the interval is \[ -0.2272727\pm1.96(0.0347165) \approx[-0.29532,-0.15923]. \] This is an asymptotic 95% interval for $\Theta_{12}$, conditional on the discovery result, under the fixed-dimensional Gaussian assumptions. Its exclusion of zero is an edgewise statement, not a simultaneous guarantee for a collection of reported intervals.

Worked Example: Checking a Three-Variable Calculation

Suppose a discovery sample selects edge $(1,3)$. On the inference sample of $m=500$ observations, let \[ S=\begin{pmatrix}2&0.4&0.2\\0.4&1.5&0.3\\0.2&0.3&1.2\end{pmatrix}. \] The leading principal minors are $2$, $2(1.5)-0.4^2=2.84$, and \[ \det(S)=2(1.8-0.09)-0.4(0.48-0.06)+0.2(0.12-0.30) =3.42-0.168-0.036=3.216. \] They are all positive, so $S$ is positive definite.

The $(1,3)$ entry of $S^{-1}$ is its $(3,1)$ cofactor divided by the determinant: \[ \widehat\Theta_{13} =\frac{(0.4)(0.3)-(0.2)(1.5)}{3.216} =\frac{0.12-0.30}{3.216} \approx-0.05597. \] The diagonal entries needed for its standard error are \[ \widehat\Theta_{11}=\frac{1.8-0.09}{3.216}\approx0.53172, \qquad \widehat\Theta_{33}=\frac{3-0.16}{3.216}\approx0.88308. \] Therefore \[ \widehat v_{13}\approx(0.53172)(0.88308)+(-0.05597)^2 \approx0.46955+0.00313=0.47268, \] and the estimated standard error is $\sqrt{0.47268/500}\approx0.03075$. The approximate 95% interval is \[ -0.0559701\pm1.96(0.0307468) \approx[-0.11623,\,0.00429]. \] The interval includes zero. Selection of the edge in the independent discovery sample does not require the inference interval to exclude zero.

Assumptions, Scope, and a Common Mistake

The inverse-covariance interval has a clear but limited scope: independent Gaussian observations, fixed $p$, positive-definite population covariance, and enough inference observations for the sample covariance to be invertible. For a growing number of variables, a valid interval may instead depend on sparsity, eigenvalue bounds, rates for nuisance estimates, and an asymptotic linear expansion. Those are properties of the chosen inference method, not consequences of splitting.

The fraction assigned to inference also matters. More inference observations usually reduce standard errors; fewer discovery observations may make edge selection less reliable. There is no universally optimal split fraction: it depends on the difficulty of discovery and the precision required for estimation. Repeating many random splits and combining the resulting intervals also changes the procedure and requires a separate justification.

Common mistake: “The edge was selected independently, so the interval is valid.” Independence removes the direct reuse-of-noise problem, but does not prove a normal approximation or a correct standard error. The correction is to establish the inference estimator's sampling result under explicit assumptions.
Another mistake: treating marginal intervals as simultaneous ones. A 95% interval for each selected entry does not imply 95% probability that every reported interval covers. Simultaneous coverage requires additional adjustment or a separate joint procedure.
Another mistake: fitting a selected graph as though it were known in advance. If discovery misses true edges, constrained estimation can be biased. Either justify consistent graph recovery for that fitting strategy or use an inference method whose validity does not rely on the selected graph being correct.

Key takeaway: Use one independent sample to decide which edges to examine and another to estimate their precision entries. Sample splitting protects against direct selection-induced bias; valid coverage still depends on the inference estimator, its standard error, and the assumptions under which its sampling distribution is justified.

Check Your Understanding

Use the assumptions and calculations in this tutorial to answer the following.

Why can choosing an edge and estimating its precision entry from the same observations make an ordinary interval misleading?
For the fixed-dimensional Gaussian inverse-covariance interval, what is the estimated variance numerator for $\widehat\Theta_{ij}$?
Does sample splitting alone justify using the inverse sample covariance when $p$ exceeds the inference-sample size? Explain.
If a selected-graph fit omits a true edge, why might its usual model-based interval fail to cover the target precision entry?
What additional claim would be needed to turn edgewise 95% intervals into a 95% simultaneous confidence statement?