Cross-Fitting High-Dimensional Nuisance Estimates

Why Evaluate Nuisance Estimates on Held-Out Observations?

In high-dimensional problems, a nuisance function may be estimated using many candidate predictors, regularization, or a flexible learning algorithm. Such methods can adapt closely to the observed data. If the same observations are then used both to fit the nuisance function and to evaluate a target-parameter score, the score can inherit patterns created by that fitting. The resulting error is not necessarily small just because the fitted nuisance predicts its training data well.

Cross-fitting separates those two tasks. Each observation contributes to the target score using a nuisance estimate trained without that observation. The construction is useful with a Neyman-orthogonal score: orthogonality reduces the leading sensitivity of the target moment to nuisance-estimation error, while held-out evaluation limits the empirical dependence created by fitting and scoring on the same observations.

The score is a function \(\psi(W;\theta,\eta)\), where \(W\) denotes an observation, \(\theta\) is the target parameter, and \(\eta\) collects nuisance functions or parameters. Let \(\eta_0\) be the true nuisance value. The target is identified by the population moment condition \(E[\psi(W;\theta_0,\eta_0)]=0\). The earlier material on Neyman-orthogonal scores concerns a key property: small perturbations to \(\eta_0\) have zero first-order effect on this population moment at \(\theta_0\).

Fold Construction and the Cross-Fitted Score

Suppose \(W_1,\ldots,W_n\) are independent observations. Divide their indices into \(K\) nonoverlapping sets \(I_1,\ldots,I_K\), whose union is \(\{1,\ldots,n\}\). For fold \(k\), fit the nuisance estimates using the complement of that fold, denoted \(I_k^c\). Write the resulting estimate as \(\hat\eta_{-k}\). For an observation \(i\in I_k\), evaluate the score using \(\hat\eta_{-k}\), not a nuisance estimate fitted using \(W_i\).

This is the cross-fitted empirical score and its estimating equation. If the score is affine in the scalar target, the equation may have an explicit solution. Otherwise, \(\hat\theta\) is obtained by solving the equation. Fold sizes are usually chosen to be roughly equal; \(K\) is commonly fixed as the sample grows. Each nuisance fit uses about \(n(K-1)/K\) observations, while every observation is used once for score evaluation.

A Numerical Cross-Fitted Score Average

The following small example makes the score calculation explicit. It is illustrative rather than an inference procedure for a real application. Consider observations \(W_i=(U_i,V_i,X_i)\), a scalar target \(\theta\), and the score \(\psi(W;\theta,\eta)=U-\theta+\eta(X)V\). Suppose the population condition \(E[V\mid X]=0\) holds. Then, for any fixed perturbation \(h(X)\), the derivative of the expected score with respect to a change from \(\eta\) to \(\eta+t h\) is \(E[h(X)V]=0\). Thus this score is locally insensitive, in expectation, to nuisance perturbations.

For six hypothetical observations, suppose the observed \(U_i\) values are \(10,12,9,11,13,10\), and the \(V_i\) values are \(1,-1,2,-2,1,-1\). Use two folds: observations 1–3 and observations 4–6. Imagine that nuisance fits trained outside those folds give held-out values \(\hat\eta_{-1}(X_i)=0.5\) for \(i=1,2,3\), and \(\hat\eta_{-2}(X_i)=-0.25\) for \(i=4,5,6\).

The score equation is the average of \(U_i+\hat\eta_{-k(i)}(X_i)V_i-\theta\). Its solution is the average of \(U_i+\hat\eta_{-k(i)}(X_i)V_i\). The first fold’s nuisance correction sums to \(0.5(1-1+2)=1.0\); the second fold’s correction sums to \(-0.25(-2+1-1)=0.5\). The sum of the \(U_i\) values is \(65\), so:

This is the root of the specified cross-fitted score equation for these hypothetical values. It is not, by itself, evidence that the nuisance fits are accurate or that a particular sampling model holds. Those questions require separate assumptions and diagnostics.

Why Cross-Fitting Can Limit Overfitting Bias

The role of cross-fitting becomes clearer by considering one evaluation fold at a time. Under independent sampling, conditional on the training observations outside fold \(k\), the fitted function \(\hat\eta_{-k}\) is fixed and the observations inside \(I_k\) are independent of that fit. Consequently, evaluation observations are not being scored using nuisance estimates that were adapted to those same observations. This does not make all score contributions across different folds independent: their training samples overlap. It does remove the direct same-observation fitting link within each fold.

Orthogonality explains why nuisance estimation error can then be manageable. Write \(\Delta_k=\hat\eta_{-k}-\eta_0\). A first-order expansion of the population score around \(\eta_0\), conditional on the training data, has the form

The first term is zero by the identifying moment condition. Neyman orthogonality makes the derivative term zero for admissible perturbations. Under suitable smoothness, the remaining remainder \(R_k\) is of second order in nuisance errors. Without orthogonality, a first-order nuisance error can remain in the target moment. Without held-out evaluation, even a population expansion does not prevent the empirical score from reflecting overfitting patterns in the same observations.

Cross-fitting therefore supports the use of flexible nuisance estimates in a score-based procedure; it does not certify the procedure by itself. A typical high-level requirement is that products of nuisance errors be sufficiently small relative to \(n^{-1/2}\), along with moment and identification conditions. The exact rates and function norms depend on the score. Valid inference also requires suitable sampling assumptions and control of the score’s variability. Cross-fitting is a way to control one source of bias and dependence, not a substitute for these conditions.

Common Pitfalls and Practical Scope

The main practical benefit of cross-fitting is easy to overstate. It prevents each observation from directly helping fit the nuisance estimate used to score that observation. It does not guarantee that a nuisance estimate generalizes well, that an orthogonal score has been chosen correctly, or that the target moment identifies the parameter.

• Using a full-sample nuisance fit for every score. This reintroduces the same-observation fitting link. For cross-fitting, the estimate applied to fold \(k\) must be trained without fold \(k\).
• Treating cross-fitting as a cure for slow rates. If nuisance errors are too large for the score’s remainder condition, held-out evaluation alone does not make the estimator accurate enough for root-\(n\) inference.
• Assuming every score is orthogonal. Cross-fitting is a sample construction, not a property of the score. First-order sensitivity to nuisance error must be addressed by the score and its assumptions.
• Ignoring the sampling structure. The argument above uses independent observations. With clusters, time dependence, or other structured sampling, folds and the corresponding theory must respect that structure; randomly splitting individual rows may create leakage or invalidate independence arguments.
• Confusing fold-specific scores with independent scores. Training samples overlap across folds, so the collection of cross-fitted contributions is generally dependent. Inference must account for the estimator’s sampling variability rather than treating fold estimates as independent replications.