Why a Predictive Pipeline Must Respect the Data Split
A predictive model is not just its final regression, classification rule, or forecast equation. It is the complete sequence that turns raw predictors into a prediction: missing-value handling, scaling, dimension reduction or feature selection, and model fitting. If any of those steps uses information from validation or test observations, the evaluation no longer represents predictions from a process fitted only on the training data.
The governing rule is simple: fit every data-dependent step using the training portion only. Then apply the fitted step to validation and test observations without recalculating its parameters. Validation data can help compare candidate pipelines; test data is held back for a final assessment. Neither should determine the preprocessing parameters used to produce its own predictions.
“Fitted parameters” include more than regression coefficients. Examples are a mean or median used for imputation, a mean and standard deviation used for scaling, the directions retained by principal components analysis (PCA), and the predictors selected by a supervised selection rule. A transformation can be unsupervised and still leak information: PCA does not use the response, but its components depend on the predictor values used to fit it.
The Fit-Then-Apply Principle
Write a fitted preprocessing transformation as \(T_{\hat{\theta}}\), where \(\hat{\theta}\) denotes parameters estimated from training data. For example, \(\hat{\theta}\) might contain an imputation mean and a scaling mean and standard deviation. The training data determine \(\hat{\theta}\); the validation and test data are transformed using that same value.
For supervised feature selection or model fitting, training outcomes may also enter the fitting operation: \(\hat{\theta}=\operatorname{Fit}(X_{\text{train}},y_{\text{train}})\). The held-out outcomes remain unavailable until evaluation. If a transformation has multiple fitted stages, such as imputation followed by scaling, fit each stage in its pipeline order on the training observations. Apply each stage in that same order to held-out observations.
Assign observations to training, validation, and test sets before calculating preprocessing quantities. When the data structure requires it, use a split that respects that structure, such as a chronological split for a time series.
Estimate imputation values, scaling parameters, PCA directions, selected predictors, and model coefficients using the training data permitted for each step.
Use the fitted training parameters to transform validation and test predictors. Do not refit a step just because a held-out set has a different mean or missingness pattern.
Use validation results for choices among candidate pipelines. Keep the test set out of those choices. If the chosen pipeline is later refitted on training plus validation data, refit its preprocessing and model steps on that combined set, then evaluate on the untouched test set.
The restriction follows directly from what a held-out evaluation is intended to measure: performance of a fitting procedure on observations that did not determine that procedure. For instance, if an imputation mean is computed from training and validation observations together, then validation values have influenced the transformation used to evaluate validation predictions. Even without using validation outcomes, this changes the procedure being assessed.
This identity shows why the combined mean is not a training-only estimate: unless the validation mean happens to equal the training mean, validation predictors change the fitted imputation value. The same logic applies to standard deviations, PCA directions, and any other data-dependent transformation.
Worked Example: Impute and Scale Without Using Validation Data
A hypothetical insurer is building a model using one numeric predictor. The three training values are \(1\), \(5\), and missing; the validation value is \(9\). Use the training mean for imputation, then standardize using the sample mean and sample standard deviation of the completed training values. Find the transformed training values and the transformed validation value.
Method. Fit the imputer on the observed training values, complete the training data, and fit the scaler on that completed training data. Apply both fitted steps in order to validation.
1. Fit the imputer. The observed training values sum to \(1+5=6\), and there are two observed values. The training imputation mean is \(6/2=3\). The completed training values are therefore \(1,5,3\).
2. Fit the scaler. The completed training mean is \((1+5+3)/3=3\). The sum of squared deviations is \((1-3)^2+(5-3)^2+(3-3)^2=4+4+0=8\). The sample variance is \(8/(3-1)=4\), so the sample standard deviation is \(s=2\).
3. Transform training values. Using \(z=(x-3)/2\), the transformed values are \((1-3)/2=-1\), \((5-3)/2=1\), and \((3-3)/2=0\).
4. Transform validation. Apply the training imputation rule if needed, then the training scaling parameters. The observed validation value is \(9\), so its standardized value is \((9-3)/2=3\). The validation value does not change either fitted parameter.
Check. The transformed training values have mean \((-1+1+0)/3=0\) and sample standard deviation \(\sqrt{(1+1+0)/2}=1\), as required. A common mistake is to include the validation value when fitting the imputer; that would give \((1+5+9)/3=5\), which is not the training imputation mean.
PCA and Feature Selection Are Also Fitted Steps
PCA estimates directions from the predictor values. To use PCA safely, calculate the training means (and, if specified, training scales), form the training covariance or correlation matrix, and estimate the component directions from that matrix. Center or scale validation and test predictors using the training quantities, then project them onto the training component directions. Recomputing components on validation data would change the coordinate system rather than apply the fitted transformation.
Feature selection requires the same discipline. An unsupervised rule, such as removing a predictor based on its training variance, is fitted from training predictors. A supervised rule, such as selecting predictors by association with the outcome, may use training outcomes as well. In either case, do not use validation or test outcomes to select predictors. Those outcomes are reserved for evaluation, not for deciding which variables the pipeline keeps.
Worked Example: Project Validation Data Using Training-Fitted PCA
A hypothetical risk team uses two numeric predictors and chooses PCA based on the training covariance matrix, without scaling. The training observations are \((1,1)\), \((3,3)\), and \((5,5)\). A validation observation is \((5,1)\). Find its coordinates on the first two training principal components.
Method. Center using the training means, calculate the training covariance matrix, find its eigenvectors, and project the centered validation observation onto those fixed directions.
1. Training means and centered observations. Both predictor means are \((1+3+5)/3=3\). The centered training observations are \((-2,-2)\), \((0,0)\), and \((2,2)\).
2. Training covariance matrix. Each variance is \((4+0+4)/(3-1)=4\), and the covariance is also \((4+0+4)/2=4\). Thus the covariance matrix is \(\begin{pmatrix}4&4\\4&4\end{pmatrix}\). Its eigenvalues are \(8\) and \(0\), with corresponding unit eigenvectors \((1,1)/\sqrt{2}\) and \((1,-1)/\sqrt{2}\).
3. Center and project validation. Center \((5,1)\) using the training means: \((5-3,1-3)=(2,-2)\). Its first-component score is \((2,-2)\cdot(1,1)/\sqrt{2}=0\). Its second-component score is \((2,-2)\cdot(1,-1)/\sqrt{2}=4/\sqrt{2}=2\sqrt{2}\), approximately \(2.8284\).
Check. The training observations all lie along the first component, so the second component has zero training variance. The validation observation lies entirely in the second direction after centering; its zero first-component score and nonzero second-component score are consistent with that geometry. The common mistake is to recalculate PCA on validation data and treat its resulting directions as if they were the training directions.
Worked Example: Select a Predictor Using Training Outcomes Only
A hypothetical portfolio model has four training observations. The candidate predictors and training outcomes are \(x_1=(0,1,2,3)\), \(x_2=(0,1,3,2)\), and \(y=(1,3,5,7)\). The prescribed selection rule keeps the predictor with the larger absolute sample correlation with \(y\), then fits a simple linear regression using that predictor. A validation observation has \(x_1=4\) and \(x_2=0\). Find the selected predictor and its validation prediction.
Method. Compute both correlations using training data only, select by the stated rule, estimate the regression on training data, and apply that fitted model to validation.
1. Training means and sums of squares. The means are \(\bar{x}_1=1.5\), \(\bar{x}_2=1.5\), and \(\bar{y}=4\). For \(x_1\), the sum of squared deviations is \(2.25+0.25+0.25+2.25=5\). For \(y\), it is \(9+1+1+9=20\).
2. Compare correlations. The cross-product sum for \(x_1\) and \(y\) is \(4.5+0.5+0.5+4.5=10\), so \(r_{x_1,y}=10/\sqrt{5\cdot20}=1\). The \(x_2\) deviations are \((-1.5,-0.5,1.5,0.5)\); their cross-product sum with the \(y\) deviations is \(4.5+0.5+1.5+1.5=8\). The sum of squared \(x_2\) deviations is \(5\), so \(r_{x_2,y}=8/\sqrt{5\cdot20}=0.8\). The rule selects \(x_1\).
3. Fit and predict. The training regression slope is \(10/5=2\), and the intercept is \(\bar{y}-2\bar{x}_1=4-3=1\). The fitted model is \(\hat{y}=1+2x_1\). For the validation observation, \(\hat{y}=1+2(4)=9\). The value \(x_2=0\) is not used because \(x_2\) was not selected.
Check. The fitted line gives predictions \(1,3,5,7\) for the four training \(x_1\) values, matching the training outcomes. The validation outcome, if later observed, must not be used to reconsider the predictor selection before reporting this validation prediction.
Common Leakage Traps and a Safe Refit
A pipeline can leak even when the final model is fitted only on the training set. The following errors are especially easy to miss:
- Imputing before the split. A mean or median calculated from all observations uses held-out predictor information.
- Scaling each data set separately. Validation values then determine their own centering and scale, and their coordinates no longer correspond to the transformation fitted on training data.
- Fitting PCA before the split. Held-out predictor patterns influence the component directions, even though PCA does not use outcomes.
- Selecting predictors using all outcomes. This lets validation or test outcomes affect the model being evaluated.
- Refitting on test data during final evaluation. Once test observations influence any fitting or selection decision, that set is no longer an untouched test set.
There is one important distinction after validation-based decisions are complete. Suppose the pipeline choices are fixed using training and validation results. It can then be reasonable to refit the chosen pipeline on the combined training and validation observations before evaluating on the test set. In that case, recompute the imputation values, scaling parameters, PCA directions, selected predictors, and model coefficients using the combined data as appropriate. Do not carry forward parameters fitted only on the original training subset if the final model is now being refitted on a larger set. The test observations remain excluded throughout.
Check Your Understanding
For each situation, identify the leakage-safe calculation or pipeline decision.
- Training predictor values are \(2\), \(6\), and missing. Impute with the training mean and standardize the completed training values using the sample standard deviation. A validation value is \(10\). What is its standardized value? (Answer: \(3\))
- Three training observations for two predictors are \((0,0)\), \((2,2)\), and \((4,4)\). PCA uses the training covariance matrix without scaling. What are the first- and second-component scores for validation observation \((4,0)\), centered using training means? (Answer: \(0\) and \(2\sqrt{2}\), respectively)
- A team calculates an imputation mean from training and validation observations, then reports validation performance. What is the problem, and what should be done instead? (Answer: Validation predictors influenced the imputer; fit the imputation value on training data only and apply it unchanged to validation)
- A chosen pipeline is to be refitted after model choices are fixed. The training set has 60 observations with predictor mean \(10\), and the validation set has 20 observations with predictor mean \(14\). What mean should be used to refit the imputer on training plus validation, and may the test set enter that calculation? (Answer: \(11\); no, the test set must remain excluded)
- A selection rule ranks predictors by their sample correlation with the outcome. Should the validation outcomes be included in calculating these correlations if validation performance is to assess the selected pipeline? State the correct data and its role. (Answer: No; calculate the selection rule using training predictors and training outcomes only, then use validation outcomes only to evaluate the resulting pipeline)