Why Audit the Workflow?
A validation score is useful only if the data used to produce it match the question the score is intended to answer. A workflow can run without errors and still give a misleading estimate: an observation may appear in both training and validation data, a transformation may use information from held-out rows, or the final test set may have influenced model choices. Auditing means tracing the data and decisions through the workflow, rather than accepting a score because the code executed.
This tutorial focuses on inspecting R workflows for four specific risks: overlapping partitions, inconsistent preprocessing, unsuitable fold structure, and accidental test-set reuse. The audit principles are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Start with the Intended Data Roles
Before reading individual R commands, state the intended role of each data subset. Training data are used to estimate model parameters and fit preprocessing steps. Validation data can support model selection or tuning, provided that the selection process is designed accordingly. A final test set is held aside for an evaluation after those choices are complete. Cross-validation repeats training and validation on folds; the roles of the rows change from fold to fold, but each fold’s validation rows must be excluded from that fold’s fitting process.
The audit should use stable observation identifiers, not just row numbers that may change after sorting or filtering. In R, expressions such as `intersect(train_id, valid_id)` and `setdiff(all_id, union(train_id, valid_id))` help identify overlaps and omissions. The identifiers must refer to the same original records in every partition. If duplicates represent the same underlying person, policy, or event, checking identifiers alone may not be enough: the audit must also check whether the intended unit of separation is the person, policy, or event.
The reason the two checks are separate can be stated precisely. Let \(I_i^{(T)}\), \(I_i^{(V)}\), and \(I_i^{(E)}\) equal 1 when observation \(i\) is assigned to training, validation, or test, respectively, and 0 otherwise. If every observation is assigned exactly once, then:
Summing over observations gives \(n_T+n_V+n_E=N\), where each \(n\) is the number of assigned observations. But equal counts do not establish the condition for each \(i\): one duplicated observation and one omitted observation can cancel in the total. Therefore, check pairwise intersections and the union of identifiers, not only partition sizes.
Worked Example: Finding Overlap and Omitted Rows
A hypothetical insurer has 20 distinct claim records, identified by integers 1 through 20. An R workflow assigns records 1–12 to training, 11–14 to validation, and 17–20 to testing. Audit the assignments and propose a correction.
Step 1: Count the assigned entries. Training has 12 entries, validation has 4, and testing has 4, for \(12+4+4=20\) entries. This count matches the number of records, but it does not yet show that the assignment is valid.
Step 2: Check intersections. Training and validation overlap at records 11 and 12, so the intersection has size 2. Training and test have no shared records, and validation and test have no shared records.
Step 3: Check coverage. The union contains records 1–14 and 17–20: 14 records from 1–14 plus 4 records from 17–20, for 18 distinct records. Records 15 and 16 are omitted. Thus the 20 assigned entries consist of 18 distinct records, with two duplicated entries; the total count concealed both defects.
Correction. One possible valid assignment is training records 1–12, validation records 13–16, and test records 17–20. The intersections are all empty, the union is records 1–20, and the counts are \(12+4+4=20\). The method is an identifier-based overlap and coverage audit. The mistake it avoids is treating a matching sum of partition sizes as proof that the partitions are correct.
Trace Every Preprocessing Step
For each data-dependent operation, ask two questions: which rows fitted it, and which rows received the resulting transformation? Imputation values, scaling means and standard deviations, category groupings, feature selection, and dimension-reduction components are all learned from data. Within a resampling split, they must be fitted using that split’s training rows and then applied to its validation rows. In a final train/test workflow, they are fitted on the training data and applied unchanged to the test data.
This does not mean that every operation is forbidden before splitting. A purely mechanical operation that uses no information from the observations, such as renaming a column according to a fixed convention, does not estimate a quantity from held-out data. The audit concern is an operation whose result can depend on the distribution, values, or outcomes of the rows meant to be held out. Check the order of commands and not only the name of a function: a transformed data frame may have been created using the complete data before the train/test split.
Identify the original record IDs and the exact code that assigns their training, validation, test, or fold roles.
For each imputation, scaling, encoding, selection, or dimension-reduction step, find the rows used to estimate its parameters.
Confirm that the fitted operation is applied consistently to held-out rows without refitting on those rows.
Determine whether any test result was used to revise features, model settings, thresholds, or the choice of model.
Worked Example: Detecting Preprocessing Leakage
A hypothetical pricing model uses one numeric predictor. Its training values are 2, 4, 6, and 8; its validation values are 10 and 100. The analyst standardizes all six values before splitting, then fits the model on the training rows. Determine whether the validation transformation is appropriate and calculate the relevant values.
Step 1: Compute the training-only mean and standard deviation. The training mean is \((2+4+6+8)/4=5\). Using the sample standard deviation, the squared deviations from 5 are 9, 1, 1, and 9. Their sum is 20, so the sample variance is \(20/(4-1)=20/3\), and the sample standard deviation is \(\sqrt{20/3}\), approximately 2.582.
Step 2: Apply those training values to validation. The training-only standardized values for 10 and 100 are \((10-5)/2.582\), approximately 1.936, and \((100-5)/2.582\), approximately 36.793. These are the validation values under the appropriate training-fitted transformation.
Step 3: Compare with the all-data transformation. The mean of all six values is \(130/6\), approximately 21.667. Their squared values sum to 10,220, so the sum of squared deviations from their mean is \(10{,}220-130^2/6\), approximately 7,403.333. The all-data sample variance is \(7{,}403.333/5\), approximately 1,480.667, and the standard deviation is approximately 38.48. Using these all-data values gives validation standardized values of approximately \((10-21.667)/38.48=-0.303\) and \((100-21.667)/38.48=2.035\).
Conclusion and correction. The two validation values influenced the mean and standard deviation used to transform themselves, so the workflow crossed the validation boundary. Fit the mean and standard deviation on the training rows only, then apply those fixed values to validation. This method avoids the common mistake of assuming that preprocessing is harmless merely because the outcome variable was not used.
Check Whether the Folds Match the Data
An apparently clean row-level split can still be unsuitable when rows have a structure that must be respected. The audit question is not simply “Are the fold IDs different?” It is “Does the split prevent information from a related observation or a future period from entering training when that would not be available in the intended prediction setting?” For repeated records on people or policies, inspect group membership across each fold. For ordered observations, inspect the time ranges in training and validation and verify that the intended forecasting direction is respected. For rare outcomes, inspect whether the fold construction has produced usable outcome representation; stratification does not remove other dependence or time-order concerns.
Check the actual fold assignments, rather than inferring them from a function name or a comment. For each fold, list the validation rows and verify that the associated training rows exclude the required units. If a person or policy must be kept together, compare group identifiers in training and validation. If the analysis concerns future periods, compare the latest training time with the validation time and check that the design does not train on the future to predict the past.
Worked Example: Auditing Folds for Repeated Policyholders
A hypothetical study has 12 claims from six policyholders, with two claims per policyholder. The target is to predict a later claim for a policyholder not represented in model fitting. In one three-fold assignment, fold 1 validates on the first claim from policyholders A, B, and C; the second claim from each of those policyholders remains in the training data. Audit this structure and give a suitable alternative.
Step 1: Identify the prediction unit. The goal concerns a policyholder not represented in training. Therefore the two claims from a policyholder must be assigned together; separating claims from the same policyholder does not match the stated prediction setting.
Step 2: Count the violation. Fold 1 has three validation claims, one each from A, B, and C. The other claim from A, the other from B, and the other from C are in training. Thus all three validation policyholders also occur in training: the policyholder-level overlap is 3, even though the individual claim rows are distinct.
Step 3: Construct a group-respecting alternative. Assign A and B to fold 1, C and D to fold 2, and E and F to fold 3. Each validation fold then contains two policyholders and four claims; its training complement contains four policyholders and eight claims. No policyholder appears on both sides of a fold. The method is grouped cross-validation, and the mistake it avoids is checking only row overlap when the relevant separation unit is the policyholder.
Protect the Final Test Set
A test set is not a reserve source of feedback during model development. If its results influence the choice among models, features, cutoffs, or other settings, the test set has participated in selection. Reporting the selected model’s score on that same set as though it were an untouched final evaluation no longer supports that interpretation. This can happen gradually: a practitioner may inspect a test score, alter the model, inspect it again, and stop when the score looks satisfactory.
In the audit, search for every point where test results become visible to the person or code making modeling decisions. Distinguish computing a final score after choices are fixed from using scores to make those choices. Also check whether test rows were used in earlier summaries, feature decisions, or preprocessing. The safer workflow is to make model-selection decisions using training and validation data or an appropriate resampling procedure, lock the procedure, and then evaluate once on the held-out test data.
Worked Example: Finding Accidental Test-Set Reuse
A hypothetical credit union compares five candidate models on a validation set, selects the one with the lowest validation error, and then evaluates that model once on a test set of 40 applications. The test errors are 12, 9, 11, 8, and 10 misclassified applications for models 1 through 5. The analyst chooses model 4 because its test error is 8, changes its settings, and evaluates all five models on the same test set again. Audit what can and cannot be claimed.
Step 1: Identify the first selection. The initial test error rates are \(12/40=0.3000\), \(9/40=0.2250\), \(11/40=0.2750\), \(8/40=0.2000\), and \(10/40=0.2500\). Choosing model 4 from these test results makes the test set part of the model-selection process. The validation-based selection did not protect the test set from this later decision.
Step 2: Identify the second selection. The analyst changes model 4 after inspecting its test performance and evaluates the candidates again on the same 40 applications. The test data have now informed both a model choice and a settings change. The second test score is not an independent final evaluation; the analyst has had additional opportunities to adapt to the test results.
Step 3: Correct the workflow. If the initial test results have already guided decisions, do not describe a later score on the same test set as an untouched final estimate. Freeze the model-selection process using training and validation data or resampling. Then use a genuinely untouched test set for a final evaluation, if one is available. The method is a decision-history audit. The mistake it avoids is treating a test set as protected simply because it was originally labelled “test.”
Common Audit Traps
A useful audit records evidence for each check instead of relying on general assurances such as “the split was random” or “preprocessing was the same.” Random assignment alone does not establish disjointness, prevent repeated-person leakage, or guarantee that the final test set remained untouched. Similarly, running the same preprocessing function on training and validation data is not necessarily consistent if the function estimates new values separately on each subset.
For example, suppose an analyst calculates the median separately within training and validation and imputes missing values using each median. The code applies “median imputation” to both subsets, but it does not apply one training-fitted imputation rule to held-out observations. In a fold-based workflow, each fold needs its own imputation values fitted from that fold’s training portion. The same principle applies to scaling and other learned transformations.
- Wrong: “Training, validation, and test sizes add to \(N\), so there is no overlap.” Correction: Check all pairwise intersections and check that the union contains the intended IDs.
- Wrong: “The test set was never used to fit model coefficients, so it is still untouched.” Correction: Decisions influenced by test performance also constitute use.
- Wrong: “No claim row is duplicated across folds, so the folds are independent.” Correction: Check whether related rows share a person, policy, or time structure that the prediction task requires the split to respect.
- Wrong: “The same preprocessing function was called on each subset, so preprocessing is consistent.” Correction: Verify that fitted quantities come from training rows and are carried unchanged to held-out rows.
Check Your Understanding
For each problem, identify the audit issue and the appropriate correction.
- A data set has 30 records. Training contains 18 IDs, validation contains 6, and test contains 6. The sizes sum to 30, but IDs 5 and 6 are in both training and validation, while IDs 29 and 30 are absent. How many distinct IDs are assigned? (Answer: 28 distinct IDs; two duplicated IDs and two omitted IDs.)
- A training subset has values 3, 5, and 7. A validation value is 11. Using the training mean and sample standard deviation, what is the standardized validation value? (Answer: The training mean is 5, the sample standard deviation is 2, and the standardized value is 3.)
- In a four-fold group split, validation contains 5 policyholders with 3 claims each. How many validation claim rows are there, and how many policyholders should be absent from the corresponding training fold? (Answer: 15 validation claim rows; all 5 validation policyholders should be absent from training.)
- A team compares three models on a test set of 50 records and picks the model with the lowest test error, 6 errors. What is that model’s observed test error rate, and can it be presented as an untouched final evaluation after this selection? (Answer: \(6/50=0.1200\); no, the test result influenced model selection.)
- Each of 24 observations is assigned to exactly one of four validation folds, with six observations per fold. In each fold, the other 18 observations are used for training. How many validation appearances should each observation have over the four folds? (Answer: Exactly one validation appearance per observation.)