Sensitivity to Modeling and Preprocessing Choices

Why Examine Sensitivity to Modeling Choices?

A reported model result can depend on decisions made before and during fitting. Examples include how missing values are handled, whether an extreme value is capped, how a categorical variable is grouped, or which model form is selected. Some choices are well supported by the data and intended use; others are merely convenient. A responsible analysis asks whether its main conclusions persist across defensible alternatives.

Sensitivity analysis here means comparing results from a primary specification with results from plausible alternatives. It is not a search for whichever specification produces the most favorable result. Nor does agreement across alternatives prove that the model is correct. It shows whether a stated conclusion is robust to the particular choices examined.

Definition: A specification is a complete set of modeling and preprocessing decisions relevant to a result, such as the predictor representation, missing-value treatment, transformation, model form, and decision threshold. A sensitivity analysis compares the result under a primary specification with results under one or more defensible alternatives, holding the evaluation target and comparison method fixed wherever possible.

The alternatives should have substantive reasons. For example, a median imputation may be reasonable for a skewed predictor, while an imputation plus missingness indicator may be reasonable when the fact that a value is missing can itself carry information. The analysis should explain why alternatives are plausible, rather than presenting arbitrary variations as equally credible.

A Fair Comparison

A useful comparison changes the choice being investigated while keeping other parts of the assessment consistent. In particular, evaluate each fitted alternative on the same evaluation observations, with the same outcome definition, metric, and units. Fit preprocessing steps using training data only; applying information from the evaluation outcomes or predictors to set preprocessing values can leak information into the assessment.

For a quantitative outcome $y_i$ and prediction $\hat{y}_i$ on evaluation observation $i$, define the error as $e_i=y_i-\hat{y}_i$. The mean absolute error and root mean squared error are

\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|e_i|, \qquad \operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}e_i^2}.

These formulas assume that all observations use the same outcome scale and that lower error is preferred. MAE gives errors a linear contribution; RMSE gives relatively more weight to large errors because it squares them. Neither is automatically the right metric for every purpose. The metric should match the modeling objective and intended use.

For classification, a fixed probability threshold turns probabilities into predicted classes. Given a defined positive class, a true positive (TP) is a positive case predicted positive; a false positive (FP) is a negative case predicted positive; a false negative (FN) is a positive case predicted negative; and a true negative (TN) is a negative case predicted negative. Accuracy is $(TP+TN)/n$, sensitivity is $TP/(TP+FN)$, and specificity is $TN/(TN+FP)$. Sensitivity is undefined if there are no actual positive cases, and specificity is undefined if there are no actual negative cases. When the consequences of errors differ, a specified cost per error can be more relevant than accuracy.

Justification for Paired Comparisons

When two specifications are evaluated on the same $n$ observations, compare their losses observation by observation. Let $L_{Ai}$ and $L_{Bi}$ be the losses for specifications A and B on observation $i$, using the same loss definition. Their mean-loss difference is

\overline{L}_A-\overline{L}_B = \frac{1}{n}\sum_{i=1}^{n}L_{Ai} - \frac{1}{n}\sum_{i=1}^{n}L_{Bi} = \frac{1}{n}\sum_{i=1}^{n}(L_{Ai}-L_{Bi}).

This follows by subtracting the two sums term by term. Pairing matters because each difference compares the specifications on the same case; it avoids confusing a change in case mix with a change in model performance. It does not make the comparison causal, remove sampling variation, or establish that small differences are meaningful. RMSE is the square root of an average squared loss, so differences in RMSE are not themselves the average of per-case RMSE differences.

Worked Example: Comparing Two Preprocessing Pipelines for Cost Prediction

A hypothetical analyst predicts claim costs in dollars. Two defensible preprocessing pipelines handle an extreme predictor value differently, and each model is evaluated on the same four held-out claims. The observed costs are $100, 200, 300,$ and $400$. Pipeline A predicts $110, 180, 330,$ and $360$; Pipeline B predicts $105, 190, 315,$ and $390$. Compare MAE and RMSE. State which pipeline performs better on these measures, and identify a common calculation trap.

Method. Calculate errors as observed minus predicted, then calculate both metrics from those errors. The same four claims and dollar scale are used for both pipelines.

For A, the errors are $100-110=-10$, $200-180=20$, $300-330=-30$, and $400-360=40$. Thus the absolute errors are $10,20,30,40$, giving

\operatorname{MAE}_A=\frac{10+20+30+40}{4}=25.00.

The squared errors are $100,400,900,$ and $1600$, with sum $3000$. Therefore

\operatorname{RMSE}_A=\sqrt{\frac{3000}{4}} =\sqrt{750} \approx 27.39.

For B, the errors are $100-105=-5$, $200-190=10$, $300-315=-15$, and $400-390=10$. The absolute errors sum to $5+10+15+10=40$, so $\operatorname{MAE}_B=40/4=10.00$. The squared errors are $25,100,225,$ and $100$, with sum $450$. Hence $\operatorname{RMSE}_B=\sqrt{450/4}=\sqrt{112.5}\approx 10.61$.

On this evaluation set, B has lower MAE and RMSE. Relative to A, its MAE is $ (25-10)/25=0.60$, a 60% reduction. Its RMSE reduction is approximately $(27.39-10.61)/27.39=0.6126$, or 61.3%. Amounts and metrics are rounded to two decimals; percentages are rounded to one decimal place.

Common mistake and correction. Dividing the RMSE difference by B's RMSE gives $(27.39-10.61)/10.61\approx 158.2\%$. That is not the reduction relative to A; it uses the alternative as the denominator. The reduction relative to the primary specification A uses $27.39$ as the denominator, giving 61.3%. This comparison describes only the specified held-out cases, not a guaranteed future improvement.

Check Whether the Main Conclusion Persists

A sensitivity analysis should connect metric changes to the conclusion that matters. A small numerical change may not alter a decision, while a change in the selected group, ranking, or action may be material even if the aggregate metric changes little. State both the quantitative result and its practical implication.

Worked Example: Comparing Missing-Value Treatments

A hypothetical team compares two ways of handling missing predictor values: median imputation alone (A), and median imputation with a missingness indicator (B). Both pipelines are fitted using training data and assessed on the same five held-out cases, with outcomes $40,50,60,70,80$. Their predictions are A: $45,55,55,75,70$; B: $42,53,62,68,78$. Compute MAE and RMSE, then say what the comparison supports.

Method. Use the quantitative-outcome metrics defined above. For A, the errors are $-5,-5,5,-5,10$. The absolute errors sum to $5+5+5+5+10=30$, so $\operatorname{MAE}_A=30/5=6.00$. The squared errors sum to $25+25+25+25+100=200$, so $\operatorname{RMSE}_A=\sqrt{200/5}=\sqrt{40}\approx 6.32$.

For B, the errors are $-2,-3,-2,2,2$. The absolute errors sum to $2+3+2+2+2=11$, giving $\operatorname{MAE}_B=11/5=2.20$. The squared errors sum to $4+9+4+4+4=25$, giving $\operatorname{RMSE}_B=\sqrt{25/5}=\sqrt{5}\approx 2.24$. Errors and metrics are in outcome units and rounded to two decimals.

B has lower error on both metrics for these cases, so the comparison supports B as the better of these two specifications on this evaluation set. It does not establish that missingness is informative in general, nor that the missingness-indicator method will be better in another population. The team should report the missing-data assumption and avoid claiming a general advantage from this one comparison.

Sensitivity analysis is most useful when the alternatives are chosen before inspecting which one wins, or are otherwise selected for a clear reason. If many alternatives are tried and only the best result is reported, the apparent performance may reflect selection rather than a reliable modeling advantage. Keep a record of the primary specification, the alternatives considered, and the reason each is defensible.

Worked Example: When Classification Conclusions Depend on Error Costs

A hypothetical screening model is refitted under two reasonable treatments of missing predictor values, producing specifications A and B. On the same six cases, actual outcomes are $1,1,0,0,0,0$, where 1 is the positive class. At a prespecified threshold of $0.50$, A produces probabilities $0.80,0.60,0.55,0.45,0.30,0.20$, and B produces $0.70,0.45,0.40,0.30,0.20,0.10$. Compare their classification measures and calculate total error cost if an FN costs \$3 and an FP costs \$1.

Method. Predict positive when the probability is at least $0.50$. For A, the first three cases are predicted positive: the first two are TP and the third is FP. The remaining three are TN. Thus $TP=2$, $FP=1$, $FN=0$, and $TN=3$. Accuracy is $(2+3)/6=0.8333$, sensitivity is $2/2=1.0000$, and specificity is $3/4=0.7500$.

For B, only the first case is predicted positive. It is TP; the second case is FN; the four negative cases are TN. Therefore $TP=1$, $FP=0$, $FN=1$, and $TN=4$. Accuracy is $(1+4)/6=0.8333$, sensitivity is $1/2=0.5000$, and specificity is $4/4=1.0000$. Probabilities and threshold are unchanged in the comparison.

With FN cost \$3 and FP cost \$1, total cost for A is $0(3)+1(1)=\$1$, while total cost for B is $1(3)+0(1)=\$3$. Under these stated costs, A has lower total error cost. Yet both have the same accuracy, and B has higher specificity. Thus “better” depends on the objective: the specifications have equal accuracy, different error profiles, and different costs. Probabilities are reported to four decimals and costs to the cent.

Reporting Material Dependence

A concise report should name the primary specification, describe the alternatives and why they are plausible, state how all alternatives were compared, and summarize what changed. Include enough detail for a reader to see whether the conclusion is stable. If a ranking or decision changes under an alternative, say so directly; do not bury the change behind an average metric.

A useful reporting structure is:

State the choice.
Identify the modeling or preprocessing decision being examined and the primary approach.

Justify the alternative.
Explain why it is a reasonable representation of the data or intended operation.

Make a comparable evaluation.
Use the same evaluation cases, outcome, metric, threshold, and units when those are relevant.

Interpret the change.
Report whether the key performance statement, ranking, or action persists, and explain any material difference.

The main limitation is that the tested alternatives do not cover every plausible specification. A conclusion can be stable across the choices examined yet still be sensitive to an unexamined choice, data limitation, or change in use. Conversely, some metric movement may be too small to affect a decision. Avoid treating a numerical difference alone as evidence of practical importance.

Exam trap: comparing alternatives on different evaluation samples, changing the metric after seeing results, or describing the lowest observed error as proof of a universally superior model. A defensible conclusion is bounded: it states what happened under the named alternatives, on the specified evaluation data, according to the chosen measure. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, compare the specified alternatives using the stated data and report the requested result.

Observed values are $12,18,20$. Specification A predicts $10,20,20$. Calculate its MAE and RMSE, rounding to three decimals. (Answer: MAE $=1.333$; RMSE $=\sqrt{8/3}\approx1.633$.)
On the same three cases, actual outcomes are $5,10,15$. Imputation A predicts $6,9,18$; imputation B predicts $5,12,14$. Which has lower MAE, and what is B's RMSE, rounded to three decimals? (Answer: B has lower MAE, $1.000$ versus $1.667$; B's RMSE is $\sqrt{5/3}\approx1.291$.)
Actual classes are $1,1,0,0$, and predicted probabilities are $0.70,0.40,0.60,0.20$. At threshold $0.50$, find $TP,FP,FN,TN$ and accuracy. (Answer: $TP=1, FP=1, FN=1, TN=1$; accuracy $=0.5000$.)
Actual classes are $1,0,0,0$. Specification A predicts $1,1,0,0$, and B predicts $0,0,0,0$. If an FN costs \$4 and an FP costs \$2, which has lower total error cost? (Answer: A costs \$2.00 and B costs \$4.00; A has lower cost.)
Two preprocessing alternatives have similar aggregate error, but one selects a different group for a consequential review action. What should the analyst report? (Answer: Report the changed group and action explicitly, describe the alternatives and comparison, and treat the decision's dependence on specification as a material limitation.)

What You'll Learn