Balancing Information Loss and Interpretability

Why Reduction Is a Trade-Off

A predictive model may begin with many candidate predictors, but a useful final representation is not necessarily the one that preserves the most detail. Keeping more detail can improve predictions, while a smaller set of predictors or a simpler representation may be easier to explain, maintain and use consistently. The practical question is not simply whether information is lost; it is which information is lost, how that affects predictions or decisions, and whether the remaining representation can be explained well enough for its purpose.

The earlier material on feature selection versus feature extraction distinguishes keeping selected original predictors from representing the data through derived features. Here, the focus is on evaluating their trade-off. A representation can be compact without being interpretable: a small number of complicated derived variables may be harder to explain than several familiar original predictors. Conversely, a list of many original variables is not automatically understandable to a decision-maker.

Definition: For this tutorial, predictive information means information in the available predictors that helps reduce prediction loss for the outcome and population of interest. Information loss is the resulting deterioration in predictive performance when a representation is simplified, assessed using a stated loss measure on appropriate evaluation data. Interpretability is the extent to which a person can understand how the representation relates to predictions and, where relevant, to decisions. Neither information loss nor interpretability has one universal measure that applies to every task.

A correlation, variance, or count of retained variables cannot by itself establish how much useful predictive information remains. A predictor may vary considerably but have little relationship to the outcome; another may contribute predictive value only in a particular population or decision setting. The earlier material on screening is useful for identifying candidates, but a screening statistic is not a substitute for evaluating the resulting representation against the intended objective.

A Consistent Basis for Comparing Choices

Compare candidate representations on the same held-out observations, with the same outcome definition and loss measure. For a numeric outcome, a common measure is the root mean squared error. If $y_j$ is the observed outcome and $\hat{y}_j$ is its prediction for evaluation record $j$, then

\operatorname{RMSE} = \sqrt{\frac{1}{n}\sum_{j=1}^{n}(y_j-\hat{y}_j)^2}.

For a binary decision, a single score such as accuracy may hide differences between the costs of false positives and false negatives. In that setting, compare performance using quantities that reflect the actual decision, such as the number and consequences of each error. In either case, the metric must be appropriate to the decision; no metric is automatically the right one merely because it is easy to calculate.

The comparison also needs an explicit account of interpretability and operational consequences. Useful questions include whether a user can state what the retained predictors mean, whether the same representation can be calculated reliably at decision time, and whether the explanation is sufficient for review. These are contextual questions, not a universal numerical score. A model used to prioritize a review queue may tolerate a different trade-off from one used to support an individual decision that must be explained.

State the decision and outcome.
Identify what will be predicted, for whom, and how a prediction will be used.

Compare predictive performance.
Use the same evaluation records and a loss measure aligned with the decision.

Describe what was discarded.
Name the detail no longer represented and consider whether it may matter for particular records or groups.

Judge the practical gain.
Decide whether the clearer explanation, simpler implementation, or other benefit justifies the measured loss.

Worked Comparisons

Worked Example: A Simpler Representation With Higher Error

A hypothetical maintenance team is comparing two models that predict a component’s remaining service life, in hours. The detailed representation uses several recorded condition measures; the compact representation uses one familiar summary measure. On the same four held-out components, the detailed model predicts 100, 115, 85 and 100 hours. The compact model predicts 105, 110, 90 and 95 hours. The observed lives are 100, 120, 80 and 100 hours. Calculate and compare RMSE, then assess the trade-off if the team will accept an RMSE no greater than 8 hours.

Method and assumptions. Treat the four held-out records as evaluation cases, use squared error in hours squared as the loss, and regard the predictions as fixed. The acceptance limit is a stated operational requirement, not a statistical result.

For the detailed model, the observed-minus-predicted errors are $100-100=0$, $120-115=5$, $80-85=-5$, and $100-100=0$. The squared errors are $0$, $25$, $25$, and $0$, so the sum is $50$, the mean squared error is $50/4=12.5$, and RMSE is $\sqrt{12.5}\approx 3.54$ hours.

For the compact model, the errors are $100-105=-5$, $120-110=10$, $80-90=-10$, and $100-95=5$. The squared errors are $25$, $100$, $100$, and $25$, giving a sum of $250$, a mean squared error of $250/4=62.5$, and RMSE $\sqrt{62.5}\approx 7.91$ hours. The compact model’s RMSE is higher, but it remains below the stated 8-hour limit.

Conclusion. If the compact representation is substantially easier to explain and the 8-hour requirement is the relevant standard, its observed performance may be an acceptable trade-off. If a decision depends on distinguishing components whose predicted lives differ by a few hours, the detailed model may be preferable. The four evaluation records do not establish that either model will meet the requirement for every future component.

Common mistake avoided: Calling the compact model “equally accurate” because it meets a limit would confuse acceptability with equal performance. Its RMSE is about 7.91 hours, compared with about 3.54 hours for the detailed model.

Worked Example: The Decision Cost of Discarded Signal

A hypothetical claims unit uses a model to flag files for additional review. In a set of 1,000 held-out files, 100 require review under the unit’s established outcome definition and 900 do not. A compact, easily explained rule produces 70 true positives, 30 false negatives, 60 false positives and 840 true negatives. A less transparent representation produces 90 true positives, 10 false negatives, 100 false positives and 800 true negatives. Assume a false negative has a decision cost of $500 and a false positive has a cost of $40; true positives and true negatives have no error cost. Compare accuracy and total error cost.

Method and assumptions. Use the provided held-out counts without adjusting a decision threshold. For each representation, accuracy is the number of correct classifications divided by 1,000. Total error cost is the number of each error type multiplied by its stated cost, then summed.

For the compact rule, correct classifications are $70+840=910$, so accuracy is $910/1{,}000=0.9100$. Its total error cost is $30\times \$500+60\times \$40=\$15{,}000+\$2{,}400=\$17{,}400$.

For the less transparent representation, correct classifications are $90+800=890$, so accuracy is $890/1{,}000=0.8900$. Its total error cost is $10\times \$500+100\times \$40=\$5{,}000+\$4{,}000=\$9{,}000$.

Conclusion. The compact rule has higher accuracy, but its total error cost is $8,400 greater under the given assumptions. If those costs adequately describe the decision, accuracy alone favors the wrong choice for minimizing error cost. If explainability is essential, the unit must still decide whether the compact rule’s explanation is worth that cost difference; the calculation makes the trade-off visible but does not make the decision automatically.

Common mistake avoided: Treating all errors as equally serious would ignore the stated cost difference. For example, the compact rule makes 90 errors and the other representation makes 110, but the compact rule’s errors are more costly in total because it has more false negatives.

Worked Example: Checking What a Pooled Representation Loses

A hypothetical service team predicts repair cost in dollars. A compact representation discards a two-level equipment-age label and assigns every case a training-set prediction of $100. A less compact but still simple representation retains the label and uses training-set predictions of $95 for the newer group and $105 for the older group. On eight held-out repairs, the observed costs are $80, $90, $100 and $110 for newer equipment, and $90, $100, $110 and $120 for older equipment. Compare held-out squared error and RMSE. Assume the training-set predictions were set before these eight outcomes were observed.

Method and assumptions. Use the same eight held-out cases and sum squared prediction errors in dollars squared. The predictions are fixed, and each record receives the prediction for its assigned representation.

With the pooled $100 prediction, the errors for newer equipment are $-\$20,-\$10,\$0,\$10$, whose squared values sum to $400+100+0+100=600$. For older equipment the errors are $-\$10,\$0,\$10,\$20$, whose squared values sum to $100+0+100+400=600$. Total squared error is $1{,}200$, mean squared error is $1{,}200/8=150$, and RMSE is $\sqrt{150}\approx \$12.25$.

With the group-specific predictions, newer-equipment errors are $-\$15,-\$5,\$5,\$15$; their squared values sum to $225+25+25+225=500$. Older-equipment errors are $-\$15,-\$5,\$5,\$15$ as well, also summing to $500$. Total squared error is $1{,}000$, mean squared error is $1{,}000/8=125$, and RMSE is $\sqrt{125}\approx \$11.18$.

Conclusion. Discarding the label raises the total squared error by $1{,}200-1{,}000=200$ dollars squared, and RMSE by about $\$12.25-\$11.18=\$1.07$. Retaining the label improves the measured predictions and still permits a straightforward explanation: the two groups receive different average predictions. Whether that improvement justifies retaining the label depends on its practical importance and on whether the group distinction can be applied consistently.

Common mistake avoided: Concluding that pooling loses no information because the overall average is still $100 overlooks the differing group predictions. The held-out errors show the consequence of that discarded distinction.

Why the Decision Depends on the Use

The comparisons above rely on a simple principle: an average loss is the average of the losses on the individual evaluation cases. If $\ell_j$ is the loss on case $j$, then

\bar{\ell}=\frac{1}{n}\sum_{j=1}^{n}\ell_j.

This follows directly from adding the $n$ case-level losses and dividing by the number of cases. For squared error, $\ell_j=(y_j-\hat{y}_j)^2$; for a decision-cost comparison, $\ell_j$ can be the cost assigned to that case’s classification result. The formula is useful only when the case-level loss represents what matters. If missing an important case is more serious than an unnecessary review, equal-weight error rates may not represent the decision well.

A defensible comparison therefore states the evaluation population, the loss or decision measure, and the practical benefit sought from a simpler representation. It also avoids claiming more than the evidence supports: lower observed validation loss does not prove that the same advantage will persist in a different population or at a later time. Performance, implementation constraints, explanation needs, and the consequences of errors all belong in the decision.

This balance is relevant to actuarial predictive work because model outputs can inform actions with different consequences. A compact representation may be easier to communicate and maintain, but a small average performance difference can conceal important loss for a particular subgroup or type of case. Conversely, retaining detail that does not change predictions or actions may add complexity without a meaningful benefit. These judgments should be documented rather than disguised as a purely mathematical rule.

Key takeaway: Choose a representation by comparing its predictive consequences on relevant evaluation data with the value of its clarity and practical usability. State what was discarded, what the chosen metric says, and which decision assumptions support the trade-off. There is no universal cutoff for acceptable information loss.

Common Traps

Using a convenient metric instead of a relevant one. Accuracy can conceal the relative costs of different errors; RMSE may not reflect a decision with asymmetric consequences.
Equating fewer predictors with greater interpretability. A small set of derived quantities can be difficult to explain, while several familiar predictors may be understandable.
Assuming that high predictor variation means useful outcome information. Variation alone does not show whether a predictor improves predictions.
Reporting performance without stating what was removed. Naming the discarded feature or distinction helps reviewers assess what may no longer be visible.
Treating a validation result as a guarantee. A measured trade-off applies to the evaluation setting and may not transfer unchanged to another population or time period.

A clear recommendation should connect the evidence to the use: “The compact rule has higher accuracy but greater expected error cost under these stated costs; we prefer it only if explanation needs outweigh that cost difference.” Such a statement is more useful than saying that one representation is simply “best.”

Check Your Understanding

For each problem, compare the stated performance with the decision context before choosing a representation.

A hypothetical property team compares two models on five held-out losses. The detailed model has squared errors $4, 9, 1, 16, 0$; the compact model has squared errors $9, 9, 4, 9, 4$. Calculate each RMSE and identify which meets a requirement of RMSE at most 3. (Answer: Detailed RMSE $=\sqrt{30/5}=\sqrt{6}\approx 2.45$; compact RMSE $=\sqrt{35/5}=\sqrt{7}\approx 2.65$; both meet the requirement.)
A hypothetical review rule has 20 false negatives and 30 false positives. Their respective costs are $300 and $20. A competing rule has 10 false negatives and 60 false positives. Calculate each total error cost and state which has the lower cost under these assumptions. (Answer: First rule costs $20(300)+30(20)=\$6{,}600$; competing rule costs $10(300)+60(20)=\$4{,}200$; the competing rule has the lower stated cost.)
A numeric prediction model has total squared error 800 on 20 held-out records when it pools two groups. A representation retaining the group label has total squared error 680 on the same records. Calculate both RMSE values and the increase in squared error from pooling. (Answer: Pooled RMSE $=\sqrt{800/20}=\sqrt{40}\approx 6.32$; retained-label RMSE $=\sqrt{680/20}=\sqrt{34}\approx 5.83$; pooling increases squared error by $800-680=120$.)
A team’s compact model has lower RMSE on a validation set, but the detailed model performs better for a small group whose cases have substantially higher error costs. What additional comparison is needed before recommending the compact model? (Answer: Compare case-level or group-specific errors using a loss measure that reflects the higher costs, and assess whether the compact model’s total decision consequences remain acceptable.)
A representation reduces 40 predictors to two derived quantities, but a user cannot explain what either quantity means or how it affects a prediction. Is the representation necessarily more interpretable, and what should be assessed? (Answer: No. Assess whether users can understand the derived quantities and their relationship to predictions and decisions; fewer quantities alone do not establish interpretability.)

What You'll Learn