Why Reduce the Number of Predictors?
A data set can contain many measurements that describe nearly the same aspect of a risk. A hypothetical claim file, for example, might include several related indicators of vehicle use, repair severity, and prior claim activity. Keeping all available measurements may be useful, but it can also make analysis slower, harder to inspect, or more difficult to communicate. Dimension reduction addresses the number of predictor variables used to represent each observation.
The reason for reducing dimensions matters. An analyst might want to reduce computation, remove redundant information, or make a high-dimensional data set easier to visualize. Those are not the same objective as selecting whichever model predicts best. A smaller representation may be useful for one purpose and still predict less accurately than a larger one.
A central distinction is between feature selection and feature extraction. Feature selection retains some original predictors and discards others. Feature extraction creates new features from information in the original predictors. Either approach can reduce dimension, but they can differ in how easy the resulting representation is to interpret. The choice should follow from the problem’s purpose rather than from a general assumption that fewer predictors are always better.
Three Different Reasons to Reduce Dimension
Computation
A smaller set of features can reduce the work required to store, inspect, or process data. In some settings, the key issue is not the cost of fitting one model but the many operations needed during exploration or model development. For example, examining all pairs of predictors requires more comparisons as the number of predictors grows. Reducing the feature count can make this work more manageable.
This benefit depends on the task. If a model fitting procedure is already fast and the data set is small, computation may not be a meaningful reason to discard predictors. If repeated calculations or limited computing resources are important constraints, it may be.
Managing redundancy
Predictors can overlap in the information they record. Redundant predictors may make it harder to explain what individual variables contribute, and can complicate some modeling procedures. The earlier material on variance inflation factors and correlated predictors provides tools for diagnosing aspects of this issue. Dimension reduction can be one response, but correlation alone does not prove that a predictor is useless: a variable may still contribute information that matters for a particular outcome or use.
The appropriate question is therefore not simply, “Are these predictors correlated?” It is, “Does reducing this set preserve the information needed for the intended task, while improving some relevant practical property?” The answer depends on the outcome, the evaluation goal, and the consequences of losing information.
Visualization
A plot is easiest to inspect when observations can be shown along a small number of axes. When each observation has many predictors, a reduced representation can help create a visual summary. Such a display is useful for exploring broad patterns, unusual observations, or possible groupings. It cannot show every detail of the original data at once. A pattern visible in a two-dimensional display may not represent every relationship in the full set of predictors, and a pattern that is not visible may still matter.
Visualization is therefore an exploratory aim, not proof that a representation is predictive or that groups in the display are real and stable. Conclusions drawn from a reduced plot should be checked against the question being studied and the original variables where possible.
Counting the Work in Pairwise Comparisons
Suppose an analyst wants to examine every distinct pair among \(p\) predictors. There are \(p\) choices for the first predictor and \(p-1\) choices for a different second predictor, giving \(p(p-1)\) ordered pairs. Each distinct pair appears twice in that count: once in each order. Dividing by two gives the number of distinct pairs.
This formula counts comparisons, not the total computational cost of every possible analysis. It shows why pairwise inspection can grow quickly as \(p\) increases. Reducing \(p\) can lower that particular workload, but it does not establish that the resulting predictors are adequate for prediction or any other purpose.
Worked Example: Reducing Pairwise Review Work
A hypothetical pricing analyst has 48 candidate measurements describing customer and policy characteristics. The analyst plans to inspect every distinct pair of predictors for redundancy. A proposed reduced data set would retain 12 features. How many pairwise comparisons are required before and after reduction, and what percentage reduction does this represent?
Method. Count the distinct unordered pairs using \(p(p-1)/2\). This count assumes that each predictor is compared with each other predictor once and that a predictor is not compared with itself.
Before reduction. With \(p=48\), the number of pairs is \(48(48-1)/2=48\times47/2=2{,}256/2=1{,}128\).
After reduction. With \(p=12\), the number is \(12(12-1)/2=12\times11/2=132/2=66\).
Change. The count falls by \(1{,}128-66=1{,}062\) comparisons. Relative to the original count, the reduction is \(1{,}062/1{,}128=0.941489\ldots\), or \(94.1\%\) to one decimal place.
Check. The remaining 66 comparisons are less than 1,128, and the reduction is just over 94%, consistent with the difference of 1,062. This calculation supports a computation-related argument for reducing the feature count. It does not show that the 12 retained features preserve the information needed for pricing. The common mistake is to treat a lower comparison count as evidence of better prediction; it is evidence only about the number of pairwise comparisons.
Dimension Reduction Is Not the Same as Predictive Model Selection
A predictive model is judged by how well it performs on data appropriate for evaluating its intended use. A dimension-reduction decision may instead be motivated by speed, easier visualization, or a more compact representation. These objectives can align, but they do not have to. In particular, a reduced model can be easier to use while having larger prediction errors.
A fair numerical comparison of prediction errors requires the same outcomes, the same evaluation cases, the same outcome units, and the same error definition for both models. For a set of \(n\) evaluation cases, the root mean squared error (RMSE) is the square root of the average squared difference between the observed and predicted outcomes. It summarizes prediction error in the outcome’s units. The calculation below uses fixed hypothetical records; no probability distribution is assumed.
Worked Example: Compactness Versus Predictive Error
A hypothetical insurer compares two models on the same four held-out claim-cost records, with amounts in thousands of dollars. Model A uses eight predictors; Model B uses three predictors created through a reduction step. The observed costs are \(10,20,30,40\). Model A predicts \(11,19,31,39\), while Model B predicts \(12,18,33,37\). Calculate each model’s RMSE and explain what the comparison supports.
Method and conditions. Use the RMSE formula on the same four cases for both models. The target and predictions are all in thousands of dollars, so the resulting RMSE is also in thousands of dollars.
Model A. The observed-minus-predicted errors are \(10-11=-1\), \(20-19=1\), \(30-31=-1\), and \(40-39=1\). Their squares are \(1,1,1,1\), so the sum of squared errors is \(4\). Thus, \(\operatorname{RMSE}_A=\sqrt{4/4}=\sqrt{1}=1.000\) thousand dollars.
Model B. Its errors are \(10-12=-2\), \(20-18=2\), \(30-33=-3\), and \(40-37=3\). Their squares are \(4,4,9,9\), with sum \(26\). Thus, \(\operatorname{RMSE}_B=\sqrt{26/4}=\sqrt{6.5}=2.5495\ldots\), or \(2.550\) thousand dollars to three decimals.
Interpretation and check. Model A has the smaller RMSE on these four cases: \(1.000<2.550\) thousand dollars. Model B is more compact, but this comparison does not support claiming that it predicts better on these cases. Nor do four held-out records establish which model will perform better in every future setting. The common mistake is to select Model B solely because it has fewer predictors. The correction is to state the objective: if predictive accuracy is the priority, compare models using an appropriate evaluation design; if compactness is the priority, describe that benefit separately and report any predictive cost.
Choosing a Goal Before Choosing a Representation
Dimension reduction is most useful when its purpose is stated in operational terms. “Reduce dimensions” is not yet a complete objective. A more useful statement identifies what should improve and what must remain adequate. For example: reduce the burden of inspecting related predictors while retaining information that matters for the intended prediction task; or create a compact display for exploration without treating the display as a final predictive model.
Decide whether the primary aim is computation, redundancy management, visualization, prediction, or a combination. Do not assume these aims are interchangeable.
For a predictive task, specify the outcome and what predictive performance must be retained. For exploration, state what patterns the display is intended to help inspect.
Check whether the reduction makes the intended work easier. If prediction matters, assess predictive performance separately rather than inferring it from the number of features.
Record which information or interpretability may be harder to recover after reduction. A compact representation can be efficient without being as transparent as the original predictors.
This process keeps the decision aligned with the use case. It also helps communicate trade-offs: an analyst can explain that a representation was reduced to aid visualization, for instance, without implying that the displayed dimensions preserve all predictive information.
Worked Example: Selecting a Reduction Goal for a Visual Review
A hypothetical reserving team has 25 numeric indicators for each of 40 claim files. Its immediate task is to produce a single scatterplot that helps reviewers look for broad patterns among the claim files. The team is not using this plot to set reserves or rank claim costs. What is the relevant goal, and what conclusion should the plot not be used to support?
Identify the task. The team wants a visual summary of 40 observations described by 25 predictors. Since its stated task is one scatterplot for broad review, the immediate goal is visualization, not prediction.
Quantify the display constraint. A conventional two-axis scatterplot shows two coordinates per observation. In this display, there are 40 plotted points and two displayed coordinates for each point. The original data have \(40\times25=1{,}000\) observation-feature entries; the display uses \(40\times2=80\) plotted coordinates. This is a display-size comparison, not proof that the 80 coordinates retain every relevant detail.
State the appropriate interpretation. Reviewers may use the plot to notice broad visual patterns or observations worth investigating. They should not conclude, based on the plot alone, that the display preserves all 25 predictors’ information, that visually close points necessarily have similar claim costs, or that the display is an adequate predictive model.
Check. The problem states no prediction or ranking task, so predictive accuracy is not the goal being evaluated. The common mistake is to treat a useful visual summary as evidence of a useful prediction representation. The correction is to judge the plot for its stated exploratory purpose and assess prediction separately if that becomes a later objective.
Common Exam Traps
- Equating fewer predictors with better prediction. A reduction can lower computation or improve a display while increasing prediction error. State which objective is being optimized.
- Treating correlation as proof of irrelevance. Correlated predictors may overlap, but one may still contain information that matters for the outcome or the intended use.
- Over-interpreting a reduced plot. A low-dimensional visual summary cannot display every feature relationship. Use it for the stated exploratory purpose, not as a complete account of the data.
- Comparing errors on different cases or scales. A predictive comparison is not interpretable if the models are evaluated on different records or if their outcomes use different units.
- Reporting a computational gain as a predictive gain. Pairwise comparison counts describe workload. They do not measure predictive accuracy.
Check Your Understanding
For each problem, identify the relevant goal and show the requested calculation where applicable.
- A hypothetical analytics team has 30 predictors and plans to inspect every distinct predictor pair. A reduced set would have 10 predictors. How many pairs are there before and after reduction, and by what percentage does the count fall? (Answer: 435 before, 45 after; a reduction of 390, or 89.7% to one decimal place.)
- A data matrix contains 40 observations and 25 predictors. A reduced display contains two plotted coordinates per observation. How many observation-feature entries are in the original matrix and how many plotted coordinates are in the display? What is the main goal if the display is intended to help reviewers see broad patterns? (Answer: 1,000 original entries and 80 plotted coordinates; visualization.)
- A reduced representation retains 6 features from an original set of 18. What fraction of the original feature count has been removed? Does that calculation establish that predictive performance is preserved? (Answer: \(12/18=66.7\%\) removed; no, feature counts alone do not establish predictive performance.)
- On the same four evaluation cases, Model C has a sum of squared errors of 36 and Model D has a sum of squared errors of 64. Calculate their RMSE values. (Answer: Model C, \(\sqrt{36/4}=3\); Model D, \(\sqrt{64/4}=4\), in the outcome’s units.)
- A team reduces 50 features to 8 to make a single exploratory chart easier to inspect, then claims the reduced representation must be the best predictor because it is easier to visualize. Identify the error in that reasoning. (Answer: visualization and predictive performance are different objectives; ease of inspection does not establish better predictive accuracy.)