Two Different Ways to Change a Predictor Set
A predictive model may begin with many recorded variables, some redundant or difficult to use together. One response is to keep a smaller subset of those variables. Another is to combine or transform the recorded variables into new ones. These approaches can both reduce the number of predictors presented to a model, but they do not mean the same thing: the first keeps selected inputs, while the second constructs a new representation.
This distinction matters in actuarial work. A retained variable such as vehicle age can be described in its original units and traced back to the data record. A constructed variable such as a claims-per-exposure rate may be useful for prediction but no longer tells us, by itself, the separate claim count and exposure. A model’s inputs therefore affect both what its results mean and what details remain available for later analysis.
Suppose the original predictor vector is \(X=(X_1,\ldots,X_p)\). Selection chooses a set of indices \(S\) and retains \(X_j\) for \(j\in S\). Extraction instead creates a representation \(Z=g(X)\), where \(g\) is a specified transformation. The output \(Z\) may have fewer, the same number, or more columns than the input \(X\); the defining feature is that its variables are constructed rather than simply retained unchanged.
A transformation need not be complicated to count as extraction. Adding two recorded amounts, dividing a claim count by exposure, or converting a date into elapsed time all create a feature from existing data. By contrast, keeping the recorded date or amount as it was collected is selection if the decision is simply whether to include it.
What Happens to the Original Information?
Selection changes which columns are available, but the columns that remain retain their original meanings. The omitted columns are no longer available to a model using only the selected set. Extraction changes the representation: the new columns may combine information from several inputs, and the original values may or may not be recoverable.
The key question for exact recovery is whether the transformation is one-to-one over the possible input values. If different input values can produce the same extracted value, then the output alone cannot tell those inputs apart. This is a property of the transformation, not a judgment about whether the extracted feature is useful.
The reason is direct. When two different inputs have the same output, any attempt to reconstruct an input using only that output receives the same value for both cases. It cannot return the correct, different input in both cases. Conversely, if each possible output corresponds to exactly one input, the inverse mapping identifies that input. This establishes recoverability of the original predictors; it does not, by itself, establish that a model built from the transformed predictors will perform better or worse.
There is also a distinction between recovering every input detail and retaining what matters for a particular outcome. A transformation can discard details that have no predictive value while preserving the information relevant to the outcome. Formally, for a response \(Y\), the transformed features retain all of the conditional predictive information in \(X\) when the conditional distribution of \(Y\) given \(X\) is the same as the conditional distribution given \(Z\), where \(Z=g(X)\). This condition is a statement about the data-generating relationship, not something guaranteed by the act of transforming variables.
Worked Example: Selecting Original Vehicle Predictors
Worked Example: Selecting Original Vehicle Predictors
A hypothetical motor insurer has three recorded predictors for each policy: \(X_1\), prior-year premium in dollars; \(X_2\), vehicle age in years; and \(X_3\), annual mileage in thousands of miles. A screening and subject-matter review leads the analyst to retain vehicle age and annual mileage for a later model. Identify the method, describe the resulting inputs, and state what information the model’s input table no longer contains.
Method. This is feature selection. The analyst retains two existing predictors and does not redefine them. The selected set is \(S=\{2,3\}\), so the model receives \((X_2,X_3)\), not a newly calculated combination.
Check the predictor count. There are 3 original predictors and 2 selected predictors. The fraction retained is \(2/3\), and the reduction in column count is \(3-2=1\). This count describes the input table, not the number of records.
Interpretation. A one-unit increase in \(X_2\) still means one additional year of vehicle age, and a one-unit increase in \(X_3\) still means one thousand additional annual miles. Their units and original interpretations have not changed.
Information no longer in the selected table. \(X_1\), the prior-year premium, is absent. A model receiving only \((X_2,X_3)\) cannot use the recorded premium as a separate predictor. The selection has not converted premium into some other feature; it has omitted that column from this model’s input.
Common mistake avoided. Calling this feature extraction because the predictor set became smaller confuses the outcome with the method. The decisive check is that both retained columns are original variables, unchanged in definition.
Selection does not guarantee that the retained variables are independent of the omitted ones, nor that the omitted variables contain no useful information. It describes an input choice. The reasons for that choice and its consequences for a model must be assessed separately. In particular, a predictor that is redundant for one modeling purpose may still have a useful interpretation or role in another.
Worked Example: An Aggregate That Cannot Recover Its Inputs
Worked Example: An Aggregate That Cannot Recover Its Inputs
A hypothetical claims team records two amounts for each claim file, \(U\) for the initial estimate and \(V\) for the later estimate, both in thousands of dollars. For a compact summary, an analyst creates \(T=U+V\) and proposes using \(T\) alone in a model. Two files have inputs \((U,V)=(3,9)\) and \((U,V)=(5,7)\). Calculate the extracted feature for each file and decide whether \(T\) alone can recover the two original amounts.
Method. This is feature extraction because \(T\) is constructed from two original predictors. Calculate it separately for each file:
Both extracted values are 12 thousand dollars, even though the original inputs differ: the first file has \(U=3,V=9\), while the second has \(U=5,V=7\). Thus the extraction maps two distinct input pairs to the same output.
Conclusion. A model using \(T\) alone cannot distinguish these two files on the basis of their original estimate split. There is no unique way to infer \(U\) and \(V\) from \(T=12\): possibilities include \((3,9)\) and \((5,7)\). The sum retains the combined amount but loses the distinction between the component amounts.
Common mistake avoided. It would be incorrect to say that the two original amounts can be recovered by “splitting the total in half.” That rule would give \((6,6)\), which is neither observed input pair. A total gives one equation in two unknowns, so it does not uniquely identify both values.
This example demonstrates exact information loss in the input representation. It does not prove that the sum is useless: if the outcome depends on the combined amount and not on how it is divided, the sum could still be a suitable predictor. If the distinction between the initial and later estimates matters, using the sum alone may discard relevant information.
Worked Example: A Rate Replaces Count and Exposure
Worked Example: A Rate Replaces Count and Exposure
A hypothetical portfolio analysis has two original predictors for each group: reported claim count \(N\) and earned exposure \(E\), measured in policy-years. The analyst creates a rate \(R=N/E\), measured in claims per policy-year, and proposes retaining \(R\) alone. Group A has \(N=6\) and \(E=1.5\); Group B has \(N=8\) and \(E=2\). Calculate the rates and explain what can and cannot be inferred from the rate-only representation.
Method. This is feature extraction: each rate is calculated from two recorded inputs. For Group A,
For Group B,
The extracted values are equal, although the inputs are not: Group A has 6 claims and 1.5 policy-years, while Group B has 8 claims and 2 policy-years. Therefore, if only \(R\) is retained, the two groups cannot be distinguished by their claim count or exposure. The rate preserves their common ratio, not their separate scale.
Interpretation. The rate gives a standardized measure of claim frequency per unit of exposure. It does not state how many claims were observed or how much exposure supported the rate. Those details can matter when interpreting the amount of data behind a rate or when a later method needs the original count and exposure.
Common mistake avoided. Equal rates do not imply equal counts and equal exposure. Here \(6/1.5=8/2\), but \(6\ne8\) and \(1.5\ne2\). Also, this ratio is undefined when \(E=0\); the calculation requires positive exposure in these examples.
How to Compare the Two Approaches
When reviewing a proposed predictor set, do not infer the method from the number of columns alone. Instead, compare the proposed inputs with the original data dictionary. If each proposed input is an unchanged member of the original list, the operation is selection. If an input is calculated, combined, recoded, or otherwise transformed from original variables, it is extraction. A workflow can use both: it may first select variables and then construct features from those selected variables.
| Question | Feature selection | Feature extraction |
|---|---|---|
| What is passed to the model? | A subset of original predictors | Predictors created by a stated transformation |
| What happens to retained meanings? | Retained variables keep their original definitions and units | Constructed variables have meanings determined by the transformation |
| Can original input values be recovered? | Selected values remain available; omitted values are not in the selected input | It depends on whether the transformation is one-to-one |
| Can the model’s input count decrease? | Yes, by retaining fewer original predictors | Yes, if several inputs are summarized or replaced by fewer outputs |
| What must be documented? | Which original predictors were kept and omitted | The transformation, its output meaning, and any information it discards |
A practical comparison should therefore answer three questions. First, what exact variables enter the model? Second, can an analyst interpret each input using the original data definitions? Third, which differences among records disappear after the operation? These checks clarify the representation without claiming that one approach is always preferable.
Write their names, definitions, units, and the observational unit to which each value refers.
Mark it as an unchanged original predictor or as a constructed feature, and specify the transformation for each constructed feature.
Check whether different original records can map to the same extracted representation, then state which distinctions are lost and which quantities remain directly interpretable.
Common Traps and Practical Implications
- Equating fewer columns with selection. A single derived rate may summarize several inputs. The column count alone does not identify whether selection or extraction occurred.
- Assuming an extracted feature is reversible. Aggregates and ratios often map multiple original values to the same output. Test this with distinct inputs rather than assuming the formula can be inverted.
- Assuming any information loss harms prediction. Losing a detail is consequential only if it matters for the analysis or prediction target. Conversely, a useful-looking summary is not guaranteed to retain all outcome-relevant information.
- Reporting a constructed variable as if it were original. A rate, total, or other derived measure needs a clear definition and units. Otherwise its interpretation and limitations are unclear.
- Confusing retaining a variable with proving its value. Selection records which original predictors were included; it does not establish that omitted predictors are irrelevant in every setting.
Check Your Understanding
For each problem, identify the operation and state what is retained or lost.
- A hypothetical underwriting file has four original predictors. The analyst keeps the recorded vehicle age and region but omits the other two predictors. How many inputs remain, and is this selection or extraction? (Answer: Two inputs remain; this is feature selection.)
- Two records have measurements \((2,10)\) and \((4,8)\). An analyst retains only their sum. Calculate both extracted values and say whether the original pair can be uniquely recovered. (Answer: Both sums are 12; the original pair cannot be uniquely recovered from the sum alone.)
- A hypothetical claim group has \(N=9\) claims over \(E=3\) policy-years. Calculate \(R=N/E\), and state whether retaining \(R\) alone preserves the separate values of \(N\) and \(E\). (Answer: \(R=3\) claims per policy-year; the rate alone does not preserve the separate count and exposure.)
- An analyst transforms two inputs \((a,b)\) into \(Z_1=a+b\) and \(Z_2=a-b\). Solve for \(a\) and \(b\) in terms of \(Z_1,Z_2\), and determine whether this transformation is one-to-one. (Answer: \(a=(Z_1+Z_2)/2\) and \(b=(Z_1-Z_2)/2\); it is one-to-one because both original inputs can be recovered.)
- A model uses an extracted summary \(Z=g(X)\). What condition on the conditional distribution of response \(Y\) says that \(Z\) retains all conditional predictive information in \(X\)? (Answer: \(P(Y\mid X)=P(Y\mid Z)\) almost surely.)