Linear Combinations as Reduced Representations

Why Represent Several Predictors With Fewer Features?

A data set may contain several predictors that overlap in the information they describe. A reduced representation replaces the original predictors with fewer features, each formed as a weighted linear combination of the originals. For example, a deliberately defined score might combine several measures of an account’s size and activity. The resulting score is easier to carry into a later analysis, but it may also conceal which original values produced it.

A linear combination is not automatically useful just because it reduces the number of columns. Its usefulness depends on the purpose of the analysis and on whether the combinations retain information relevant to that purpose. The key questions are: How is the transformation calculated? What does it retain? What distinctions does it discard?

Definition: Let $x$ be a column vector of $p$ original predictor values, and let $A$ be a fixed $p \times k$ matrix of weights, where $k<p$. The reduced feature vector is $z=A^\mathsf{T}x$, a vector with $k$ entries. Each entry of $z$ is a weighted linear combination of the original predictors. For a data matrix $X$ with $n$ rows and $p$ columns, the transformed data matrix is $Z=XA$, with $n$ rows and $k$ columns.

The weights in $A$ specify exactly how each reduced feature is calculated. They might be set in advance for a subject-matter reason or determined by a separate procedure. The definition itself does not say that the weights are good, that the reduced features predict an outcome well, or that the original predictors have comparable measurement scales.

Calculating Reduced Features

To calculate a reduced representation, take each row of original predictor values and apply the same weights to that row. The transformation is fixed across observations: a particular reduced feature must mean the same calculation for every row. In the example below, the weights are given, and the calculation produces one score from three predictors.

Worked Example: Building a Single Account Score

A hypothetical insurer wants a simple screening score from three account characteristics. For each account, $x_1$ is annual payroll measured in units of $100,000, $x_2$ is the number of operating locations, and $x_3$ is the number of years in operation. The specified score is $z=0.6x_1+1.5x_2+0.2x_3$. Calculate the score for each account and identify what the reduction does not tell the analyst.

Method. Substitute each account’s three recorded values into the stated weighted sum. The calculation uses the provided weights; it does not estimate them.

For account A, $(x_1,x_2,x_3)=(8,2,5)$:

$$ z_A=0.6(8)+1.5(2)+0.2(5)=4.8+3.0+1.0=8.8. $$

For account B, $(x_1,x_2,x_3)=(5,4,3)$:

$$ z_B=0.6(5)+1.5(4)+0.2(3)=3.0+6.0+0.6=9.6. $$

For account C, $(x_1,x_2,x_3)=(10,1,8)$:

$$ z_C=0.6(10)+1.5(1)+0.2(8)=6.0+1.5+1.6=9.1. $$

The reduced data consist of the scores $8.8, 9.6,$ and $9.1$, rather than the three original values for each account. Account B has the largest score under this specified rule. However, the score alone does not reveal whether an account’s value came mainly from payroll, number of locations, or years in operation. For example, different combinations of the three original values can produce the same score. A common mistake is to treat the score as a probability or as a naturally meaningful quantity; here it is only the weighted sum defined in the problem.

What a Linear Combination Can Preserve

A reduction is exactly adequate for a particular linear score when that score can itself be calculated from the reduced features. Suppose a score on the original predictors has coefficient vector $\beta$, so its value for observation $x$ is $x^\mathsf{T}\beta$. If there is a vector $\gamma$ such that $\beta=A\gamma$, then:

x^\mathsf{T}\beta =x^\mathsf{T}A\gamma =(A^\mathsf{T}x)^\mathsf{T}\gamma =z^\mathsf{T}\gamma.

This identity is the basic preservation result: when $\beta$ is a linear combination of the columns of $A$, the score using all original predictors can be computed exactly from the reduced features. No original values are needed for that score once $z$ is known. If $\beta$ cannot be written as $A\gamma$, that exact representation is not available from these reduced features alone.

Result: The linear score $x^\mathsf{T}\beta$ is exactly computable from $z=A^\mathsf{T}x$ for every $x$ if and only if $\beta=A\gamma$ for some vector $\gamma$. This condition is specific to the score being considered; a reduction may preserve one score exactly and fail to preserve another.

Worked Example: Checking Whether a Score Is Preserved

Suppose two original predictors are $x_1$ and $x_2$, and the retained feature is $z=x_1+x_2$. A hypothetical linear scoring rule is $\eta=3x_1+3x_2$. Determine whether $z$ is sufficient to calculate $\eta$, and compare it with the score $\eta'=3x_1+x_2$.

Method. Check whether each score’s coefficient vector is a multiple of the weight vector defining $z$. Here $A=(1,1)^\mathsf{T}$, so any score exactly expressible from $z$ must have coefficient vector $A\gamma=(\gamma,\gamma)^\mathsf{T}$.

For $\eta$, the coefficients are $(3,3)^\mathsf{T}=A(3)$. Therefore $\gamma=3$, and the score can be calculated from the reduced feature:

\eta=3x_1+3x_2=3(x_1+x_2)=3z.

For a numerical check, take $(x_1,x_2)=(1,5)$, $(3,3)$, and $(6,0)$. Each gives $z=6$, and each gives $\eta=18$. The reduced feature preserves this score exactly.

For $\eta'=3x_1+x_2$, the coefficient vector $(3,1)^\mathsf{T}$ is not a multiple of $(1,1)^\mathsf{T}$. The reduction therefore cannot determine $\eta'$ exactly. In particular, $(1,5)$ and $(3,3)$ both give $z=6$, but their scores are $3(1)+5=8$ and $3(3)+3=12$, respectively. The same reduced value corresponds to different original scores. The trap to avoid is assuming that because the retained feature is calculated from the originals, every score based on the originals can also be recovered.

What a Reduction Can Obscure

When $k<p$, the transformation maps more original values to fewer reduced values. In general, it cannot uniquely recover all $p$ original predictors. More formally, the map $x\mapsto A^\mathsf{T}x$ sends a $p$-dimensional input to a $k$-dimensional output. Since $k<p$, there is at least one nonzero vector $h$ such that $A^\mathsf{T}h=0$. Consequently, $x$ and $x+h$ have the same reduced representation:

A^\mathsf{T}(x+h)=A^\mathsf{T}x+A^\mathsf{T}h=A^\mathsf{T}x.

Thus, the transformation necessarily hides at least some differences among original predictor values. Whether those hidden differences matter depends on the intended use. If they affect an outcome-relevant score, losing them may be consequential. If they do not affect the analysis of interest, a reduced representation may be adequate for that purpose.

Worked Example: Two Features From Four Predictors

A hypothetical portfolio summary replaces four predictors with two features: $z_1=x_1+x_2$ and $z_2=x_3+x_4$. Compare the original records $(2,1,4,0)$ and $(0,3,1,3)$. Then determine whether each of two stated linear scores can be calculated exactly from $z_1,z_2$.

Method. First calculate both retained features for each record. Then check whether each original score’s coefficient vector has equal coefficients on $x_1,x_2$ and equal coefficients on $x_3,x_4$, as required for a combination of $z_1$ and $z_2$.

For the first record:

z_1=2+1=3,\qquad z_2=4+0=4.

For the second record:

z_1=0+3=3,\qquad z_2=1+3=4.

Both records therefore reduce to $(3,4)$, even though their original predictor values differ. Consider first the score $\eta_1=2x_1+2x_2-x_3-x_4$. Its coefficients match $2z_1-z_2$, so it is preserved:

\eta_1=2z_1-z_2=2(3)-4=2.

Direct calculation confirms the result for the first record: $2(2)+2(1)-4-0=2$. For the second record it is $2(0)+2(3)-1-3=2$.

Now consider $\eta_2=2x_1-x_3+x_4$. Its coefficients on $x_3$ and $x_4$ differ, so it cannot be written using only $z_1,z_2$. Direct calculation gives $2(2)-4+0=0$ for the first record and $2(0)-1+3=2$ for the second. Since both records have the same reduced features but different $\eta_2$, the reduced representation obscures information needed for this score. The common mistake is to check only that both original scores use the same four predictors; exact preservation depends on how their coefficients align with the specified combinations.

Using a Reduced Representation Carefully

A reduced representation replaces selected columns with constructed features; it does not simply delete columns while leaving the rest unchanged. This is feature extraction: the retained features are newly constructed from the originals. In feature selection, by contrast, the retained predictors are chosen from the original columns. The two approaches can both reduce the number of variables, but only a linear-combination representation changes what each retained column means.

Before using a reduced representation, state the weights and the intended purpose. Then ask what information that purpose requires. The exact preservation condition above answers one precise question: whether a particular linear score can be calculated without the original predictors. It does not establish that all outcome-relevant information has been retained, or that a model using the reduced features will perform well.

In applied work, the interpretation of a reduced feature also depends on its weights and on how the original predictors are recorded. A weight is not automatically a measure of importance, and a larger weight alone does not show that its predictor matters more. A weighted combination may be useful as a compact representation while still being less transparent than the original columns.

Write the transformation.
Specify the original predictor vector, the weight matrix, and the resulting reduced features.

Calculate the transformed values.
Apply the same weights to each observation, showing the products and their sum.

Check the intended use.
For an exact linear score, test whether its coefficient vector can be expressed as a combination of the transformation’s weight vectors.

State what is lost.
Identify distinctions among original predictor values that identical reduced features no longer reveal.

A frequent exam trap is to claim that a reduced representation “preserves the information” without saying which information. A defensible statement is narrower: for example, “this transformation preserves the specified linear score because its coefficient vector is a linear combination of the transformation weights.” If the condition does not hold, use two different original vectors with the same reduced features but different scores to demonstrate the loss.

The central idea is that dimension reduction through linear combinations is purpose-dependent. A representation with fewer features can simplify data and preserve selected calculations exactly, but it cannot generally preserve every distinction present in the original predictors. Understanding both sides is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, calculate or assess the reduced representation before checking the answer.

A screening feature is $z=2x_1-x_2+0.5x_3$. For $(x_1,x_2,x_3)=(4,2,5)$, calculate $z$. (Answer: $8.5$)
A feature is $z=x_1+x_2$. Two records are $(2,7)$ and $(5,4)$. Do they have the same reduced value, and can $x_1-x_2$ be determined from that value alone? (Answer: Both have $z=9$; no, since $x_1-x_2=-5$ and $1$, respectively.)
Let $z_1=x_1+x_2$ and $z_2=x_3$. Can the score $4x_1+4x_2-2x_3$ be calculated from the reduced features? If so, write it in terms of $z_1,z_2$. (Answer: Yes; it equals $4z_1-2z_2$.)
A transformation retains $z_1=x_1+x_2$ and $z_2=x_3+x_4$. For $(x_1,x_2,x_3,x_4)=(1,4,2,3)$, calculate both features and then calculate $3z_1-2z_2$. (Answer: $z_1=5$, $z_2=5$, and $3z_1-2z_2=5$.)
Using $z_1=x_1+x_2$ and $z_2=x_3+x_4$, decide whether $\eta=x_1+2x_2-x_3-x_4$ can be recovered exactly. Compare records $(1,0,0,0)$ and $(0,1,0,0)$. (Answer: No; both give $(z_1,z_2)=(1,0)$, but their $\eta$ values are $1$ and $2$.)

What You'll Learn