Why a Preprocessing Plan Matters
A data set can contain a long-tailed positive predictor, a predictor with many zeros, and measurements expressed on very different scales. Each feature raises a different question: should it be transformed, standardized, or binned? A sound plan answers these questions before fitting a model, records the choices, and applies them consistently to new data.
The aim is not to transform every predictor. A transformation can make a scale easier to work with, but it also changes how differences in the original values are represented. Binning can simplify interpretation, but it discards within-bin distinctions. The earlier material on choosing between raw, transformed, and binned predictors motivates these trade-offs; here the emphasis is on carrying them out as one reproducible workflow.
Choose a Transformation That Matches the Values
For a strictly positive predictor \(x\), the natural-log transformation is \(g(x)=\ln(x)\), defined only for \(x>0\). It preserves the ordering of observations while compressing large values more than small ones. The transformed difference between two positive values is \(\ln(x_2)-\ln(x_1)=\ln(x_2/x_1)\), so differences correspond to ratios on the original scale.
If zero is a valid observed value, \(\ln(x)\) is not defined at zero. One option is \(g(x)=\ln(1+x)\), often written as `log1p(x)` in R. This is defined for \(x\geq 0\), maps zero to zero, and still compresses larger values. The added constant is a modeling choice tied to the units: changing units can change the effect of adding 1. Do not use this formula for negative values without a separate, justified treatment.
When zero has a distinct practical meaning, a transformed numeric predictor alone may not fully express that distinction. For example, zero use of a service may differ substantively from a small positive amount of use. A plan can retain both \(I(x>0)\), an indicator of positive use, and \(\ln(1+x)\). The indicator records whether the value is zero; the transformed predictor describes variation in the amount. Whether both are useful is a modeling decision, not an automatic rule.
Worked Example: Log-Transforming a Skewed Positive Predictor
A hypothetical portfolio has a positive training predictor with values \(1,2,8\). The analyst proposes a natural-log transformation, followed by standardization using the training mean and sample standard deviation. Calculate the parameters and standardize a future value of \(16\).
Method. Transform first, then calculate the training-set mean and sample standard deviation on the transformed values. This avoids the common mistake of standardizing the original values and then treating those parameters as if they applied after transformation.
The transformed training values are \(0,\ln(2),\ln(8)\). Since \(\ln(8)=3\ln(2)\), write them as \(0,a,3a\), where \(a=\ln(2)\). Their mean is
The sum of squared deviations from this mean is
There are \(3-1=2\) degrees of freedom, so the sample variance is \(7a^2/3\) and the sample standard deviation is \(a\sqrt{7/3}\approx 1.0588\). These parameters are calculated from the transformed training data, not from the future value.
For the future value \(16\), the transformed value is \(\ln(16)=4a\). Using the training parameters, its standardized value is
Check. The result is positive because \(16\) exceeds every training value, and the calculation uses the training mean and sample standard deviation. The common mistake avoided is refitting the mean or standard deviation after seeing the future observation.
Standardize With Training-Set Parameters
For transformed training values \(g(x_1),\ldots,g(x_n)\), define their sample mean \(\bar{g}\) and sample standard deviation \(s_g\). If \(s_g>0\), standardize an observation by \(z=(g(x)-\bar{g})/s_g\). The same \(\bar{g}\) and \(s_g\) must be used for training and subsequent data. This makes the training transformed values have mean zero and sample standard deviation one, up to numerical precision.
The reason the formula centers and scales as intended follows directly from the definitions. Subtracting the training mean gives deviations whose sum is zero. Dividing each deviation by \(s_g\) divides the sum of squared deviations by \(s_g^2\); because \(s_g^2\) is the original sum of squared deviations divided by \(n-1\), the standardized sample variance is one.
This result describes the training data. A test set need not have mean zero or standard deviation one under the training parameters, and it should not be forced to. Recalculating parameters on test data would make the preprocessing depend on information that was meant to be held out.
Worked Example: Keeping Zero Values and Their Distinction
For a hypothetical predictor measuring a nonnegative quantity, the training values are \(0,0,1,3\). The analyst wants a log transformation that is defined at zero, plus a separate indicator for positive values. Calculate the transformed training mean and sample standard deviation, then standardize a future value of \(7\).
Method. Apply \(\ln(1+x)\) to the numeric feature and calculate its parameters from training observations only. Separately record \(I(x>0)\). This avoids the mistake of applying \(\ln(x)\) to zero or replacing zero with an arbitrary small positive number.
The transformed values are \(0,0,\ln(2),\ln(4)\). Let \(a=\ln(2)\), so these are \(0,0,a,2a\). The transformed mean is \(3a/4\), approximately \(0.5199\). The sum of squared deviations is
Dividing by \(4-1=3\) gives sample variance \(11a^2/12\), and sample standard deviation \(a\sqrt{11/12}\), approximately \(0.6636\). The indicator values for the four training observations are \(0,0,1,1\).
For \(x=7\), the transformed value is \(\ln(8)=3a\), and the indicator is \(I(7>0)=1\). Standardizing with the training parameters gives
Check. A future value of \(7\) is above all training values, so its standardized value is positive and exceeds the training maximum’s standardized value. The indicator remains a separate record of positivity; it is not the same quantity as the standardized transformed value.
Make Binning and Scale Decisions Explicit
Binning replaces a numeric value with the interval into which it falls. It can be useful when a small number of ranges is easier to communicate or when a model needs a simpler representation. However, observations in the same bin become indistinguishable on that representation. If within-bin variation may matter, retain the continuous predictor as well or compare the representations during model development.
A binning decision is incomplete unless it states the cut points and which interval includes a value exactly on a boundary. In R, `cut(x, breaks = c(-Inf, 35, 50, Inf), right = FALSE)` uses intervals that include the left endpoint and exclude the right endpoint: \((-\infty,35)\), \([35,50)\), and \([50,\infty)\). Thus 35 enters the second bin and 50 the third. If cut points are selected from the data, estimate them on training data and apply the same cut points later.
Standardization addresses scale, not skewness: it subtracts a constant and divides by a positive constant but does not remove a long right tail. Conversely, a logarithm changes spacing but does not ensure that two predictors are on comparable scales. A plan may therefore transform a predictor and then standardize it, but each step should have a stated purpose.
Worked Example: A Mixed-Scale Plan With Scaling and Bins
A hypothetical training set contains travel distance in kilometres with values \(100,200,300\), service duration in hours with values \(0.2,0.4,0.6\), and ages \(27,35,48,61\). The analyst wants standardized numeric predictors and a three-bin age representation using cut points 35 and 50. Determine the parameters, the bin assignments, and how a future distance of \(400\) km and duration of \(0.5\) hours are processed.
Method. Standardize each numeric predictor using its own training mean and sample standard deviation. Keep age cut points fixed and state the interval closure. This avoids using one predictor’s scale parameters for another or allowing an ambiguous boundary assignment.
For distance, the training mean is \((100+200+300)/3=200\) km. Its sample variance is \(\{(-100)^2+0^2+100^2\}/2=10{,}000\), so its sample standard deviation is \(100\) km. The standardized training values are \(-1,0,1\).
For duration, the training mean is \((0.2+0.4+0.6)/3=0.4\) hours. Its sample variance is \(\{(-0.2)^2+0^2+0.2^2\}/2=0.04\), so its sample standard deviation is \(0.2\) hours. The standardized training values are again \(-1,0,1\), although the original units and parameters differ.
For ages, use the intervals \((-\infty,35)\), \([35,50)\), and \([50,\infty)\). The training ages 27, 35, 48, and 61 are assigned respectively to the first, second, second, and third bins. In particular, 35 is included in the second bin and 50 would be included in the third.
The future distance of 400 km is standardized as \((400-200)/100=2\). The future duration of 0.5 hours is standardized as \((0.5-0.4)/0.2=0.5\). Neither future observation changes the stored training parameters. The common mistake avoided is treating the identical standardized training values as evidence that the original predictors had interchangeable units or needed the same parameters.
Implement and Record the Plan in R
For a positive predictor, check that all values meet the domain requirement before using `log(x)`. For a nonnegative predictor, `log1p(x)` computes \(\ln(1+x)\). A direct, auditable R workflow is to create the transformed training variable, calculate `mu <- mean(train_g)` and `s <- sd(train_g)`, and then create `train_z <- (train_g - mu) / s`. For later data, use `test_z <- (log1p(test_x) - mu) / s` when that was the chosen transformation. The test data must not be used to recompute `mu` or `s`.
For an untransformed numeric predictor, apply the same recipe without the transformation: calculate its training mean and `sd`, then subtract and divide using those stored values. R’s `sd()` uses the sample standard deviation with denominator \(n-1\), matching the formulas above. If using `scale()`, inspect and save its centering and scaling attributes so that later data use exactly the training parameters; do not call it independently on the test set.
A concise preprocessing record should name the variable, transformation, training mean and standard deviation where applicable, any zero indicator, bin cut points, and interval closure. Also record the permitted input domain and what happens to invalid or missing values. Binning cut points specified in advance from subject-matter requirements are not estimated from the outcomes; data-driven cut points must be selected without using held-out data.
Confirm whether zeros and negatives are possible, identify units, and check for invalid values before transforming.
Use a justified raw, transformed, or binned form. For meaningful zeros, consider whether a positivity indicator should accompany a transformed amount.
Calculate every data-dependent mean, standard deviation, or cut point using training observations only; check that any scale parameter is usable.
Reuse the stored parameters and exact bin boundaries on validation, test, and future data. Record the choices so the transformation can be reproduced.
Common Mistakes and Practical Checks
- Taking a logarithm of zero. The natural log requires a positive input. Use a justified zero-compatible transformation such as \(\ln(1+x)\) for nonnegative data, or use another clearly specified approach.
- Refitting preprocessing on test data. This changes the transformation relative to the training process and leaks information. Store and reuse training parameters.
- Assuming standardization fixes skewness. Centering and scaling change location and scale, not the relative shape of a distribution. Choose a transformation separately if skewness is a concern.
- Leaving bin boundaries vague. Values on a cut point must go somewhere. Specify both the cut points and whether intervals include their left or right endpoint.
- Assuming every predictor needs every operation. A transformation, standardization, and binning each have a purpose and a cost. Apply only the steps justified for the predictor and modeling task.
A planned workflow makes preprocessing part of the model specification rather than an undocumented sequence of edits. Clear choices also make it possible to reproduce predictions and explain what information a transformed or binned predictor retains. This workflow is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each scenario, identify the appropriate calculation or preprocessing decision and check your result against the final answer.
- A strictly positive training predictor is \(2,8,32\). Apply the natural logarithm and calculate the transformed training mean and sample standard deviation to four decimals. (Answer: mean \(=2.0794\); sample standard deviation \(=1.3863\).)
- A nonnegative training predictor is \(0,0,3\). Apply \(\ln(1+x)\), then calculate the transformed training mean and sample standard deviation to four decimals. (Answer: mean \(=0.4621\); sample standard deviation \(=0.8004\).)
- A raw predictor has training values \(10,20,30\). Using the training mean and sample standard deviation, standardize a test value of 40. Show the training parameters before giving the standardized value. (Answer: training mean \(=20\), sample standard deviation \(=10\), test standardized value \(=2\).)
- In R, bins are created using `cut(x, breaks = c(-Inf, 5, 10, Inf), right = FALSE)`. Which bins contain values 5 and 10? State the intervals. (Answer: 5 is in \([5,10)\); 10 is in \([10,\infty)\).)
- A nonnegative training predictor has values \(0,1,3\), and a future observation is 7. Use `log1p` and training-only standardization; also state the future positivity-indicator value. (Answer: transformed training mean \(=\ln(2)\), sample standard deviation \(=\ln(2)\), standardized future value \(=2\), positivity indicator \(=1\).)