Why a Straight-Line Screen Can Miss a Useful Predictor
A numeric predictor can carry substantial information about an outcome even when its marginal relationship is not well described by a straight line. A predictor with a U-shaped relationship, for example, may be strongly associated with the outcome at both low and high values, while a linear summary reports little association. Conversely, a large Pearson correlation does not establish that the relationship is actually linear.
A marginal screen examines one predictor and the outcome at a time. Pearson correlation summarizes linear association, while rank correlation can summarize monotone association. Neither is a complete description of every possible pattern. This tutorial focuses on using plots and simple nonlinear summaries to notice when a straight-line screen may be inadequate. These are screening diagnostics, not evidence that a predictor has a causal effect or will add predictive value after other predictors are considered.
Start With the Shape of the Data
For a numeric outcome and numeric predictor, begin with a scatterplot of the observed pairs \((x_i,y_i)\). Look for systematic shape rather than judging association from a single number. A smooth curve can help reveal gradual curvature, but with a small sample it can also suggest patterns that are not well supported by the data. Grouping the predictor into a few ordered intervals and plotting the mean outcome in each group is another useful descriptive check.
Grouped means make the pattern easier to see, but they discard detail: the chosen cut points affect the display, and a group mean can conceal wide variation within that group. Show the number of observations in each group when possible. Do not treat a jagged sequence of small-group means as proof of a complicated relationship. The purpose is to identify a plausible departure from a straight-line description that merits further modeling.
A scatterplot also helps distinguish curvature from outliers, clusters, or changes in spread. A single extreme observation can materially affect correlation and fitted lines. A pattern that appears only because of one point should be investigated as a data-quality or population question before it is interpreted as a nonlinear association.
Worked Example: A U-Shape With Zero Pearson Correlation
A hypothetical analyst examines a numeric risk score \(X\) and a numeric service cost \(Y\), in hundreds of dollars, for five records. The observed pairs are \((-2,4),(-1,1),(0,0),(1,1),(2,4)\). Calculate Pearson correlation and compare a straight-line screen with a quadratic summary. The purpose is descriptive; no sampling inference is requested.
Method. Calculate the centered cross-product and sums of squares for Pearson correlation, then inspect the relationship with a quadratic model. This example avoids the mistake of treating zero correlation as proof of no association.
The predictor mean is \(\bar{x}=(-2-1+0+1+2)/5=0\). The outcome mean is \(\bar{y}=(4+1+0+1+4)/5=2\). The centered cross-product is \(S_{XY}=\sum (x_i-\bar{x})(y_i-\bar{y})\): \((-2)(2)+(-1)(-1)+(0)(-2)+(1)(-1)+(2)(2)=-4+1+0-1+4=0\).
The predictor sum of squares is \(S_{XX}=4+1+0+1+4=10\). The outcome sum of squares is \(S_{YY}=2^2+(-1)^2+(-2)^2+(-1)^2+2^2=14\). Therefore \(r=S_{XY}/\sqrt{S_{XX}S_{YY}}=0/\sqrt{140}=0\).
The scatterplot is nevertheless a clear U-shape: each observed outcome equals \(x^2\). The straight-line least-squares fit has slope \(S_{XY}/S_{XX}=0/10=0\) and intercept \(\bar{y}=2\), so it predicts 2 for every \(x\). Its residual sum of squares is \(14\), equal to the total sum of squares, and its \(R^2\) is \(0\).
The quadratic model \(Y=\beta_0+\beta_1X+\beta_2X^2+\varepsilon\) fits with \(\hat{\beta}_0=0\), \(\hat{\beta}_1=0\), and \(\hat{\beta}_2=1\). Every fitted value equals the observed value, so the residual sum of squares is 0 and the descriptive \(R^2\) is 1. This exact fit is a feature of the five constructed observations, not a guarantee of perfect prediction in new data.
Check. The zero Pearson correlation follows from the symmetry: the positive and negative \(x\) values cancel in the centered cross-product. The U-shaped plot and quadratic fit show why the predictor cannot be dismissed on the basis of Pearson correlation alone.
Simple Nonlinear Summaries
One practical extension of a straight-line screen is a quadratic regression. For a numeric outcome, the model is \(Y=\beta_0+\beta_1X+\beta_2X^2+\varepsilon\). Here \(\beta_2\) allows the fitted mean to bend. The fitted curve is still a simplified description: it can represent a U-shape or an inverted U-shape, but it is not appropriate for every nonlinear pattern.
For descriptive least-squares fitting, the coefficients are chosen to minimize the residual sum of squares. The common \(R^2\) summary is the proportion of the outcome’s total variation around its sample mean accounted for by fitted values in that data set. In a nested comparison, the quadratic model includes the linear model as the special case \(\beta_2=0\); its training residual sum of squares therefore cannot be larger. An improvement in training fit alone does not establish that the curve is reliable or useful for prediction.
The quadratic term depends on the scale and origin of \(X\). Centering \(X\) can make the coefficients easier to interpret and reduce numerical difficulties, but it does not change the fitted values when the model contains an intercept, linear term, and squared term. A quadratic coefficient is not a universal measure of “nonlinearity”; it describes curvature under this particular model and scale.
Why the Linear Fit and Pearson Correlation Agree
With an intercept and one predictor, the least-squares straight-line fit has slope \(S_{XY}/S_{XX}\). Its fitted deviations from the outcome mean are proportional to the centered predictor values. This explains why Pearson correlation and the linear fit summarize the same marginal linear signal in this setting.
For an intercept model, \(\hat{y}_i=\bar{y}+\hat{\beta}_1(x_i-\bar{x})\), where \(\hat{\beta}_1=S_{XY}/S_{XX}\).
The sum of squared fitted deviations is \(\sum(\hat{y}_i-\bar{y})^2=\hat{\beta}_1^2S_{XX}=S_{XY}^2/S_{XX}\).
Since \(r^2=S_{XY}^2/(S_{XX}S_{YY})\), the linear model’s \(R^2\), which is explained variation divided by \(S_{YY}\), equals \(r^2\).
Thus, a small Pearson correlation corresponds to a small \(R^2\) for the simple straight-line fit with an intercept. It says that the linear fit explains little variation in the observed outcome. It does not say that a quadratic, threshold, or other nonlinear pattern is absent.
Worked Example: A Strong Linear Screen Can Still Hide Curvature
A hypothetical analyst records a predictor \(X\) and a numeric outcome \(Y\) for four units. The pairs are \((0,0),(1,1),(2,4),(3,9)\). Compare the Pearson correlation and straight-line fit with the quadratic pattern. This example avoids the mistake of interpreting a large correlation as proof of linearity.
Method. Compute centered sums for Pearson correlation and the straight-line regression, then check whether a quadratic expression exactly describes these observations.
The means are \(\bar{x}=(0+1+2+3)/4=1.5\) and \(\bar{y}=(0+1+4+9)/4=3.5\). The centered predictor values are \(-1.5,-0.5,0.5,1.5\), and the centered outcome values are \(-3.5,-2.5,0.5,5.5\). Therefore \(S_{XY}=5.25+1.25+0.25+8.25=15\), \(S_{XX}=2.25+0.25+0.25+2.25=5\), and \(S_{YY}=12.25+6.25+0.25+30.25=49\).
Pearson correlation is \(r=15/\sqrt{5\cdot49}=15/\sqrt{245}\approx0.9578\). The straight-line slope is \(15/5=3\), and the intercept is \(3.5-(3)(1.5)=-1\). Its fitted values at \(x=0,1,2,3\) are \(-1,2,5,8\), giving residuals \(1,-1,-1,1\). Thus, \(\operatorname{SSE}=1+1+1+1=4\), and \(R^2=1-4/49=45/49\approx0.9184\).
The observations also satisfy \(y=x^2\), so the quadratic fit is exact and has \(\operatorname{SSE}=0\). The correlation and linear \(R^2\) are both large, but the relationship is not a straight line. With so few observations, these calculations describe the observed data; they do not establish that the quadratic form will continue beyond the observed range.
Check. The predictor is strongly associated with the outcome, but the scatterplot and zero-residual quadratic fit reveal curvature. A high linear screen can identify association while still giving an incomplete account of its shape.
Grouped Means Can Reveal a Threshold
Not every departure from a straight line is smoothly curved. An outcome may remain approximately level over a range of predictor values and then change. A quadratic model can sometimes approximate such a pattern, but a plot of grouped means may make the threshold-like shape more apparent. In a screening context, describe the observed pattern rather than asserting that a precise threshold exists.
Worked Example: Spotting a Step Pattern That a Line Smooths Over
For a hypothetical operational study, a numeric predictor \(X\) takes the values 0, 1, 2, 3, and 4. The corresponding outcome values are 2, 2, 2, 8, and 8. Calculate the straight-line fit and use the ordered observations to assess whether a line adequately describes the pattern. This example avoids the mistake of treating a reasonably large correlation as a complete description.
Method. Calculate the sample means and centered sums to obtain the linear fit, then compare its residuals with the observed sequence.
The means are \(\bar{x}=(0+1+2+3+4)/5=2\) and \(\bar{y}=(2+2+2+8+8)/5=22/5=4.4\). The centered \(x\) values are \(-2,-1,0,1,2\), and the centered \(y\) values are \(-2.4,-2.4,-2.4,3.6,3.6\). Hence \(S_{XY}=4.8+2.4+0+3.6+7.2=18\), \(S_{XX}=4+1+0+1+4=10\), and \(S_{YY}=3(2.4^2)+2(3.6^2)=17.28+25.92=43.2\).
The line has slope \(18/10=1.8\) and intercept \(4.4-(1.8)(2)=0.8\). Its fitted values are \(0.8,2.6,4.4,6.2,8.0\). The residuals are \(1.2,-0.6,-2.4,1.8,0\), so \(\operatorname{SSE}=1.44+0.36+5.76+3.24+0=10.8\). Thus \(R^2=1-10.8/43.2=0.75\). Pearson correlation is positive, \(r=18/\sqrt{10\cdot43.2}\approx0.8660\).
The ordered outcomes, \(2,2,2,8,8\), show a flat section followed by a jump. The line reports a positive association but smooths the change across the entire predictor range. Because there is only one observation at each \(x\), this small constructed example cannot establish how much within-group variability exists or where a real threshold lies.
Check. The positive correlation is compatible with a step pattern, not just a steadily increasing line. A plot or ordered group summary adds shape information that the correlation does not contain.
Applying the Diagnostics Carefully
In R, a scatterplot can be created with a plotting function such as plot using the outcome against the predictor. A quadratic fit can be specified with a formula of the form outcome ~ predictor + I(predictor^2); the I() ensures the square is evaluated as arithmetic. Compare its fitted curve and residual pattern with the straight-line fit. A smooth added to a scatterplot can be a useful visual guide, but its appearance depends on the smoothing choices and should not be treated as a formal finding.
For a binary outcome, the same diagnostic principle applies to observed event proportions across ordered predictor groups: plot the proportions and the group sizes. If modeling the probability directly, a quadratic term can be included in a logistic regression, whose fitted probabilities remain between 0 and 1. A quadratic term on the log-odds scale does not imply a quadratic curve in probability; the link function changes the shape. Avoid applying ordinary least squares to binary outcomes as if they were unrestricted continuous measurements.
A marginal relationship can also be altered or obscured by other predictors. The diagnostics here describe one predictor at a time, so they do not determine conditional importance, causation, or the final form of a predictive model. They are useful for recognizing when a straight-line screen may be too narrow and for identifying patterns that deserve careful follow-up.
Common Mistakes
- Rejecting a predictor because Pearson correlation is near zero. A symmetric U-shape can have zero linear correlation and still show a clear relationship. Inspect the plot and consider a simple nonlinear summary.
- Assuming a large correlation proves linearity. A curved, steadily increasing pattern can have a high Pearson correlation. Check whether the fitted line leaves a systematic residual pattern.
- Interpreting a better quadratic fit as proof of predictive improvement. A quadratic model cannot have worse in-sample least-squares fit than the nested straight-line model, but this fact alone says nothing about performance on new observations.
- Overinterpreting grouped means. The bins are chosen by the analyst, and sparse bins can be unstable. Show group sizes and use the original scatterplot as context.
- Calling a marginal association causal or conditionally important. A one-predictor screen does not account for other variables or establish what would happen under an intervention.
Check Your Understanding
For each problem, identify what the linear summary does and what a simple nonlinear check adds.
- A predictor takes values \(-3,-1,1,3\), and the outcomes are \(9,1,1,9\). What is the Pearson correlation, and what simple model fits the four points exactly? (Answer: \(r=0\); \(Y=X^2\) fits exactly.)
- For \(X=0,1,2,3\), the outcomes are \(1,2,5,10\). Calculate Pearson correlation using \(S_{XY}=15\), \(S_{XX}=5\), and \(S_{YY}=49\), then state the exact quadratic pattern. (Answer: \(r=15/\sqrt{245}\approx0.9578\); \(Y=X^2+1\) fits exactly.)
- A plot of grouped outcome means across five ordered predictor groups is \(3,3,3,7,7\). Describe the shape and name one reason not to claim an exact threshold from this summary alone. (Answer: A plateau followed by a jump; group sizes, within-group variation, or the chosen cut points may make the apparent threshold uncertain.)
- A quadratic regression has a smaller training SSE than a straight-line regression on the same data. Does this alone show that the quadratic model predicts better on new data? (Answer: No; the quadratic model contains the line as a special case, so its training SSE cannot be larger, but the decrease alone does not establish better prediction on new data.)
- A numeric predictor has \(r=0.03\) with a numeric outcome, while the scatterplot shows low outcomes near the middle of the predictor range and higher outcomes at both extremes. Should it be screened out based only on \(r\), and what simple summary could check the pattern? (Answer: No; inspect the scatterplot and assess a quadratic fit or ordered grouped means.)