When a Curve Levels Off, Is It Turning?
A relationship that rises quickly and then changes more slowly can look like it is approaching a ceiling. But a curve that bends downward may also flatten over the observed range before eventually reaching a maximum and declining. These shapes can be hard to distinguish when data cover only a limited range of the predictor.
The distinction matters because the two explanations imply different behavior. A saturating relationship approaches a limiting level; a polynomial with a maximum eventually reverses direction. A fitted curve can also suggest a turning point that is outside the observed range, or so close to its edge that the data show little more than leveling. The curve’s shape, the location of any turning point, and the observations around it all deserve attention.
Two Different Shape Claims
Write the conditional mean response at predictor value \(x\) as \(m(x)=E(Y\mid X=x)\). A regression curve describes this mean; individual responses can vary around it. For comparing shapes, suppose observations follow \(Y_i=m(x_i)+\varepsilon_i\), where the errors have mean zero conditional on the predictor values. If we compare candidate fits using ordinary sums of squared residuals, we also assume independent errors with a common variance and use the same observations and outcome scale for both candidates. These assumptions make that particular comparison interpretable; they do not establish that either curve is the true relationship.
One common increasing saturation model is \(m(x)=a+Lx/(K+x)\), for \(x\geq 0\), \(L>0\), and \(K>0\). It starts at \(a\), increases, and approaches \(a+L\) as \(x\) grows. Another is an exponential approach to a plateau, \(m(x)=a+L(1-e^{-kx})\), with \(L>0\) and \(k>0\). These are examples, not universal models: subject knowledge and the observed pattern help determine which shapes are reasonable.
A quadratic curve has the form \(m(x)=\beta_0+\beta_1x+\beta_2x^2\). When \(\beta_2<0\), it has a maximum and bends downward; when \(\beta_2>0\), it has a minimum and bends upward. Unlike a saturating curve, a quadratic does not approach a horizontal plateau as \(x\) grows. It continues along its parabolic path and eventually moves away from its turning point.
Finding the Turning Point of a Quadratic
For \(m(x)=\beta_0+\beta_1x+\beta_2x^2\), the slope at \(x\) is \(m'(x)=\beta_1+2\beta_2x\). At a turning point, this slope is zero. Solving gives \(x^*=-\beta_1/(2\beta_2)\), provided \(\beta_2\neq 0\). The second derivative is \(m''(x)=2\beta_2\): a negative value means the curve bends down and the turning point is a maximum; a positive value means it bends up and the point is a minimum.
The same result can be seen by completing the square:
The squared term is zero at \(x=-\beta_1/(2\beta_2)\). If \(\beta_2<0\), multiplying the squared term by \(\beta_2\) makes it nonpositive, so this location gives the maximum. If \(\beta_2>0\), it makes the term nonnegative, so the location gives the minimum. This establishes both the turning-point formula and its direction.
A calculated turning point is a feature of the fitted equation. To judge its relevance to the data, compare it with the observed predictor range, from \(x_{\min}\) to \(x_{\max}\). If it falls outside that range, the fitted curve is monotone throughout the observed range, even though it may turn elsewhere. If it falls inside, the observations should be examined on both sides of it. Merely being inside the range does not establish that the turn is well supported.
Worked Example: A Quadratic That Is Still Rising in the Data Range
Suppose a hypothetical data set has predictor values \(x=0,2,4,6,8\), and consider two candidate mean curves: \(f_q(x)=4+8x-0.4x^2\) and \(f_s(x)=4+60x/(10+x)\). The quadratic might eventually turn downward, while the second curve approaches a plateau. First check what each curve predicts over the observed range.
For the quadratic, the turning point is \(x^*=-8/[2(-0.4)]=10\). Its slope is \(f_q'(x)=8-0.8x\), which is positive for every \(x\) from 0 to 8. Thus it is increasing throughout the observed range; its maximum at \(x=10\) is outside that range.
Substituting the observed predictor values into the quadratic gives:
The final value is \(42.40\): \(4+64-25.6=42.4\). For the saturation curve, the predictions are \(4.00,\;14.00,\;21.14,\;26.50,\;30.67\), respectively. For example, \(f_s(8)=4+60(8)/(10+8)=4+480/18=30.67\). The saturation curve increases by progressively smaller amounts and approaches 64.
For a concrete comparison, suppose the hypothetical observed responses are \(4,14,21,27,31\), in outcome units. The sum of squared residuals for the stated saturation curve is approximately \(0^2+0^2+0.143^2+(-0.500)^2+(-0.333)^2=0.382\). For the quadratic it is \(0^2+4.4^2+8.6^2+10.6^2+11.4^2=335.64\). On these observations, the saturation candidate describes the pattern much more closely. This comparison illustrates the shapes; it does not prove that the population relationship follows a saturation model. The quadratic is also still rising across the observed range, so it does not describe an observed turning point.
Look for Evidence on Both Sides of a Proposed Turn
A plausible turning point should make sense in context and be visible as a change in direction in the part of the predictor range where observations exist. For a proposed maximum, ask whether responses tend to rise before the location and fall after it. For a minimum, ask whether they tend to fall before and rise after. Data concentrated on one side of the proposed point cannot directly show both parts of that pattern.
The amount and distribution of data matter as well. A handful of observations at the high end, or a single unusual response, can strongly affect a quadratic’s curvature and vertex. Repeated coverage on both sides makes a reversal more credible as a description of the observed pattern, though it does not by itself resolve uncertainty about the underlying mean curve. A smooth-looking fitted line is not a substitute for checking the observations.
Worked Example: A Turning Point With Observations on Both Sides
Imagine a hypothetical experiment relating a controlled dose \(x\), measured in dose units, to a response measured in outcome units. A fitted quadratic is \(f(x)=20+6x-0.75x^2\), and observations are available at \(x=0,2,4,6,8\). The fitted turning point is \(x^*=-6/[2(-0.75)]=4\). Since the quadratic coefficient is negative, this is a maximum.
Its predictions are \(20,29,32,29,20\) at those five dose values. Suppose the observed responses are \(20,28,31,28,21\). The observations rise from 20 at dose 0 to 31 at dose 4, then fall to 21 at dose 8. They are lower on both sides of the center, and the observed predictor values extend on each side of the fitted maximum. Here, a peak is a plausible descriptive summary of the observed pattern; a curve that only approaches a higher plateau would not express that decline after dose 4.
This is not enough to conclude that the underlying mean response truly has a maximum at dose 4. The five observations may be noisy, and a different plausible curve could describe the pattern nearly as well. The example supports a cautious shape judgment, not a formal claim about how certain the turning point is. That judgment would require considering the variability and how well the pattern is determined by the data.
When Flattening Looks Like a Turn
A quadratic can have a turning point inside the observed range yet show only a tiny reversal near the boundary. Such a fit may capture flattening rather than a convincing decline. Check how far observations extend beyond the proposed turn and how much the fitted response changes there. A slight bend at the last one or two predictor values is weaker evidence of a meaningful reversal than a clear rise-and-fall pattern across a substantial part of the range.
Worked Example: A Boundary Turn That May Only Be Flattening
Consider the hypothetical quadratic \(g(x)=10+5x-0.6x^2\), fitted to observations with \(x\) between 0 and 4.5. Its maximum occurs at \(x^*=-5/[2(-0.6)]=4.167\), which is just inside the upper boundary. Its slope is \(g'(x)=5-1.2x\), so it is positive at \(x=4\), where the slope is \(0.2\), and negative at \(x=4.5\), where the slope is \(-0.4\).
The fitted responses at the top of the range are \(g(4)=10+20-9.6=20.40\), \(g(4.167)\approx20.42\), and \(g(4.5)=10+22.5-12.15=20.35\). The predicted drop from the maximum to the end of the observed range is only about \(0.07\) outcome units. Although the equation has an in-range turning point, the fitted reversal is very small and close to the boundary.
If there are few observations near the upper end, or their responses do not clearly fall, this pattern is better described cautiously as flattening with a quadratic vertex near the boundary than as a well-established peak. A saturating curve could plausibly describe the same visible leveling. The formula alone cannot decide which explanation is better supported.
A Practical Comparison
Compare candidates over the same observed range and on the same outcome scale. Overlay their predictions on the data, then ask what each model implies beyond the plotted points. A saturation curve approaches a limit under its stated parameter restrictions; a quadratic eventually turns and continues along a parabola. If the two candidates look similar in the observed range, their very different behavior elsewhere is not evidence that the data distinguish them there.
| Question | Saturating curve | Quadratic alternative |
|---|---|---|
| What is the shape claim? | Change diminishes toward a limiting level | Curvature can produce a maximum or minimum |
| Does it necessarily reverse direction? | No, not under the usual increasing or decreasing saturation restrictions | Yes, if the turning point is considered over a sufficiently broad range |
| What should be checked in the observed range? | Whether the apparent leveling is consistent with approaching a plateau | Whether the turning point is in range and observations show a reversal on both sides |
| What is a key limitation? | A plateau may not be distinguishable if observations do not extend far enough | A fitted vertex may be weakly informed, especially near an edge |
A candidate’s numerical fit is one part of the decision, not the entire decision. If using residual sums of squares, compare them on the same observations and outcome scale, and remember that a smaller value describes fit to those observations, not proof of the curve’s long-term shape. Subject knowledge can rule out implausible limits or reversals, while plots and observations help reveal whether the candidate captures the pattern that is actually present.
Common Mistake
Wrong reasoning: “The fitted quadratic has a maximum inside the observed range, so the data establish a real peak.” The vertex is calculated from the fitted coefficients, but those coefficients can be influenced by noise, sparse coverage, and observations near the range boundary.
Correction: Treat the vertex as a feature of the fitted curve, then check whether the observed responses show a rise before it and a decline after it, with meaningful coverage on both sides. If the reversal is tiny or confined to the edge, describe the data as showing leveling or a possible turn rather than claiming a well-supported peak. A curve’s mathematical shape and the strength of the data supporting that shape are different questions.
Check Your Understanding
Use the curve shape and observed predictor range to answer each question.
- A fitted quadratic has a maximum at \(x=12\), but observations cover only \(x=2\) to \(x=9\). Is the fitted relationship increasing or decreasing throughout the observed range? What can you say about whether the data show a turn?
- For \(m(x)=7+4x-0.5x^2\), calculate the turning point and identify whether it is a maximum or minimum.
- What observed pattern would make a fitted maximum more plausible than a saturating increase?
- A quadratic has a vertex just inside the highest observed predictor value, but the predicted response barely decreases afterward. Why should the reversal be described cautiously?
- Why does a smaller residual sum of squares for one candidate not establish that its implied behavior outside the observed range is correct?