Why Direction of Change Matters
Subject knowledge may tell us that a mean response should keep increasing or keep decreasing across the predictor range of interest. For example, a larger cumulative exposure may plausibly correspond to a higher mean response throughout the observed range. That knowledge does not determine the exact curve: the increase might be nearly constant, slow down, or speed up. It does, however, provide a useful constraint when comparing candidate regression forms.
A fitted curve that bends is not necessarily a problem. A curve can change steepness while continuing in the same direction. The relevant question is whether the estimated mean response actually reverses direction, and, if it does, whether the data provide convincing support for that reversal in the region where it occurs.
The range matters. A curve may be increasing over the observed data range but turn outside it; that does not make the curve nonmonotone on the range being studied. Conversely, a curve that reverses slightly near a boundary is still nonmonotone there, even if its overall trend is strong. The response itself need not increase for every observation: monotonicity here describes the conditional mean, not the individual data points.
Checking a Candidate Curve
For a differentiable fitted mean curve \(\hat m(x)\), its derivative describes its instantaneous direction of change. If \(\hat m'(x)\geq 0\) throughout the range, the fitted curve is nondecreasing there; if \(\hat m'(x)\leq 0\) throughout, it is nonincreasing. A derivative that changes sign indicates a reversal. For a curve without a convenient derivative, compare fitted values across ordered predictor values, or check the slopes of its segments.
This check identifies the shape of the fitted curve; it does not establish that the population mean has that shape. Coefficients and plots are estimates based on a finite sample. A small fitted reversal may reflect sampling variability, especially where observations are sparse. Assessing whether it is supported therefore also requires attention to the data near the reversal, residual behavior, and predictive performance. A curve’s overall fit statistics alone cannot show that every local feature is reliable.
As in ordinary regression, a candidate model describes the conditional mean and leaves deviations around it to residual variation. For the usual least-squares comparisons, assume the observations are independent, the errors have mean zero conditional on the predictor, and the candidate mean is evaluated on the intended predictor range. Constant error variance is needed for standard least-squares uncertainty calculations. Normal errors are an additional assumption when using small-sample normal-theory inference; they are not required simply to inspect fitted values or calculate squared prediction errors.
Worked Example: Curvature Without a Reversal
Suppose a hypothetical experiment records a continuous response \(Y\) at predictor values from 0 to 6. Subject knowledge suggests that the mean response should be nondecreasing on this range. Two least-squares fits to the hypothetical data are
The quadratic coefficient is negative, so the quadratic fit bends downward. But bending downward is not the same as decreasing. Differentiate the fitted quadratic:
On the stated range, the smallest derivative is at \(x=6\): \(1.60-0.16(6)=0.64\). Since this is positive, the fitted quadratic is increasing throughout 0 to 6. For example, its fitted values at the endpoints are \(\hat m_Q(0)=4.00\) and \(\hat m_Q(6)=4.00+9.60-2.88=10.72\). The linear fit is also increasing, with slope 1.20.
Suppose the quadratic has lower validation mean squared error than the line, 0.92 rather than 1.15 outcome-units squared, while residual plots show no remaining systematic pattern. These results favor the curved form for prediction among these candidates, without conflicting with the known direction of change. The conclusion is not that the mean rises by a constant amount: the quadratic’s increase slows across the range.
Comparing Monotone and Unconstrained Candidates
A candidate form can encode monotonicity directly, or it can allow the fitted curve to reverse. Examples of monotone candidates include a line with a slope constrained to the known direction, a suitably chosen transformation, or a constrained spline whose fitted segments cannot run against that direction. Unconstrained polynomials and splines can be useful candidates too, but their flexibility means that the estimated curve should be checked for reversals rather than assumed to respect subject knowledge.
Compare candidates on the same response scale and over the same predictor range. Consider the shape, residual structure, and out-of-sample performance together; use the prediction loss that matches the intended task. A monotone constraint may prevent implausible fitted reversals, but it can also prevent the model from representing a real reversal if the subject-matter premise is wrong. Thus, the premise should be stated and its range made explicit. A close match to the training data alone is not evidence that every local bend is reproducible.
Worked Example: Is a Small Reversal Near the Boundary Supported?
A hypothetical dose-response analysis has observations across doses from 0 to 8 units, with relatively few observations at doses 7 and 8. Subject knowledge suggests a nondecreasing mean response over this measured range. An unconstrained quadratic fit is
Its derivative is \(3.00-0.44x\), which is zero at \(x=3.00/0.44=6.82\) and negative for larger \(x\). Therefore, the fitted curve reverses within the observed range. The fitted values at doses 6, 7, and 8 are \(5+18-7.92=15.08\), \(5+21-10.78=15.22\), and \(5+24-14.08=14.92\). The fitted drop from dose 7 to dose 8 is \(14.92-15.22=-0.30\) response units.
Now suppose the hypothetical observed group means at those doses are 15.0, 15.2, and 15.1, respectively, and a monotone constrained candidate predicts 15.00, 15.10, and 15.20. The observed means do not show a clear decline at the boundary. In a prespecified validation set, suppose the quadratic’s mean squared error is 0.84 and the constrained candidate’s is 0.82. The difference, \(0.84-0.82=0.02\) outcome-units squared, is small and favors neither curve decisively without considering the variability of the validation result.
Here the quadratic’s fitted reversal is weak: it is small in magnitude, located where data are sparse, and does not improve the stated validation score. The evidence does not justify presenting that decline as a well-supported feature. Given the stated subject knowledge, the monotone candidate is a defensible choice, while the limited boundary data should be acknowledged. This is not proof that the true mean cannot decline; it is a judgment about what this data set supports.
Checking Flexible Fits Carefully
For a flexible fitted curve, monotonicity can be checked over a grid of predictor values covering the range of interest. Calculate consecutive fitted differences,
If every \(\Delta_j\geq 0\), the fit is nondecreasing at the checked grid points; if every \(\Delta_j\leq 0\), it is nonincreasing at those points. This grid check is useful for locating a possible reversal, but it does not by itself guarantee monotonicity between grid points. For a differentiable form, inspect its derivative analytically where feasible. For a piecewise-linear form, inspect the slope of each segment. For other flexible forms, use an adequately fine grid and, if necessary, a direct shape constraint.
Worked Example: Comparing an Unconstrained Fit With a Monotone Fit
Suppose a hypothetical study records mean response over predictor values from 0 to 5. The scientific premise is that the mean should increase across this range. At four consecutive grid points, an unconstrained flexible fit gives fitted values 2.0, 3.1, 3.0, and 4.2. A monotone constrained fit gives 2.1, 3.0, 3.2, and 4.0.
For the unconstrained fit, the consecutive changes are \(3.1-2.0=1.1\), \(3.0-3.1=-0.1\), and \(4.2-3.0=1.2\) response units. The negative change flags a fitted reversal between the second and third grid points. For the constrained fit, the changes are \(3.0-2.1=0.9\), \(3.2-3.0=0.2\), and \(4.0-3.2=0.8\); all are nonnegative, as required by the constraint.
Suppose the unconstrained fit has training residual sum of squares 8.4, compared with 8.7 for the monotone fit, but their validation mean squared errors are 1.06 and 1.02, respectively. The unconstrained fit’s training advantage is \(8.7-8.4=0.3\) squared response units, while its validation error is \(1.06-1.02=0.04\) squared response units higher. The tiny reversal does not correspond to an improvement in validation performance. In light of the premise, the monotone curve is preferable among these two candidates, though its shape and performance should not be treated as established beyond the observed range.
A Short Justification for the Derivative Check
The derivative criterion follows from the definition of monotonicity. If a differentiable function has \(m'(x)\geq0\) throughout an interval, then for any \(x_1<x_2\) in that interval, the mean value theorem gives a point \(c\) between them such that
The final inequality holds because both \(m'(c)\geq0\) and \(x_2-x_1>0\). Thus \(m(x_1)\leq m(x_2)\), which is precisely nondecreasing behavior. The corresponding argument with \(m'(x)\leq0\) establishes nonincreasing behavior. A derivative changing sign warns of a possible reversal, but checking the derivative’s sign over the entire range is what determines whether the curve is monotone there.
Common Mistake: Treating Every Fitted Reversal as a Discovery
A polynomial or flexible curve may turn because its fitted shape is estimated from noisy data. The wrong reasoning is: “The derivative becomes negative, so the mean response truly declines at high values.” The derivative only establishes that the estimated curve declines there. To assess whether that feature is supported, examine the observations and their density in that region, compare plausible monotone candidates, and check whether the reversal improves performance on data not used to fit the curve. A small training-fit improvement is not enough to establish a real reversal.
The opposite mistake is also possible: forcing a curve to be monotone solely because that direction seems plausible, without checking whether the premise is appropriate over the chosen range. State the scientific premise, define the range, and report how the candidate fits behave. If evidence for a reversal is ambiguous, describe that uncertainty rather than declaring either a genuine turning feature or an absolute absence of one.
Check Your Understanding
Use the definitions and checks in this tutorial to answer the following questions.
- A differentiable fitted curve has derivative \(0.8-0.1x\) over \(0\leq x\leq 6\). Is it nondecreasing throughout this range? Show the minimum derivative.
- Explain why a downward-bending curve can still represent an increasing mean relationship.
- A flexible fit decreases slightly between two grid points where only a few observations were collected. Name two checks that would help assess whether that reversal is supported.
- Why does a lower training residual sum of squares not by itself show that a fitted reversal is useful?
- What does a monotone constraint guarantee about a fitted mean curve, and what does it not establish about the true population mean?