Why Knot Placement and Smoothing Matter
A spline lets a rating model represent a nonlinear relationship between a continuous predictor and expected loss or claim frequency. Its flexibility is useful only when the data support the detail it can express. Too few knots may miss a credible change in the rating pattern; too many, or poorly placed knots, may let a small and noisy part of the portfolio dictate a local effect.
The earlier material on spline bases introduced basis functions as columns in a model matrix. Here the practical question is how to choose where those functions can change the fitted curve, and how much curvature to allow. The aim is not to make a fitted curve follow every fluctuation. It is to estimate a pattern that is supported by experience and useful for rating.
How Knot Location Affects the Fitted Shape
Consider a log-link frequency model with exposure \(e_i\) and a continuous rating predictor \(x_i\):
Here \(N_i\) is the claim count, \(\lambda_i\) is expected frequency per unit of exposure, and the logarithm of exposure is an offset. The function \(f\) is represented by spline basis terms. A knot gives the basis additional capacity to change the curve's behavior around that predictor value. It does not require the fitted curve to turn there: the estimated coefficients may make the curve quite smooth through the knot.
For example, a cubic truncated-power basis includes terms of the form \((x-k)_+^3\), where \((x-k)_+=\max(x-k,0)\). This term is zero for \(x\leq k\) and grows as \(x\) moves above \(k\). Moving \(k\) changes which observations can contribute to that basis term and how strongly they contribute. Other spline bases encode the same broad idea differently; the practical implication remains that knot locations affect where the model can adapt.
Knots are usually placed inside the observed predictor range, with boundary behavior handled by the chosen spline construction. In practice, candidate interior knots may be based on exposure-weighted quantiles so that the model has flexibility where a meaningful amount of risk is observed. Equally spaced knots can be reasonable when exposure is spread fairly evenly across the predictor range. Neither convention is automatically best: the observed exposure distribution and the purpose of the rating factor matter.
Worked Example: What Moving a Knot Changes
A hypothetical insurer models claim frequency using the Poisson log-link model above. Let \(r=x/10\), use knots at \(r=3\) and \(r=5\), and define \(B_k(r)=(r-k)_+^3\). For illustration, suppose the fitted linear predictor is \(\eta=-3+0.08r-0.02B_3(r)+0.03B_5(r)\). Calculate its spline terms and linear predictor at \(x=20,40,60\). This is a basis-calculation example: no interest rate or mortality assumption applies, and the Poisson mean is exposure times the modeled frequency.
At \(x=20\): \(r=20/10=2\). Then \(B_3=(2-3)_+^3=0\) and \(B_5=(2-5)_+^3=0\). Thus \(\eta=-3+0.08(2)-0.02(0)+0.03(0)=-3+0.16=-2.84\).
At \(x=40\): \(r=4\). Then \(B_3=(4-3)^3=1\) and \(B_5=(4-5)_+^3=0\). Thus \(\eta=-3+0.08(4)-0.02(1)+0.03(0)=-3+0.32-0.02=-2.70\).
At \(x=60\): \(r=6\). Then \(B_3=(6-3)^3=27\) and \(B_5=(6-5)^3=1\). Thus \(\eta=-3+0.08(6)-0.02(27)+0.03(1)=-3+0.48-0.54+0.03=-3.03\).
The checked linear predictors are \(-2.84,-2.70,-3.03\). The basis terms change their contribution as \(r\) passes each knot. Moving a knot would change the basis values and therefore the fitted curve for the same coefficients; refitting would also generally change the coefficients. This example avoids a common mistake: treating a knot as a required turning point rather than as a location where the model gains flexibility.
Placing Knots Where Experience Can Support Them
Exposure matters because observations with little exposure generally provide less information about expected frequency. Placing knots according to the raw number of records can be misleading when records have very different exposures. An exposure-weighted quantile places a candidate knot where a selected fraction of total exposure lies at or below that predictor value. Ties can make several quantiles coincide; in that case, using every suggested knot may add little or no useful flexibility.
Quantile placement is a starting point, not a rule. Review the amount of exposure on both sides of each proposed knot, the number of distinct predictor values, and whether a local feature is credible rather than an isolated fluctuation. When the tails have little experience, a knot near an extreme can allow an unstable tail pattern. A knot in a dense region can be better supported, although too many closely spaced knots may still create redundant or weakly estimated effects.
Worked Example: Exposure-Weighted Candidate Knots
A hypothetical portfolio has five predictor bands. The exposure weights are 10, 20, 40, 20, and 10 units at \(x=10,20,30,40,50\), respectively. The model is a Poisson frequency GLM with mean count \(e_j\lambda_j\) and a log exposure offset. No interest or mortality assumption applies. Find the 25th, 50th, and 75th percentile predictor values using the inverse cumulative exposure distribution, and decide what these candidates imply.
Step 1: Total exposure. The total is \(10+20+40+20+10=100\) units. The target cumulative exposures are \(0.25(100)=25\), \(0.50(100)=50\), and \(0.75(100)=75\).
Step 2: Cumulative exposure. Through \(x=10\), cumulative exposure is 10. Through \(x=20\), it is \(10+20=30\). Through \(x=30\), it is \(30+40=70\). Through \(x=40\), it is \(70+20=90\). Through \(x=50\), it is 100.
Step 3: Read off the quantiles. The first cumulative value at least 25 is 30, at \(x=20\), so the 25th percentile is 20. The first cumulative value at least 50 is 70, at \(x=30\), so the median is 30. The first cumulative value at least 75 is 90, at \(x=40\), so the 75th percentile is 40. The proposed interior knots are therefore 20, 30, and 40.
Check and interpretation. The exposure below or at the selected values is respectively 30, 70, and 90 units, each meeting its target of 25, 50, and 75 units. The candidates distribute flexibility across the central exposure rather than adding a knot at either extreme. With only five bands, three interior knots may be too flexible for a reliable fit; these values are candidates to assess, not an automatic prescription. The common error is to treat quantiles as guaranteed optimal knots, without checking how many distinct predictor values and how much information support the resulting basis.
More Knots Versus More Smoothing
Increasing the number of knots expands the set of curves the model can represent. A smoothing penalty instead allows a flexible basis but charges the model for excessive curvature. One common conceptual form is to minimize the model deviance plus a penalty proportional to integrated squared curvature:
Here \(D\) measures lack of fit, \(J(f)\) measures curvature, and \(\lambda\) controls the trade-off. The exact deviance scaling and penalty implementation depend on the fitting method. A larger \(\lambda\) favors less curvature; at \(\lambda=0\), this penalty imposes no restriction. For the stated curvature penalty, a function with \(J(f)=0\) has \(f''(x)=0\) throughout the domain and is linear there. In many spline implementations, the linear component is consequently unpenalized.
The reason for the trade-off follows directly from minimization. If a candidate curve reduces deviance by allowing a local bend but increases \(J(f)\), it is preferred only when the deviance improvement exceeds the added penalty. At an interior optimum, differentiating the objective with respect to the coefficients gives the first-order condition \(\nabla D+\lambda\nabla J=0\). As \(\lambda\) increases, the curvature contribution to that condition becomes more influential, so the selected curve generally bends less. This does not mean every coefficient shrinks by the same amount or that every fitted value must move in the same direction.
The number of knots and the penalty should be considered together. A large basis with smoothing can offer flexibility while controlling roughness. A small basis may be too restrictive even with no penalty. Candidate settings can be compared using a suitable validation approach, alongside stability and actuarial judgment. Validation should respect the data structure: for example, repeated observations for the same risk should not be split across training and validation sets if that would leak information.
Worked Example: Choosing Between Two Smoothing Levels
Suppose two spline fits to a hypothetical Poisson frequency portfolio are summarized by deviance \(D\) and curvature measure \(J\). Fit A has \(D=84\) and \(J=10\); Fit B has \(D=90\) and \(J=2\). Use the objective \(D+\lambda J\) to compare them at \(\lambda=0.5\) and \(\lambda=2\). The response model is Poisson with a log exposure offset; the stated deviance and curvature values are assumed outputs, not raw data. Interest and mortality assumptions do not apply.
At \(\lambda=0.5\): Fit A's objective is \(84+0.5(10)=84+5=89\). Fit B's objective is \(90+0.5(2)=90+1=91\). The smaller objective is 89, so Fit A is preferred under this criterion.
At \(\lambda=2\): Fit A's objective is \(84+2(10)=84+20=104\). Fit B's objective is \(90+2(2)=90+4=94\). The smaller objective is 94, so Fit B is preferred.
Check and interpretation. Fit A has lower deviance but more curvature; Fit B sacrifices 6 units of deviance while reducing curvature by 8 units. The difference in objective, A minus B, is \((84-90)+\lambda(10-2)=-6+8\lambda\), which is positive when \(\lambda>0.75\). Thus the preference changes at \(\lambda=0.75\), consistent with the two calculations. The common mistake is to choose the smallest deviance regardless of the penalty; the smoothing objective explicitly balances fit against roughness.
Diagnosing the Risk of Unstable Local Effects
A flexible spline can react strongly to a small amount of experience, particularly near a tail or around a knot supported by little exposure. An extreme observed frequency in a thin band is not, by itself, persuasive evidence of a sharp underlying rating change. The fitted curve borrows structure through its basis and any smoothing penalty, but a highly flexible specification can still produce pronounced local effects.
A useful review is to map exposure across the predictor range, examine how fitted relativities change under modest changes in knot placement or smoothing, and check whether local changes persist in validation data. This is a stability assessment, not a claim that one particular curve is correct. If a local indication changes substantially when one small portion of the data is removed, the data may not support that degree of local detail.
Worked Example: A Knot Near a Sparse Tail
A hypothetical insurer estimates claim frequency by age of equipment using a Poisson model with an exposure offset. Four bands have the following exposure and observed counts: ages near 20, exposure 500 and 20 claims; near 40, exposure 400 and 24 claims; near 60, exposure 90 and 8 claims; near 80, exposure 10 and 2 claims. Calculate each observed frequency per exposure unit and assess whether a new knot near the last band is well supported. No interest or mortality assumption applies; the conditional count distribution is Poisson.
Frequency calculations. For the first band, \(20/500=0.0400\). For the second, \(24/400=0.0600\). For the third, \(8/90=0.088888\ldots\), or 0.0889 to four decimals. For the final band, \(2/10=0.2000\). Thus the sequence of observed frequencies is \(0.0400,0.0600,0.0889,0.2000\).
Assess tail support. The last band's rate is based on only 10 exposure units and 2 claims. If its count were 1 instead, the observed rate would be \(1/10=0.1000\); if it were 3, the rate would be \(3/10=0.3000\). A one-claim change moves the observed frequency by 0.1000. By comparison, a one-claim change in the first band moves its frequency by \(1/500=0.0020\).
Conclusion and check. The rates are arithmetically correct, but the tail estimate is much more sensitive to a small count change. A knot near the final band could grant extra local flexibility where support is weak, so it should not be added simply because the observed frequency is high. Examine stability under alternative specifications and validation before accepting a tail effect. The trap is to read the 0.2000 rate as a precise indication without considering its small exposure denominator.
Practical Selection and Common Traps
Identify gaps, ties, concentration, and thin tails. Use exposure rather than row count when exposure varies materially.
Use a defensible number of knots and candidate locations that the available data can support. Quantiles are a starting design, not a proof of optimality.
Assess fit and curvature together, using a consistent objective or validation method. Do not choose a penalty based on deviance alone.
Review whether reasonable changes in knot locations or smoothing cause large local changes, especially in low-exposure regions.
Three common errors are worth distinguishing. First, equally spaced knots do not guarantee equal statistical support. Second, a knot is not a discontinuity or a required bend in the fitted curve. Third, adding knots and increasing the smoothing penalty are not equivalent operations: one expands the representable function space, while the other discourages roughness within a chosen basis. A well-supported rating curve balances flexibility with evidence, rather than maximizing either complexity or smoothness by default.
Check Your Understanding
Work through these original problems in order, checking both the calculations and the modeling interpretation.
- A portfolio has predictor values 10, 20, 30, and 40 with exposure weights 15, 35, 30, and 20. Using the inverse cumulative exposure definition, find the 50th percentile candidate knot. (Answer: 20)
- Let \(B_k(r)=(r-k)_+^3\). For \(r=5.5\) and knots at 2.5 and 4.5, calculate both basis values. (Answer: \(B_{2.5}=27\), \(B_{4.5}=1\))
- Two fits have \((D,J)=(72,8)\) and \((78,2)\). Under \(D+\lambda J\), which fit is preferred at \(\lambda=2\)? Show the two objectives. (Answer: the second fit; objectives are 88 and 82)
- A tail band has exposure 25 and 4 claims. Calculate its observed frequency, then recalculate it if one claim is removed. What does the difference suggest about placing a knot that isolates this band? (Answer: frequencies are \(4/25=0.1600\) and \(3/25=0.1200\); the 0.0400 change indicates sensitivity that warrants checking whether the local effect is supported.)
- Explain why a knot at the predictor value with the highest observed frequency is not automatically the best knot placement. (Answer: a high observed frequency may be driven by low exposure or random variation; knot placement should reflect support across the predictor range and be assessed for stability and validation performance.)