Why Validate Demand Estimates Over Time?
A demand model is used to estimate how likely a customer is to choose or buy a product under specified conditions, including its offered price. Its fit to the data used to build it does not establish that its predictions will remain useful later. Customer composition, competitive conditions, distribution channels, and the way prices are assigned can change. An out-of-time validation tests predictions against outcomes that occurred after the model was developed.
The validation is informative only if the prediction is genuinely prospective in design: model form, fitted parameters, and data-processing decisions are fixed using the development period, and predictions are made for later-period cases without using their outcomes. The later data then provide evidence about forecast accuracy and response stability, not a guarantee that the relationship will persist indefinitely.
The prediction unit and outcome window must be consistent in development and validation. For example, if one prediction is for a single quote and the outcome is purchase within a specified period, the validation should not count multiple records for the same quote as independent purchases. Likewise, features must reflect information available at prediction time. A variable updated after the customer responds can leak outcome information into the prediction.
Calibration, Discrimination, and Probability Loss
These measures answer different questions. Calibration asks whether predicted probabilities agree with observed frequencies. Discrimination asks whether cases that purchase tend to receive higher scores than cases that do not. Probability-based losses measure the accuracy of individual probability forecasts and penalize overconfident errors.
A calibration table can group cases by predicted probability and compare each group’s mean prediction with its observed purchase rate. The group’s observed rate is an estimate, so small groups can show substantial random variation. A calibration plot is a visual version of the same comparison. A more formal calibration assessment may fit a logistic model to the validation outcomes:
Ideal calibration corresponds to \(\alpha=0\) and \(\beta=1\). A nonzero intercept indicates systematic underprediction or overprediction when the slope is held fixed; a slope below one commonly indicates predictions that are too extreme. The fitted coefficients are estimates and should be interpreted with uncertainty, especially when the validation sample is small. Probabilities of exactly zero or one require special handling before taking logits.
For discrimination, the area under the receiver operating characteristic curve (AUC) has a pairwise interpretation: it is the probability that a randomly selected purchaser has a higher score than a randomly selected nonpurchaser, with ties counted as one-half. An AUC near 0.5 indicates little ranking ability; a higher AUC indicates stronger ranking. AUC does not show whether a predicted probability of, say, 0.60 corresponds to a 60% purchase rate, and it can remain unchanged when probabilities are systematically shifted.
The Brier score is the mean squared error of probability forecasts. Log loss gives a particularly severe penalty to a confident forecast contradicted by the outcome. If a prediction is exactly zero or one and is wrong, its log loss is infinite; in practice, models should avoid impossible probabilities, or any probability clipping rule should be specified before comparing results.
Why the Brier Score Rewards Accurate Probabilities
Suppose a case has a true purchase probability \(q\), and the forecast is \(p\). The outcome is Bernoulli with probability \(q\), so the expected squared loss can be derived directly:
The term \(q(1-q)\) does not depend on the forecast. The remaining term, \((p-q)^2\), is nonnegative and is zero only when \(p=q\). Thus, in expectation, the Brier score is minimized by forecasting the true probability. This explains why it assesses probability accuracy rather than only whether the model chose the right class at an arbitrary cutoff.
Worked Example: Aggregate Calibration Can Hide Weak Ranking
A hypothetical insurer evaluates four later-period purchase forecasts. The forecast probabilities and outcomes are: \(p=0.80,Y=1\); \(p=0.70,Y=0\); \(p=0.40,Y=1\); and \(p=0.10,Y=0\). Calculate the overall calibration gap, Brier score, log loss, and AUC. Use natural logarithms and report losses to four decimals.
Method. Calculate probability calibration and proper scoring losses from the four forecasts, then compare each purchaser’s score with each nonpurchaser’s score for AUC.
Calibration. The mean forecast is \((0.80+0.70+0.40+0.10)/4=2.00/4=0.50\). There are two purchases, so the observed rate is \(2/4=0.50\). Therefore, \(\bar y-\bar p=0.50-0.50=0.00\).
Brier score. The four squared errors are \((1-0.80)^2=0.04\), \((0-0.70)^2=0.49\), \((1-0.40)^2=0.36\), and \((0-0.10)^2=0.01\). Their sum is \(0.90\), giving a Brier score of \(0.90/4=0.2250\).
Log loss. The losses are \(-\ln(0.80)=0.2231\), \(-\ln(0.30)=1.2040\), \(-\ln(0.40)=0.9163\), and \(-\ln(0.90)=0.1054\). Their sum is \(2.4488\); the mean is \(2.4488/4=0.6122\).
AUC. The purchasers have scores 0.80 and 0.40; the nonpurchasers have 0.70 and 0.10. Of the four purchaser–nonpurchaser pairs, 0.80 beats both scores, while 0.40 beats 0.10 but not 0.70. Thus AUC is \(3/4=0.7500\). Overall calibration is exact, but the score of 0.40 for a purchaser falls below the 0.70 score for a nonpurchaser, illustrating that aggregate calibration and discrimination are distinct.
Check. The observed and forecast totals are both \(2.00\), and all four losses are nonnegative. A common mistake is to infer strong ranking from matching overall averages; the pairwise AUC calculation shows why that inference is not justified.
Compare Segments Before Drawing Conclusions
An overall validation result can conceal a serious problem in one customer segment. For each segment, compare observed and predicted demand, and calculate scoring losses using the same outcome and forecast definitions. Segment comparisons should also account for sample size and uncertainty: a small group’s observed rate can vary substantially even when its underlying demand has not changed.
Price mix matters as well. If one period has more high-price offers than another, its overall purchase rate can fall even if demand at each price is unchanged. For a fair temporal comparison, compare like with like: use common price ranges and relevant customer characteristics, or standardize both periods to a common reference mix. If the later data have little or no overlap with the development data in price or customer characteristics, validation there requires extrapolation and is less dependable.
Worked Example: Segment-Level Calibration Reveals a Miss
A hypothetical demand model is evaluated on two equally sized customer segments. In each segment there are four quotes with forecasts \(0.20,0.40,0.60,0.80\). Segment Cedar has outcomes \(0,0,1,1\), while segment Flint has outcomes \(0,0,0,0\). For each segment, calculate mean predicted demand, observed demand, the calibration gap, and Brier score.
Method. Compute segment metrics separately rather than relying on a combined result. This prevents one group’s calibration from masking another’s.
Cedar. The mean prediction is \((0.20+0.40+0.60+0.80)/4=2.00/4=0.50\). The observed rate is \(2/4=0.50\), so the gap is \(0.50-0.50=0.00\). The squared errors are \(0.20^2=0.04\), \(0.40^2=0.16\), \((1-0.60)^2=0.16\), and \((1-0.80)^2=0.04\). Their mean is \(0.40/4=0.1000\).
Flint. The mean prediction is again \(0.50\), but the observed rate is \(0/4=0.00\), so the gap is \(0.00-0.50=-0.50\). The squared errors are \(0.20^2=0.04\), \(0.40^2=0.16\), \(0.60^2=0.36\), and \(0.80^2=0.64\). Their mean is \(1.20/4=0.3000\).
Combined check. Across both segments the mean forecast remains \(0.50\), while the observed rate is \(2/8=0.25\), for a gap of \(-0.25\). The combined Brier score is \((0.40+1.20)/8=0.2000\), between the two segment scores. The overall number alone would not identify Flint’s much larger overprediction. With only four cases in each segment, however, this pattern is a signal to investigate rather than conclusive evidence of a persistent segment effect.
Common mistake. Reporting only the pooled gap loses important information. The correction is to report segment metrics alongside the pooled result, and to state the sample sizes so that a striking but imprecise segment rate is not overinterpreted.
Test Price-Response Stability on Later Data
For price-sensitive demand, validation should examine more than the later period’s overall purchase rate. Compare predicted and observed demand across comparable prices and customer segments. A model can match overall demand while missing the way demand changes with price; conversely, an overall rate can shift because the validation sample contains a different mix of prices or customers.
The comparison is descriptive unless the price assignment supports a causal interpretation. If prices were assigned in response to expected demand or other customer characteristics, a change in observed purchase rates across prices may reflect those differences as well as price response. Earlier material on confounding and adjustment and instrumental variables addresses the conditions for causal price effects. For validation, the practical requirement is to make the compared periods as comparable as possible and avoid treating an unadjusted change as proof of a causal change.
Worked Example: Later-Period Drift at Comparable Prices
A hypothetical insurer freezes a demand model before a later period. In the later period, 100 comparable quotes at price 100 receive forecast probability 0.40, and 100 at price 120 receive forecast probability 0.25. There are 35 purchases at price 100 and 15 at price 120. Assume the quote mix within these price groups is comparable to the model’s intended validation population. Calculate calibration by price, overall calibration, the observed and predicted price contrasts, Brier score, and log loss. Report losses to four decimals.
Method. Use the group purchase rates to compare forecasts at matched prices, then weight each group equally because each contains 100 quotes. For grouped Brier and log loss, each quote in a group has the stated forecast probability.
Calibration by price. At price 100, observed demand is \(35/100=0.35\), so the gap is \(0.35-0.40=-0.05\). At price 120, observed demand is \(15/100=0.15\), so the gap is \(0.15-0.25=-0.10\).
Overall calibration. The mean forecast is \((100(0.40)+100(0.25))/200=(40+25)/200=0.325\). The observed rate is \((35+15)/200=50/200=0.250\). The overall gap is \(0.250-0.325=-0.075\), indicating overprediction by 7.5 percentage points in this validation sample.
Price contrast. The predicted decrease from price 100 to 120 is \(0.40-0.25=0.15\), or 15 percentage points. The observed decrease is \(0.35-0.15=0.20\), or 20 percentage points. The observed contrast is 5 percentage points larger in magnitude than the predicted contrast. This is evidence of a mismatch in the later-period pattern at these prices, not by itself proof that the causal price effect changed.
Brier score. For price 100, the mean squared error is \((35(1-0.40)^2+65(0-0.40)^2)/100=(35(0.36)+65(0.16))/100=(12.60+10.40)/100=0.2300\). For price 120 it is \((15(0.75)^2+85(0.25)^2)/100=(15(0.5625)+85(0.0625))/100=(8.4375+5.3125)/100=0.1375\). The overall score is \((0.2300+0.1375)/2=0.18375\), or \(0.1838\) to four decimals.
Log loss. At price 100 it is \(-[0.35\ln(0.40)+0.65\ln(0.60)]=0.6527\). At price 120 it is \(-[0.15\ln(0.25)+0.85\ln(0.75)]=0.4525\). The overall score is \((0.6527+0.4525)/2=0.5526\), using unrounded group values.
Check. The total predicted purchases are 65 and the observed purchases are 50, matching the overall rates of 0.325 and 0.250. A common mistake is to compare only the total rates and overlook the price pattern; another is to call the 5-point contrast difference a causal change without checking price assignment and comparability.
Design and Interpret the Validation
A sound out-of-time validation follows a disciplined sequence. The central idea is to keep the later period independent of model development and then assess both probability accuracy and the conditions under which the forecasts were made.
Specify the prediction unit, purchase definition, outcome window, and information available at the quote or pricing decision.
Fit the model and choose transformations, segment definitions, and probability-processing rules using development data only. Do not use validation outcomes to tune those choices.
Compare the periods’ customer, product, channel, and price distributions. Identify regions in which the validation data have limited development-period support.
Show calibration overall and by relevant groups, a proper probability loss, and a discrimination measure when ranking is relevant. Include sample sizes and uncertainty.
Separate changes in customer and price mix from changes in within-group performance. Decide whether monitoring, recalibration, or refitting is warranted based on the pattern and business context.
A random train-test split can place near-contemporaneous records in both samples, concealing changes that occur over time. Out-of-time validation instead preserves chronology. Even then, one later period does not establish that drift is permanent: seasonality, temporary market events, sampling variation, and operational changes can all affect results. Repeated monitoring across later windows provides a stronger basis for decisions.
Calibration deterioration with stable discrimination suggests that rankings may remain useful while the probability level has shifted. A decline in discrimination means the model is also less effective at ordering cases. Neither pattern alone determines the remedy. Recalibration adjusts predicted probability levels; refitting may be needed when predictor relationships or the price-response shape have changed. Any remedy should be evaluated on a subsequent untouched period rather than judged on the same outcomes used to make the change.
Check Your Understanding
For each problem, calculate the requested validation quantity and interpret it in context.
- A later-period sample has four forecasts and outcomes: \((0.30,0),(0.50,1),(0.70,1),(0.90,0)\). Calculate mean predicted demand, observed demand, the calibration gap, and the Brier score. (Answer: mean prediction \(=0.6000\), observed rate \(=0.5000\), gap \(=-0.1000\), Brier score \(=0.3100\).)
- Two equally weighted validation cases have \(p=0.80,Y=1\) and \(p=0.25,Y=0\). Calculate mean log loss using natural logarithms. (Answer: \(-[\ln(0.80)+\ln(0.75)]/2=0.2554\).)
- Among four purchaser–nonpurchaser pairs, purchaser scores are 0.70 and 0.40, while nonpurchaser scores are 0.60 and 0.20. Calculate AUC, counting a purchaser score above a nonpurchaser score as a win. (Answer: three of four pairs are wins, so AUC \(=0.7500\).)
- A frozen model predicts 0.30 for each of 20 quotes in Segment Pine; 4 purchase. It predicts 0.50 for each of 20 quotes in Segment Lake; 12 purchase. Calculate the overall mean prediction, observed rate, calibration gap, and each segment’s gap. (Answer: overall prediction \(=0.4000\), observed rate \(=0.4000\), overall gap \(=0.0000\); Pine gap \(=0.20-0.30=-0.1000\), Lake gap \(=0.60-0.50=0.1000\).)
- In a later period with comparable customer mix, a model predicts purchase rates of 0.45 at price 90 and 0.30 at price 110. Observed rates are 0.40 and 0.20. Find the predicted and observed decreases between prices and their difference. State one reason the result does not alone establish a causal change. (Answer: predicted decrease \(=0.15\), observed decrease \(=0.20\), observed decrease is 0.05 larger; nonrandom or confounded price assignment could contribute to the observed contrast.)