Introduction
A home-price model can fit the properties used to build it quite closely and still make disappointing predictions for homes in a neighborhood it has never seen. The distinction matters when the intended use is to estimate prices in a new area rather than to describe the properties already analyzed. A held-out neighborhood provides a direct, though limited, check of that task.
The central design choice is to keep every property from the reserved neighborhood out of model fitting. The model is built using property characteristics from other neighborhoods, then its predictions are compared with the reserved homes' observed prices. This tests performance under a change in location that a random split of individual homes might conceal.
Model and Held-Out Neighborhood
Suppose the response \(Y_i\) is the sale price of home \(i\), measured in thousands of dollars. Available characteristics are floor area in square feet, number of bedrooms, age in years, and distance from a city center in miles. One possible working regression model is
Here, \(\beta_0,\ldots,\beta_4\) are model coefficients and \(\varepsilon_i\) represents the part of price not captured by the included characteristics. The model describes an assumed linear relationship in the conditional mean: at fixed values of the listed characteristics, expected price changes according to the corresponding coefficients. It is a predictive description, not evidence that changing a home's area or age would cause its price to change by the coefficient amount.
For this evaluation to say something useful about prediction, the test homes must not influence the fitted model or choices made while developing it. Also, the reserved neighborhood should represent the kind of location to which the model will be applied. The working model should capture the relevant price patterns well enough to transfer across those neighborhoods. These conditions cannot be guaranteed by the split itself. If the reserved neighborhood differs greatly from the training neighborhoods, the test can reveal that limitation rather than provide a reliable estimate for every possible neighborhood.
A location label used only to identify the held-out neighborhood cannot be treated as an ordinary predictor for that unseen neighborhood: the model would have no fitted neighborhood effect for a category never present in training. Instead, the example uses characteristics available for every home. Geographic information that can be measured consistently, such as distance from a shared center, may be a predictor, but it does not necessarily capture every neighborhood difference.
Worked Example: Reserving a Neighborhood Before Fitting
A hypothetical file contains 240 home sales from five neighborhoods, labeled A through E, with sale price, area, bedrooms, age, and distance recorded for every home. The goal is to evaluate predictions for homes in neighborhood E.
Split. Set aside all 48 neighborhood E sales as the test set. Use the remaining 192 homes from neighborhoods A through D as the training set. No neighborhood E price can be used to estimate coefficients, choose among candidate models, or adjust the predictor list.
Check the design. The response and four characteristics are recorded for both sets, so the model can be fitted on training observations and used to calculate test predictions. The split is at the neighborhood level, not the home level: no property in E is included in fitting.
State what the evaluation targets. The resulting test errors describe predictions for the observed homes in E. They are relevant to predicting in a new neighborhood only insofar as E resembles the future target locations. A single reserved neighborhood cannot establish performance for every neighborhood.
Fitting and Making a Prediction
Ordinary least squares estimates the coefficients by choosing values that minimize the sum of squared differences between training prices and the corresponding fitted values. Write the fitted model as \(\widehat{Y}_i\), where the coefficient estimates are marked with hats. Once fitted, the model predicts a price by substituting a home's recorded characteristics. The result is in thousands of dollars here, so multiplying by 1,000 expresses it in dollars.
Worked Example: Predicting the Price of a Held-Out Home
Suppose fitting the training data gives the following hypothetical equation, with price in thousands of dollars:
A home in the reserved neighborhood has area 1,800 square feet, 3 bedrooms, age 20 years, and distance 5 miles. Its predicted price is
The prediction is \(502.5\) thousand dollars, or $502,500. The calculation uses only the coefficients estimated from neighborhoods A through D and the characteristics of this home. The intercept is the model's predicted price when every numerical predictor equals zero; that combination is not a realistic home in this setting, so the intercept need not have a useful standalone interpretation.
Before relying on the prediction, compare the home's characteristics with values in the training data. If, for example, its area is far beyond every training home's area, the model is extrapolating. A calculated prediction can still be produced, but the observed data provide little support for trusting the assumed linear relationship that far outside the training range.
Measuring Held-Out Prediction Errors
For test home \(i\), define its prediction error as observed price minus predicted price. A positive error means the model underpredicted; a negative error means it overpredicted. The error keeps the outcome's units, which in this example are thousands of dollars.
Several summaries answer different questions. The mean error measures average direction of error: a negative value indicates average overprediction under this sign convention. The mean absolute error (MAE) averages error magnitudes and is less dominated by a few large errors than a squared-error measure. The root mean squared error (RMSE) squares errors before averaging, then takes a square root; it gives relatively greater weight to large misses. All three are in the same units as the response.
These are descriptive summaries of the test predictions. They do not require a normal-error assumption to be calculated. However, generalizing their values to a broader set of neighborhoods requires that the test neighborhood and homes be relevant to that broader target.
Worked Example: Calculating Test-Set Error Summaries
Four hypothetical homes in the held-out neighborhood have the following observed and predicted prices. Amounts are in thousands of dollars.
| Home | Observed price \(Y_i\) | Predicted price \(\widehat{Y}_i\) | Error \(e_i\) |
|---|---|---|---|
| 1 | 400 | 420 | -20 |
| 2 | 540 | 510 | 30 |
| 3 | 380 | 365 | 15 |
| 4 | 560 | 600 | -40 |
For example, home 1 has error \(400-420=-20\), meaning the model overpredicted by $20,000. The mean error is
thousand dollars. Thus the model overpredicted by an average of $3,750 across these four homes, though individual errors went in both directions.
The MAE is
thousand dollars, or $26,250. The RMSE is
thousand dollars. The RMSE exceeds the MAE because squaring gives extra weight to the larger errors. With only four homes, these values illustrate the calculations but are too limited to summarize reliably all future home predictions.
Calibration and the Difference From Training Fit
A model is well calibrated in a broad sense when predicted prices agree with observed prices on average for the group being assessed. A useful first check is to compare the mean predicted price with the mean observed price. Calibration should also be examined across ranges of predicted price: an overall average can conceal overprediction among lower-priced homes and underprediction among higher-priced homes, or the reverse.
For the four homes above, the mean predicted price is \((420+510+365+600)/4=473.75\) thousand dollars. The mean observed price is \((400+540+380+560)/4=470\) thousand dollars. Their difference, observed minus predicted, is \(-3.75\) thousand dollars, matching the mean error. Grouping the two lower predictions gives mean predicted price \((365+420)/2=392.5\) and mean observed price \((380+400)/2=390\), a difference of \(-2.5\). For the two higher predictions, the corresponding means are \(555\) and \(550\), a difference of \(-5\). In this tiny example, both groups show modest average overprediction. Such small groups do not establish stable calibration.
Worked Example: Comparing Training and Test RMSE
Suppose the training set contains 100 homes and the sum of squared training residuals is \(32{,}400\) in squared thousands of dollars. Its descriptive training RMSE, using the number of homes as the divisor, is
thousand dollars. The test RMSE calculated above is about \(27.95\) thousand dollars. The test RMSE is therefore about \(27.95-18=9.95\) thousand dollars larger.
This difference is plausible: ordinary least squares chose coefficients to reduce squared errors in the training data, not in neighborhood E. The training value describes fit to homes used for estimating coefficients; the test value describes errors for homes excluded from that fitting. The gap may reflect the advantage of fitting on the training observations, random variation in which homes were sampled, different price patterns in E, or model misspecification. It does not by itself identify which explanation is responsible.
The training RMSE here is a predictive summary, not the usual residual standard error: its denominator is the number of observations rather than a degrees-of-freedom adjustment. The test RMSE uses the same descriptive divisor convention for the test homes. Comparing the two is informative as a performance contrast, not a formal test that the model has changed.
Why the Neighborhood Split Matters
A random split of homes can place properties from every neighborhood in both training and test sets. If homes within a neighborhood share unrecorded influences, such as local amenities or market conditions, the model may benefit from having seen other homes from that same area during fitting. Its test results would then answer a different question: prediction for more homes from represented neighborhoods. Holding out every home in one neighborhood instead asks whether the fitted relationship transfers to an unrepresented area.
The split is deliberately challenging, but a single neighborhood is not a random sample of all possible locations. If E has unusually high prices or a different relationship between area and price, its errors may be larger for reasons specific to E. Conversely, good performance in E does not prove that every new neighborhood will be predicted equally well. The report should identify the held-out neighborhood, the number of test homes, the predictor set, and the error summaries, and should limit its conclusions to the evaluation performed.
A further practical requirement is to keep test outcomes out of development decisions. If the model is changed repeatedly in response to its errors on E, those prices have effectively influenced the modeling process; the resulting test summaries no longer provide an untouched check. The held-out results can be used to describe shortcomings and guide future work, but the reported performance should be clear about whether the test set remained independent of such decisions.
Common Mistake
- Wrong reasoning: “The model's training RMSE is $18,000, so its predictions in a new neighborhood will typically be within $18,000 of the sale price.”
- Correction: Training RMSE summarizes errors on the observations used for fitting and is generally an optimistic guide to performance on unseen homes. Report the held-out test errors as well, specify that the test set was one neighborhood, and avoid promising a typical error for other locations without evidence that the test neighborhood represents them.
- Wrong reasoning: “The mean test error is close to zero, so predictions are accurate.”
- Correction: Positive and negative errors can cancel. Here, the mean error is \(-3.75\) thousand dollars, while MAE and RMSE are about \(26.25\) and \(27.95\) thousand dollars. Report a directional summary together with a measure of error magnitude.
Check Your Understanding
Use the definitions and calculations in this tutorial to answer the following questions.
- Why should every home from the reserved neighborhood be excluded from coefficient estimation?
- A test home has observed price $455,000 and predicted price $480,000. What is its prediction error under the convention used here, and what does its sign mean?
- Why can RMSE be larger than MAE for the same test set?
- What limitation should be stated when test performance is measured using only one held-out neighborhood?
- How could the mean error be close to zero even when the model makes some large prediction errors?