Selection and Estimation Answer Different Questions
A time-series model-selection procedure has two distinct tasks. Selection chooses the forecasting specification, such as an AR(1) rather than a constant-mean model. Estimation determines the parameter values for the chosen specification, such as an autoregressive coefficient or a mean. Keeping those tasks separate helps prevent a common inefficiency: choosing a model using some observations, then forecasting with parameter estimates that ignore useful history that has since become available.
A typical chronological design divides observations into training, validation, and test periods. Fit candidate specifications on the training period and compare their forecasts on the validation period. Once the specification is chosen and the selection decisions are fixed, refit that specification using the combined training and validation history. The test period remains unused until final evaluation. For a forecast made after the development history ends, the refitted model uses all observations available at that forecast origin, but no future test observations.
Refitting is not the same as selecting again. If validation results lead to choosing an AR(1), refitting means estimating the AR(1) parameters on the expanded development history. It does not mean using the new estimates to search again among many specifications and choosing whichever now looks best. The distinction is important because the validation period has already influenced the choice.
How the Chronological Workflow Works
Estimate each candidate using only observations available in the training period.
Generate validation forecasts according to the intended forecasting procedure and compare them using the selected error metric.
Record the chosen specification and the forecasting choices that go with it. Do not treat refitting as a fresh validation search.
Re-estimate the chosen specification using training and validation observations available at the forecast origin.
Forecast beyond the refit sample. Compare with test observations only when evaluating forecast performance.
A validation score estimates how candidate procedures performed during the selection exercise; it is not the final test score. A test set serves a different purpose: it provides a final evaluation after the model choice and refitting procedure have been specified. If a test observation is used to change the model or its parameters, it is no longer an untouched test observation for that evaluation.
Refitting a Zero-Mean AR(1)
For a concrete calculation, consider the zero-mean AR(1) model
Assume the innovations \(\varepsilon_t\) are independent and identically distributed with mean zero and finite variance. For conditional least squares, condition on the initial observation and estimate \(\phi\) by minimizing the sum of squared one-step residuals over the observed transitions. If Gaussian innovations are also assumed, the same coefficient estimate is the conditional Gaussian maximum-likelihood estimate. These formulas do not apply unchanged to a model with an intercept, a different error structure, or a different estimation method.
The ratio follows by differentiating the objective with respect to \(\phi\). Its derivative is \(-2\sum_{t=2}^{n}Y_{t-1}(Y_t-\phi Y_{t-1})\). Setting it to zero gives \(\sum Y_{t-1}Y_t=\phi\sum Y_{t-1}^2\), and solving gives the displayed estimate, provided the denominator is positive. The denominator contains the lagged values in the fitted transitions, not every observed value indiscriminately.
Worked Example: Select on Validation Data, Then Refit
Suppose a hypothetical monthly index has training observations \(Y_1=1\), \(Y_2=3\), and \(Y_3=1\). The validation observations are \(Y_4=0.5\) and \(Y_5=0.4\), and \(Y_6=0.3\) is reserved as a test observation. Compare a zero-mean AR(1) with the zero-forecast benchmark. Use one-step validation forecasts that update with each newly observed value. If the AR(1) is selected, refit it on \(Y_1,\ldots,Y_5\) and forecast \(Y_6\).
Method and assumptions. Use conditional least squares for a zero-mean AR(1), so each one-step forecast is the estimated coefficient times the latest observed value. The zero-forecast benchmark predicts zero at every step. Compare the candidates by validation mean squared error (MSE).
1. Fit on training data. There are two training transitions: \((Y_1,Y_2)\) and \((Y_2,Y_3)\). Thus the numerator is \(1(3)+3(1)=6\), and the denominator is \(1^2+3^2=10\). The training estimate is \(6/10=0.6\).
2. Compare validation forecasts. The AR(1) forecast for \(Y_4\) is \(0.6(1)=0.6\), so its error is \(0.5-0.6=-0.1\). After observing \(Y_4=0.5\), the forecast for \(Y_5\) is \(0.6(0.5)=0.3\), so its error is \(0.4-0.3=0.1\). The AR(1) validation MSE is \(((-0.1)^2+(0.1)^2)/2=0.02/2=0.01\). The zero-forecast errors are \(0.5\) and \(0.4\); its MSE is \((0.25+0.16)/2=0.205\). The AR(1) has the lower validation MSE, so select it.
3. Refit the selected specification. Using the transitions from \(Y_1,\ldots,Y_5\), the numerator is \(1(3)+3(1)+1(0.5)+0.5(0.4)=6+0.5+0.2=6.7\). The denominator is the sum of squared lag values \(Y_1,\ldots,Y_4\): \(1^2+3^2+1^2+0.5^2=1+9+1+0.25=11.25\). Therefore \(\widehat{\phi}_{\text{refit}}=6.7/11.25=134/225\approx0.5956\).
4. Forecast and check on the test observation. The forecast is \((134/225)(0.4)=268/1125\approx0.2382\). The test error is \(0.3-0.2382\approx0.0618\). Using the unrounded forecast, the squared error is \((139/2250)^2\approx0.003817\), rounded to six decimal places. The test value was not used in either fitting step.
Common mistake avoided. Adding \(Y_5^2\) to the refitting denominator is wrong: \(Y_5\) is a response in the transition \(Y_4\) to \(Y_5\), not a lagged value. The denominator is \(Y_1^2+\cdots+Y_4^2=11.25\), not \(11.25+0.4^2\). That mistake would change the fitted coefficient and the forecast.
Refitting Also Updates the Innovation Variance
For a Gaussian AR(1), refitting includes estimating the innovation variance, not just the autoregressive coefficient. Under conditional Gaussian maximum likelihood, with \(Y_1\) treated as fixed and \(n-1\) transitions, the variance estimate is the residual sum of squares divided by the number of transitions. This is the maximum-likelihood divisor; an unbiased sample-variance divisor is a different convention and is not the conditional maximum-likelihood estimate.
Worked Example: Re-estimate the Coefficient and Variance
A hypothetical equipment index has \(Y_1=2\), \(Y_2=1\), \(Y_3=2\), and \(Y_4=1\) in training, followed by validation value \(Y_5=1.2\). The test value is \(Y_6=1.0\). Assume a zero-mean AR(1) with independent Gaussian innovations \(N(0,\sigma^2)\), where \(\sigma^2\) is the innovation variance. Compare the AR(1) with forecasts of zero, select by validation squared error, then refit the AR(1) and estimate its conditional Gaussian variance.
1. Fit the training model and select. The training numerator is \(2(1)+1(2)+2(1)=6\), and the denominator is \(2^2+1^2+2^2=9\). Thus \(\widehat{\phi}_{\text{train}}=6/9=2/3\). The AR(1) validation forecast is \((2/3)(1)=2/3\), giving error \(1.2-2/3=8/15\) and squared error \(64/225\approx0.2844\). The zero forecast has squared error \(1.2^2=1.44\). The AR(1) is selected.
2. Refit on all five development observations. The four lag-response products sum to \(2(1)+1(2)+2(1)+1(1.2)=2+2+2+1.2=7.2\). The four lagged values are \(2,1,2,1\), so the denominator is \(2^2+1^2+2^2+1^2=10\). Hence \(\widehat{\phi}_{\text{refit}}=7.2/10=0.72\).
3. Calculate residuals and variance. In order, the fitted residuals are \(1-0.72(2)=-0.44\), \(2-0.72(1)=1.28\), \(1-0.72(2)=-0.44\), and \(1.2-0.72(1)=0.48\). Their squared sum is \(0.1936+1.6384+0.1936+0.2304=2.256\). There are four transitions, so the conditional Gaussian maximum-likelihood estimate is \(\widehat{\sigma}^2=2.256/4=0.564\).
4. Forecast and evaluate. The forecast is \(0.72(1.2)=0.864\). The test error is \(1.0-0.864=0.136\), and the squared error is \(0.136^2=0.018496\). The variance estimate describes fitted one-step innovation variation; it does not change the point forecast in this zero-mean AR(1).
Common mistake avoided. The denominator for the coefficient is \(10\), the sum of squares of the four lag values. Including \(Y_5^2\) would incorrectly add a response value. Also, dividing the residual sum of squares by five rather than four would not give the conditional Gaussian maximum-likelihood variance estimate for these four transitions.
Refitting a Constant-Mean Forecast
The same separation applies when the selected model is not autoregressive. Suppose a series follows a constant-mean model \(Y_t=\mu+\varepsilon_t\), where the errors are independent with mean zero and finite variance. The least-squares estimate of \(\mu\) is the sample average, and the forecast at every future horizon is that estimated mean. This specification assumes no time-varying level or serial dependence; it should not be used merely because its arithmetic is convenient.
Worked Example: Refit a Selected Mean Model
A hypothetical monthly service-volume series has training values \(10,12,8\), validation values \(9,11\), and a test value \(12\), all measured in thousands of transactions. Compare a constant-mean model, estimated on training data, with a last-observation forecast. Use sequential one-step validation forecasts. If the constant-mean model is selected, refit its mean on training and validation data and evaluate the next forecast.
1. State the candidate forecasts. Assume the constant-mean specification \(Y_t=\mu+\varepsilon_t\), with independent mean-zero errors, and estimate \(\mu\) by the training average. That average is \((10+12+8)/3=30/3=10\), so the model predicts 10 for each validation observation. The last-observation benchmark uses the most recently observed value.
2. Compare validation errors. For the constant-mean model, the errors are \(9-10=-1\) and \(11-10=1\). Its MSE is \((1+1)/2=1\). For the last-observation benchmark, the first forecast is 8, giving error \(9-8=1\); after observing 9, the next forecast is 9, giving error \(11-9=2\). Its MSE is \((1^2+2^2)/2=5/2=2.5\). Select the constant-mean specification.
3. Refit and forecast. The refitted mean uses all five development observations: \((10+12+8+9+11)/5=50/5=10\). The forecast for the next observation is therefore 10 thousand transactions.
4. Evaluate on the reserved test value. The test error is \(12-10=2\) thousand transactions, and the squared error is \(2^2=4\) thousand-squared transactions. The test observation is used only for this final evaluation, not to update the estimated mean.
Common mistake avoided. Using only the training average after validation data have become available leaves observed development history unused. Conversely, incorporating the test value into the average would make the reported test error an in-sample result rather than a final evaluation of the forecast.
What Refitting Does—and Does Not—Change
A refit can change coefficient estimates, fitted residuals, innovation variance estimates, and forecasts. It does not automatically change the selected model form. For example, after choosing a particular ARIMA specification, its parameters can be estimated again on the expanded development history. The selected orders and other specification choices remain fixed for that refit.
This distinction also clarifies what a forecast represents. A validation forecast is generated using the information and fitted values prescribed by the validation design. A final forecast after refitting uses the larger development sample available at its forecast origin. The two forecasts can differ even though they use the same specification, because their parameter estimates and latest observed values can differ.
| Stage | Purpose | Data used |
|---|---|---|
| Training fit | Estimate candidate models | Training observations |
| Validation comparison | Choose a specification | Validation outcomes, forecast using the specified procedure |
| Final refit | Estimate the chosen specification for forecasting | Training and validation history available at the forecast origin |
| Test evaluation | Assess final forecast performance | Test outcomes, not used to fit or choose the evaluated forecast |
A validation score can favor one candidate because of the particular validation period, so it should not be interpreted as a guarantee of future accuracy. The role of refitting is different: once the choice is made, it uses the available development history to estimate that chosen model. The test score then evaluates the resulting forecasting procedure on observations held back from those decisions.
Common Traps
- Refitting the wrong model. Re-estimating all candidates and selecting again after validation is not simply a refit of the selected specification. Keep the selection decision distinct from parameter estimation.
- Using the wrong denominator. In an AR(1) conditional least-squares estimate, sum squared lag values for the fitted transitions. Do not add the final response value as if it were another lag.
- Discarding validation history without a reason. For a final forecast beyond the development period, the selected model is commonly re-estimated using both training and validation observations that are now available.
- Using test data too early. A test observation can evaluate a completed forecast. If it is used to estimate parameters or revise the specification, it no longer provides an untouched final evaluation.
- Confusing validation and final forecast protocols. A rolling one-step validation forecast may use each newly observed validation value for the next forecast. A multi-step forecast made before any future outcomes are observed must instead follow its specified multi-step procedure.
Check Your Understanding
For each problem, distinguish selection, refitting, and test evaluation. Assume the stated model and forecasting rules apply.
- A zero-mean AR(1) has training values \(Y_1=2,Y_2=1,Y_3=3\), validation value \(Y_4=2\), and test value \(Y_5=1\). Conditional least squares gives what training coefficient, what validation forecast, and what coefficient after refitting on \(Y_1,\ldots,Y_4\)? (Answer: training coefficient \(5/10=0.5\); validation forecast \(1.5\); refit coefficient \(11/14\approx0.7857\).)
- A constant-mean model has training values \(4,6,8\) and validation values \(7,5\). After selecting this specification, what is its refitted mean using all five development observations, and what is its next forecast? (Answer: refitted mean \(30/5=6\); forecast \(6\).)
- For a zero-mean AR(1), the refitted coefficient is \(0.5\) and the development series ends at \(Y_n=4\). The reserved test value is \(1.5\). Find the forecast and test squared error. (Answer: forecast \(0.5(4)=2\); squared error \((1.5-2)^2=0.25\).)
- A conditional Gaussian AR(1) refit has three transition residuals \(0.2,-0.4,0.2\). What is the conditional maximum-likelihood innovation variance estimate? (Answer: \((0.2^2+(-0.4)^2+0.2^2)/3=0.24/3=0.08\).)
- A model was chosen using validation data and then refitted on training plus validation observations. An analyst now changes its order after inspecting test errors. Can the original test score still be described as an untouched final evaluation of the revised procedure? (Answer: No; the test outcomes influenced the revised model choice, so they are no longer untouched for evaluating that revised procedure.)