Estimating Candidate ARIMA Models for Comparison

Why Estimation Details Matter

Screening candidate orders with the ACF and PACF narrows the models to be fitted; it does not estimate their parameters. Estimation supplies the numerical coefficients and innovation variance for each candidate, along with evidence about how reliably the numerical procedure found a solution. A coefficient alone is not a complete fit record: the optimization method, convergence status, standard error, and any boundary warning also matter.

This tutorial focuses on fitting and documenting candidate ARIMA models. It does not use fitted likelihoods to rank models. In particular, likelihood values should not be compared until the models’ likelihood calculations and data treatments are known to be comparable.

Definition: An ARIMA$(p,d,q)$ model applies $d$ differences to a time series and represents the result using $p$ autoregressive terms and $q$ moving-average terms. One common convention is $$ \phi(B)(1-B)^d y_t=c+\theta(B)\varepsilon_t, $$ where $B y_t=y_{t-1}$, $\phi(B)=1-\phi_1B-\cdots-\phi_pB^p$, $\theta(B)=1+\theta_1B+\cdots+\theta_qB^q$, and $c$ is an included constant or drift term. The innovations are commonly modeled as independent $N(0,\sigma^2)$ variables. The constant’s interpretation depends on the differencing order and model specification.

The sign convention for moving-average coefficients must be checked. Under the convention above, an MA(1) equation is $\varepsilon_t+\theta_1\varepsilon_{t-1}$. Some software uses a minus sign instead; the same fitted process can therefore be reported with the opposite coefficient sign. For an AR(1), stationarity requires $|\phi_1|<1$. For an MA(1) under the plus-sign convention above, invertibility requires $|\theta_1|<1$. Higher-order models have corresponding root conditions: the autoregressive and moving-average polynomials must have roots outside the unit circle.

What Maximum Likelihood Estimates

Maximum likelihood chooses parameter values that make the observed series most plausible under the specified model. In a Gaussian ARIMA fit, the parameters usually include autoregressive and moving-average coefficients, any included constant or drift, and the innovation variance $\sigma^2$. The likelihood depends on how the initial observations and unobserved past innovations are handled. Conditional likelihood conditions on initial values and computes subsequent innovations recursively; exact likelihood also accounts for the distribution of the initial state under its assumptions. Software should identify the likelihood method it uses.

For a fixed parameter vector and a sequence of $n$ Gaussian innovations $e_t$, the conditional log-likelihood is

\ell=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\sigma^2) -\frac{1}{2\sigma^2}\sum_{t=1}^{n}e_t^2.

This expression explains why estimation is generally numerical for ARMA models: the innovations depend recursively on the parameters, and the likelihood must be optimized over the admissible parameter region. For a simple conditional Gaussian AR(1), however, the estimate has a closed form.

Derivation: Suppose $y_t=\phi y_{t-1}+\varepsilon_t$, with zero mean, independent $\varepsilon_t\sim N(0,\sigma^2)$, and a fixed initial value $y_1$. Conditional on $y_1$, define $e_t(\phi)=y_t-\phi y_{t-1}$ for $t=2,\ldots,T$. The conditional log-likelihood is a constant minus $\frac{1}{2\sigma^2}\sum_{t=2}^{T}(y_t-\phi y_{t-1})^2$. For fixed $\sigma^2$, maximizing it is equivalent to minimizing this sum of squares. Differentiating with respect to $\phi$ and setting the derivative to zero gives $\sum y_{t-1}(y_t-\phi y_{t-1})=0$, hence $$ \widehat{\phi}= \frac{\sum_{t=2}^{T}y_{t-1}y_t}{\sum_{t=2}^{T}y_{t-1}^{\,2}}. $$ For this conditional fit, the maximum-likelihood variance estimate is $\widehat{\sigma}^2=\sum e_t(\widehat{\phi})^2/(T-1)$. This result assumes the stated zero-mean AR(1) model and fixed initial value; it is not a general closed-form estimator for ARMA models or for a model with an estimated intercept.

Worked Example: Conditional Gaussian Estimation of an AR(1)

A hypothetical sequence has observed values $y_1=2,\ y_2=1,\ y_3=2,\ y_4=0,\ y_5=1$. Fit the zero-mean conditional AR(1) model $y_t=\phi y_{t-1}+\varepsilon_t$, conditioning on $y_1$. Assume independent Gaussian innovations with unknown variance, and impose stationarity $|\phi|<1$.

Method. Use the conditional maximum likelihood formulas derived above. The four lagged values are $2,1,2,0$, and the corresponding current values are $1,2,0,1$. Thus

\sum_{t=2}^{5}y_{t-1}y_t=(2)(1)+(1)(2)+(2)(0)+(0)(1)=4,

\sum_{t=2}^{5}y_{t-1}^{\,2}=2^2+1^2+2^2+0^2=9, \qquad \widehat{\phi}=\frac{4}{9}=0.4444.

The estimate satisfies the stationarity restriction because $0.4444<1$. The fitted conditional errors, retaining exact fractions, are

e_2=1-\frac{4}{9}(2)=\frac{1}{9},\quad e_3=2-\frac{4}{9}(1)=\frac{14}{9},\quad e_4=0-\frac{4}{9}(2)=-\frac{8}{9},\quad e_5=1-\frac{4}{9}(0)=1.

Therefore,

\operatorname{SSE} =\left(\frac{1}{9}\right)^2+\left(\frac{14}{9}\right)^2+ \left(-\frac{8}{9}\right)^2+1^2 =\frac{1+196+64+81}{81} =\frac{38}{9}=4.2222.

There are four conditional errors, so the maximum-likelihood variance estimate is $\widehat{\sigma}^2=(38/9)/4=19/18=1.0556$. The conditional Gaussian log-likelihood at these estimates is $\ell=-2[\log(2\pi)+1+\log(19/18)]\approx-5.7839$. Using the usual local-curvature approximation for this conditional regression, the estimated standard error of $\widehat{\phi}$ is $\sqrt{\widehat{\sigma}^2/9}=\sqrt{19/162}\approx0.3425$.

Check. The squared errors sum to $38/9$, and dividing by the four conditional observations gives $19/18$, not $38/27$. A common mistake is to divide the maximum-likelihood variance by $n-1$, as for an unbiased regression variance estimate. Here the requested Gaussian maximum-likelihood estimate divides by $n=4$.

Reading Coefficients and Standard Errors

A fitted autoregressive coefficient describes dependence in the modeled series, which may be a differenced series rather than the original observations. A fitted moving-average coefficient describes the contribution of a prior innovation under the stated sign convention. An estimated innovation variance describes the scale of the one-step innovations on the modeled scale. It is a variance, not a standard deviation; the estimated innovation standard deviation is $\sqrt{\widehat{\sigma}^2}$.

A standard error estimates local uncertainty in a fitted parameter. In maximum likelihood, it is commonly obtained from the inverse observed information matrix, derived from the curvature of the log-likelihood near the estimate. This is a local approximation: it can be unreliable when the likelihood is very flat, parameters are strongly dependent, or an estimate lies close to a constraint. Standard errors are reported on a particular parameter scale; if software optimizes a transformed parameter to enforce a constraint, check whether its output has been converted back to the model’s coefficient scale.

Worked Example: Interpreting an MA(1) Fit and Its Warning

For an invented series of 24 monthly changes, software fits an ARIMA$(0,1,1)$ model with no drift. It reports the equation $\Delta y_t=\varepsilon_t+\theta\varepsilon_{t-1}$, $\widehat{\theta}=-0.82$, standard error $0.18$, $\widehat{\sigma}^2=4.00$, and successful optimizer termination. It also reports a possible invertibility-boundary warning. Interpret the output under the stated plus-sign convention.

Coefficient and scale. The fitted change equation includes the prior innovation with coefficient $-0.82$. The estimated innovation standard deviation is $\sqrt{4.00}=2.00$, in the units of the monthly change.

Uncertainty and boundary distance. The point estimate is inside the MA(1) invertibility region because $|-0.82|=0.82<1$. Its absolute distance from the nearer boundary is $1-0.82=0.18$. Relative to the reported standard error, that distance is $0.18/0.18=1.00$ standard error. This calculation does not itself establish that the estimate is invalid; it shows why a boundary warning deserves to be recorded and investigated.

Method and trap avoided. Interpret the coefficient using the software’s written equation, then check its admissible region and distance to the boundary. Reading $-0.82$ as a positive coefficient because another convention uses a minus sign would misstate the fitted model. Successful termination means the optimizer stopped according to its stopping rule; it does not erase the separate boundary warning.

Convergence and Boundary Checks

Maximum likelihood for ARIMA models is typically found by an iterative optimizer. A fit record should distinguish whether the algorithm terminated successfully from whether the solution is numerically and statistically well behaved. Useful items to retain include the software and likelihood method, parameter estimates, standard errors, estimated innovation variance, convergence code or message, iteration limit, gradient or tolerance information when available, and any constraint or boundary warning.

A coefficient close to a stationarity or invertibility boundary may indicate weak identification, a nearly nonstationary or noninvertible fitted component, or sensitivity to initialization and optimization. A boundary estimate can also arise because the candidate specification is approximating a simpler process. Do not silently clip the estimate, discard the warning, or assume a small standard error guarantees a reliable fit. Record the warning and, where appropriate, refit with different starting values or inspect the reported optimization details.

Specify the candidate precisely.
Record $p,d,q$, any seasonal terms, whether a constant or drift is included, the coefficient sign convention, and the innovation distribution assumed for the fit.

Fit with a stated likelihood method.
Use the software’s maximum-likelihood procedure and note whether it conditions on initial values or uses an exact likelihood treatment. The treatment can affect the numerical results.

Read the full parameter output.
Record coefficient estimates, their standard errors, the innovation variance, and any estimated constant or drift. Interpret all values on the modeled scale.

Review numerical status and constraints.
Check the termination message, convergence criteria, and any warnings. Verify that the fitted coefficients meet the required root conditions and note their distance from the boundary.

Keep an auditable fit record.
Preserve the specification and warnings with the estimates. A converged fit is a candidate result to assess further, not by itself a conclusion that the model is adequate or preferred.

Worked Example: Recording Two Candidate Fits

An analyst fits two candidates to a hypothetical sequence of monthly index changes. Both models use the same stated Gaussian likelihood method. The first is ARIMA$(1,1,0)$, with no drift; the second is ARIMA$(0,1,1)$, with no drift. The reported output is:

Candidate	Coefficient (SE)	$\widehat{\sigma}^2$	Optimizer record
ARIMA(1,1,0)	$\widehat{\phi}=0.97\ (0.04)$	1.21	Termination code indicates success; maximum gradient component 0.014; stated tolerance 0.001
ARIMA(0,1,1)	$\widehat{\theta}=-0.96\ (0.08)$	1.18	Termination code indicates success; maximum gradient component 0.0002; stated tolerance 0.001; boundary warning

First candidate. For this AR(1) coefficient, the stationarity boundary is at magnitude 1. Its distance is $1-0.97=0.03$, which is $0.03/0.04=0.75$ reported standard errors. Although the termination code says success, the maximum gradient component $0.014$ exceeds the stated tolerance $0.001$. Record this discrepancy rather than describing the numerical convergence as unqualified.

Second candidate. Under the plus-sign MA convention, the invertibility boundary is also at magnitude 1. The distance is $1-|-0.96|=0.04$, or $0.04/0.08=0.50$ standard errors. The reported gradient, $0.0002$, is below the tolerance $0.001$, so the optimizer’s stated criterion is met; the separate boundary warning still needs to be retained.

Method and trap avoided. Check numerical stopping and parameter admissibility separately for every fit. Here, a successful termination code does not make the first gradient acceptable under the stated tolerance, and a small gradient does not remove the second model’s boundary warning. This example records fit quality information; it does not use the likelihoods to select between candidates.

Practical Interpretation

A disciplined estimation workflow prevents several common errors:

Confusing variance with standard deviation. If the output reports $\widehat{\sigma}^2=4$, the innovation standard deviation is $2$, not $4$.
Ignoring the model’s scale. Coefficients in a model for differenced observations describe the differenced series, not directly the original level.
Assuming every optimizer message means the same thing. A termination code, gradient tolerance, and constraint warning convey different information and should be read together.
Forgetting conventions. An MA coefficient’s sign is meaningful only relative to the equation used by the fitting software.
Treating a standard error as a guarantee. The reported value is a local approximation and may be less informative near a parameter boundary.

For an actuarial application, these distinctions matter when a time-series model is used to represent a hypothetical claim count index, expense measure, or economic series. The estimated model is only as interpretable as its stated scale, assumptions, and fitting record. This estimation discipline is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

Work each problem using the stated model convention and assumptions.

A zero-mean conditional AR(1) fit has lagged values $1,2,1$ and corresponding current values $0.5,1.5,0.5$. Find the conditional maximum likelihood estimate of $\phi$. (Answer: $\widehat{\phi}=2/3\approx0.6667$.)
A Gaussian conditional fit has six innovations and residual sum of squares $7.2$. What is the maximum-likelihood estimate of the innovation variance? (Answer: $\widehat{\sigma}^2=7.2/6=1.2$.)
Under $\Delta y_t=\varepsilon_t+\theta\varepsilon_{t-1}$, software reports $\widehat{\theta}=-0.91$ with standard error $0.05$. What is its distance from the invertibility boundary in standard-error units, and what should be noted? (Answer: The distance is $1-0.91=0.09$, or $0.09/0.05=1.8$ standard errors; record that it is inside but relatively close to the boundary.)
An optimizer reports successful termination, maximum gradient component $0.006$, and tolerance $0.001$. Does the gradient meet the stated tolerance? (Answer: No; $0.006>0.001$, so record the discrepancy even though the termination code indicates success.)
A fitted ARIMA$(0,1,0)$ model includes drift $c=1.4$ and innovations with mean zero. What does the drift represent under this specification? (Answer: It is the fitted mean change per time period, $E[\Delta y_t]=1.4$, in the units of $y_t$ per period.)

Candidate	Coefficient (SE)	\(\widehat{\sigma}^2\)	Optimizer record
ARIMA(1,1,0)	\(\widehat{\phi}=0.97\ (0.04)\)	1.21	Termination code indicates success; maximum gradient component 0.014; stated tolerance 0.001
ARIMA(0,1,1)	\(\widehat{\theta}=-0.96\ (0.08)\)	1.18	Termination code indicates success; maximum gradient component 0.0002; stated tolerance 0.001; boundary warning

What You'll Learn