Why Differencing Enters Time-Series Models
An ARMA model describes a series whose mean and covariance structure are stable over time. Many observed series instead have a changing level: a random walk can wander without bound, and a deterministic trend can shift the mean from one period to the next. Applying an ARMA model directly to such levels generally violates the stationarity assumptions used to identify and interpret ARMA behavior.
Differencing replaces each level by its change from the preceding time. When nonstationarity arises because a stable process accumulates in the level, this operation can remove the accumulation and leave a stationary process. ARIMA models formalize that construction: the differenced series has ARMA dynamics, while the “integrated” part describes how those differences accumulate back into the original level.
A standard source of nonstationarity is an accumulating, or unit-root, component. If \(w_t\) is stationary and \(y_t=y_{t-1}+w_t\), then the level \(y_t\) is a cumulative sum of the \(w_t\)'s. Taking a first difference gives \(\Delta y_t=w_t\), recovering the stationary process. This does not mean that every nonstationary series becomes stationary after differencing: the reason for the nonstationarity matters.
From a Stationary Process to an Integrated Level
Let \(B\) denote the lag operator, defined by \(By_t=y_{t-1}\). Then the first difference can be written \(\Delta y_t=(1-B)y_t\). If the differenced series follows a stationary ARMA process, the level is obtained by accumulating those differences. In general terms, an ARIMA model says that after \(d\) differences, the series follows an ARMA model. The integration component records this differencing and accumulation relationship; it does not assert that the original levels are stationary.
Here \(\phi(B)\) and \(\theta(B)\) describe the autoregressive and moving-average dynamics of the differenced process, \(d\) is the number of differences, \(c\) is a constant when included, and \(\varepsilon_t\) is a zero-mean white-noise innovation with finite variance. The equation is useful only when the differenced process is stationary and the ARMA specification is appropriate. A deterministic trend, a variance that changes over time, or a structural break may require different treatment rather than simply increasing \(d\).
The underlying cancellation is direct. If \(y_t=y_{t-1}+w_t\), subtracting \(y_{t-1}\) from both sides gives \(\Delta y_t=w_t\). In lag notation, \((1-B)y_t=w_t\). The accumulating factor in the level is removed, exposing the stationary dynamics.
Worked Example: A Random Walk With Drift
A hypothetical monthly index follows \(y_t= y_{t-1}+12+\varepsilon_t\), where the drift is 12 index points per month and the innovations are independent and identically distributed with mean 0 and variance 4. Let \(y_0=100\), and suppose the four observed innovations are \(2,-1,0,3\). Find the observed levels, the differenced values, and the mean and variance of the differenced process. Explain why the level is nonstationary.
Method. Rearrange the model to express the increment, then compute the path using the stated innovations. The model assumptions are independent innovations with constant mean 0 and variance 4; no normal distribution is assumed or needed.
For each month, \(\Delta y_t=y_t-y_{t-1}=12+\varepsilon_t\). Thus the four increments are \(12+2=14\), \(12-1=11\), \(12+0=12\), and \(12+3=15\). Adding each increment to the preceding level gives \(y_1=100+14=114\), \(y_2=114+11=125\), \(y_3=125+12=137\), and \(y_4=137+15=152\).
The differenced process is \(12+\varepsilon_t\), so its mean is \(12+0=12\), its variance is \(4\), and its autocovariance is zero at every nonzero lag because the innovations are independent. It is stationary. In contrast, \(E[y_t]=100+12t\), which changes with time. Also, since \(y_0\) is fixed, \(\operatorname{Var}(y_t)=4t\), which grows with time. The level therefore fails the weak-stationarity conditions.
Check. The computed increments are \(114-100=14\), \(125-114=11\), \(137-125=12\), and \(152-137=15\), matching the model. The common mistake is to call the level stationary because its increments have a stable distribution; stationarity of the increments does not make the accumulating levels stationary.
Differencing Can Reveal ARMA Dependence
The stationary process revealed by differencing need not be white noise. It may itself have autoregressive or moving-average dependence. The central ARIMA construction is therefore broader than a random walk: a nonstationary level can be the accumulated history of a stationary ARMA process. The differenced series then retains that process's dependence structure.
Worked Example: A Level Built From a Stationary AR Process
A hypothetical reserve index satisfies \(y_t=y_{t-1}+x_t\), where \(x_t=0.5x_{t-1}+\varepsilon_t\). Assume \(\varepsilon_t\) is independent white noise with mean 0 and variance 3, and \(x_t\) is initialized from its stationary distribution. For illustration, also take \(x_0=0\) for a conditional path calculation and use innovations \(2,-1,0,1\). Identify the differenced model, calculate its stationary variance and lag-one autocovariance, and compute the path from \(y_0=50\).
Method. First identify the process for the increment \(y_t-y_{t-1}\). Then use the stationary AR(1) moment formulas. The theoretical moments use stationary initialization; the specified path is a separate conditional calculation starting at \(x_0=0\).
Since \(y_t-y_{t-1}=x_t\), the first difference follows \(x_t=0.5x_{t-1}+\varepsilon_t\), a stationary AR(1) process because the autoregressive coefficient has absolute value less than 1. Its mean is 0. Its variance \(\gamma(0)\) satisfies \(\gamma(0)=0.5^2\gamma(0)+3\), so \(0.75\gamma(0)=3\) and \(\gamma(0)=4\). The lag-one autocovariance is \(\gamma(1)=0.5\gamma(0)=2\), and the lag-one autocorrelation is \(2/4=0.5\).
For the path, use the recursion for \(x_t\): \(x_1=0.5(0)+2=2\); \(x_2=0.5(2)-1=0\); \(x_3=0.5(0)+0=0\); and \(x_4=0.5(0)+1=1\). These are the increments. Thus \(y_1=50+2=52\), \(y_2=52+0=52\), \(y_3=52+0=52\), and \(y_4=52+1=53\).
The differenced process is stationary AR(1), not white noise: its lag-one autocovariance is 2. This is precisely the role of the ARMA part of an ARIMA model. The common mistake is to conclude that differencing must remove all serial dependence; it removes the accumulating level component, while stationary dependence in the increments remains.
Why a Deterministic Trend Can Also Differ to Stationarity
Differencing can also remove a deterministic linear trend, although the resulting process need not have the same structure as the original noise. For a stationary process \(z_t\), the differenced noise \(z_t-z_{t-1}\) remains stationary. In particular, any fixed finite linear combination of a weakly stationary process has a constant mean and a covariance that depends only on lag. This explains why a linear trend plus stationary fluctuations can become stationary after one difference.
To justify this formula, write \(\Delta z_t=z_t-z_{t-1}\) and \(\Delta z_{t-k}=z_{t-k}-z_{t-k-1}\). Expanding their covariance gives \(\operatorname{Cov}(z_t,z_{t-k})-\operatorname{Cov}(z_t,z_{t-k-1})-\operatorname{Cov}(z_{t-1},z_{t-k})+\operatorname{Cov}(z_{t-1},z_{t-k-1})\). By weak stationarity these terms are, respectively, \(\gamma(k)\), \(\gamma(k+1)\), \(\gamma(k-1)\), and \(\gamma(k)\). Combining them yields the displayed expression, which depends on \(k\) alone. Thus differencing a stationary process does not make it nonstationary, though it changes its autocovariances.
Worked Example: A Linear Trend Plus Independent Noise
A hypothetical quarterly workload is \(y_t=5+2t+\eta_t\), where \(\eta_t\) are independent and identically distributed with mean 0 and variance 9. For a possible observed path, let \(\eta_0=1,\eta_1=-2,\eta_2=0,\eta_3=2,\eta_4=-2\). Determine the mean, variance, and autocovariances of the differenced series, and verify its observed values for \(t=1,\ldots,4\).
Method. Difference the trend and noise separately. Independence over time makes all covariances between distinct \(\eta_t\)'s zero. No distributional form beyond the given moments is required.
The difference is \(\Delta y_t=2+\eta_t-\eta_{t-1}\), since \(5+2t-[5+2(t-1)]=2\). Its mean is 2. Its variance is \(\operatorname{Var}(\eta_t-\eta_{t-1})=9+9=18\), because the two noise terms are independent. At lag one, the only shared noise term in \(\Delta y_t\) and \(\Delta y_{t-1}\) is \(\eta_{t-1}\), with opposite signs, so \(\operatorname{Cov}(\Delta y_t,\Delta y_{t-1})=-9\). At every lag of two or more, the noise terms do not overlap, so the covariance is 0.
For the observed path, the levels at \(t=0,1,2,3,4\) are \(6,5,9,13,11\), respectively. The differences are \(5-6=-1\), \(9-5=4\), \(13-9=4\), and \(11-13=-2\). The model gives \(2+(-2-1)=-1\), \(2+(0-(-2))=4\), \(2+(2-0)=4\), and \(2+(-2-2)=-2\), matching every difference.
The level has mean \(5+2t\), so it is nonstationary. The differenced series has constant mean and covariance depending only on lag; it is a stationary moving-average process. The common mistake is to omit the shared noise term when calculating lag-one covariance. Its coefficient is positive in one difference and negative in the adjacent difference, giving \(-9\), not 0 or \(+9\).
When the ARIMA Connection Is Useful—and Its Limits
The integration idea separates two features of a time series: the behavior of the level and the dependence in its changes. If the level accumulates a stationary ARMA process, differencing isolates that stationary process. The ARMA dynamics then describe how changes relate across time, while accumulation explains why levels can wander or trend. This separation is useful because ARMA identification and inference rely on stationary behavior.
Differencing is not a universal repair. A changing variance is not removed merely by subtracting successive levels. A series with a one-time level shift or a nonlinear trend may not become stationary under a simple difference. Also, the fact that a difference looks less trending in a short observed sample does not establish that the differenced process is stationary. The proposed model and the time-series behavior must support that conclusion.
A practical check is to state explicitly what process is assumed stationary. For a random walk with drift, it is the increments after removing the drift. For a level formed by accumulating an ARMA process, it is that stationary ARMA process. For a deterministic linear trend plus stationary fluctuations, differencing removes the linear trend but also filters the fluctuations. These are related constructions, but they are not interchangeable explanations of the data.
Check Your Understanding
For each problem, identify the relevant differenced process and use the stated assumptions; answers are provided for checking.
- A hypothetical account balance follows \(y_t=y_{t-1}+4+\varepsilon_t\), where \(\varepsilon_t\) are independent with mean 0 and variance 2. What are the mean and variance of the first differences, and is the level weakly stationary? (Answer: Difference mean 4, variance 2; the level is nonstationary because its mean changes by 4 each period and its variance grows by 2 per period when the initial level is fixed.)
- Let \(y_t=y_{t-1}+x_t\), where \(x_t=0.25x_{t-1}+\varepsilon_t\) is stationary and the independent innovations have mean 0 and variance 2. Find the variance and lag-one autocovariance of \(\Delta y_t\). (Answer: \(\operatorname{Var}(\Delta y_t)=2/(1-0.25^2)=32/15\); lag-one autocovariance \(=(1/4)(32/15)=8/15\).)
- Suppose \(y_t=8-t+\eta_t\), where \(\eta_t\) are independent with mean 0 and variance 5. Find the mean, variance, and lag-one autocovariance of \(\Delta y_t\). (Answer: Mean \(-1\), variance \(10\), and lag-one autocovariance \(-5\).)
- A level is defined by \(y_t=y_{t-1}+w_t\), where \(w_t=0.6w_{t-1}+\varepsilon_t\), and the independent innovations have variance 1. What process is the first difference, and what are its variance and lag-one autocovariance? (Answer: A stationary AR(1) process; variance \(1/(1-0.6^2)=25/16\), lag-one autocovariance \(0.6(25/16)=15/16\).)
- A series has the form \(y_t=3t+z_t\), where \(z_t\) is weakly stationary with autocovariance function \(\gamma(k)\). Explain why one difference is stationary, and state its mean and autocovariance at lag \(k\) if \(E[z_t]=0\). (Answer: \(\Delta y_t=3+\Delta z_t\) has constant mean 3 and lag-dependent covariance \(2\gamma(k)-\gamma(k-1)-\gamma(k+1)\).)