Covariance Between Time Points

Why Compare Observations at Two Times?

A time series records a variable as it evolves, but knowing the mean and variance at each time does not tell us whether observations at different times move together. For example, a process might tend to be above its usual level at one time when it is also above its usual level at another. Covariance provides a numerical summary of that tendency. It describes a pair of time points, not a whole time-series model.

The covariance depends on the joint behavior of the two observations. Their separate distributions determine their means and variances, but those marginal summaries alone generally do not determine covariance. To calculate it, we need information about which values of the two observations occur together.

Definition: Let $X_s$ and $X_t$ be observations from a time-indexed stochastic process at times $s$ and $t$, with finite second moments. Their covariance is $$ \operatorname{Cov}(X_s,X_t) = E\left[(X_s-E[X_s])(X_t-E[X_t])\right]. $$ It measures the expected product of their deviations from their respective means. The covariance is positive when same-direction deviations contribute more on average, negative when opposite-direction deviations contribute more, and zero when the expected centered product is zero.

The finite-second-moment condition ensures the expectations needed in the definition exist and are finite. The times $s$ and $t$ can be any two specified time points; they need not be consecutive or equally spaced from any other pair. No stationarity assumption is required to define this covariance.

Calculating Covariance From Joint Outcomes

For a finite set of possible paired outcomes, multiply each possible value of $X_s$ by the value of $X_t$ that occurs with it, then average those products using their probabilities. This gives $E[X_sX_t]$. The means $E[X_s]$ and $E[X_t]$ must be calculated separately from their own possible values.

\operatorname{Cov}(X_s,X_t) = E[X_sX_t]-E[X_s]E[X_t]

This equivalent formula is often the most direct for a joint probability table. It also makes clear why the two marginal means are not enough: the expected product $E[X_sX_t]$ depends on the way the outcomes are paired.

Worked Example: Negative Covariance Across Two Time Points

In a hypothetical process, $X_3$ is a workload index at time 3 and $X_7$ is the index at time 7. Three paired outcomes are equally likely: $(X_3,X_7)=(6,6),(10,4),(14,2)$. Find their covariance and interpret its sign.

Method. Calculate each time-specific mean, then calculate the expected product and subtract the product of the means. Because the three paired outcomes are equally likely, each average is a sum divided by 3.

Mean at time 3. $E[X_3]=(6+10+14)/3=30/3=10.$

Mean at time 7. $E[X_7]=(6+4+2)/3=12/3=4.$

Expected product. Keep each pair together when multiplying: $E[X_3X_7]=(6\cdot6+10\cdot4+14\cdot2)/3=(36+40+28)/3=104/3.$

Covariance. $\operatorname{Cov}(X_3,X_7)=104/3-(10)(4)=104/3-120/3=-16/3.$

The covariance is $-16/3$ index-squared units. Its negative sign indicates that opposite-direction deviations contribute more on average: in these outcomes, larger-than-mean values at time 3 are paired with smaller-than-mean values at time 7. The method avoids treating the two time-point values as if they were independently paired.

Why the Shortcut Formula Holds

The raw-product formula follows by expanding the centered product in the definition. Write $\mu_s=E[X_s]$ and $\mu_t=E[X_t]$. Then use linearity of expectation and the fact that a nonrandom mean can be taken outside an expectation:

\begin{aligned} \operatorname{Cov}(X_s,X_t) &=E[(X_s-\mu_s)(X_t-\mu_t)]\\ &=E[X_sX_t-\mu_tX_s-\mu_sX_t+\mu_s\mu_t]\\ &=E[X_sX_t]-\mu_tE[X_s]-\mu_sE[X_t]+\mu_s\mu_t\\ &=E[X_sX_t]-\mu_s\mu_t-\mu_s\mu_t+\mu_s\mu_t\\ &=E[X_sX_t]-E[X_s]E[X_t]. \end{aligned}

In the fourth line, $E[X_s]=\mu_s$ and $E[X_t]=\mu_t$, so the terms combine as shown. The result is useful computationally, but the centered-product definition remains the clearest guide to interpretation: it compares each value with the mean at its own time.

Worked Example: Positive Covariance and Its Scale

A hypothetical reserve indicator is recorded at times 2 and 5. Across three equally likely process scenarios, the paired values $(X_2,X_5)$, measured in thousands of dollars, are $(2,1),(4,3),(6,5)$. Calculate the covariance and explain what its magnitude means.

Method. Use the expected-product formula, with all three scenarios assigned probability $1/3$.

Means. $E[X_2]=(2+4+6)/3=12/3=4$, and $E[X_5]=(1+3+5)/3=9/3=3.$

Expected product. $E[X_2X_5]=(2\cdot1+4\cdot3+6\cdot5)/3=(2+12+30)/3=44/3.$

Covariance. $\operatorname{Cov}(X_2,X_5)=44/3-(4)(3)=44/3-36/3=8/3.$

The positive sign indicates that same-direction deviations contribute more on average in this set of paired scenarios. The value $8/3$ is in squared thousands-of-dollars units, not in thousands of dollars. It is a scale-dependent measure, so its numerical magnitude is not a direct percentage or a scale-free measure of strength.

Check. The centered products are $(2-4)(1-3)=4$, $(4-4)(3-3)=0$, and $(6-4)(5-3)=4$. Their average is $(4+0+4)/3=8/3$, agreeing with the shortcut calculation.

Sign, Magnitude, and What Covariance Does Not Say

A positive covariance means that the expected centered product is positive; a negative covariance means it is negative. This is a statement about the average of products of deviations, not a guarantee about every possible pair of observations. Individual outcomes may depart from the overall tendency.

Covariance has the product of the two variables’ units. If one time-point measurement is expressed in dollars and the other in thousands of dollars, the covariance’s numerical value changes when the units are converted. Multiplying one variable by a positive constant multiplies covariance by that constant; multiplying it by a negative constant reverses the sign as well. Consequently, a large absolute covariance need not mean stronger dependence than a smaller covariance calculated on different scales.

A covariance of zero says that positive and negative centered products balance in expectation. It does not, in general, say that the two observations are independent. Independence means the joint behavior factors into the separate behaviors; zero covariance is only a statement about one expected product. A special case is a pair of jointly normally distributed variables, for which zero covariance does imply independence, but that additional distributional condition must be established. It is not a general rule for time-indexed observations.

Worked Example: Zero Covariance Without Independence

For a hypothetical process, three equally likely paired outcomes at times 1 and 4 are $(X_1,X_4)=(-1,1),(0,-2),(1,1)$. Determine the covariance and decide whether zero covariance means the observations are independent.

Calculate the means. $E[X_1]=(-1+0+1)/3=0$, and $E[X_4]=(1-2+1)/3=0.$

Calculate the expected product. $E[X_1X_4]=((-1)(1)+(0)(-2)+(1)(1))/3=(-1+0+1)/3=0.$

Calculate covariance. $\operatorname{Cov}(X_1,X_4)=0-(0)(0)=0.$

Check dependence. Whenever $X_1=0$, $X_4=-2$; whenever $X_1$ is either $-1$ or $1$, $X_4=1$. Thus the value at time 4 is determined by whether the value at time 1 is zero, so the observations are dependent. The covariance is zero because the centered products cancel, not because the observations are unrelated.

Common Calculation Traps

Pairing incorrectly. In $E[X_sX_t]$, multiply values that occur together in the same joint outcome. Combining values from different rows changes the joint distribution.
Using the wrong means. Subtract $E[X_s]E[X_t]$, not the square of one mean unless the two means happen to be equal.
Dropping the centering step. $E[X_sX_t]$ alone is generally not covariance. The product of the means must be subtracted.
Reading zero covariance as independence. Zero covariance means a zero expected centered product; it does not rule out other forms of dependence.
Interpreting the magnitude without units. Covariance is measured in the product of the original units and changes with rescaling.

Arithmetic check: For the outcomes $(6,6),(10,4),(14,2)$, an incorrect product sum of $96$ would give $E[X_3X_7]=32$ and the incorrect covariance $32-40=-8$. The products are $36,40,28$, whose sum is $104$, not $96$. Therefore $E[X_3X_7]=104/3$ and the correct covariance is $104/3-40=-16/3$.

For a calculation, a reliable order is to identify the paired outcomes and probabilities, compute each time-specific mean, compute the expected paired product, and then subtract the product of means. Finally, check the arithmetic and state the units and sign in context. This makes the result interpretable without implying more than covariance establishes.

Check Your Understanding

For each problem, use the paired outcomes and probabilities given; keep the two time-point values matched within each outcome.

At times 2 and 6, three equally likely outcomes are $(1,2),(3,4),(5,6)$. Calculate the covariance. (Answer: $8/3$)
A process has outcomes $(X_1,X_3)=(0,4)$ with probability $0.25$ and $(4,0)$ with probability $0.75$. Find the covariance. (Answer: $-3$)
At two specified times, the equally likely outcomes are $(1,0)$ and $(3,2)$. Calculate the covariance. (Answer: $1$)
Suppose $\operatorname{Cov}(A,B)=-2$, and define $U=3A+1$, $V=2B-5$. What is $\operatorname{Cov}(U,V)$? (Answer: $-12$)
Three equally likely paired outcomes are $(-2,4),(0,0),(2,4)$. Calculate the covariance and state whether the variables are independent. (Answer: $0$; they are dependent)

What You'll Learn