Why Likelihood Comparisons Need a Common Basis
A larger maximized likelihood can make one time-series model look preferable, but only if the likelihoods refer to the same observed data under compatible definitions. A model scored on fewer response observations may have a higher log likelihood simply because fewer density contributions were included. Likewise, a conditional likelihood and an exact likelihood need not be values of the same comparison criterion.
This issue often arises when candidate ARIMA models use different differencing orders, initial conditions, or treatments of missing observations. Before comparing likelihood values, identify exactly which random variables are scored and which are conditioned on. The relevant check is not merely whether the models were fitted to the same data file.
“Compatible definitions” requires care. Two models can use different probability distributions and still be compared by likelihood if both assign a joint density to the same response vector, under the same conditioning convention. But a joint density for \(y_2,\ldots,y_n\) conditional on \(y_1\) is not automatically comparable to a likelihood that scores only \(\Delta y_3,\ldots,\Delta y_n\) conditional on \(y_1,y_2\). The latter omits a response contribution and conditions on additional data.
What Must Match
For a direct likelihood comparison, check these parts of the likelihood specification:
- Response observations: Both likelihoods must score the same response vector, or a mathematically equivalent representation of that complete vector.
- Conditioning information: The observations treated as fixed initial values or other conditioning data must agree.
- Likelihood construction: Both calculations should use the same convention, such as exact likelihood or conditional likelihood, and handle initial states consistently.
- Missing observations: Both models must account for missingness using compatible rules. Dropping, imputing, or integrating over missing values changes what probability is evaluated.
Exact likelihoods include the probability distribution of the observed vector under the model, including any required initial-state distribution. Conditional likelihoods instead condition on specified observations, often to avoid modeling an initial distribution. Conditional likelihood can be useful, but a change in what is conditioned on changes the likelihood being calculated.
Differencing and the Observations Being Scored
For a series \(y_1,\ldots,y_n\), first differencing gives \(\Delta y_t=y_t-y_{t-1}\) for \(t=2,\ldots,n\). If \(y_1\) is fixed and the full vector \((y_2,\ldots,y_n)\) is transformed into \((\Delta y_2,\ldots,\Delta y_n)\), the transformation is one-to-one. In matrix form, each difference is the current level minus the preceding level, so the transformation matrix is triangular with diagonal entries equal to 1. Its determinant is 1, and therefore the absolute Jacobian determinant is 1.
That result applies to the full vector of differences. It does not authorize adding a density for an omitted difference, nor does it make a likelihood based only on later differences equivalent to a likelihood for the levels. If the candidate likelihood conditions on \(y_1\) and \(y_2\) and scores only \(\Delta y_3,\ldots,\Delta y_n\), it has not specified the probability contribution for \(\Delta y_2\). To compare it with a likelihood for \(y_2,\ldots,y_n\) conditional on \(y_1\), the model must also specify the distribution of \(\Delta y_2\) and its joint relationship with the later differences.
List the times or observations whose probability is included in each reported likelihood.
State which initial values or other observations are treated as given rather than scored.
Verify that any transformation includes the full relevant vector, and that missing values are handled under a compatible probability model.
If the scored data or likelihood definition differs, recompute both likelihoods on a common basis or do not interpret their difference as a direct model comparison.
Worked Example: Checking a Conditional AR(1) Likelihood
Suppose a candidate model is \(y_t=\phi y_{t-1}+\varepsilon_t\), where \(\varepsilon_t\) are independent \(N(0,\sigma^2)\) errors. The model is scored conditionally on \(y_1\), with known \(\phi=0.5\) and \(\sigma^2=1\). For observed values \(y_1=2,\ y_2=3,\ y_3=2\), calculate the conditional log likelihood and identify what it scores.
Method. Given \(y_1\), the scored responses are \(y_2\) and \(y_3\). Their conditional means are \(\phi y_1\) and \(\phi y_2\), so the innovations are the observations minus those means.
At time 2, the conditional mean is \(0.5(2)=1\), so the innovation is \(3-1=2\). At time 3, the conditional mean is \(0.5(3)=1.5\), so the innovation is \(2-1.5=0.5\). The sum of squared innovations is \(2^2+0.5^2=4.25\).
For \(m=2\) independent Gaussian innovations with variance 1, the conditional log likelihood is
The calculation scores \(y_2,y_3\) conditional on \(y_1\); it does not include a density for \(y_1\). The result is checked by using two innovations and a squared-error total of 4.25. Common mistake: subtracting \(\phi y_2\) from \(y_2\) for the time-2 innovation, or claiming the likelihood includes the initial observation. The time-2 forecast uses \(y_1\), and \(y_1\) is conditioned on, not scored.
Worked Comparisons Involving Missing Data and Differencing
Worked Example: Comparing Models on the Same Observed Times
A hypothetical monthly series is observed at times 1, 2, 4, and 6, with values \(2,4,3,5\), respectively. Times 3 and 5 are missing. Two candidate models specify independent Gaussian observations at the observed times with known variance \(\sigma^2=1\). Model A has conditional means \(3,3,3,3\); Model B has conditional means \(2,3,4,4\). Calculate both log likelihoods using only the observed responses, and compare them.
Method. Use the Gaussian log likelihood for the same four observed values under each model. With variance 1, each model has the common constant \(-\frac{4}{2}\log(2\pi)=-2\log(2\pi)\), and the remaining term is minus one-half the residual sum of squares.
For Model A, the residuals are \(2-3=-1,\ 4-3=1,\ 3-3=0,\ 5-3=2\). Their squares sum to \(1+1+0+4=6\). Thus
For Model B, the residuals are \(2-2=0,\ 4-3=1,\ 3-4=-1,\ 5-4=1\). Their squares sum to \(0+1+1+1=3\). Thus
Model B has the higher log likelihood by \(1.5000\). Both values score the same four observed responses, with the same variance and missing-data rule: the missing times contribute no observations in this specified likelihood. The comparison is valid for these stated models. It would not automatically remain valid if one candidate instead imputed the missing values or used a joint likelihood that integrated over them. Common mistake: treating a missing value as an observed response with a guessed value, thereby changing the data and likelihood.
Worked Example: Why Scoring Only Later Differences Is Not Equivalent
Consider observed levels \(y_1=10,\ y_2=12,\ y_3=11,\ y_4=13\). Candidate A is a Gaussian random walk with independent increments \(\Delta y_t\sim N(0,1)\) for \(t=2,3,4\), conditional on \(y_1\). Candidate B has independent increments \(\Delta y_t\sim N(1,1)\). First compare their full conditional likelihoods for \(y_2,y_3,y_4\) given \(y_1\). Then calculate the likelihood for Candidate B if it is instead conditioned on \(y_1,y_2\) and scores only \(\Delta y_3,\Delta y_4\).
Method. Compute all three observed increments: \(\Delta y_2=12-10=2\), \(\Delta y_3=11-12=-1\), and \(\Delta y_4=13-11=2\). The transformation from \((y_2,y_3,y_4)\) to these three increments, conditional on \(y_1\), has Jacobian determinant 1.
For Candidate A, the increments have mean 0, so the squared deviations are \(2^2,(-1)^2,2^2\), summing to \(9\). For Candidate B, the increments have mean 1, so the deviations are \(1,-2,1\), whose squares sum to \(6\). Each full likelihood has three Gaussian contributions with variance 1:
On this common basis, Candidate B has the higher log likelihood by \(1.5000\). Now change the calculation for Candidate B: condition on both \(y_1\) and \(y_2\), and score only \(\Delta y_3=-1\) and \(\Delta y_4=2\). Their deviations from mean 1 are \(-2\) and \(1\), with squared sum \(5\). The resulting conditional log likelihood is
This last value is not a re-expression of Candidate B’s full likelihood: it has only two density contributions and conditions on \(y_2\), rather than scoring \(\Delta y_2\). The determinant-one transformation does not supply the missing probability for \(\Delta y_2\). To obtain a likelihood for \(y_2,y_3,y_4\) conditional on \(y_1\), the model must specify the distribution of \(\Delta y_2\) and its joint relationship with the later increments. Common mistake: adding a density for \(\Delta y_2\) after the fact without specifying that part of the model.
Common Traps and Practical Interpretation
Likelihood values are not standardized scores whose magnitudes can be compared across arbitrary data sets. For example, the log likelihood generally accumulates contributions as the number of scored observations changes. A higher value from a shorter series or a likelihood conditioned on more observations does not establish that the corresponding model describes the same data better.
Similarly, using the same dates in a spreadsheet is not sufficient if the probability calculations differ. An exact likelihood may account for an initial distribution, while a conditional likelihood may fix initial observations. A differenced model may score differences beginning at a later time, while a level model scores the level responses. A missing-data procedure may omit a time point in one fit but integrate over it in another. Each difference must be made explicit before interpreting likelihood values.
When candidates use different differencing orders, the practical question is whether both have been formulated as densities for the same observed response vector under compatible conditioning. If not, their raw likelihood values should not be compared directly. Refit or reformulate the candidates on a common basis where justified; otherwise, report that the likelihood comparison is not valid as stated. Later model-selection criteria also rely on a clearly specified likelihood, so alignment at this stage matters.
Check Your Understanding
For each problem, check both the numerical calculation and exactly which observations the likelihood scores.
- Two Gaussian models score the same five residuals, with known variance \(\sigma^2=4\). Model A has residual sum of squares 20, and Model B has residual sum of squares 12. Which log likelihood is larger, and by how much? (Answer: Model B, by \(1.0000\) log-likelihood unit.)
- A series \(y_1,\ldots,y_8\) is transformed to first differences while conditioning on \(y_1\). How many differences must be included to represent the full transformed response vector, and which are they? (Answer: Seven, \(\Delta y_2,\ldots,\Delta y_8\).)
- One candidate scores \(y_2,\ldots,y_8\) conditional on \(y_1\); another scores only \(\Delta y_3,\ldots,\Delta y_8\) conditional on \(y_1,y_2\). Can their reported likelihoods be directly compared? State the missing requirement. (Answer: No; they score different response vectors and use different conditioning. A model for \(\Delta y_2\) and its joint relationship with later differences is needed to construct the full likelihood conditional on \(y_1\).)
- Two models are fitted to a series with observations at times 1, 2, 4, and 5, while time 3 is missing. One omits time 3; the other replaces it with an interpolated value and treats that value as observed. Are their likelihoods directly comparable? (Answer: No; they use different response data and missing-data treatments.)
- With \(y_1\) fixed, the full vector of levels \(y_2,\ldots,y_6\) is transformed into \(\Delta y_2,\ldots,\Delta y_6\). If the transformation is one-to-one, what is the absolute Jacobian determinant, and what does that imply? (Answer: It is 1; the full-vector density has no Jacobian adjustment beyond a factor of 1, provided the model is expressed for all five differences.)