Likelihood Quantities Used In Information Criteria

Introduction

Information criteria summarize a fitted model’s likelihood while accounting for model complexity. Before comparing criteria, you must be able to locate three quantities correctly: the maximized log likelihood, the sample size, and the number of free parameters estimated. A mistake in any one of these can change a comparison, even when the model output is otherwise read correctly.

This tutorial focuses on identifying those quantities, not on choosing a model from criterion values. The key discipline is to separate the likelihood calculation from other output that describes fit, such as residual sums of squares, deviance, or a goodness-of-fit p-value.

Definition: For a model with parameter vector \(\theta\) and observed data \(y\), the likelihood is \(L(\theta;y)\), viewed as a function of \(\theta\) with the data held fixed. The log likelihood is \(\ell(\theta;y)=\log L(\theta;y)\). The maximized log likelihood is \(\ell(\hat{\theta};y)\), where \(\hat{\theta}\) is the maximum-likelihood estimate. It is not the log likelihood at an arbitrary parameter value.

The Three Quantities to Identify

The standard forms of the Akaike information criterion and Bayesian information criterion use the maximized log likelihood, a parameter count, and—in BIC—the sample size:

\mathrm{AIC}=-2\ell(\hat{\theta};y)+2k, \qquad \mathrm{BIC}=-2\ell(\hat{\theta};y)+k\log n.

Here \(k\) is the number of free parameters estimated in the fitted model, and \(n\) is the number of observational units represented by the likelihood under the stated sampling model. These formulas help identify what the output must supply. Their terms are not interchangeable: a residual statistic is not \(\ell(\hat{\theta};y)\), and a p-value is not \(n\) or \(k\).

Maximized log likelihood: use the fitted model’s log likelihood evaluated at its maximum-likelihood estimates. It is often reported as “logLik,” “log likelihood,” or “−2 log likelihood.” If output gives \(-2\ell\), the maximized log likelihood is its negative one-half.
Sample size \(n\): count observational units under the likelihood—not automatically the number of rows printed in a summary table, the number of categories, or the number of events.
Parameter count \(k\): count distinct free parameters estimated in fitting the model. A parameter known and fixed in advance is not estimated. An estimated nuisance parameter, such as a variance or scale parameter, does count.

For independent observations, the log likelihood is the sum of the individual log-likelihood contributions. This explains why the unit of observation matters: each independent record contributes according to the model, while a table may merely summarize many such records. For grouped data, the likelihood may be written directly in terms of counts, and the total number of observations represented by those counts is the relevant sample size.

Why the Likelihood Adds Across Independent Records

Suppose \(Y_1,\ldots,Y_n\) are independent observations with model densities or probability functions \(f(y_j;\theta)\). Independence gives a product likelihood. Taking logarithms turns that product into a sum:

L(\theta;y)=\prod_{j=1}^{n} f(y_j;\theta) \quad\Longrightarrow\quad \ell(\theta;y)=\sum_{j=1}^{n}\log f(y_j;\theta).

The maximized log likelihood is this sum evaluated at the parameter value \(\hat{\theta}\) that maximizes it. This also clarifies why an event count need not equal \(n\): a record with no event may still contribute information through its likelihood contribution. It is the observational unit defined by the model—not merely the number of observed events—that determines the count.

Worked Example: Maximizing an Exponential Severity Likelihood

A hypothetical portfolio has four independent, fully observed claim severities, measured in thousands of dollars: \(1,2,3,4\). Assume each severity has an exponential distribution with rate \(\lambda>0\), density \(f(x;\lambda)=\lambda e^{-\lambda x}\) for \(x>0\). Identify the maximized log likelihood, \(n\), and \(k\). A software summary also reports a residual sum of squares of \(5.0\).

Method. Write the likelihood from the assumed density, maximize it, and then count observations and estimated parameters. For the four severities, \(\sum x_j=1+2+3+4=10\), so

\ell(\lambda)=\sum_{j=1}^{4}\log(\lambda e^{-\lambda x_j}) =4\log\lambda-10\lambda.

Differentiate with respect to \(\lambda\): \(\ell'(\lambda)=4/\lambda-10\). Setting this equal to zero gives \(\hat{\lambda}=4/10=0.4\). The second derivative is \(\ell''(\lambda)=-4/\lambda^2<0\) for \(\lambda>0\), confirming a maximum.

Substitute the estimate:

\ell(\hat{\lambda})=4\log(0.4)-10(0.4) =4(-0.916291)-4 =-7.665164.

Thus the maximized log likelihood is approximately \(-7.6652\), \(n=4\) claim records, and \(k=1\) because only \(\lambda\) was estimated. The residual sum of squares \(5.0\) is not a likelihood and does not replace any of these quantities. Common mistake avoided: counting the four severity values as four parameters. They are observations; the fitted model has one free parameter.

Counting Units and Parameters in Less Direct Output

The sample size is determined by the likelihood’s observational structure. If a data set contains one record per policy, then the record count is generally the relevant \(n\), even if only some policies experienced an event. If an analysis uses grouped frequency counts, the number of categories is not the sample size: the category counts represent the observational units. The model and the way the likelihood was formed must make the unit clear.

Likewise, the parameter count is based on what was estimated, not on how many coefficients appear under a particular heading. For example, an estimated intercept, an estimated slope, and an estimated variance give \(k=3\). A known offset or a fixed threshold does not add to \(k\). A parameter estimated from the same data counts even if it is called a nuisance parameter and is not the main quantity of interest.

When censoring or truncation is part of the data structure, the likelihood contribution must reflect that structure. For instance, an individual right-censored record contributes a survival-probability term rather than a density at an unobserved event time. It still represents an observed record. The likelihood must be the one appropriate to the sampling scheme; do not substitute a likelihood for fully observed data. The details of constructing such likelihoods are separate from identifying the criterion quantities once the fitted likelihood is specified.

Worked Example: Event Count Is Not the Sample Size

A hypothetical study follows five independent equipment leases. Three leases end with a recorded failure, and two are still operating when observation stops. Under an exponential time-to-failure model with rate \(\lambda\), the total exposure time is 8 years. The log likelihood for these event and right-censoring records is \(\ell(\lambda)=3\log\lambda-8\lambda\). Identify the maximized log likelihood, sample size, and parameter count.

Method. The likelihood has three observed failures and eight total years of exposure. Differentiate: \(\ell'(\lambda)=3/\lambda-8\). The maximum occurs at \(\hat{\lambda}=3/8=0.375\), and \(\ell''(\lambda)=-3/\lambda^2<0\), so it is a maximum.

Evaluate:

\ell(\hat{\lambda})=3\log(0.375)-8(0.375) =3(-0.980829)-3 =-5.942487.

The maximized log likelihood is approximately \(-5.9425\). There are \(n=5\) lease records, not \(n=3\): the two right-censored leases are observed units and contribute to the likelihood through their exposure. There is one estimated parameter, so \(k=1\). Common mistake avoided: using the number of failures as the sample size. The event count appears in the likelihood expression, but the study contains five observational records.

Likelihood Output Is Not a Fit Diagnostic

A residual statistic summarizes discrepancies between observations and fitted values under a particular definition. A goodness-of-fit p-value is a probability calculated under a specified test and its assumptions. Neither quantity is the maximized log likelihood. A model can have a small residual statistic or a large p-value without either being a criterion input.

Check the label and definition before copying a software output. If it reports “−2 log likelihood,” convert it to \(\ell(\hat{\theta};y)\) by multiplying by \(-1/2\). If it reports deviance, residual sum of squares, a test statistic, or a p-value, do not relabel it as a log likelihood. Output conventions differ, so confirm what a reported number means rather than relying on its position in a table.

Worked Example: Separating Model Output From Criterion Inputs

A hypothetical log-severity regression is fitted to 12 independent observed claims. The model estimates an intercept, a slope, and an error variance; a deductible adjustment is supplied as a known fixed offset. The software reports logLik \(=-28.40\), residual sum of squares \(=9.6\), and a goodness-of-fit p-value of \(0.31\). Identify the required quantities and correct two proposed interpretations: “\(k=4\), because the offset is in the model” and “the p-value is the likelihood input.”

Method. The output explicitly labels the maximized log likelihood as \(-28.40\), so \(\ell(\hat{\theta};y)=-28.40\). There are 12 observed claim records, hence \(n=12\). The intercept, slope, and error variance are three estimated free parameters, so \(k=3\). The deductible offset is fixed, not estimated, and is excluded from \(k\).

The residual sum of squares \(9.6\) and the p-value \(0.31\) are diagnostics, not substitutes for the maximized log likelihood. Therefore both proposed interpretations are incorrect: the offset does not make \(k=4\), and the p-value is not the likelihood input. Common mistake avoided: counting every quantity mentioned in a model description as an estimated parameter, rather than counting only free parameters fitted from the data.

Common Exam Traps

The following checks prevent the most frequent misreadings:

Using the raw likelihood instead of its logarithm. The criterion formula uses \(\ell(\hat{\theta};y)\), not \(L(\hat{\theta};y)\). A likelihood is nonnegative; a log likelihood may be negative.
Using an unmaximized value. The required log likelihood is evaluated at the fitted maximum-likelihood estimate, not at a convenient or null parameter value unless that value is in fact the maximum under the model being assessed.
Confusing events with records. Count the observational units represented by the likelihood. An observed unit can contribute information even if no event occurred during follow-up.
Counting fixed quantities as estimated parameters. Known values and fixed offsets are not free fitted parameters. Estimated nuisance parameters are.
Substituting a diagnostic. Residual sums, deviance, test statistics, and p-values answer different questions. Match each output to its definition.

A defensible identification should state the model’s likelihood convention, name the unit counted by \(n\), and list the free estimated parameters that make up \(k\). This is especially important when comparing model output assembled in different formats: the numerical labels alone do not establish that the quantities have the required meanings.

Key takeaway: Identify \(\ell(\hat{\theta};y)\) from the fitted likelihood, \(n\) from the observational units represented in that likelihood, and \(k\) from the free parameters estimated. Keep these distinct from residual statistics and goodness-of-fit p-values.

Check Your Understanding

For each situation, identify the requested likelihood quantities and avoid substituting a diagnostic or event count.

A hypothetical exponential severity model is fitted to five claims totaling 15 thousand dollars. Its rate estimate is \(1/3\) per thousand dollars. The log likelihood is \(5\log\lambda-15\lambda\). Give the maximized log likelihood, \(n\), and \(k\), to four decimal places for the log likelihood. (Answer: \(\ell(\hat{\lambda})=5\log(1/3)-5=-10.4931\), \(n=5\), \(k=1\).)
A study has nine independent subjects, of whom four experience the event; the other five are right-censored. A model estimates one hazard parameter. What is \(n\), and what is \(k\)? (Answer: \(n=9\), \(k=1\); the event count is four.)
A fitted regression estimates two coefficients and one variance parameter; a known exposure offset is also included. It is fitted to 16 independent records. Identify \(n\) and \(k\). (Answer: \(n=16\), \(k=3\); the fixed offset is not estimated.)
Software reports “−2 log likelihood \(=42.8\),” a residual deviance of \(17.1\), and a p-value of \(0.08\). What is the maximized log likelihood? (Answer: \(\ell(\hat{\theta};y)=-42.8/2=-21.4\); neither the deviance nor the p-value is the log likelihood.)
A multinomial table summarizes 30 independent policies across four outcome categories. The category probabilities are fixed in advance, and the likelihood is based on the four category counts. What sample size and parameter count should be identified? (Answer: \(n=30\), \(k=0\); four categories do not mean four observations, and no probabilities were estimated.)

What You'll Learn