Why Work With Observation-Level Contributions?
In nonlinear econometric models, the probability assigned to an observation depends on unknown parameters through a response function, a distributional parameter, or both. A useful way to build the sample likelihood is to begin with the probability model for one observation and then combine those contributions across observations. This makes clear which parts of the objective depend on the parameters and how the model’s nonlinear structure enters the calculation.
The central operation is simple but requires care: when observations are independent conditional on the explanatory variables, their joint likelihood is a product. Taking logarithms turns that product into a sum. Each term in the sum is an observation-level log likelihood contribution. The sum is the sample log likelihood, not an average unless we explicitly divide it by the sample size.
Here, \(x_i\) is treated as observed, and the likelihood is a function of \(\theta\) after the data have been observed. A probability mass function is used for discrete outcomes, while a density is used for continuous outcomes. The independence condition is conditional on the explanatory variables in these examples; the contributions need not have identical distributions or identical covariates. If observations are dependent, the joint model must instead be factorized appropriately, for example into conditional densities given earlier observations. Summing marginal log densities alone is not generally valid.
From a Joint Model to a Sum
The product-to-sum result follows directly from the definition of conditional independence and the logarithm of a product. For a sample with conditionally independent outcomes, the joint probability mass or density is the product of the individual terms. Taking its logarithm gives:
This is more than a notational convenience. It gives a direct way to construct the objective from the model’s probability statement, and it helps locate mistakes: omitting one observation, using the wrong distribution, or dropping a term that depends on \(\theta\) changes the objective. The contribution \(\ell_i(\theta)\) is allowed to vary across observations because outcomes and explanatory variables vary.
Worked Example: Bernoulli Logit Contributions
Worked Example: Bernoulli Logit Contributions
Suppose a binary outcome \(y_i\) is conditionally Bernoulli with success probability \(p_i\), and the model specifies \(p_i=\exp(\eta_i)/(1+\exp(\eta_i))\), where \(\eta_i=\beta_0+\beta_1x_i\). Assume the outcomes are conditionally independent given the observed \(x_i\). For three hypothetical observations, take \(x=(0,1,2)\), \(y=(0,1,0)\), and evaluate the log likelihood at \(\beta_0=-0.4\) and \(\beta_1=0.8\).
For a Bernoulli observation, \(f_i(y_i\mid x_i;\beta)=p_i^{y_i}(1-p_i)^{1-y_i}\), where \(\beta=(\beta_0,\beta_1)\). Taking logs gives the individual contribution:
The second equality follows by substituting the logistic probability and collecting terms. The linear predictor is \(\eta_i=-0.4+0.8x_i\), so its three values are \(-0.4\), \(0.4\), and \(1.2\). The corresponding probabilities are approximately \(0.4013\), \(0.5987\), and \(0.7685\).
For observation 1, \(y_1=0\), so \(\ell_1=-\log(1+\exp(-0.4))\approx-0.5130\). For observation 2, \(y_2=1\), so \(\ell_2=0.4-\log(1+\exp(0.4))\approx-0.5130\). For observation 3, \(y_3=0\), so \(\ell_3=-\log(1+\exp(1.2))\approx-1.4633\). Thus:
This is the sample log likelihood at the specified parameter values, not an estimate of the parameters. The example uses the Bernoulli model and conditional independence as stated; it does not assert that the chosen values maximize the likelihood. The nonlinear dependence appears because the probabilities are logistic functions of the parameters, and the log contributions include \(\log(1+\exp(\eta_i))\).
What Counts as a Constant?
A likelihood contribution can contain both parameter-dependent terms and terms that are constant with respect to \(\theta\). A term may depend on the observed data and still be constant for likelihood optimization: once the data are fixed, it does not change as \(\theta\) changes. Such a term can be omitted when finding a maximizer, provided it is genuinely independent of every parameter being estimated. It should be retained when reporting the full log likelihood or comparing quantities for which its value matters.
The distinction is about parameter dependence, not whether a term looks complicated. In a Poisson model, \(-\log(y_i!)\) depends on the observed count but not on the Poisson mean, so it is constant in the mean parameters. By contrast, the term \(-\lambda_i(\theta)\) depends on the parameters through the mean and cannot be dropped. In a normal model with an unknown variance, the log-normalizing term involving that variance is parameter-dependent and must be retained.
Worked Example: Poisson Counts With a Nonlinear Mean
Consider conditionally independent counts \(y_i\) with Poisson means \(\lambda_i\). Suppose the mean function is \(\lambda_i=\exp(\theta_0)x_i/(\theta_1+x_i)\), with \(x_i>0\), \(\exp(\theta_0)>0\), and \(\theta_1+x_i>0\). This is a specified nonlinear response function: the unknown parameters enter the mean through both an exponential and a ratio. For two hypothetical observations, let \(x=(1,2)\), \(y=(2,5)\), and evaluate at \(\theta_0=\log 6\) and \(\theta_1=1\).
The Poisson mass function is \(f_i(y_i\mid x_i;\theta)=\exp(-\lambda_i)\lambda_i^{y_i}/y_i!\), for nonnegative integer \(y_i\). Therefore the observation-level log contribution is:
At the chosen parameter values, \(\exp(\theta_0)=6\). For \(x_1=1\), \(\lambda_1=6(1)/(1+1)=3\); for \(x_2=2\), \(\lambda_2=6(2)/(1+2)=4\). The first contribution is \(2\log 3-3-\log(2!)\approx2.1972-3-0.6931=-1.4959\). The second is \(5\log 4-4-\log(5!)\approx6.9315-4-4.7875=-1.8560\). Hence the sample log likelihood is:
The factorial terms are constants with respect to \(\theta_0\) and \(\theta_1\), but \(\log\lambda_i\) and \(-\lambda_i\) both depend on those parameters. The independence assumption permits the two contributions to be added; the Poisson specification supplies the correct count probabilities. The calculation illustrates how a nonlinear mean function is inserted into the likelihood without replacing the probability model with a least-squares objective.
Nonlinear Dependence Can Enter Through the Mean
A probability model’s parameters need not enter only through a linear predictor. For a continuous outcome, a nonlinear mean function can be substituted into a density in the same way. The density’s normalizing terms and residual terms must both be included, with careful attention to whether the variance is fixed or estimated.
Worked Example: Gaussian Outcomes With a Nonlinear Mean
Suppose \(y_i\), conditional on \(x_i\), is normally distributed with mean \(m_i(\theta)=\theta_1/[1+\exp(-\theta_2x_i)]\) and known standard deviation \(\sigma=1.5\). Assume conditional independence. This model uses a nonlinear function of the parameters for the conditional mean. For two hypothetical observations, take \(x=(0,1)\), \(y=(1.5,4)\), and evaluate at \(\theta_1=4\), \(\theta_2=\log 3\).
The conditional density is normal with variance \(\sigma^2\), so the individual log likelihood contribution is:
At \(x_1=0\), \(m_1=4/(1+1)=2\). At \(x_2=1\), \(\exp(-\theta_2)=1/3\), so \(m_2=4/(1+1/3)=3\). The shared normalizing term is \(-\tfrac12\log(2\pi(1.5)^2)\approx-1.3241\). The first squared residual is \((1.5-2)^2=0.25\), giving \(\ell_1\approx-1.3241-0.25/4.5=-1.3797\). The second squared residual is \((4-3)^2=1\), giving \(\ell_2\approx-1.3241-1/4.5=-1.5463\). Therefore:
The normalizing term is constant with respect to \(\theta_1\) and \(\theta_2\) because \(\sigma\) is known and fixed here. The squared residual terms depend on these parameters through \(m_i(\theta)\). If \(\sigma\) were an unknown parameter, the normalizing term would no longer be constant with respect to the full parameter vector and could not be omitted in optimization.
A Reliable Construction Routine
When constructing a likelihood contribution, the main safeguard is to proceed from the probability model rather than guessing an objective based on a familiar regression formula. Keep the contribution tied to the particular observation, including its covariates and observed outcome. The following sequence makes the assumptions and parameter dependence visible:
Specify the probability mass function or density for each outcome given its covariates and parameters, including any distributional assumptions.
Substitute the observation’s \(y_i\) and \(x_i\) into the model’s mass or density, keeping the unknown parameters as arguments.
Expand the expression carefully and identify every term that depends on any parameter in the model.
For conditional independence, sum the log contributions. For dependent observations, use the appropriate joint or sequential conditional factorization instead.
Common Pitfalls and Why the Objective Matters
A frequent mistake is to treat every term involving observed data as disposable. For example, \(-\log(y_i!)\) in the Poisson example can be omitted when maximizing over the mean parameters, but \(-\lambda_i(\theta)\) cannot. Another mistake is to drop a normalizing term because it is constant in the outcome; if it depends on an unknown variance, it is not constant in the parameters. Always ask: does this term change when any parameter being estimated changes?
A second pitfall is adding marginal log densities when observations are dependent. The sum formula follows from a product factorization of the joint model; it is not a general identity for dependent data. A third is confusing a sample log likelihood with its average. The average log likelihood is \(\ell(\theta)/n\); for a fixed sample size, it has the same maximizer as \(\ell(\theta)\), but the two quantities have different numerical scales and should be named accurately.
Observation-level contributions are useful because the full objective is assembled transparently from the stated model. They also make it possible to inspect which observations contribute more or less to the total at particular parameter values. But a contribution is not by itself a measure of causal effect or a guarantee that the model is correctly specified: its interpretation remains conditional on the assumptions used to define the probability model.
Check Your Understanding
Answer each question by referring to the probability model and its parameter dependence.
- Why does conditional independence allow a sample log likelihood to be written as a sum?
- In a Bernoulli logit model, which part of the individual contribution reflects the nonlinear transformation of the linear predictor?
- In a Poisson model, why may \(-\log(y_i!)\) be omitted when maximizing over mean parameters, while \(-\lambda_i(\theta)\) may not?
- In the Gaussian example, would the normalizing term remain constant if \(\sigma\) were unknown? Explain.
- What additional factorization is needed before individual terms can be summed for dependent outcomes?