From Poisson Responses to a Likelihood
A regression model specifies how the distribution of a response depends on covariates. To use that model to assess a particular set of regression coefficients, we need the probability of the observed responses under those coefficients. For independent Poisson responses, this probability is a product of individual Poisson probabilities. Taking its logarithm turns that product into a sum, with one transparent term for each observation.
The tutorial on the canonical exponential-family form for Poisson responses gives the Poisson mass function, and the Poisson mean and variance are both equal to its parameter. Here the focus is narrower: construct the likelihood for a collection of independent responses when their means are determined by a log-linear regression model.
\(L(\beta)=P(Y_1=y_1,\ldots,Y_n=y_n\mid x_1,\ldots,x_n)\).
The design vectors \(x_i\) and observed counts \(y_i\) are held fixed when writing \(L(\beta)\). Define the linear predictor \(\eta_i=x_i^T\beta\). The model mean is then \(\mu_i=\exp(x_i^T\beta)\), which is positive for every finite \(\beta\). The likelihood records how compatible each candidate coefficient vector is with the observed counts under the stated model; it is not a probability distribution over coefficient vectors.
Deriving the Joint Likelihood
For a Poisson random variable with mean \(\mu_i\), the probability of observing the nonnegative integer \(y_i\) is \(e^{-\mu_i}\mu_i^{y_i}/y_i!\). Independence means the probability of observing the entire response vector is the product of these individual probabilities. Substituting the log-linear mean gives the regression likelihood.
\(L(\beta)=\displaystyle\prod_{i=1}^n \frac{\exp\{-\exp(x_i^T\beta)\}\exp\{y_i x_i^T\beta\}}{y_i!}\).
Equivalently, \(L(\beta)=\displaystyle\prod_{i=1}^n \frac{\exp\{y_i x_i^T\beta-\exp(x_i^T\beta)\}}{y_i!}\).
To verify the result directly, independence gives \(P(Y_1=y_1,\ldots,Y_n=y_n\mid x_1,\ldots,x_n)=\prod_{i=1}^n P(Y_i=y_i\mid x_i)\). For each \(i\), the Poisson probability at \(y_i\) is \(\exp(-\mu_i)\mu_i^{y_i}/y_i!\). Replacing \(\mu_i\) by \(\exp(x_i^T\beta)\), and using \(\{\exp(x_i^T\beta)\}^{y_i}=\exp\{y_i x_i^T\beta\}\), yields the displayed product. This derivation requires independent responses conditional on the fixed design and the specified Poisson distributions; without independence, the product need not be the joint probability. \(\square\)
The log-likelihood is the natural logarithm of this likelihood. Since the likelihood is a product of positive terms, the logarithm of the product is the sum of their logarithms.
\(\ell(\beta)=\log L(\beta)=\displaystyle\sum_{i=1}^n \left[y_i x_i^T\beta-\exp(x_i^T\beta)-\log(y_i!)\right]\).
The contribution from observation \(i\) is \(\ell_i(\beta)=y_i\eta_i-\exp(\eta_i)-\log(y_i!)\), where \(\eta_i=x_i^T\beta\).
For the derivation, take the logarithm of the product in the likelihood: \(\log L(\beta)=\sum_{i=1}^n\log\left[\exp\{y_i x_i^T\beta-\exp(x_i^T\beta)\}/y_i!\right]\). For each summand, the logarithm of the numerator is \(y_i x_i^T\beta-\exp(x_i^T\beta)\), and the logarithm of the denominator is \(\log(y_i!)\). Subtracting the latter gives the stated expression. Thus the full likelihood contribution of one observation has three pieces: a count times its linear predictor, a negative fitted mean, and a negative log-factorial. \(\square\)
Reading the Observation-Level Contribution
The linear predictor enters the contribution in two places. Its first appearance is \(y_i\eta_i\); its second is through the mean \(\exp(\eta_i)\). The factorial term depends on the observed count but not on \(\beta\). It is still part of the full log-likelihood and should be retained when calculating the actual likelihood or the log-probability of the observed data.
When comparing coefficient vectors for the same observed data, the factorial terms are identical for every candidate. They therefore cancel in a difference of log-likelihoods, or in a likelihood ratio. That cancellation does not make them absent from the full likelihood. Keeping the full expression is a reliable way to avoid confusion about what quantity is being calculated.
Worked Example: Three Independent Counts
Suppose there are three independent responses, with design vectors \(x_1=(1,0)^T\), \(x_2=(1,1)^T\), and \(x_3=(1,0)^T\). Let \(\beta=(0,\log 2)^T\), and suppose the observed counts are \(y_1=0\), \(y_2=2\), and \(y_3=1\). Construct the likelihood and log-likelihood at this coefficient vector.
First calculate each linear predictor and mean: \( \eta_1=0,\ \eta_2=\log 2,\ \eta_3=0\), so \( \mu_1=1,\ \mu_2=2,\ \mu_3=1\). The individual Poisson probabilities are \(P(Y_1=0)=e^{-1}\), \(P(Y_2=2)=e^{-2}2^2/2!=2e^{-2}\), and \(P(Y_3=1)=e^{-1}/1!=e^{-1}\).
Independence gives \(L(\beta)=e^{-1}(2e^{-2})(e^{-1})=2e^{-4}\). The observation-level log contributions are \(\ell_1=0(0)-1-\log(0!)=-1\), \(\ell_2=2\log 2-2-\log(2!)=\log 2-2\), and \(\ell_3=1(0)-1-\log(1!)=-1\). Here \(0!=1\), \(1!=1\), and \(2!=2\). Summing gives \(\ell(\beta)=-1+(\log 2-2)-1=\log 2-4\), which agrees with \(\log(2e^{-4})=\log 2-4\).
Worked Example: Calculating Contributions From Linear Predictors
Consider two independent responses with \(x_1=(1,0)^T\), \(x_2=(1,1)^T\), coefficient vector \(\beta=(\log 3,\log 2)^T\), and observed counts \(y_1=2\), \(y_2=0\). Find the means, each log-likelihood contribution, and the total likelihood.
The linear predictors are \(\eta_1=\log 3\) and \(\eta_2=\log 3+\log 2=\log 6\). Thus the means are \(\mu_1=3\) and \(\mu_2=6\). For the first observation, \(\ell_1=2\log 3-3-\log(2!)=2\log 3-3-\log 2\). For the second, \(\ell_2=0(\log 6)-6-\log(0!)=-6\). Therefore \(\ell(\beta)=2\log 3-\log 2-9\).
The likelihood calculated directly is \(P(Y_1=2)P(Y_2=0)=(e^{-3}3^2/2!)(e^{-6})=(9/2)e^{-9}\). Its logarithm is \(\log(9/2)-9=2\log 3-\log 2-9\), matching the sum of contributions.
Comparing Candidate Coefficient Vectors
The likelihood is a function of the coefficients, so it can be evaluated at different candidate vectors while keeping the data and design fixed. This comparison illustrates why the factorial terms can be omitted from a likelihood difference, even though they belong in the full log-likelihood.
Worked Example: A Likelihood Ratio for Two Candidates
Let \(x_1=(1,0)^T\), \(x_2=(1,1)^T\), with observed counts \(y_1=1\) and \(y_2=3\). Compare \(\beta_A=(0,\log 2)^T\) and \(\beta_B=(\log 2,0)^T\) by calculating \(L(\beta_B)/L(\beta_A)\).
Under \(\beta_A\), the linear predictors are \(0,\log 2\), giving means \(1,2\). Hence \(L(\beta_A)=(e^{-1})(e^{-2}2^3/3!)=(4/3)e^{-3}\). The corresponding log-likelihood contributions are \(\ell_{1,A}=1(0)-1-\log(1!)=-1\) and \(\ell_{2,A}=3\log 2-2-\log(3!)=3\log 2-2-\log 6\). Their sum is \(\log(4/3)-3\), as required.
Under \(\beta_B\), both linear predictors equal \(\log 2\), so both means equal \(2\). Then \(L(\beta_B)=(e^{-2}2)(e^{-2}2^3/3!)=(8/3)e^{-4}\). Therefore \(L(\beta_B)/L(\beta_A)=[(8/3)e^{-4}]/[(4/3)e^{-3}]=2/e\). Equivalently, the log-likelihood difference is \(\ell(\beta_B)-\ell(\beta_A)=\log(2/e)=\log 2-1\). The terms \(-\log(1!)\) and \(-\log(3!)\) are the same under both candidates and cancel from this difference.
Common Mistakes and a Practical Check
A frequent error is to write the Poisson mass for each response correctly but then add the probabilities rather than multiply them. Independence gives a product for the joint probability. It is only after taking logarithms that the contributions add. Another error is to use \(y_i\eta_i-\eta_i\) as the contribution: the model subtracts the mean \(\exp(\eta_i)\), not the linear predictor \(\eta_i\).
- Wrong: “The log-likelihood contribution is \(y_i\eta_i-\eta_i\), because the mean is represented by the linear predictor.” Correction: The linear predictor is \(\eta_i\), but the mean is \(\exp(\eta_i)\); the correct contribution is \(y_i\eta_i-\exp(\eta_i)-\log(y_i!)\).
- Wrong: “The factorial can always be removed because it is constant.” Correction: It is constant in \(\beta\) for fixed data, but it is needed for the full likelihood and its numerical value.
- Wrong: “A likelihood is a probability distribution over \(\beta\).” Correction: The likelihood is the joint probability of the observed data, viewed as a function of \(\beta\); it does not assign probabilities to coefficient values.
Check Your Understanding
Use the independent Poisson model and the observation-level contribution to answer these questions.
- State the assumption that permits the joint likelihood to be written as a product of individual Poisson probabilities.
- For one observed count \(y_i\) and linear predictor \(\eta_i\), write the complete log-likelihood contribution.
- If \(y_i=0\) and \(\eta_i=\log 4\), calculate the contribution, including the factorial term.
- Explain why the factorial terms cancel when comparing two coefficient vectors for the same data.
- Explain the difference between a likelihood as a function of \(\beta\) and a probability distribution over \(\beta\).