Conditional Likelihoods For Independent Responses

Why Conditional Independence Gives a Product Likelihood

In a response model, each observation comes with covariate information. The response distributions may differ from one observation to another because their covariates differ, even when all observations share the same unknown parameter. If the responses are conditionally independent given the observed covariates, their joint conditional distribution can be built from separate observation-level contributions. That factorization is what makes the conditional likelihood manageable.

The condition matters: an observed collection of responses does not, by itself, justify multiplying their individual probabilities or densities. The product follows from a model assumption about how the responses depend on one another after the covariates are given. Conditional independence is also distinct from identical distribution: each response can have a different conditional distribution.

Definition: Let $Y_1,\ldots,Y_n$ be responses and let $X=(X_1,\ldots,X_n)$ denote the covariates. The responses are conditionally independent given $X$ if, for every collection of measurable sets $A_1,\ldots,A_n$, $$ P(Y_1\in A_1,\ldots,Y_n\in A_n\mid X) = \prod_{i=1}^n P(Y_i\in A_i\mid X) $$ with probability one. If the conditional distribution of $Y_i$ depends only on its own covariate $X_i$, we may write its mass function or density as $f_i(y_i\mid x_i;\theta)$. The dependence on $i$ allows observations to have different conditional distributions.

The assumption that the conditional law of $Y_i$ depends only on $X_i$ is separate from conditional independence. In some designs a response may depend on the entire covariate vector; then its contribution should be written $f_i(y_i\mid x;\theta)$. In either case, the factorization uses the conditional distribution of each response given the covariates specified by the model.

From Conditional Independence to a Likelihood

Suppose the observed covariates are $x=(x_1,\ldots,x_n)$, and the observed responses are $y=(y_1,\ldots,y_n)$. Assume that, conditional on $X=x$, the responses are independent and that each has a mass function or density $f_i(\,\cdot\mid x_i;\theta)$ with respect to an appropriate dominating measure. A mass function is used for a discrete response; a density is used for a continuous one. In either case, the joint conditional model factors into the individual contributions.

Theorem: Under conditional independence given $X$, if the conditional law of each $Y_i$ has density or mass function $f_i(y_i\mid x_i;\theta)$, then the joint conditional density or mass function is $$ f(y_1,\ldots,y_n\mid x_1,\ldots,x_n;\theta) = \prod_{i=1}^n f_i(y_i\mid x_i;\theta). $$ Consequently, for observed data, the likelihood conditional on the covariates is $$ L(\theta;y\mid x) = \prod_{i=1}^n f_i(y_i\mid x_i;\theta). $$

For discrete responses, the factorization follows directly by taking $A_i=\{y_i\}$ in the definition of conditional independence. For responses with densities, the same product rule applies to their joint conditional density. More generally, the density statement is with respect to the product of the response-specific dominating measures and holds almost everywhere under that measure.

To see the density argument, fix a covariate value $x$ for which the conditional independence and density assumptions hold. The product of the individual conditional laws defines a probability measure on response vectors. On every measurable rectangle $A_1\times\cdots\times A_n$, this product measure assigns probability $$ \prod_{i=1}^n P(Y_i\in A_i\mid X=x), $$ which is the joint conditional probability by conditional independence. Rectangles determine the joint distribution, so the product measure is the joint conditional law. Its density with respect to the product dominating measure is the product of the individual densities, giving the stated factorization. This also proves the discrete case when counting measure is used. $\square$

Once the observed data are fixed, each factor is viewed as a function of $\theta$. The conditional likelihood is therefore a product of observation-level likelihood contributions. It is conditional on $x$: it describes the probability model for the responses given those covariates, not a model for how the covariates themselves were generated.

Worked Example: Bernoulli Responses With Different Covariates

Let $X_i$ be a binary covariate, and suppose that, conditional on the covariates, the responses are independent Bernoulli variables with $$ P(Y_i=1\mid X_i=0;\theta)=\theta,\qquad P(Y_i=1\mid X_i=1;\theta)=1-\theta,\qquad 0<\theta<1. $$ The observed covariates are $x=(0,1,1)$, and the observed responses are $y=(1,0,1)$. Find the conditional likelihood.

For a Bernoulli response, the contribution is $p_i(\theta)^{y_i}\{1-p_i(\theta)\}^{1-y_i}$, where $p_i(\theta)=P(Y_i=1\mid x_i;\theta)$. The three success probabilities are $\theta$, $1-\theta$, and $1-\theta$, respectively. Thus the observed contributions are $$ \theta,\qquad 1-(1-\theta)=\theta,\qquad 1-\theta. $$

Conditional independence gives the joint conditional likelihood: $$ L(\theta;y\mid x)=\theta\cdot\theta\cdot(1-\theta)=\theta^2(1-\theta). $$ At $\theta=0.4$, the value is $0.4^2(0.6)=0.16(0.6)=0.096$. The covariates make the response distributions differ across observations, but the likelihood still factors because of the assumed conditional independence.

Conditional Likelihood Need Not Mean Identically Distributed Responses

A product likelihood does not require one common mass function or density for every observation. The factors can vary with covariates or with known observation-specific quantities. What is shared may be the parameter $\theta$, while the conditional distributions themselves differ. The next example makes this distinction explicit: the Poisson means vary by observation, but all are determined by one rate parameter and known covariate multipliers.

Worked Example: Poisson Counts With Observation-Specific Means

Suppose the responses are conditionally independent and $Y_i\mid x_i$ has a Poisson distribution with mean $\lambda c_i$, where the $c_i$ are known positive covariate multipliers and $\lambda>0$. Take $c=(1,2,1)$ and observed counts $y=(1,0,2)$. Find the likelihood and evaluate it at $\lambda=0.5$.

The three means are $\lambda,2\lambda,\lambda$. A Poisson response with mean $\mu$ has mass function $e^{-\mu}\mu^y/y!$. The observation-level contributions are therefore $$ e^{-\lambda}\lambda,\qquad e^{-2\lambda},\qquad e^{-\lambda}\frac{\lambda^2}{2}. $$

Multiplying them gives $$ L(\lambda;y\mid x) = e^{-4\lambda}\frac{\lambda^3}{2}. $$ The exponent uses the sum of the three means, $\lambda+2\lambda+\lambda=4\lambda$; the factor $1/2$ comes from the factorial $2!$ for the third count. At $\lambda=0.5$, $$ L(0.5;y\mid x) = e^{-2}\frac{(0.5)^3}{2} = e^{-2}\frac{0.125}{2} = \frac{e^{-2}}{16} \approx 0.00846. $$ The Poisson distributions are not identical because their means differ, yet the conditional joint mass function is the product of their three contributions.

Log-Likelihood Contributions

Products can be inconvenient for calculation, especially when there are many observations. When the likelihood is positive, taking its logarithm turns the product into a sum. Each summand is the log-likelihood contribution of one response.

Result: If $L(\theta;y\mid x)=\prod_{i=1}^n f_i(y_i\mid x_i;\theta)$ and all factors are positive, then $$ \ell(\theta;y\mid x) = \log L(\theta;y\mid x) = \sum_{i=1}^n \log f_i(y_i\mid x_i;\theta). $$

Proof. Apply the logarithm to both sides of the likelihood factorization. The logarithm of a finite product of positive numbers is the sum of their logarithms, so $$ \log\left\{\prod_{i=1}^n f_i(y_i\mid x_i;\theta)\right\} = \sum_{i=1}^n \log f_i(y_i\mid x_i;\theta). $$ This gives the stated expression. If any contribution is zero, the likelihood is zero and its log is $-\infty$; the sum representation remains valid with the convention $\log 0=-\infty$. $\square$

Worked Example: Normal Responses With a Shared Parameter

Suppose that, conditional on $x=(0,1,2)$, the responses are independent with $Y_i\mid x_i\sim N(\theta x_i,1)$, where the second argument denotes variance. The observed responses are $y=(0,1,1)$. Derive the conditional likelihood and calculate it at $\theta=0.5$.

The Normal density with mean $\theta x_i$ and variance $1$ is $$ f_i(y_i\mid x_i;\theta) = (2\pi)^{-1/2}\exp\left\{-\frac{(y_i-\theta x_i)^2}{2}\right\}. $$ Conditional independence gives $$ L(\theta;y\mid x) = (2\pi)^{-3/2} \exp\left\{-\frac{1}{2}\sum_{i=1}^3 (y_i-\theta x_i)^2\right\}. $$ At $\theta=0.5$, the fitted means are $0,0.5,1$. The residuals are $0,0.5,0$, so the sum of squared residuals is $0^2+0.5^2+0^2=0.25$. Hence $$ L(0.5;y\mid x) = (2\pi)^{-3/2}\exp(-0.25/2) = (2\pi)^{-3/2}e^{-1/8} \approx 0.0560. $$ Each factor here is a density value. The product is the joint conditional density at the observed response vector, not the probability of observing that exact vector.

What Conditioning Does—and a Common Mistake

The conditional likelihood treats the covariates as given and models the responses conditional on them. If a joint probability model for covariates and responses is used instead, it includes both the response distribution given the covariates and the distribution of the covariates. Conditioning removes the latter from the likelihood being calculated here. This distinction is useful whenever covariates are part of the observed data but the inferential model is specifically for responses given those covariates.

A common mistake is to multiply the individual conditional contributions merely because there is one response recorded per observation. The correction is to check the dependence assumption: the product is justified when the responses are conditionally independent given the covariates. If they remain dependent after conditioning, their joint conditional model must account for that dependence; multiplying separate marginal conditional contributions would not give the joint likelihood.

A second mistake is to interpret a product likelihood as evidence that responses are identically distributed. The examples show otherwise: observation-specific probabilities, means, or conditional means can differ. Conditional independence concerns factorization of the joint conditional distribution; identical distribution concerns whether the individual distributions have the same form and parameters.

Key takeaway: Given covariates, conditional independence turns the joint response model into a product of observation-level mass functions or densities. The factors may differ across observations, and the resulting likelihood is a function of the unknown parameter with the covariates held fixed.

Check Your Understanding

Use the conditional model and factorization rules in this tutorial to answer the following.

In one sentence, state what conditional independence of $Y_1,\ldots,Y_n$ given $X$ means.
Why does conditional independence not imply that all responses have the same conditional distribution?
For independent Bernoulli responses with success probabilities $0.3$ and $0.7$, and observed values $1$ and $0$, write the joint conditional likelihood contribution.
For independent Poisson responses with means $2\lambda$ and $\lambda$, and observed counts $0$ and $1$, write the likelihood as a function of $\lambda$.
What additional factor would appear in a joint model for covariates and responses that is not part of the conditional response likelihood?