Building A Response Model From Support To Likelihood

From Possible Responses to a Likelihood

A response model is more than a formula for a probability or density. To use it coherently, we need to say which response values are possible, how probabilities or densities are assigned to those values, and how the observed responses were sampled. Only then can we write a likelihood for the data. This sequence matters in generalized linear models because the response distribution is the starting point for modeling how outcomes relate to covariates.

The support specifies the set of possible response values. For a discrete response, a probability mass function assigns probabilities to those values and sums to one. For a continuous response, a density is nonnegative and integrates to one; its value at a point is not the probability of observing that exact point. Once the sampling assumptions are stated, the individual probability or density contributions can be combined into a likelihood.

Definition: A response model specifies a support for a random variable $Y$, a probability mass function or density on that support, and a parameter space for the unknown model parameter. Given observed responses $y_1,\ldots,y_n$, the likelihood $L(\theta)$ is their joint probability mass or joint density, evaluated at those observations and regarded as a function of $\theta$.

The likelihood is a function of the parameter for fixed observed data. Its numerical value is not generally the probability that the parameter is true, nor—when responses are continuous—the probability of the observed data.

Three Ingredients of a Response Model

A useful construction keeps three choices separate. First, identify the response support. Second, choose a distribution whose mass function or density is defined on that support. Third, state the sampling relationship among observations, including whether distributions can vary across observations. Identical distributions are not required for independence, and independence should not be silently inferred from identical marginal distributions.

State what can be observed.
Specify the support, such as $\{0,1\}$, the nonnegative integers, or the real line.

Assign probabilities or density.
Give a mass function or density, its parameter space, and any boundary cases needed to make it well defined.

State the sampling assumptions.
For example, responses may be independent conditional on fixed covariates, while having different response distributions at different covariate values.

Evaluate the joint model at the data.
Use the joint mass or density to obtain the likelihood as a function of the unknown parameters.

In a covariate-based model, write $Y_i\mid x_i$ for the response distribution at a given covariate value $x_i$. Treating the observed covariates as fixed, one can specify a separate mass function or density for each response. If the responses are conditionally independent given the covariates, their conditional likelihood is the product of those contributions.

Worked Response Models

Worked Example: Binary Responses and Bernoulli Likelihood

Suppose five patients have binary outcomes, where $Y_i=1$ denotes improvement and $Y_i=0$ denotes no improvement. Assume independent responses with a common improvement probability $p$, where $0\leq p\leq 1$. The support is $\{0,1\}$, and the Bernoulli mass function is specified at each support value by

f(1;p)=p,\qquad f(0;p)=1-p.

This piecewise definition also makes the endpoint cases unambiguous: at $p=0$, the response is zero with probability one; at $p=1$, it is one with probability one. Suppose the observed responses are $1,0,1,1,0$. Independence gives

$$ L(p)=f(1;p)f(0;p)f(1;p)f(1;p)f(0;p) =p^3(1-p)^2. $$

Here the expression is defined at both endpoints: $L(0)=0$ and $L(1)=0$, as the data include both outcomes. For an interior value such as $p=0.6$,

$$ L(0.6)=(0.6)^3(0.4)^2 =0.216(0.16)=0.03456. $$

This is the probability of this particular ordered sequence under the model with $p=0.6$. If the sequence is summarized only by its count of successes, the probability of observing three successes in five trials is $\binom{5}{3}p^3(1-p)^2$. That binomial probability includes all ten possible sequences with three successes; it is not the likelihood contribution for the one ordered sequence unless the data are recorded only as a count.

Worked Example: Count Responses and a Poisson Likelihood

Suppose three independent observation periods produce event counts $Y_1,Y_2,Y_3$, each with a Poisson distribution having rate $\lambda$. The support is $\{0,1,2,\ldots\}$, and take $\lambda>0$. For every support value $y$, the probability mass is

f(y;\lambda)=\frac{e^{-\lambda}\lambda^y}{y!},\qquad y=0,1,2,\ldots.

The parameter restriction $\lambda>0$ makes the displayed expression well defined under ordinary exponentiation. A model allowing $\lambda=0$ would define that boundary case separately: $P(Y=0)=1$ and $P(Y=y)=0$ for $y\geq1$.

For observed counts $0,2,1$, independence gives the likelihood

\begin{aligned} L(\lambda) &=\frac{e^{-\lambda}\lambda^0}{0!} \frac{e^{-\lambda}\lambda^2}{2!} \frac{e^{-\lambda}\lambda^1}{1!}\\ &=\frac{1}{2}e^{-3\lambda}\lambda^3. \end{aligned}

At $\lambda=1$, this is $e^{-3}/2$, approximately $0.0249$. In this discrete example, that value is the probability of exactly the ordered observations $0,2,1$ under the model. The corresponding log-likelihood for $\lambda>0$ is $\ell(\lambda)=-3\lambda+3\log(\lambda)-\log(2)$, obtained by taking the logarithm of the likelihood.

Worked Example: Continuous Responses and a Normal Likelihood

Suppose two independent measurements have distributions $Y_i\sim N(\mu,1)$, where the variance is known to be one and $\mu\in\mathbb{R}$ is unknown. Each response has support $\mathbb{R}$ and density

f(y;\mu)=\frac{1}{\sqrt{2\pi}} \exp\left\{-\frac{(y-\mu)^2}{2}\right\},\qquad y\in\mathbb{R}.

For observed measurements $y_1=1$ and $y_2=3$, the independent-sampling likelihood is

\begin{aligned} L(\mu) &=f(1;\mu)f(3;\mu)\\ &=\frac{1}{2\pi} \exp\left\{-\frac{(1-\mu)^2+(3-\mu)^2}{2}\right\}. \end{aligned}

At $\mu=2$, the squared deviations are $1$ and $1$, so $L(2)=(2\pi)^{-1}e^{-1}$, approximately $0.0585$. This is a joint density value, not the probability that the measurements equal exactly 1 and 3. For example, the probability that each measurement lies within $\epsilon$ of its observed value is the integral of the joint density over the rectangle $[1-\epsilon,1+\epsilon]\times[3-\epsilon,3+\epsilon]$. For small $\epsilon$, this probability is approximately $L(\mu)(2\epsilon)^2$, not $L(\mu)$ itself.

Why Independent Contributions Multiply

Theorem: Let $Y_1,\ldots,Y_n$ be independent random variables. Suppose each $Y_i$ has a probability mass function or density $f_i$ with respect to a measure $\nu_i$ on its support. Then their joint mass function or density with respect to the product measure $\nu_1\times\cdots\times\nu_n$ is $$ f(y_1,\ldots,y_n)=\prod_{i=1}^n f_i(y_i). $$ Thus, for observed values $y_1,\ldots,y_n$, the likelihood is $L(\theta)=\prod_{i=1}^n f_i(y_i;\theta)$ when each contribution depends on a common parameter $\theta$.

Proof. For measurable sets $A_i$ in the respective supports, independence gives

P(Y_1\in A_1,\ldots,Y_n\in A_n) =\prod_{i=1}^n P(Y_i\in A_i) =\prod_{i=1}^n\int_{A_i}f_i(y_i)\,d\nu_i(y_i).

Because the integrands are nonnegative and measurable, Tonelli's theorem applies to their product, so the right-hand side equals

\int_{A_1\times\cdots\times A_n} \prod_{i=1}^n f_i(y_i)\, d(\nu_1\times\cdots\times\nu_n)(y_1,\ldots,y_n).

Rectangles of this form determine the product probability measure. Therefore the product of the individual mass functions or densities is a density for the joint distribution with respect to the product measure. Evaluating that joint mass or density at the observed data gives the stated likelihood. The same reasoning applies conditional on fixed covariates when the responses are conditionally independent and each conditional distribution has its specified mass function or density. $\square$

Identical distributions are a separate assumption. If the responses are independent but have different probabilities or densities, the same result gives a product of different contributions. For instance, binary responses with probabilities $p_i$ have likelihood $\prod_i f(y_i;p_i)$; there is no reason to replace the $p_i$'s by a common probability unless the model specifies one.

Log-Likelihood Additivity

Result: If the independent-response likelihood has positive contributions $f_i(y_i;\theta)$, then its log-likelihood is the sum of the individual log contributions: $$ \ell(\theta)=\log L(\theta)=\sum_{i=1}^n\log f_i(y_i;\theta). $$

Derivation. Independence gives $L(\theta)=\prod_{i=1}^n f_i(y_i;\theta)$. When every factor is positive, the logarithm is defined, and the logarithm product rule yields

\ell(\theta) =\log\left\{\prod_{i=1}^n f_i(y_i;\theta)\right\} =\sum_{i=1}^n\log f_i(y_i;\theta).

If a contribution is zero, then $L(\theta)=0$ and the extended log-likelihood is $-\infty$; the finite-sum formula applies only where all contributions are positive. This condition matters at boundary parameter values. For example, the Bernoulli likelihood for mixed zero and one outcomes vanishes at $p=0$ and $p=1$, even though the piecewise mass function itself is well defined there.

The sum form is convenient for calculation and interpretation, but it does not change what the likelihood is: it remains the joint model for the observed responses, viewed as a function of its parameter.

Common Mistakes and Why the Setup Matters

Writing a mass formula outside its domain. A formula involving $0^0$ is not automatically defined under usual exponentiation. State boundary cases separately, restrict the parameter space, or explicitly declare the convention being used. The Bernoulli mass function above is piecewise, and the Poisson example specifies $\lambda>0$ while explaining how a model could include $\lambda=0$.
Multiplying contributions without an independence assumption. Marginal models alone do not determine the joint distribution. The product likelihood requires independent responses, or conditional independence when conditioning on covariates.
Treating a density value as a probability. A density can exceed one and depends on the units of measurement. Probabilities for continuous responses come from integrating the density over sets.
Confusing a response sequence with a count summary. For binary data, the probability of one ordered sequence differs from the probability of all sequences with the same number of successes. The likelihood must match the data actually observed.

Careful model construction makes the meaning of each likelihood contribution explicit. In later modeling work, covariates can change the response distribution from one observation to another; the support, probability or density, and sampling assumption still need to be clear before the likelihood can be interpreted or used.

Check Your Understanding

Use the support, probability or density, and sampling assumptions to justify each answer.

For a Bernoulli response with parameter $p\in[0,1]$, why is a piecewise definition of the mass function useful at the parameter boundaries?
Three independent Poisson counts have observed values $1,0,2$. Write their likelihood as a function of $\lambda>0$, leaving it either factored or simplified.
For independent continuous observations, explain why the product of their densities at the observed values is not generally the probability of those exact observations.
What assumption justifies replacing a joint response density by a product of individual densities? Does identical distribution alone justify it?
When can the log-likelihood be written as a finite sum of log contributions, and what happens if one contribution is zero?