Local Likelihood for Conditional Densities

Why Local Likelihood?

A conditional density describes the distribution of a response $Y$ at a covariate value $x$. Conditional kernel density estimation uses weights to combine nearby responses. Local likelihood takes a different route: choose a probability model for $Y\mid X=x$, then fit its parameters using observations near a target value $x_0$. The fitted model is a valid conditional distribution at $x_0$, provided its parameters belong to the model's allowable parameter space.

The approach is useful when a distributional model has a meaningful structure, such as a Bernoulli, Poisson, or Gaussian family, but its parameters vary with covariates. Rather than fit one model to all observations, local likelihood emphasizes observations with covariates near $x_0$. A local polynomial extension allows the parameters to change across that neighborhood.

Definition of Local Likelihood

Suppose $(X_i,Y_i)$, $i=1,\ldots,n$, are independent observations from a common distribution, with $X_i\in\mathbb R^d$. Let $p(y;\theta)$ be a conditional density or probability mass function, relative to a fixed measure, and suppose the conditional law of $Y$ given $X=x$ is represented by $p(y;\theta(x))$. For a target $x_0$, choose a nonnegative kernel $K$ and componentwise bandwidths $h_j>0$. Define $w_i=K((X_i-x_0)/h)$, where division is componentwise.

Definition: The local log-likelihood at $x_0$, for a proposed parameter function $\theta(\cdot)$, is $$ \ell_{x_0}(\theta)=\sum_{i=1}^n w_i\log p(Y_i;\theta(X_i)). $$ A local likelihood estimator maximizes this criterion over a specified class of parameter functions, subject to the requirement that each evaluated parameter gives a valid conditional distribution.

Multiplying all weights by the same positive constant does not change the maximizer. Usually $K$ is chosen to be zero outside a bounded neighborhood, or to decrease with distance, so distant observations contribute little or nothing. The weights must be nonnegative for the usual concavity and weighted-likelihood interpretations. The criterion is a weighted sum of conditional log-likelihood contributions; it is not a joint likelihood for the covariates.

Worked Example: A Local Constant Bernoulli Model

At a target $x_0$, suppose three observations have binary responses $0,1,1$ and local weights $0.25,0.50,0.25$, respectively. Fit a Bernoulli conditional model whose success probability is constant within this local fit, denoted by $p$.

The weighted log-likelihood is $$ \ell(p)=0.25\log(1-p)+0.50\log p+0.25\log p =0.25\log(1-p)+0.75\log p,\qquad 0<p<1. $$ Differentiating gives $$ \ell'(p)=-\frac{0.25}{1-p}+\frac{0.75}{p}. $$ Setting the derivative to zero yields $0.75(1-p)=0.25p$, hence $p=0.75$. The second derivative is $-0.25/(1-p)^2-0.75/p^2<0$, so this is the unique maximizer. The fitted conditional distribution at $x_0$ assigns probability $0.75$ to 1 and $0.25$ to 0.

Local Constant and Local Polynomial Forms

A local constant fit treats the parameter as fixed over the neighborhood used in the likelihood: $\theta(X_i)=\theta_0$. It therefore maximizes $\sum_i w_i\log p(Y_i;\theta_0)$. The fit estimates the parameter at $x_0$ by pooling nearby observations, with more influence assigned to observations receiving larger weights.

For a local polynomial fit, it is often convenient to model a parameter that is unconstrained or to use a link that maps to its allowed space. For exponential families, the natural parameter is particularly useful. Let $z_i=(X_i-x_0)/h$, and let $r(z)$ be a vector of polynomial basis functions. For example, a local constant basis is $r(z)=1$; a local linear basis in $d$ covariates is $r(z)=(1,z_1,\ldots,z_d)^\mathsf{T}$. Approximate the natural parameter by $\eta_\beta(x)=B^\mathsf{T}r((x-x_0)/h)$, where $B$ is a matrix of coefficients. The local constant fit is the intercept-only case; the local linear fit also estimates first-order changes in $\eta$ near $x_0$.

Definition: A degree-$p$ local polynomial likelihood uses as $r(z)$ all monomials in $z$ of total degree at most $p$, including the constant term. It maximizes the weighted conditional log-likelihood over the polynomial coefficients, subject to the condition that the resulting model parameters are valid wherever the fitted conditional density is required.

The local polynomial is an approximation in a neighborhood, not a claim that the parameter function is globally polynomial. In particular, a polynomial natural parameter may leave its allowable parameter space outside the fitting region. The model must enforce validity at least at the covariate values where fitted densities will be interpreted; a link or constrained optimization can be used when the natural parameter space does not contain every polynomial value.

Worked Example: Local Constant Gaussian Mean

Suppose $Y\mid X=x$ is modeled as Gaussian with known variance $4$ and a locally constant mean. At $x_0$, three responses are $1,3,6$, with weights $1,2,1$. The log-likelihood, up to terms independent of the mean $\mu$, is $$ \ell(\mu)=-\frac{1}{2(4)}\{1(1-\mu)^2+2(3-\mu)^2+1(6-\mu)^2\}. $$ Differentiating and setting the result to zero gives $$ \frac{1}{4}\{(1-\mu)+2(3-\mu)+(6-\mu)\}=0, \qquad \hat\mu=\frac{1+6+6}{1+2+1}=\frac{13}{4}=3.25. $$ The weighted squared-error criterion is strictly concave in $\mu$ after multiplication by the negative variance factor, so this is its unique maximizer. In the Gaussian exponential-family representation, the natural parameter is $\eta=\mu/\sigma^2$, not generally $\mu$. Here $\sigma^2=4$, so the fitted natural parameter is $\hat\eta=3.25/4=13/16$. Every real mean corresponds to a valid natural parameter; the mean itself is the natural parameter only when the variance is 1.

Exponential Families Preserve Valid Densities

A one-parameter or multiparameter exponential family can be written, relative to a fixed measure, as $$ p(y;\eta)=b(y)\exp\{\eta^\mathsf{T}T(y)-A(\eta)\}. $$ Here $T(y)$ is a vector of sufficient statistics, $b(y)\geq 0$ is a base density or mass function, and the natural parameter space consists of those $\eta$ for which the normalizing integral is finite. The log-partition function is $A(\eta)=\log\int b(y)\exp\{\eta^\mathsf{T}T(y)\}\,d\nu(y)$, where $\nu$ is the reference measure.

Result: For every $\eta$ in the natural parameter space, $p(y;\eta)$ is nonnegative and integrates (or sums) to one. Thus, if a covariate-dependent natural parameter $\eta(x)$ remains in that space, the model defines a valid conditional density at each such $x$.

Proof. Since $b(y)\geq0$ and the exponential is positive, $p(y;\eta)\geq0$. By the definition of $A(\eta)$, $$ \int p(y;\eta)\,d\nu(y) =\exp\{-A(\eta)\}\int b(y)\exp\{\eta^\mathsf{T}T(y)\}\,d\nu(y) =\exp\{-A(\eta)\}\exp\{A(\eta)\}=1. $$ The same argument applies to a discrete distribution with the integral replaced by a sum. Substituting any allowable $\eta(x)$ proves the conditional statement. $\square$

This is an important advantage over directly approximating a density by a polynomial in $y$: such an approximation can become negative or fail to integrate to one. An exponential-family model is nonnegative by construction and normalized by its log-partition function. These guarantees do not, by themselves, ensure that the fitted parameter is estimable or that the model is well matched to the data.

Worked Example: A Local Linear Poisson Model

Consider Poisson responses at standardized covariate locations $z=-1,0,1$, with observed counts $y=1,2,4$, and equal positive local weights. The Poisson natural parameter is $\eta=\log\lambda$, where $\lambda$ is the mean, and its log-partition function is $A(\eta)=e^\eta$. Use the local linear form $\eta(z)=a+bz$. The three observed log-counts are $0,\log 2,\log 4$, which lie exactly on the line $\eta(z)=\log 2+(\log 2)z$. Thus $a=\log 2$ and $b=\log 2$, giving fitted means $e^0=1$, $e^{\log 2}=2$, and $e^{\log 4}=4$ at the three locations.

To check the likelihood equations, the weighted score for $(a,b)$ is $$ \sum_{i=1}^3 (y_i-e^{a+bz_i}) \begin{pmatrix}1\\z_i\end{pmatrix}. $$ At the stated coefficients, each residual is zero, so both score components are zero. The Hessian is $$ -\sum_{i=1}^3 e^{a+bz_i} \begin{pmatrix}1&z_i\\z_i&z_i^2\end{pmatrix}. $$ For any nonzero vector $v=(v_0,v_1)^\mathsf{T}$, its quadratic form is $-\sum_i e^{a+bz_i}(v_0+v_1z_i)^2<0$: a nonzero linear polynomial cannot vanish at all three distinct locations. Hence the log-likelihood is strictly concave, and the score-zero coefficients give its unique maximum. At any other covariate value where the model is used, $\eta(z)$ remains real, so $\lambda=e^{\eta(z)}>0$ and the Poisson distribution remains normalized.

Score Equations and Optimization

For an exponential-family response with natural parameter $\eta_i=B^\mathsf{T}r(z_i)$, the weighted objective is $$ \ell(B)=\sum_{i=1}^n w_i\left\{\log b(Y_i)+\eta_i^\mathsf{T}T(Y_i)-A(\eta_i)\right\}. $$ When $A$ is differentiable and the parameter values are in the interior of the natural parameter space, differentiation gives the score equation $$ \sum_{i=1}^n w_i r(z_i)\{T(Y_i)-\nabla A(\eta_i)\}^\mathsf{T}=0. $$ The identity $E_\eta[T(Y)]=\nabla A(\eta)$ holds when differentiation can be interchanged with the normalizing integral, for example under the usual regularity conditions for a regular exponential family in the interior of its natural parameter space. The score therefore equates weighted observed sufficient statistics to their fitted conditional expectations, in the directions represented by the polynomial basis.

Result: If the natural parameter space is convex, $A$ is convex there, and $w_i\geq0$, then the local polynomial log-likelihood above is concave in $B$. If it is differentiable and an interior score-zero point exists, that point is a global maximizer.

Proof. The terms $\log b(Y_i)$ do not depend on $B$, and each $\eta_i=B^\mathsf{T}r(z_i)$ is affine in $B$. The term $\eta_i^\mathsf{T}T(Y_i)$ is therefore affine in $B$. Since $A$ is convex, $A(\eta_i)$ is convex as a function of $B$, so $-w_iA(\eta_i)$ is concave for $w_i\geq0$. Summing these terms and the affine terms proves concavity of $\ell(B)$. For a differentiable concave function, at any point $B_*$ its value at another feasible point $B$ is at most $\ell(B_*)+\langle\nabla\ell(B_*),B-B_*\rangle$. If $\nabla\ell(B_*)=0$, this is at most $\ell(B_*)$; hence $B_*$ is a global maximizer. $\square$

Concavity does not guarantee a finite maximizer. For example, a Bernoulli sample containing only successes can make the log-likelihood increase as the fitted success probability approaches 1, without attaining a maximum at a finite logit. Local fits can also be unstable when few observations have positive weights or when the local design does not support all polynomial coefficients. Optimization should therefore check convergence, parameter validity, and whether the fitted solution is finite.

Common Pitfall and Practical Interpretation

A common mistake is to assume that any polynomial fit to a distribution parameter automatically produces a valid conditional density. That is false if the parameter is constrained. For instance, a Gaussian variance must be positive, and a Bernoulli probability must lie between 0 and 1. A direct polynomial for either quantity can leave its allowable range. The correction is to model a suitable unconstrained parameter, such as the log variance or the Bernoulli logit, or to impose explicit constraints. For a Gaussian with known variance, use the natural parameter $\eta=\mu/\sigma^2$; do not identify it with the mean unless $\sigma^2=1$.

Local likelihood matters when the conditional distribution's shape, not just its mean, is the target. A conditional Gaussian model can track a changing mean, while a richer family can allow other distributional features to vary. The local constant and local polynomial choices control how that variation is approximated near $x_0$. They do not remove the need to select a suitable family, bandwidth, and parameter constraints.

Check Your Understanding

Use the definitions and derivations above to answer the following.

Why does multiplying all local weights by one positive constant leave the local likelihood maximizer unchanged?
For a Bernoulli exponential family, what is the natural parameter in terms of the success probability, and what parameter space does it occupy?
In a local linear Poisson model with $\eta(z)=a+bz$, how do you obtain the fitted mean at a specified $z$?
Explain why the exponential-family form guarantees normalization, and state the condition on the natural parameter that is needed.
Why can a concave local log-likelihood still fail to have a finite maximizer?