Conditional Kernel Density Estimation

From Conditional Distributions to Conditional Densities

A conditional density describes the distribution of an outcome at a specified covariate value. Let $X\in\mathbb{R}^d$ be a covariate vector and $Y\in\mathbb{R}$ a scalar outcome. Suppose $(X_i,Y_i)$, $i=1,\ldots,n$, are independent and identically distributed observations from a joint distribution with density $f_{X,Y}$. Where the marginal density $f_X(x)$ is positive, the target conditional density is $f(y\mid x)=f_{X,Y}(x,y)/f_X(x)$.

The ratio suggests a direct estimator: smooth the joint observations in both $X$ and $Y$, smooth the covariates in $X$, and divide the first estimate by the second. This construction is useful because it gives an estimate of a density in $y$ at each fixed $x$, rather than estimating a single conditional mean or quantile. Its normalization and its support still need careful attention.

Definition: Let $K:\mathbb{R}^d\to[0,\infty)$ and $L:\mathbb{R}\to[0,\infty)$ be kernels that each integrate to one. For bandwidths $h>0$ and $b>0$, define $K_h(u)=h^{-d}K(u/h)$ and $L_b(v)=b^{-1}L(v/b)$. The conditional kernel density estimator at a covariate value $x$ is $$ \widehat f(y\mid x) = \frac{\displaystyle \frac{1}{n}\sum_{i=1}^n K_h(x-X_i)L_b(y-Y_i)} {\displaystyle \frac{1}{n}\sum_{i=1}^n K_h(x-X_i)} = \frac{\displaystyle\sum_{i=1}^n K_h(x-X_i)L_b(y-Y_i)} {\displaystyle\sum_{i=1}^n K_h(x-X_i)}, $$ provided the denominator is positive. Here $h$ controls smoothing over covariates and $b$ controls smoothing over outcomes.

The displayed form uses a common bandwidth $h$ in every covariate direction and a scalar bandwidth $b$ for the outcome. Different covariate bandwidths can be used in different directions, but the interpretation is unchanged. Nonnegative kernels are important: they make the estimate a nonnegative weighted average of outcome densities. In this tutorial, kernels are assumed to be measurable and integrable; wherever an integral or expectation is used below, its stated finiteness conditions apply.

Why the Estimate Is a Density

The ratio has a particularly useful finite-sample representation. Define the normalized covariate weights $w_i(x)=K_h(x-X_i)/\sum_{j=1}^n K_h(x-X_j)$. When the denominator is positive, these weights are nonnegative and sum to one. The estimator is therefore a weighted mixture of outcome kernels centered at the observed outcomes.

Theorem: If $K_h(x-X_i)\geq 0$ for every $i$, the denominator $\sum_iK_h(x-X_i)$ is positive, and $L_b$ is a nonnegative density on $\mathbb{R}$, then $\widehat f(y\mid x)$ is a density in $y$: it is nonnegative and integrates to one.

Proof. Define $w_i(x)$ as above. Then $w_i(x)\geq0$, and $\sum_iw_i(x)=1$. Rearranging the estimator gives $\widehat f(y\mid x)=\sum_iw_i(x)L_b(y-Y_i)$, so it is nonnegative. Since each translated and rescaled kernel integrates to one, finite linearity of integration gives

\int_{\mathbb{R}}\widehat f(y\mid x)\,dy = \sum_{i=1}^n w_i(x)\int_{\mathbb{R}}L_b(y-Y_i)\,dy = \sum_{i=1}^n w_i(x) = 1.

There are finitely many terms, so no interchange of an infinite sum and an integral is involved. This proves the claim. $\square$

Worked Example: A Gaussian Conditional Density Estimate

Take three observations $(X_i,Y_i)=(-1,1),(0,0),(1,-1)$. Use Gaussian kernels for both variables, with covariate bandwidth $h=1$ and outcome bandwidth $b=0.5$. Estimate the conditional density at $x=0,y=0$.

For the covariate weights, the Gaussian kernel values are proportional to $e^{-1/2},1,e^{-1/2}$. The common Gaussian normalizing constant cancels from the ratio. For the outcome kernel, $L_b(v)=\{b\sqrt{2\pi}\}^{-1}\exp\{-v^2/(2b^2)\}$. Thus $L_{0.5}(0)=0.7978846$, while $L_{0.5}(1)=L_{0.5}(-1)=0.7978846e^{-2}\approx0.1079819$.

Substitution gives the numerator and denominator, using unrounded kernel values:

\begin{aligned} \text{numerator} &=2e^{-1/2}(0.1079819)+e^{-1}(0.7978846) \approx 0.424514,\\ \text{denominator} &=2e^{-1/2}+e^{-1} \approx 1.580941,\\ \widehat f(0\mid 0) &=\frac{0.424514}{1.580941} \approx 0.268520. \end{aligned}

The two observations with $X_i=\pm1$ contribute less to the estimate at $x=0$ than the observation with $X_i=0$, because their covariate weights are smaller. The result is a density value at $y=0$, not a probability of observing exactly zero.

What the Bandwidths Do

The covariate bandwidth $h$ determines how quickly observations are downweighted as their covariates move away from the target $x$. A small $h$ emphasizes nearby covariate values, giving a more local estimate but potentially leaving only a few observations with appreciable weight. A large $h$ borrows information from a broader covariate neighborhood and can blur differences between conditional distributions at distinct values of $X$.

The outcome bandwidth $b$ controls the width of each component density $L_b(y-Y_i)$. A small $b$ produces a more detailed, potentially irregular density estimate. A large $b$ smooths more heavily and can obscure features of the conditional outcome distribution. The two bandwidths act in different spaces: $h$ changes which observations contribute strongly at a given $x$, while $b$ changes the shape of each observation’s contribution across $y$.

Worked Example: Covariate Weights Form a Mixture

Suppose the observations have covariates $-0.5,0,0.5,2$, outcomes $0,1,2,10$, and we estimate at $x=0$. Use the triangular kernel $K(u)=(1-|u|)_+$, where $a_+=\max(a,0)$, with $h=1$. The unnormalized covariate weights are $0.5,1,0.5,0$, so the normalized weights are $1/4,1/2,1/4,0$.

Now use a standard Gaussian outcome kernel with $b=1$, and evaluate at $y=1$. The relevant component values are $L_1(1)=\phi(1)\approx0.241971$, $L_1(0)=\phi(0)\approx0.398942$, and $L_1(-1)=\phi(1)\approx0.241971$, where $\phi$ is the standard normal density. Therefore

\widehat f(1\mid 0) = \frac14(0.241971)+\frac12(0.398942)+\frac14(0.241971) \approx 0.320456.

The observation with covariate 2 has weight zero because the compactly supported triangular kernel gives it no contribution at this $x$. Changing $h$ could change these weights; changing $b$ would change the component density values. Regardless of these choices, the mixture integrates to one as long as its weights are defined and the outcome kernel is a density.

Population Smoothing and the Support Condition

The estimator is a ratio of random quantities. For fixed bandwidths, its large-sample target is generally a smoothed conditional density, not exactly $f(y\mid x)$. The following result makes that distinction explicit.

Theorem: Fix $x,y,h,b$. Suppose $(X_i,Y_i)$ are iid, $E[K_h(x-X)]$ is finite and positive, and $E[K_h(x-X)L_b(y-Y)]$ is finite. Then, as $n\to\infty$, $$ \widehat f(y\mid x) \xrightarrow{a.s.} \frac{E[K_h(x-X)L_b(y-Y)]}{E[K_h(x-X)]}. $$ If $f_X$ and $f(y\mid u)$ are densities for which the following integrals are defined, the ratio on the right equals $$ \frac{\displaystyle\int K_h(x-u)f_X(u) \left\{\int L_b(y-v)f(v\mid u)\,dv\right\}du} {\displaystyle\int K_h(x-u)f_X(u)\,du}. $$

Proof. The denominator of the estimator divided by $n$ is the sample average of the iid random variables $K_h(x-X_i)$. By the strong law of large numbers, it converges almost surely to $E[K_h(x-X)]$. The numerator divided by $n$ is the sample average of the iid random variables $K_h(x-X_i)L_b(y-Y_i)$, so it converges almost surely to $E[K_h(x-X)L_b(y-Y)]$. The limiting denominator is positive by assumption. The ratio therefore converges almost surely to the ratio of these expectations.

For the integral expression, write the expectation using the joint density and then use $f_{X,Y}(u,v)=f_X(u)f(v\mid u)$ wherever this factorization is defined. The integrand is nonnegative, so Tonelli’s theorem permits the iterated integration. The denominator follows by integrating the joint density over $v$, using $\int f(v\mid u)\,dv=1$. This yields the stated expression. $\square$

The limit averages outcome-smoothed conditional densities over covariate values $u$, with weights proportional to $K_h(x-u)f_X(u)$. It therefore depends on the covariate density as well as on the conditional distributions nearby. As bandwidths shrink under suitable additional regularity conditions, this smoothing can become more local; the fixed-bandwidth result itself does not assert such a limit.

A central support requirement is that the covariate denominator be positive. If no observation receives positive kernel weight at $x$, the estimator is undefined there. At a population level, the corresponding condition is $E[K_h(x-X)]>0$. Estimating a conditional law where the covariate distribution has no local support is an extrapolation problem, not something the ratio automatically solves. Even when the denominator is positive, very small total weight can make the estimate unstable.

Boundary Correction for the Outcome

Ordinary symmetric outcome kernels can place mass outside the known support of $Y$. For example, if $Y\geq0$, a Gaussian kernel centered at an observed value near zero assigns positive density to negative outcomes. Simply restricting the displayed estimate to $y\geq0$ loses probability mass; the restricted curve no longer integrates to one.

One direct correction is to truncate and renormalize each outcome-kernel contribution. For a support set $S$, define $C_i=\int_S L_b(v-Y_i)\,dv$. Provided $C_i>0$, replace $L_b(y-Y_i)$ by $\mathbf{1}\{y\in S\}L_b(y-Y_i)/C_i$. Each corrected component is a density on $S$, so any nonnegative mixture of them with weights summing to one is also a density on $S$. This correction fixes normalization on the specified support; it does not, by itself, remove all boundary bias.

Worked Example: Truncating at a Zero Boundary

Suppose $Y\geq0$, and at a fixed covariate value the estimator assigns equal weights to two observations, $Y_1=0$ and $Y_2=1$. Use Gaussian outcome kernels with $b=1$. The uncorrected kernel centered at zero has only half its mass on the nonnegative half-line: $\int_0^\infty\phi(y)\,dy=0.5$. The kernel centered at one has mass $\int_0^\infty\phi(y-1)\,dy=\Phi(1)\approx0.841345$, where $\Phi$ is the standard normal distribution function.

The corrected estimate is

\widehat f_{\!S}(y\mid x) = \frac12\frac{\mathbf{1}\{y\geq0\}\phi(y)}{0.5} + \frac12\frac{\mathbf{1}\{y\geq0\}\phi(y-1)}{\Phi(1)}.

Its integral over the support is $\frac12(0.5/0.5)+\frac12(0.841345/0.841345)=1$. At the boundary $y=0$, its value is approximately $\frac12(0.398942/0.5)+\frac12(0.241971/0.841345)\approx0.5427$. The correction preserves a valid density on the allowed support, although it changes the component shapes near the boundary.

Common Mistakes and Practical Interpretation

Dividing by the sample size instead of the covariate weight sum. The joint density estimate divided by the marginal covariate density estimate gives the ratio in the definition. Dividing the joint estimate by $n$ alone does not generally produce a conditional density.
Assuming a density estimate is a point probability. $\widehat f(y\mid x)$ is a density value. Probabilities come from integrating it over outcome sets.
Ignoring zero or tiny denominators. A zero denominator makes the estimator undefined; a tiny denominator signals that very little local covariate information supports the estimate.
Clipping negative outcomes without renormalizing. Removing the part of a kernel outside the support reduces its integral. Truncate and normalize the component kernels, or use another method that explicitly preserves support and mass.
Treating $h$ and $b$ as interchangeable. They smooth different variables and have distinct units. Tuning one does not substitute for choosing the other.

Conditional kernel density estimation offers a transparent construction: the covariate kernel chooses a local mixture of observed outcomes, and the outcome kernel turns those outcomes into a smooth density. This transparency also makes its limits visible. Support, local data availability, boundary behavior, and both bandwidths must be considered when interpreting the resulting conditional distribution.

Check Your Understanding

Use the definitions and derivations above to answer the following questions.

Why does the conditional kernel density estimator integrate to one even though it is formed as a ratio?
In the mixture representation, what determines the weight of observation $i$, and what determines the shape of its outcome contribution?
What does the fixed-bandwidth almost-sure limit average over, and why is it not generally the exact conditional density at $x$?
What happens if every covariate-kernel weight is zero at the target $x$?
For a nonnegative outcome, why is restricting an ordinary Gaussian-kernel estimate to nonnegative values insufficient to produce a density on that support?