Forest Weights as Adaptive Kernels

Forest Predictions as Local Averages

A regression forest prediction at a query point \(x\) is a weighted average of training responses. The weights encode how the collection of trees places \(x\) and the training predictors into terminal leaves. This perspective connects forest prediction to kernel smoothing, but with an important distinction: the forest’s weights are learned from the data and depend on the query point. They are not generally evaluations of one fixed function of the distance between \(x\) and \(X_i\).

The useful question is therefore not whether a forest is literally a classical kernel estimator. It is how its weights distribute mass over the observed predictors. If most weight falls near \(x\), the prediction behaves like a local average. If substantial weight remains far away, the prediction may mix responses from regions where the regression function differs from its value at \(x\).

For example, a tree that predicts by averaging the responses in the leaf containing \(x\) assigns equal weight to observations in that leaf and zero weight to observations outside it. A forest averages such tree weights across trees. Thus a forest weight can be positive even when \(X_i\) is not in the same leaf as \(x\) in every tree. The resulting weights summarize the forest’s aggregate notion of which observations are relevant to \(x\).

From Leaf Membership to Adaptive Weights

Suppose tree \(b\) has a terminal leaf \(L_b(x)\) containing \(x\), and let \(N_b(x)\) be the number of training predictors in that leaf. If the tree predicts by the within-leaf response average, its weight on observation \(i\) is \(\mathbf{1}\{X_i\in L_b(x)\}/N_b(x)\), provided \(N_b(x)>0\). Averaging over \(B\) trees gives

Each tree’s weights are nonnegative and sum to one, so their average also has those properties. More generally, the interpretation applies whenever the forest’s prediction has nonnegative normalized weights, including suitable randomized or subsampled constructions. The weights can depend on the training predictors, the tree randomization, and—in response-adaptive procedures—possibly the responses as well.

A classical kernel estimator typically uses a fixed kernel shape and a bandwidth to assign weight according to a scaled distance from \(x\), followed by normalization. Forest weights instead arise from a collection of data-adaptive partitions. The word kernel here describes the role of the weights as a localized averaging rule; it does not imply that the forest uses a fixed radial kernel, or that its weights depend only on pairwise distances.

Neighborhood Mass and Local Averaging

Choose a metric \(d\) on the predictor space. For radius \(r>0\), define the forest’s mass within the neighborhood of \(x\) by \(A_{n,r}(x)=\sum_{i=1}^n W_{ni}(x)\mathbf{1}\{d(X_i,x)\leq r\}\). Since the weights sum to one, \(1-A_{n,r}(x)\) is the weight outside that neighborhood. This quantity gives a direct measure of localization at the chosen radius.

Proof. Subtract \(m(x)\) from the weighted average and use that the weights sum to one: \(\sum_i W_{ni}(x)m(X_i)-m(x)=\sum_i W_{ni}(x)\{m(X_i)-m(x)\}\). Apply the triangle inequality and split the sum into indices with \(d(X_i,x)\leq r\) and those with \(d(X_i,x)>r\). For the first set, the absolute difference is at most \(\omega(r)\); its total weight is \(A_{n,r}(x)\). For the second set, boundedness gives \(|m(X_i)-m(x)|\leq |m(X_i)|+|m(x)|\leq2M\); its total weight is \(1-A_{n,r}(x)\). Adding the two bounds proves the inequality. \(\square\)

The result separates two sources of error in the weighted signal average. Weight placed near \(x\) incurs at most the local variation \(\omega(r)\), while weight placed farther away is controlled only by the global bound \(2M\). In particular, if radii \(r_n\) shrink to zero and \(1-A_{n,r_n}(x)\) tends to zero, then both terms tend to zero: \(\omega(r_n)\to0\), and the outside mass vanishes. This is a deterministic implication for any realized set of weights and predictors; probabilistic claims require showing that the mass condition holds in the relevant mode of convergence.

Effective Support: How Concentrated Are the Weights?

Neighborhood mass measures localization relative to a chosen radius. A different summary measures concentration without choosing a radius: the effective support, also called the effective sample size of the weights, is the reciprocal of the sum of their squares.

If exactly \(k\) observations have positive weight, then \(1\leq N_{\mathrm{eff}}(x)\leq k\). To verify this, nonnegativity and normalization imply \(\sum_i W_{ni}(x)^2\leq(\sum_i W_{ni}(x))^2=1\), so the effective support is at least one. For the other direction, apply Cauchy–Schwarz to the \(k\) positive weights: \(1=(\sum_i W_{ni}(x))^2\leq k\sum_i W_{ni}(x)^2\). Taking reciprocals yields \(N_{\mathrm{eff}}(x)\leq k\). Equality at the upper bound occurs when those \(k\) weights are equal. Thus a large effective support requires weight to be spread out, but it does not by itself show that the observations receiving weight are close to \(x\).

When Effective Support Gives a Variance Formula

Effective support is often linked to the noise variance of a weighted average. That link requires care: if the weights use the responses, conditioning on the weights can change the conditional distribution of the errors. The formula below is valid under explicit conditional assumptions, not merely because the weights sum to one.

Proof. Conditional on the predictors and weights, the signal \(\sum_i W_{ni}(x)m(X_i)\) is fixed, so only \(\sum_i W_{ni}(x)\varepsilon_i\) contributes to the conditional variance. Expanding the variance gives

The cross-covariance terms vanish by assumption. The definition of \(N_{\mathrm{eff}}(x)\) gives the final equality. The conditional mean-zero assumption also ensures that the conditional mean prediction is the weighted signal average. \(\square\)

Common Mistake: Conditioning on Response-Dependent Weights

A common error is to assume that errors which are uncorrelated and mean zero conditional on the predictors must retain those properties after conditioning on weights. That conclusion is not automatic when the weights depend on the responses. Conditioning on a response-dependent weight can reveal information about the errors and change their conditional means, variances, or covariances.

For a direct counterexample, let \(X_1,X_2\) be fixed and let \(\varepsilon_1,\varepsilon_2\) be independent Rademacher variables, each equal to \(1\) or \(-1\) with probability \(1/2\). Set \(m(X_i)=0\), and choose weights \((1,0)\) if \(\varepsilon_1=1\), and \((0,1)\) otherwise. Given the predictors alone, both errors have mean zero and variance one, and their covariance is zero. But conditional on weights \((1,0)\), the prediction is \(\varepsilon_1=1\) with probability one, so its conditional variance is zero—not \(1\cdot(1^2+0^2)=1\). The weighted-variance identity is therefore not justified by predictor-conditional error assumptions alone. It can be used when its conditional assumptions hold, for example when weights are fixed independently of the estimation errors under the conditioning, but response adaptation must be addressed explicitly.