Histogram Density Estimators

From Bin Counts to a Density Estimate

A histogram density estimator turns observations into a piecewise-constant estimate of a density. The central calculation is simple but easy to mis-scale: a bin’s count must be divided by both the sample size and the bin’s volume. Dividing only by the sample size estimates the probability in the bin, not a density. The volume adjustment converts that probability into density per unit volume.

Let \(X_1,\ldots,X_n\) be independent and identically distributed random vectors in \(\mathbb{R}^d\), each with density \(f\) with respect to \(d\)-dimensional Lebesgue measure. Suppose the distribution is supported on the bounded rectangle \(D=\prod_{k=1}^d[a_k,b_k]\), meaning \(P(X_i\in D)=1\). Partition each coordinate interval into a finite number of equal subintervals. Their Cartesian products form a regular rectangular partition of \(D\). Use a disjoint boundary convention, such as half-open cells except for the final cell on each coordinate, so that every point of \(D\) belongs to exactly one cell.

For a regular rectangular grid, if the cell widths along the \(d\) coordinates are \(h_1,\ldots,h_d\), then \(v=\prod_{k=1}^d h_k\). In one dimension, the volume is simply the bin width. In multiple dimensions, it is the product of the side lengths, not their sum. The estimate is constant within each cell and changes only when \(x\) crosses a cell boundary.

The partition and \(v\) are fixed when calculating the estimator’s sampling properties below. In particular, the results do not require the density to be continuous or bounded. For each cell define its probability \(\pi_j=P(X_i\in B_j)=\int_{B_j}f(u)\,du\). The probability \(\pi_j\) is dimensionless; dividing it by \(v\) gives a quantity with the units of a density.

Normalization: Every Sample Produces Total Mass One

A density must be nonnegative and integrate to one. A histogram estimate is nonnegative by construction. Its normalization is also exact, not merely true on average.

Proof. On cell \(B_j\), the estimator is \(N_j/(nv)\), and the cell has volume \(v\). The integral over that cell is therefore \(\int_{B_j}\hat f(x)\,dx=(N_j/(nv))v=N_j/n\). The estimator is zero outside \(D\), so summing over the cells gives \[ \int_{\mathbb{R}^d}\hat f(x)\,dx =\sum_{j=1}^M\frac{N_j}{n} =\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^M\mathbf{1}\{X_i\in B_j\}. \] Each observation belongs to exactly one cell, so the inner sum equals one for every \(i\). The expression is consequently \(n/n=1\). Since the observations lie in \(D\) with probability one, this normalization holds almost surely under the stated model. \(\square\)

This calculation explains why the bin volume is essential: each cell contributes its estimated height multiplied by its volume, giving \(N_j/n\), the observed fraction of the sample in that cell. The contributions across cells add to one because the observed fractions do.

Expectation and Variance Within a Bin

Fix a point \(x\in D\), and let \(B_j\) be the cell to which the boundary convention assigns it. The random value \(\hat f(x)\) depends on the sample only through \(N_j\). Because each observation independently falls in \(B_j\) with probability \(\pi_j\), the count \(N_j\) has a binomial distribution with sample size \(n\) and success probability \(\pi_j\). This gives both the mean and variance of the estimate.

Proof. Define \(I_{ij}=\mathbf{1}\{X_i\in B_j\}\), so \(N_j=\sum_{i=1}^n I_{ij}\). Each indicator has expectation \(E[I_{ij}]=\pi_j\) and variance \(\operatorname{Var}(I_{ij})=\pi_j(1-\pi_j)\). Independence of the observations makes the indicators independent as \(i\) varies. Therefore \(E[N_j]=n\pi_j\) and \(\operatorname{Var}(N_j)=n\pi_j(1-\pi_j)\). Since \(\hat f(x)=N_j/(nv)\), scaling gives \[ E[\hat f(x)]=\frac{E[N_j]}{nv}=\frac{\pi_j}{v} \quad\text{and}\quad \operatorname{Var}(\hat f(x)) =\frac{\operatorname{Var}(N_j)}{n^2v^2} =\frac{\pi_j(1-\pi_j)}{nv^2}. \] Finally, \(\pi_j=\int_{B_j}f(u)\,du\), so \(\pi_j/v\) is the integral of \(f\) over the cell divided by its volume. This is precisely the cell average of \(f\). \(\square\)

The expectation is not, in general, the true density value at every point of the bin. It is the same cell average for every \(x\in B_j\), because the estimate itself is constant there. The formula holds for any integrable density supported on \(D\); interpreting \(f(x)\) at a particular point additionally depends on the chosen representative of a density, which is defined only up to changes on sets of measure zero.

Interpreting the Formulas and Avoiding a Scaling Error

The variance formula can also be expressed in terms of the expected height. If \(\bar f_j=\pi_j/v\), then substituting \(\pi_j=v\bar f_j\) gives \(\operatorname{Var}(\hat f(x))=\bar f_j(1-\pi_j)/(nv)\) for \(x\in B_j\). This form makes the role of volume visible: a smaller cell generally has fewer observations, so the estimate can be more variable. The factor \(1-\pi_j\) also reflects that a bin count is binomial rather than an unconstrained count.

A frequent mistake is to estimate the density in \(B_j\) by \(N_j/n\). That is the empirical fraction of observations in the bin, and it estimates \(\pi_j\), the bin’s probability mass. To obtain a density height, divide that fraction by the bin volume: \((N_j/n)/v=N_j/(nv)\). Another error is to use the side length in place of volume in multiple dimensions. For a rectangle with side lengths \(h_1,\ldots,h_d\), its volume is their product, so the correct denominator is \(n\prod_{k=1}^d h_k\).