Nonparametric Estimation of the Alternative Density

Why Estimate the Alternative Density Nonparametrically?

In a two-group model, test statistics arise from a null component and an alternative component. The null density may be specified or estimated from central test statistics, but the alternative distribution can have an irregular shape: effects may vary in magnitude, and the resulting test statistics need not follow one convenient parametric curve. A nonparametric estimate lets the observed data inform that shape, while smoothing reduces the noise that would arise from using a raw histogram.

The key challenge is that the alternative status of each test is not observed. The estimation procedure must account for this uncertainty rather than simply label every test with a small p-value as alternative. The resulting density is then used in a plug-in estimate of local FDR, so its smoothing choices can change posterior null probabilities.

Definition: Let $Z_1,\ldots,Z_m$ be test statistics with common marginal density $f$. In the two-group model, $$ f(z)=\pi_0 f_0(z)+(1-\pi_0)f_1(z), $$ where $\pi_0$ is the null proportion, $f_0$ is the null density, and $f_1$ is the alternative density. Given fitted mixture parameters and component densities, the posterior alternative responsibility for test $i$ is $$ \tau_i=P(\text{alternative}\mid Z_i)= \frac{(1-\pi_0)f_1(Z_i)} {\pi_0f_0(Z_i)+(1-\pi_0)f_1(Z_i)}. $$ A responsibility-weighted kernel estimate of $f_1$ is $$ \widehat f_1(z)= \frac{\sum_{i=1}^m \tau_i K_h(z-Z_i)} {\sum_{i=1}^m \tau_i}, \qquad K_h(u)=\frac{1}{h}K\!\left(\frac{u}{h}\right), $$ where $K$ is a nonnegative kernel integrating to one and $h>0$ is the bandwidth.

Model, Assumptions, and the Smoothing Trade-Off

This is a smoothed nonparametric update for the alternative component, not the exact unrestricted M-step of the ordinary mixture likelihood: the unrestricted update places responsibility-weighted point masses at the observations. Each observation contributes in proportion to its fitted probability of belonging to that component. In practice, the responsibilities and the density estimate depend on each other. An EM-style procedure alternates between computing responsibilities under the current model and updating the density with the weighted estimate. Smoothing is essential: an unsmoothed weighted empirical distribution puts point masses at observed statistics rather than estimating a continuous density.

The basic interpretation assumes that tests share a common marginal mixture distribution, the null density and null proportion are specified or estimated, and the alternative component has a density that can be meaningfully smoothed. For standard kernel bias and variance reasoning, observations are often treated as independent draws from that mixture, and $f_1$ is assumed sufficiently smooth near the point of interest. Genomic test statistics can be dependent; in that case, the density estimate can still describe the marginal distribution, but uncertainty calculations that assume independent tests may be too optimistic.

Bandwidth $h$ controls the amount of smoothing. A large bandwidth combines observations across a wide range and can hide peaks, shoulders, or asymmetry in $f_1$. A small bandwidth follows local fluctuations, including random noise, and may produce a jagged density with unstable local FDR values. Under the usual smoothness conditions, for a symmetric kernel with finite second moment, the leading smoothing bias at an interior point is approximately proportional to $h^2 f_1''(z)$. The variance is approximately proportional to $1/(n_{\mathrm{eff}}h)$, where the responsibility-based effective sample size is $$ n_{\mathrm{eff}}=\frac{(\sum_i\tau_i)^2}{\sum_i\tau_i^2}. $$ This quantity is at most the number of tests with positive responsibility. Since $0\leq \tau_i\leq 1$, $\tau_i^2\leq\tau_i$, so $n_{\mathrm{eff}}\geq\sum_i\tau_i$. It can be much smaller than the total number of tests when only a small fraction of tests have appreciable alternative responsibility.

Worked Example: A Responsibility-Weighted Estimate

Suppose five hypothetical test statistics are $-2,-1,0,1,2$, and a fitted mixture assigns them alternative responsibilities $0.1,0.2,0.4,0.8,0.9$, respectively. Estimate the alternative density at $z=1$ using the uniform kernel $K(u)=\tfrac12$ for $|u|\leq1$, zero otherwise, and bandwidth $h=1$.

Here $K_h(z-Z_i)=1/(2h)=1/2$ when $|z-Z_i|\leq1$, and is zero otherwise. At $z=1$, the statistics $0,1,2$ are within bandwidth one, with responsibilities $0.4,0.8,0.9$. The total responsibility is $0.1+0.2+0.4+0.8+0.9=2.4$. Therefore, $$ \widehat f_1(1)= \frac{(0.4+0.8+0.9)(1/2)}{2.4} =\frac{1.05}{2.4}=0.4375. $$ The estimate is a density, not a probability: probabilities correspond to areas over intervals. Its units are inverse units of the test statistic.

With the same data but $h=0.5$, only the statistic $1$ contributes at $z=1$, giving $$ \widehat f_1(1)=\frac{0.8(1/(2(0.5)))}{2.4} =\frac{0.8}{2.4}=0.3333. $$ This difference illustrates the role of bandwidth: the smaller bandwidth uses fewer nearby observations and produces a different local estimate.

Bandwidth Selection and Diagnostics

A practical bandwidth should be chosen for the density feature that matters to the analysis, not solely for visual smoothness. A pilot bandwidth can be selected by weighted cross-validation or a rule-of-thumb calculation, then checked across a reasonable range. Because the responsibilities themselves depend on the fitted density, a rigorous comparison refits the mixture for each candidate bandwidth rather than treating the weights as permanently known. If a small change in $h$ causes large changes in the density near important test statistics or changes many local FDR estimates, those estimates are bandwidth-sensitive and should be reported as such.

Inspect the estimated density over the full statistic range, along with the fitted mixture density and null component. Warning signs include artificial narrow spikes, a density that does not integrate approximately to one over its support, pronounced oscillations from a very small bandwidth, or an alternative estimate that is implausibly broad because null-like observations received substantial alternative weight. A useful diagnostic is to compare the observed statistic distribution with the fitted mixture, while remembering that a good fit to the marginal density does not by itself guarantee that the null and alternative components are separately well identified.

Bandwidth choice is not a substitute for checking the null model. If the assumed $f_0$ is poorly calibrated, responsibility weights may be wrong, and smoothing cannot repair that error. Nor should a visually attractive curve be interpreted as evidence that every local feature is real; in a high-dimensional analysis, many apparent bumps can be sampling variation.

Worked Example: Boundary Correction for a Bounded Statistic

Suppose the statistic is a p-value, so its support is $[0,1]$, and two hypothetical observations are $0.1$ and $0.2$, each with alternative responsibility 1. Use a uniform kernel with $h=0.2$ and estimate the density at the boundary $z=0$. Without correction, each kernel has height $1/(2h)=2.5$ at zero, so the ordinary estimate is $2.5$.

That kernel places part of its mass below zero, outside the possible p-value range. One boundary correction is to truncate each kernel to $[0,1]$ and renormalize it there. For an observation $p_i$, the corrected kernel is $$ K_{h,i}^{*}(z)= \frac{K_h(z-p_i)} {\int_0^1 K_h(u-p_i)\,du},\qquad 0\leq z\leq1. $$ For $p_1=0.1$, the original uniform kernel extends from $-0.1$ to $0.3$; its in-range width is $0.3$, so its corrected height at zero is $1/0.3=3.333$. For $p_2=0.2$, the support is $[0,0.4]$, of width $0.4$, giving corrected height $1/0.4=2.5$. The equally weighted estimate is therefore $(3.333+2.5)/2=2.917$.

This calculation illustrates why naive symmetric smoothing can distort a density near a finite boundary. The corrected estimate integrates to one on $[0,1]$ because each observation's truncated kernel is normalized on that interval. The correction is not needed for a statistic whose support is the whole real line, such as an ordinary z-score.

From the Alternative Density to Local FDR

Once $\widehat f_1$ is available, the fitted mixture density is $\widehat f(z)=\widehat\pi_0 f_0(z)+(1-\widehat\pi_0)\widehat f_1(z)$. The plug-in local FDR is the fitted posterior probability that a test at $z$ is null: $$ \widehat{\operatorname{lfdr}}(z)= \frac{\widehat\pi_0 f_0(z)} {\widehat\pi_0 f_0(z)+(1-\widehat\pi_0)\widehat f_1(z)}. $$ It lies between zero and one when the fitted quantities are nonnegative and the denominator is positive. If the estimated alternative density is unusually large at a statistic, the fitted posterior null probability tends to decrease; if it is small, the local FDR tends to increase. This is a pointwise posterior probability under the fitted mixture model, not the tail-area false discovery rate for a selected set.

Worked Example: Density Uncertainty and a Local FDR

At a hypothetical statistic $z=2$, suppose the null is standard normal, $\widehat\pi_0=0.8$, and $\widehat f_1(2)=0.20$ per unit of $z$. The standard normal density at 2 is approximately $f_0(2)=0.0540$ per unit of $z$. The fitted mixture density is $$ \widehat f(2)=0.8(0.0540)+0.2(0.20)=0.0432+0.0400=0.0832. $$ Thus, $$ \widehat{\operatorname{lfdr}}(2)=\frac{0.0432}{0.0832}=0.5192. $$ Under this fitted model, a test with statistic 2 has an estimated posterior null probability of about 0.519, or 51.92%.

For an illustration of uncertainty, suppose a bootstrap gives a standard error of 0.030 for $\widehat f_1(2)$, while temporarily treating $\widehat\pi_0$ and $f_0(2)$ as fixed. Write $a=\widehat\pi_0 f_0(2)=0.0432$, $b=1-\widehat\pi_0=0.2$, and $g(y)=a/(a+by)$. Then $$ g'(y)=-\frac{ab}{(a+by)^2} =-\frac{0.0432(0.2)}{0.0832^2}\approx-1.248. $$ A first-order standard-error approximation is $1.248(0.030)=0.0374$ for the local FDR. A rough normal 95% interval is $0.5192\pm1.96(0.0374)$, or approximately $[0.446,0.593]$. This interval accounts only for the stated alternative-density uncertainty; a full analysis should also account for uncertainty in the null model and $\pi_0$, and should respect the local FDR's $[0,1]$ range.

Uncertainty, Alternatives, and Limitations

A useful way to quantify uncertainty is to bootstrap the entire estimation procedure: resample test statistics in a way appropriate to their dependence structure, refit the mixture, recompute responsibilities, re-estimate $f_1$, and recalculate local FDR. Resampling only the final density while holding responsibilities fixed misses uncertainty from the mixture fit. If tests are dependent, ordinary resampling of individual tests can understate variability; resampling suitable blocks or other dependence-aware units may be more appropriate. Pointwise intervals also do not automatically provide simultaneous coverage over a range of statistics.

A second approach estimates the marginal density $f$ and subtracts the null contribution: $$ \widehat f_1(z)= \frac{\widehat f(z)-\widehat\pi_0 f_0(z)} {1-\widehat\pi_0}. $$ This can be unstable because small errors in either term are amplified by division, especially when $1-\widehat\pi_0$ is small. It can also produce negative values, which cannot be a density, and generally requires constraints or regularization. Responsibility-weighted smoothing is often more naturally nonnegative, though it remains dependent on the fitted mixture and is not free of identification problems.

A fixed parametric alternative can be more stable and economical when its assumptions are credible, but may miss departures from its chosen shape. Nonparametric smoothing is more flexible, at the cost of bandwidth sensitivity and higher variance when the effective alternative sample is small. Neither method can recover details unsupported by the data. In particular, strong overlap between null and alternative components can make the estimated alternative density and local FDR uncertain even when the overall mixture density looks well estimated.

Common Mistakes

Treating responsibilities as observed labels. The $\tau_i$ values are fitted probabilities, not known truth. Omitting their uncertainty can make estimates appear more certain than they are.
Choosing a bandwidth by appearance alone. A smooth curve can erase real structure, and a jagged one can reflect noise. Check sensitivity and fit the mixture for each bandwidth considered.
Ignoring a finite support boundary. Symmetric kernels can place mass outside the allowable range. Use an appropriate boundary correction or a transformation, and verify normalization.
Calling density a probability. $\widehat f_1(z)$ has density units. It is not the probability that a test is alternative; that probability is $\tau_i$ for an observed statistic under the fitted model.
Reporting local FDR without its model dependence. It is a posterior probability conditional on the fitted null, mixture proportion, and alternative density, not a model-free property of the statistic.

Key takeaway: A responsibility-weighted kernel estimate provides a flexible estimate of the alternative density, but its usefulness depends on the mixture fit, bandwidth, support-aware smoothing, and honest treatment of uncertainty. Local FDR inherits uncertainty from all these ingredients.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

If all responsibilities are multiplied by the same positive constant, does the normalized weighted kernel estimate change? Why?
What is the main risk of using a very large bandwidth for $\widehat f_1$?
Why can an ordinary symmetric kernel be problematic for p-value density estimation near zero?
Using $\widehat\pi_0=0.9$, $f_0(z)=0.04$, and $\widehat f_1(z)=0.10$, calculate the plug-in local FDR.

Solutions. (1) No. Both the numerator and denominator are multiplied by that constant, so it cancels. (2) Oversmoothing can hide peaks or other features and bias the alternative density at a statistic of interest. (3) It assigns kernel mass below zero, outside the p-value support, and can distort the estimated density near the boundary unless corrected. (4) The null contribution is $0.9(0.04)=0.036$, and the alternative contribution is $0.1(0.10)=0.010$. Thus the mixture density is $0.046$, and the local FDR is $0.036/0.046=0.7826$, approximately 78.26% under the fitted model.