Uncertainty and Reporting for Local FDR Estimates

Why Uncertainty Belongs in a Local FDR Report

A local false discovery rate is often reported as a single number for each test. That number can look precise even when it depends strongly on estimated mixture parameters, an uncertain null model, or a fitting procedure that is unstable near a boundary. Reporting only the plug-in estimates hides these sources of uncertainty and can make threshold-based decisions appear more certain than they are.

For a test statistic \(Z_i=z\), a two-component mixture model writes its marginal density as

Here \(\pi_0\) is the mixture proportion of null tests, \(f_0\) and \(f_1\) are the null and alternative densities, and \(\eta_0,\eta_1\) denote their parameters. The local FDR at \(z\) is the posterior probability of the null component under this model:

In practice, the parameters are estimated, producing the plug-in value \(\widehat{\operatorname{lfdr}}(z)=\operatorname{lfdr}(z;\hat{\theta})\). This is a model-based estimate, not a direct observation of whether the individual null hypothesis is true. Uncertainty in \(\hat{\theta}\) induces uncertainty in the estimated local FDR. The data value \(z\) is ordinarily treated as observed and fixed when summarizing that parameter-estimation uncertainty.

Assumptions and Two Ways to Quantify Uncertainty

Uncertainty calculations are only as credible as the model and sampling assumptions behind them. A likelihood-based calculation typically assumes that the chosen mixture describes the distribution of the test statistics, that the mixture is identifiable in the parameter region being used, and that the estimator is sufficiently regular for its large-sample approximation. A simple likelihood for all \(Z_i\) also assumes independence. If test statistics are dependent, treating them as independent can understate uncertainty; a dependence-aware covariance estimate or a defensible joint resampling model is needed.

Mixture models require particular care. If a component proportion is near zero, two components are poorly separated, or parameters are not identifiable, standard large-sample approximations may be unreliable. The earlier material on identifiability and boundary estimates in mixtures is relevant here: a small standard error from a routine Hessian calculation does not resolve a nonregular or weakly identified fit.

When regularity is plausible, the delta method gives a local approximation. Let \(\hat{\theta}\) estimate \(\theta\), let \(\widehat{\Sigma}\) estimate the covariance matrix of \(\hat{\theta}\), and define \(g(\theta)=\operatorname{lfdr}(z;\theta)\) at a fixed observed \(z\). If \(g\) is differentiable and \(\hat{\theta}\) is approximately normal, then

The square root is the estimated standard error. A normal-approximation interval is the estimate plus or minus 1.96 standard errors. Because a local FDR must lie between 0 and 1, this interval can be awkward near either boundary. One option is to form an interval on the logit scale and transform its endpoints back to the probability scale. This does not repair a poor model or a poor approximation; it only respects the parameter’s range.

A refitted bootstrap is another approach. Each bootstrap replicate must repeat the fitting steps used for the original estimate, including estimating mixture parameters and any data-dependent tuning choices. A parametric bootstrap generates replicate data from the fitted sampling model; it therefore inherits that model’s assumptions. A nonparametric or block-based resampling method requires its own justification, especially when test statistics are dependent. Resampling only the final local FDR values without refitting the model omits an important source of uncertainty.

Bootstrap Summaries and Stability of Decisions

A bootstrap distribution can reveal asymmetry, boundary pile-ups, and unstable estimates that a single standard error may conceal. For each replicate, refit the mixture and evaluate the local FDR at the same observed statistic, or repeat the full analysis if the target is a data-dependent decision. Summarize the distribution with a standard deviation, quantiles, or both. State how replicates were generated, how many fits failed, and whether failed fits were excluded or handled by a prespecified rule.

A percentile interval uses selected quantiles of the bootstrap estimates. It is a useful summary when the bootstrap is appropriate, but it is not automatically reliable: it cannot correct misspecification, dependence ignored in the resampling scheme, or a fitting algorithm that routinely finds only one of several plausible solutions. Report the point estimate and interval together, and describe substantial skew or boundary concentration rather than reducing everything to a symmetric standard error.

Thresholds, Expected False Discoveries, and Sensitivity

A local FDR threshold is a decision rule, not a property of the data alone. If a fixed set \(S\) of tests is selected using local FDRs, then, conditional on the observed data and the fitted model, the expected number of null tests in that set is the sum of their local FDRs:

If the selected set is nonempty and fixed after conditioning on the data, dividing this sum by its size gives the model-based expected false discovery proportion. This is an expectation, not a guarantee that the realized proportion will equal that value. Uncertainty in fitted mixture parameters can change the local FDRs and even change which tests pass a threshold. Therefore, report both the threshold and the resulting selection, and consider how the decision changes across bootstrap refits or defensible model specifications.

Sensitivity analysis asks whether the scientific conclusion depends on reasonable modeling choices. Relevant alternatives can include a plausible null distribution, alternative-distribution family, treatment of extreme statistics, or fitting options. These should be scientifically defensible choices, not merely a search for a preferred result. A compact table can show the estimated local FDR at key statistics and the number of tests passing the stated threshold under each specification.

Diagnostics, Comparisons, and Common Pitfalls

Useful diagnostics include checking the fitted marginal distribution against the observed test-statistic distribution, inspecting local FDR calibration plots and null-model diagnostics, and examining optimizer convergence, boundary estimates, and bootstrap fit failures. A bootstrap distribution with multiple modes or a pile-up at zero or one is a warning that a single symmetric interval may be inadequate. Also compare local FDR estimates and threshold selections across reasonable starting values when optimization is sensitive to initialization. These checks can expose problems; passing them does not prove that the mixture model is correct.

The delta method is fast and gives a direct account of parameter covariance, but it relies on a sufficiently smooth, regular fit and can misrepresent skewed or boundary-constrained uncertainty. A refitted bootstrap can describe nonlinearity and asymmetry more directly, but it can be computationally costly and depends on the resampling model. A Bayesian analysis can instead summarize posterior uncertainty in parameters and local FDRs, but its summaries depend on the prior and the specified likelihood. These approaches answer related but not identical inferential questions; label the method and its assumptions rather than presenting intervals as interchangeable.

A frequent mistake is to report a local FDR of \(0.05\) as “a 5% chance that this finding is false” without qualification. The correction is to say that, under the fitted mixture model and conditional on the observed statistic, the estimated posterior probability of the null component is \(0.05\). Another mistake is to call a bootstrap interval a posterior probability interval, or to claim a threshold controls the realized false discovery proportion exactly. State which model-based quantity is being reported and what uncertainty procedure produced it.