Why Mixture Identifiability Matters
In a two-group model for genomic test statistics, the observed distribution is represented as a mixture of a null component and an alternative component. The estimated null proportion can affect downstream assessments of which tests are likely null. But a numerical estimate is not automatically informative: if the component distributions are identical, or nearly so, the data cannot determine their relative proportions reliably. In that case, an optimizer may return a precise-looking number even though many mixture proportions fit almost equally well.
This tutorial focuses on that distinction. Identifiability is a property of the model: can different parameter values produce exactly the same distribution? Practical distinguishability concerns the data at hand: even if the model is identifiable in principle, is the information in a finite sample sufficient to estimate its parameters stably? A boundary estimate is one symptom to investigate, not a verdict about how many tests are truly null.
A standard likelihood fit treats the observed test statistics \(X_1,\ldots,X_m\) as independent draws from this mixture, giving likelihood \(L(\pi_0)=\prod_{i=1}^{m}f(X_i)\). That independence assumption must be appropriate for the likelihood being used; if dependence is present, a suitable joint model or a justified approximation is needed. The identifiability arguments below concern the mixture distribution itself and do not remove the need to address dependence.
When Is the Mixture Proportion Identifiable?
First suppose both component densities are known and fixed. If \(f_0\) and \(f_1\) differ on a set with positive probability, then the mixture proportion is identifiable. To see why, suppose two proportions \(\pi_0\) and \(\pi_0'\) produce the same density:
Rearranging gives
If \(f_0\) and \(f_1\) differ on a set of positive probability, the second factor is not zero almost everywhere, so equality of the distributions requires \(\pi_0=\pi_0'\). If instead \(f_0=f_1\) almost everywhere, then \(f(x)=f_0(x)\) for every value of \(\pi_0\); the mixture proportion is not identifiable at all.
This argument has an important qualification. It assumes the component densities are fixed and known. In applications, parameters of \(f_1\), or even parameters describing the null distribution, may also be estimated. Then different combinations of the mixture proportion and component parameters can produce the same or nearly the same mixture density. Identifiability must be considered for the whole parameterization, not just for \(\pi_0\) in isolation.
Worked Example: Exact Non-Identifiability When Components Coincide
Suppose a model uses \(f_0=N(0,1)\) and \(f_1=N(\mu,1)\), where \(\mu\) is also estimated. Consider the parameter value \(\mu=0\).
At that value, both components have the same density, so for every \(\pi_0\) between 0 and 1,
Thus \((\pi_0,\mu)=(0.2,0)\) and \((0.8,0)\), for example, give exactly the same distribution of observations. No sample size can distinguish those parameter values using this model: the non-identifiability is exact. If the fitted \(\mu\) is merely close to zero, rather than exactly zero, the components differ but may still be difficult to distinguish with a finite sample. A sample that appears consistent with \(\mu\) near zero does not prove the components coincide; it signals that estimates of the mixture proportion may be sensitive to the data and model.
Boundary Estimates for the Null Proportion
Because \(0\leq\pi_0\leq1\), a likelihood maximization is constrained. A boundary estimate occurs when the maximum is attained at \(\hat{\pi}_0=0\) or \(\hat{\pi}_0=1\). For empirical null-proportion estimation, \(\hat{\pi}_0=1\) means that the fitted model assigns zero mixture weight to the alternative component. It does not prove that every test in the population is null: a finite sample can look more null-like than the assumed alternative predicts, and model misspecification can have the same effect.
A simple discrete example makes the mechanism explicit. Suppose a statistic has two possible outcomes, 0 and 1. Under the null, the probability of 1 is 0.50; under the alternative, it is 0.80. The mixture probability of observing 1 is therefore \(q=0.50\pi_0+0.80(1-\pi_0)=0.80-0.30\pi_0\), which is restricted to the interval from 0.50 to 0.80. If the observed fraction of 1s is below 0.50, the unconstrained best-fitting probability is outside the model's allowable range. The constrained likelihood then maximizes at \(q=0.50\), or \(\pi_0=1\).
Worked Example: A Boundary Maximum at the All-Null Fit
Suppose 10 independent hypothetical test outcomes are observed, with 4 equal to 1 and 6 equal to 0. Assume the null and alternative probabilities above are correctly specified and fixed. The likelihood, omitting the binomial coefficient that does not depend on the parameters, is
For an unconstrained binomial probability, the likelihood is maximized at the observed fraction, \(q=4/10=0.40\). But 0.40 is outside the allowable interval. To verify the constrained maximum, differentiate the log likelihood:
For every allowable \(q\geq0.50\), the denominator is positive and \(4-10q\leq-1\), so the log likelihood is decreasing throughout the allowable interval. Its maximum is therefore at \(q=0.50\). Solving \(0.50=0.80-0.30\pi_0\) gives \(\hat{\pi}_0=1\). This is a boundary estimate under the specified model; it is not evidence that the true alternative proportion is necessarily zero.
A boundary maximum can arise because the data favor an endpoint, because the components are poorly distinguished, or because the fitted component shapes do not represent the data well. These explanations are not interchangeable. A likelihood maximum at an endpoint is a result of the optimization; deciding what it means requires checking the component assumptions and the behavior of the likelihood nearby.
Weak Separation and Unstable Estimates
Even when \(f_0\ne f_1\), their difference may be small relative to sampling variability. One way to see the consequence is through a two-category model. Let the probability of outcome 1 be \(q=a\pi_0+b(1-\pi_0)\), where \(a\) and \(b\) are the null and alternative probabilities. If \(a\ne b\), the population proportion can be recovered from \(q\):
When \(b-a\) is close to zero, small sampling changes in the estimated \(q\) are magnified by division by a small number. The model is identifiable for \(a\ne b\), but the proportion can be estimated very imprecisely. In a smooth likelihood, the same issue appears as a shallow profile: a broad range of \(\pi_0\) values has nearly the same likelihood.
Worked Example: A Formally Identifiable but Weakly Separated Mixture
Suppose the null probability of outcome 1 is 0.50 and the alternative probability is 0.55. In 100 independent hypothetical observations, 52 outcomes are 1. The components are distinct, so the mixture is identifiable when their probabilities are known. The fitted mixture probability is \(q=52/100=0.52\), giving
To gauge the instability, the binomial standard error for the observed fraction is approximately \(\sqrt{q(1-q)/100}=\sqrt{0.52(0.48)/100}=0.0500\). Since \(\pi_0=(0.55-q)/0.05\), changes in \(q\) are multiplied by \(1/0.05=20\); the corresponding approximate standard error for \(\hat{\pi}_0\) is \(0.0500/0.05=1.00\). This is an approximation for the unconstrained estimate, not a valid uncertainty interval respecting the parameter bounds. Its size relative to the entire allowable range from 0 to 1 makes the key point: the estimate 0.600 is highly unstable, despite exact identifiability. A small change in the observed fraction could move the constrained fit to an endpoint.
Diagnosing a Difficult Fit
A single reported estimate does not show whether the mixture is well identified. Examine the fitted likelihood and the component assumptions directly. The following checks help distinguish a stable interior estimate from an endpoint or a broad region of near-equivalent fits.
- Inspect a profile likelihood. Fix \(\pi_0\) at a grid of allowable values and maximize over any other fitted parameters at each value. A sharply peaked profile supports a more stable estimate than a flat profile or one with several nearly equal maxima.
- Check component separation. Plot or otherwise compare the fitted null and alternative densities over the region where observations occur. Components that overlap almost completely provide little information about group membership or mixture weight.
- Repeat the fit from different starting values. Materially different final parameter estimates with nearly equal likelihoods suggest a ridge, multiple maxima, or numerical instability. A converged optimizer alone does not establish identifiability.
- Vary plausible modelling choices. Refit under reasonable alternative specifications for the alternative density or null parameters. Large changes in \(\hat{\pi}_0\) indicate that the estimate depends strongly on assumptions that the data may not resolve.
- Check the boundary and nearby values. If \(\hat{\pi}_0=1\), compare the likelihood there with values just below 1. A near-tie indicates a weakly determined alternative weight; a steep drop is stronger evidence that the specified likelihood favors the endpoint, although it still does not establish that the specification is correct.
These checks complement the calibration diagnostics and sensitivity analyses introduced in earlier material on empirical null models. No plot or optimization diagnostic can establish that a component model is scientifically correct; diagnostics identify ways in which the fitted model may be uninformative or inadequate.
Model Choices, Comparisons, and Limitations
With fixed, well-separated component densities, estimating a single mixture proportion is comparatively constrained. If the alternative density is allowed to be highly flexible, it may adapt to the observed distribution in ways that trade off against \(\pi_0\). A parametric alternative can reduce that flexibility, but a poor parametric shape can force observations into the wrong component and push the null estimate toward a boundary. A nonparametric alternative avoids some shape restrictions but does not by itself solve the identification problem: flexibility can make distinct allocations fit similarly well.
These approaches answer different modelling needs, not a universal ranking. Central matching, maximum likelihood, and nonparametric alternative estimation each depend on assumptions about which part of the observed distribution is informative and how the component densities are represented. When the components overlap or the assumptions are weak, agreement across reasonable fits is reassuring but not proof of identification; disagreement is a warning that the estimate should not be treated as a stable data feature.
Standard regular large-sample approximations also need caution at a boundary. At \(\pi_0=1\), the alternative weight is zero, and parameters describing the alternative may then have no effect on the likelihood. This violates the usual setting in which all parameters are locally estimable and interior. Consequently, a routine symmetric standard error or a routine likelihood-ratio reference distribution may be inappropriate without further justification. A boundary fit deserves explicit reporting and a method of uncertainty assessment designed for the fitted model, rather than an automatic regular approximation.
Check Your Understanding
Answer each question, then compare with the concise solutions below.
- If \(f_0=f_1\), what does the mixture density imply about the identifiability of \(\pi_0\)?
- Known null and alternative probabilities of outcome 1 are 0.30 and 0.70. The observed fraction is 0.50. Calculate the fitted \(\pi_0\).
- In the preceding question, if the observed fraction were 0.20, what would the constrained maximum-likelihood estimate be?
- Why can a mixture proportion be identifiable but still have a highly unstable estimate?
- Name two diagnostics that can reveal a weakly identified or numerically unstable mixture fit.
Solutions. (1) Since \(f(x)=f_0(x)\) for every \(\pi_0\), the proportion is not identifiable. (2) \(\hat{\pi}_0=(0.70-0.50)/(0.70-0.30)=0.20/0.40=0.500\). (3) The allowable mixture probability ranges from 0.30 to 0.70, so the unconstrained value 0.20 is outside the model. The likelihood is maximized at the nearest allowable endpoint, \(q=0.30\), which gives \(\hat{\pi}_0=(0.70-0.30)/0.40=1\). (4) Distinct parameter values may imply distinct distributions but those distributions can be so similar that sampling variability produces large changes in the estimate. (5) Examples include profiling the likelihood over \(\pi_0\), comparing fitted component densities for overlap, checking sensitivity to starting values, and refitting under plausible alternative density specifications.