Two Different Reasons an Estimate May Be Uncertain
An imprecise estimate can arise because a sample is small, because the observed data contain little information about a parameter, or because the data cannot distinguish some parameter values at all. These situations are not interchangeable. More observations can reduce ordinary sampling uncertainty and can help when identification is weak. More observations drawn from the same observed-data process cannot separate parameter values that imply exactly the same observed-data distribution.
This distinction is especially important when outcomes can be missing. The observed-data law describes both which outcomes are observed and the values recorded when they are. A model for the complete data may contain features of the unobserved outcomes that do not affect this law. The introductory material on identified functionals distinguishes targets determined by the observed-data law from targets that are not; here the emphasis is on why a target may fail to be determined, or may be determined only very imprecisely.
Structural nonidentification is a property of the model and the observation process, not a synonym for a large standard error. It persists even if the observed-data distribution is known exactly. By contrast, a parameter is weakly identified when distinct values imply distinct observed-data laws, but the differences among those laws are small enough that realistic samples provide little precision. Weak identification is often described through a likelihood that changes slowly across a range of parameter values, or through small information in a parameter direction.
A model can also be structurally nonidentified in some directions and informative in others. For example, the observed response rate and the outcome distribution among respondents may be determined by the observed law even when the outcome distribution among nonrespondents is not. It is therefore important to name the target, not simply declare a whole model “identified” or “unidentified.”
Worked Example: Two Complete-Data Laws, One Observed Law
Worked Example: Two Complete-Data Laws, One Observed Law
Suppose each unit has a binary outcome \(Y\), but \(Y\) is observed only when the response indicator \(R=1\). Assume independent units with a common distribution. In a hypothetical population, let \(P(R=1)=0.60\), and among respondents let \(P(Y=1\mid R=1)=0.70\). The observed data reveal these quantities, but not the outcome distribution among nonrespondents.
Consider two complete-data models. In both, \(P(R=1)=0.60\) and \(P(Y=1\mid R=1)=0.70\), but in the first let \(P(Y=1\mid R=0)=0.20\), and in the second let it equal \(0.80\). The observed probabilities are the same in both models:
The observed record for a nonrespondent contains \(R=0\), not that unit’s \(Y\). Thus the two values of \(P(Y=1\mid R=0)\) do not change any observed-data probability. The observed-data laws are exactly equal.
However, the population mean of the binary outcome differs. In the first model it is \(0.60(0.70)+0.40(0.20)=0.50\). In the second it is \(0.60(0.70)+0.40(0.80)=0.74\). Therefore the population mean is not determined by the observed-data law in this unrestricted model, even though the response rate and respondent outcome probability are.
Why Exact Equivalence Cannot Be Fixed by a Larger Sample
For independent observations, the observed-data likelihood is the product of the observed-data probabilities (or densities) for the records. If two parameter values give the same observed-data law, each record has the same probability under both values. Their likelihoods are consequently equal for every possible observed sample. This yields a direct justification for why more observations cannot resolve exact nonidentification.
In the binary-outcome example, changing \(P(Y=1\mid R=0)\) leaves the likelihood unchanged. Increasing the number of respondents may estimate the response rate and respondent outcome distribution more precisely, but no respondent’s outcome supplies information about the outcome distribution among nonrespondents. A larger sample from the same observation process does not alter that fact.
The distinction between a parameter and a target functional matters here. Even when the full parameter contains nonidentified components, a particular function of those components may still be constant across all parameter values compatible with the observed law. Conversely, a target such as the population mean in the example varies across compatible complete-data models. Identification must be assessed for the stated target.
Assumptions Can Select a Value Without Making It Data-Identified
Worked Example: An Assumption Resolves a Choice
Use the same hypothetical response and outcome data: \(P(R=1)=0.60\) and \(P(Y=1\mid R=1)=0.70\). Now impose the restriction \(P(Y=1\mid R=0)=P(Y=1\mid R=1)\). Under this restriction, the nonrespondent success probability is set to \(0.70\), so the population mean is
The calculation is correct under the imposed equality. But the observed data in the previous example were also compatible with nonrespondent probabilities \(0.20\) and \(0.80\). The equality is therefore a modeling assumption that selects one complete-data law from several observationally equivalent laws; it is not evidence in the observed data that the equality holds. A reported estimate of \(0.70\) should be understood as conditional on that restriction.
This example illustrates a general caution: a model can produce a unique estimate after restrictions are imposed even though the unrestricted observed-data problem does not identify the target. Additional assumptions may be scientifically defensible, but the source of the resulting information must be described accurately. Estimation within a restricted model does not by itself demonstrate that the restriction is true.
Weak Identification: Distinct Laws That Are Hard to Tell Apart
Weak identification is different. Here the mapping from parameter values to observed-data laws is one-to-one over the parameter space: distinct values do imply distinct laws. Yet the laws can be so similar that finite data do not distinguish the values precisely. The effect is not a perfectly flat likelihood, but a shallow one.
A simple way to see the distinction is to examine information. In a regular one-parameter model, Fisher information measures how rapidly the observed-data distribution changes with the parameter. Small information means that even a correctly specified, identifiable model can yield a large standard error. A zero-information direction can indicate a local failure of identification, so information calculations should be interpreted alongside the model’s global mapping, not used as a stand-alone proof of identification.
Worked Example: A Weakly Informative Observed Pattern
Consider a hypothetical model in which each unit has an observed binary pattern indicator \(O\), with \(P(O=1\mid\theta)=r(\theta)=0.50+0.001\theta\), where \(-400\leq\theta\leq400\). Think of \(O\) as a recorded response pattern whose probability depends on a model parameter. Assume independent units and that this probability model is correctly specified. Because \(r(\theta)\) increases strictly with \(\theta\), each value of \(\theta\) gives a different observed-data law: the parameter is identifiable.
Suppose a hypothetical sample has \(n=100\) units and \(x=51\) with \(O=1\). The observed proportion is \(\hat{p}=51/100=0.51\). Setting \(r(\hat{\theta})=\hat{p}\) gives
The estimate is identifiable, but the probability changes by only \(0.001\) for a one-unit change in \(\theta\). For a Bernoulli observation with success probability \(r(\theta)\), the score is \(r'(\theta)\{O-r(\theta)\}/[r(\theta)\{1-r(\theta)\}]\). Since the variance of \(O\) is \(r(\theta)\{1-r(\theta)\}\), the information in \(n\) independent observations is
At \(\hat{\theta}=10\), \(r=0.51\) and \(r'=0.001\). Thus \(I_n=100(0.001)^2/[0.51(0.49)]\approx0.0004002\), giving the approximate standard error \(1/\sqrt{I_n}\approx49.99\) parameter units. A rough 95% Wald interval is \(10\pm1.96(49.99)\), or approximately \([-87.98,107.98]\). It is wide because this sample provides little precision, not because the model assigns exactly the same law to different values of \(\theta\).
Diagnostics and Practical Comparisons
A useful diagnosis begins with the model’s observed-data map: ask which observed probabilities or densities depend on each parameter, and whether distinct parameter values can give exactly the same collection of probabilities. Then examine how sensitive those probabilities and the likelihood are to changes in the parameter. The first question concerns structural identification; the second concerns the strength of information in the data.
Write down which quantities are actually recorded, including the response pattern, and state the parameter or functional to be learned.
Look for distinct parameter values that leave every observed-data probability unchanged. If they do, the affected parameter or target is structurally nonidentified.
Inspect likelihood profiles, information, and uncertainty across relevant parameter directions. A unique but shallow profile signals weak information, not exact equivalence.
State whether a unique answer comes from observed-data variation, a model restriction, or both. Assess whether conclusions change under plausible alternative restrictions.
These checks have limitations. A numerical optimizer that returns one solution does not establish identification; it may select one point on a flat ridge. A large standard error does not prove structural nonidentification either: it may reflect a small sample or weak information in an otherwise identifiable model. A singular or nearly singular information matrix can reveal problematic directions, but finite-sample numerical behavior and parameter scaling also matter. Conclusions should be checked against the model equations and the observed-data likelihood.
| Feature | Structural nonidentification | Weak identification |
|---|---|---|
| Observed-data laws | At least two parameter values give exactly the same law | Distinct values give distinct laws, but some are very similar |
| Likelihood behavior | Exactly equal along observationally equivalent values | Changes slowly over a range of values |
| Effect of more same-process data | Cannot distinguish exactly equivalent values | Can improve precision if information accumulates |
| Typical concern | Target is not determined without extra restrictions or data | Estimates may be unstable or intervals wide in realistic samples |
Alternative responses address different problems. Collecting more observations can strengthen weak identification, but it cannot break exact equivalence if the observation process and model remain unchanged. Collecting new types of information or imposing a scientifically motivated restriction can distinguish otherwise equivalent models, but the restriction’s role must be made explicit. Neither a narrow interval under strong assumptions nor a stable optimizer output is, by itself, a diagnostic that the observed data identify the target.
Common Mistake
- Wrong reasoning: “The estimate has a large standard error, so the parameter is not identified.”
- Correction: A large standard error is evidence of imprecision in that analysis, not proof of exact nonidentification. Check whether distinct parameter values induce the same observed-data law. If not, the parameter may be weakly identified or the sample may simply be small.
- Wrong reasoning: “The model returns one estimate, so the data identify the target.”
- Correction: A unique output may reflect a restriction, a numerical convention, or a prior choice. Verify whether the observed-data law itself separates the relevant parameter values.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In the binary-outcome example, which quantities are determined by the observed-data law, and why is the population mean not determined?
- If two parameter values give the same observed-data law, what happens to their likelihoods for any observed sample?
- In the weak-identification example, what is \(\hat{\theta}\) when the observed proportion is \(0.52\), and does that alone mean the estimate is precise?
- Would increasing the sample size from 100 to 10,000 resolve the exact nonidentification in the first example? Explain.
- Suppose an equality restriction produces one estimate of a missing-outcome mean. What must be reported about the source of that estimate?
Solutions. (1) The response probability \(0.60\) and respondent success probability \(0.70\) are determined by the observed law. The nonrespondent success probability is not observed, and changing it changes the population mean without changing the observed law. (2) They are equal for every possible observed sample because each record has the same probability under both parameter values. (3) Solving \(0.50+0.001\hat{\theta}=0.52\) gives \(\hat{\theta}=20\). This estimate alone does not establish precision; precision also depends on the sample size and the rate at which the observed law changes with \(\theta\). (4) No. More observations can estimate the observed response and respondent outcome distributions more accurately, but the likelihood remains unchanged across the observationally equivalent nonrespondent outcome probabilities. (5) The analysis must state that the estimate depends on the imposed restriction, distinguish that assumption from information supplied by observed data, and explain its scientific basis.