Two Different Sources of Variation
A treatment-effect analysis can report a confidence interval for an estimate calculated under one set of assumptions. It can also report how the estimated effect changes when those assumptions change. These quantities are both relevant, but they answer different questions. The interval describes sampling uncertainty conditional on an assumption; variation across assumptions describes sensitivity of the result to the assumption.
The distinction matters when the data do not determine the treatment effect without additional assumptions. For example, in a hypothetical trial with missing outcomes, a particular assumption about those outcomes may produce one estimated treatment effect, while a different assumption produces another. The earlier material on conditional treatment estimates and standard errors develops the estimate and its standard error under a specified assumption. Here the focus is how to interpret that conditional uncertainty alongside changes across assumptions.
The assumption \(a\) is not automatically a random variable. A set of sensitivity scenarios is often used to examine how conclusions depend on alternative assumptions, not to claim that each scenario has a known probability of being true. Thus, a range of estimates across scenarios is not, by itself, a confidence interval or a probability distribution for the treatment effect.
What a Conditional Confidence Interval Says
Suppose that, under a fixed assumption \(a\), the estimator is approximately unbiased and normally distributed with standard error \(SE(a)\). A large-sample two-sided 95% confidence interval is then
Its frequentist interpretation is conditional: if the assumption \(a\) holds and the study were repeatedly conducted under the relevant sampling model, intervals computed in this way would contain \(\theta(a)\) approximately 95% of the time. It is not a statement that the assumption is correct with 95% probability, nor that the true effect has a 95% probability of lying in this particular realized interval.
The interval’s width reflects sampling uncertainty as represented by the model at \(a\). Its center, \(\hat{\theta}(a)\), may move when \(a\) changes. The amount of that movement is not included merely by calculating an interval at one value of \(a\).
Worked Example: A Conditional Interval at Two Assumptions
Consider a hypothetical trial estimating the mean treatment effect on systolic blood pressure, in mmHg, with negative values favoring treatment. Under assumption \(a_1\), the estimated effect is \(-4.0\) mmHg and its standard error is \(1.2\) mmHg. Under assumption \(a_2\), the estimate is \(-1.0\) mmHg and its standard error is also \(1.2\) mmHg. Suppose the normal approximation is suitable under each assumption.
Under \(a_1\), the 95% interval is
Under \(a_2\), it is
Each interval describes sampling uncertainty conditional on its own assumption. The point estimates differ by \(3.0\) mmHg, which indicates sensitivity to the choice between these assumptions. The difference between the estimates is not another standard error, and the pair of intervals is not an interval that accounts for all assumptions. The first interval lies entirely below zero, while the second includes zero; thus, the conditional statistical conclusion differs across the two assumptions.
Why the Two Quantities Must Not Be Combined by Inspection
A confidence interval has a sampling interpretation because it is tied to a repeated-sampling procedure and its coverage under specified conditions. A spread of estimates across assumptions is instead a comparison of results obtained under different conditions. It has no universal probability interpretation. For instance, the smallest and largest estimates in a chosen scenario table do not automatically form a 95% interval.
A useful way to make the distinction precise is to write the interval for each assumption separately:
where \(z_{1-\alpha/2}\) is the standard normal quantile used for the desired nominal confidence level. Changing \(a\) can change the estimand \(\theta(a)\), the estimate \(\hat{\theta}(a)\), and possibly its standard error. A conditional interval quantifies uncertainty in estimating the effect defined at that particular \(a\); it does not quantify uncertainty about which \(a\) should be used.
If \(A\) is instead assigned a genuine probability distribution, a different calculation may be meaningful. For example, suppose \(A\) is random, the target is explicitly the random quantity \(\theta(A)\), and the estimator is conditionally unbiased, so that \(E\{\hat{\theta}(A)\mid A\}=\theta(A)\). The law of total variance gives
where the last equality treats \(SE(A)^2\) as the conditional sampling variance. The first term represents average within-assumption sampling variance; the second represents variation in the target across the probability distribution for \(A\). This decomposition requires the stated random-\(A\) model and target. It does not justify treating an arbitrary list of sensitivity scenarios as random or equally likely.
Worked Example: Scenario Spread Is Not a Standard Error
Suppose a hypothetical analysis reports estimated treatment effects, in points on a symptom scale, under three fixed assumptions: \(2.0\), \(3.5\), and \(5.0\). Their corresponding standard errors are \(0.8\), \(0.9\), and \(1.1\) points. These are scenario results, not draws from a probability distribution over assumptions.
The range of the three estimates is
That \(3.0\)-point span describes how far the reported estimates vary across these selected assumptions. It is not an estimated sampling standard deviation: the standard errors, which describe sampling uncertainty conditional on each assumption, are \(0.8\), \(0.9\), and \(1.1\) points.
For comparison, the conditional 95% interval under the first assumption is
This interval concerns the effect under the first assumption, not all three assumptions. The \(3.0\)-point scenario range concerns changes across assumptions, not repeated-sample variation. Without a probability model assigning meaning to the three assumptions, neither the range nor an average of their estimates has a probability-based interpretation.
When Averaging Across Assumptions Is—and Is Not—Meaningful
It can be tempting to average scenario estimates and describe the result as a compromise that includes assumption uncertainty. Arithmetic alone does not give that average a statistical interpretation. If assumptions have no probability weights and the estimand is not defined as an average over them, the average may correspond to no specified scientific question.
Worked Example: An Average Requires a Defined Target
Suppose a hypothetical analysis gives estimated effects of \(-2\), \(-5\), and \(-8\) mmHg under three fixed assumptions. Their arithmetic mean is
The calculation is correct, but it does not establish that \(-5\) mmHg is the effect under any particular assumption, or that it is a best estimate accounting for uncertainty about assumptions. It is simply the unweighted mean of three scenario estimates.
An average could have a defined interpretation if a probability distribution over assumptions were justified and the target were explicitly the mean effect under that distribution. For example, if the three assumptions each had probability \(1/3\), the target would be the probability-weighted average of their effects. Those weights are additional modeling information, not something inferred from having three scenarios. Even then, uncertainty for the averaged target must be calculated using a method appropriate to the joint model; averaging the three conditional confidence intervals is not that method.
Diagnostics, Alternatives, and Limitations
A clear report keeps the two kinds of information visible rather than compressing them into one number. For each assumption being examined, report the assumption’s meaning, the estimate, its conditional standard error or interval, and the effect scale and units. Compare the point estimates across assumptions separately. This makes it possible to see both how much the estimate moves and how precise it is conditional on each scenario.
Warning signs include calling a scenario range a confidence interval, treating the difference between two estimates as a standard error, or describing a conditional confidence interval as if it included uncertainty about the assumptions. Also check whether standard errors are being compared on the same scale and whether the interval procedure is appropriate under each assumption. A normal-based interval may be unreliable with a small sample, a strongly skewed estimator, or a standard error that does not reflect the actual sampling design.
The main alternatives answer distinct questions. A conditional confidence interval gives sampling uncertainty at a specified assumption. A scenario comparison shows how estimates change over the assumptions selected for examination. A probabilistic model for assumptions can support probability-weighted summaries, but only if its distribution and target are substantively justified. None of these alternatives automatically replaces the others. In particular, a conditional interval can be narrow while the estimate changes substantially across assumptions, or wide while estimates remain similar across assumptions.
Sensitivity analysis is limited by the assumptions it examines. Stability across a small or poorly chosen set does not demonstrate stability under every plausible assumption. Conversely, a large change across scenarios does not identify which scenario is correct. The results should be interpreted as conditional on the assumptions and analysis procedures actually used.
State which assumption defines the treatment effect being estimated.
Report the estimate and its standard error or confidence interval under that assumption.
Describe changes in point estimates as sensitivity to assumptions, not as sampling variation.
Average or combine across assumptions probabilistically only when a defensible distribution and corresponding target have been defined.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An estimate under a fixed assumption is \(4.0\) units with \(SE=0.5\). What is its approximate 95% normal-based confidence interval, and what uncertainty does it represent?
- Two assumptions produce estimates of \(1\) and \(6\) units. Is their difference of \(5\) units a standard error? Explain.
- A researcher lists four sensitivity scenarios and reports the range of their estimates as a “95% confidence interval.” What is wrong with this interpretation?
- When could averaging effects across assumptions have a probability-based interpretation?
Solutions. 1. The interval is \(4.0\pm1.96(0.5)=4.0\pm0.98=(3.02,4.98)\) units. It represents sampling uncertainty conditional on the fixed assumption, under the normal approximation. 2. No. The \(5\)-unit difference measures the change between two scenario estimates; it is not a standard error unless a separate sampling model establishes that interpretation. 3. The range across four selected scenarios has no automatic 95% coverage property. A confidence interval requires a procedure with an appropriate repeated-sampling interpretation under stated conditions. 4. Averaging can be probability-based when a defensible probability distribution over assumptions is specified and the target is explicitly defined as an average over that distribution; the averaging procedure must also account for the relevant uncertainty.