Why Multiple Outcomes Need a Joint Conclusion
A treatment can appear beneficial on one outcome and uncertain, negligible, or harmful on another. Under missing-data sensitivity analysis, each outcome’s result may also change as assumptions about unobserved outcomes change. A useful summary must therefore answer two different questions: how robust is each outcome-specific conclusion? And do the outcomes support one coherent overall claim under the same plausible assumptions?
This tutorial treats the outcomes as a specified set of \(K\) outcomes and the sensitivity assumptions as a shared scenario \(\eta\) in a plausible set \(S\). It does not combine different outcomes into a single score. Instead, it makes explicit what can be concluded about each outcome and whether those conclusions hold together. Effects should first be oriented so positive values consistently represent benefit; for an outcome where lower values are desirable, this may mean reversing the sign of the treatment contrast.
For a particular scenario, a confidence interval entirely above \(B_j\) supports a benefit exceeding the chosen threshold at the interval’s stated confidence level, conditional on that scenario and the interval procedure’s assumptions. An interval entirely below \(-H_j\) supports harm exceeding its threshold. An interval that overlaps either threshold does not establish that the effect lies on one side of that threshold. The threshold is a clinical or scientific criterion, not a substitute for statistical uncertainty.
Outcome-Specific and Joint Robustness
An outcome-specific robust benefit conclusion requires that the relevant result hold throughout the plausible set, not just under a selected scenario. For interval-based conclusions, the pointwise condition is \(L_j(\eta)>B_j\) for every \(\eta\in S\). Likewise, an outcome-specific robust harm conclusion requires \(U_j(\eta)<-H_j\) for every scenario. These conditions are stronger than finding one favorable scenario, and they do not imply that all other outcomes support the same conclusion.
An all-outcome robust benefit claim requires the benefit condition to hold for every outcome and every scenario. This is a demanding statement: even if most outcomes support benefit, one outcome whose interval crosses its threshold prevents this particular all-outcome claim. The same logic applies to an all-outcome harm claim. More generally, an overall report can state which outcomes are robustly beneficial, harmful, or inconclusive, without forcing a single label onto all outcomes.
For a finite set of outcomes, the all-outcome benefit condition can be written directly as
It is tempting to replace this pointwise requirement with \(\inf_{\eta\in S}L_j(\eta)>B_j\). That is a sufficient uniform-margin criterion, but it is not equivalent in general: the infimum may equal \(B_j\) even though every individual value is strictly greater. If \(S\) is compact and \(L_j\) is continuous, the infimum is attained, and the equivalence does hold. Without such conditions, report the pointwise condition or use a non-strict bound that matches the intended claim.
Worked Examples Across Outcomes
Worked Example: Classifying Three Outcomes Across Two Scenarios
Suppose a hypothetical trial reports effects oriented so positive values favor treatment. The outcomes are symptom improvement in points, functional score improvement in points, and serious adverse events avoided per 100 patients. Let the respective benefit thresholds be \(B_1=2\), \(B_2=3\), and \(B_3=1\), in those units. There are two plausible sensitivity scenarios, A and B. The conditional intervals are:
| Outcome | Scenario A interval | Scenario B interval | Benefit threshold |
|---|---|---|---|
| Symptoms (points) | [2.4, 5.0] | [2.2, 4.7] | 2 points |
| Function (points) | [3.4, 6.1] | [2.8, 5.5] | 3 points |
| Events avoided per 100 | [1.3, 3.0] | [1.1, 2.8] | 1 per 100 |
For symptoms, the lower limits exceed 2 in both scenarios: \(2.4>2\) and \(2.2>2\). Thus the interval-based benefit conclusion is robust for this outcome over the two listed scenarios. For function, scenario A exceeds its threshold, \(3.4>3\), but scenario B does not, because \(2.8<3\). Function is not robustly above the clinical threshold. For events avoided, both lower limits exceed 1: \(1.3>1\) and \(1.1>1\), so the conclusion is robust for this outcome.
The results support robust threshold-exceeding benefit for symptoms and events avoided, but not for all three outcomes. It would be inaccurate to call the overall result an all-outcome robust benefit. This conclusion concerns only the two scenarios specified; it extends to a larger sensitivity set only if the condition is checked throughout that set.
Worked Example: A Strict Pointwise Claim Without a Uniform Margin
For one hypothetical outcome, suppose the plausible sensitivity set is \(S=(0,1]\) and the conditional lower confidence limit is \(L(\eta)=B+\eta\), where \(B\) is the clinical benefit threshold. For any permitted scenario, \(\eta>0\), so \(L(\eta)=B+\eta>B\). The pointwise condition for benefit holds at every scenario.
However,
Therefore \(\inf_{\eta\in S}L(\eta)>B\) is false. The infimum is approached as \(\eta\) approaches zero, but zero is not in the set, so it is not attained. The correct conclusion is that the lower limit exceeds the threshold at every allowed scenario, with no positive uniform margin separating it from the threshold. If a report needs to guarantee a margin of at least \(\delta>0\), it must instead establish \(L(\eta)\geq B+\delta\) throughout \(S\).
Worked Example: Separating Benefit, Harm, and Inconclusive Outcomes
Consider two hypothetical outcomes with effects oriented so positive values favor treatment. Across every scenario in a specified plausible set, outcome A has intervals contained within \([1.2,4.0]\), and outcome B has intervals contained within \([-5.0,-2.4]\). Let \(B_A=1\), \(H_A=1\), \(B_B=2\), and \(H_B=2\), each in its outcome’s units.
For outcome A, the lower limit is at least 1.2, so \(L_A(\eta)\geq1.2>1=B_A\) for every scenario. It meets the robust benefit criterion. For outcome B, the upper limit is at most \(-2.4\), so \(U_B(\eta)\leq-2.4<-2=-H_B\) for every scenario. It meets the robust harm criterion. The outcomes therefore support opposing robust conclusions: benefit on A and harm on B.
This does not justify an unqualified overall benefit or harm claim. It calls for an outcome-by-outcome report that preserves the conflict and states the units and thresholds. A single average across these differently scaled outcomes would require an explicitly defined composite and defensible weighting; it cannot be inferred from these intervals alone.
Why the Joint Summary Is Not a Joint Confidence Interval
A sensitivity-robust conclusion and a joint statistical confidence statement are different. The first asks whether an assumption-conditional conclusion persists as \(\eta\) varies over \(S\). The second asks about simultaneous coverage of multiple unknown effects under repeated sampling. If each outcome has a separate 95% confidence interval, that alone does not establish 95% simultaneous coverage of all \(K\) effects. Do not describe an all-outcome conclusion based on marginal intervals as having a joint 95% confidence level unless the intervals or procedure were constructed to provide that joint coverage.
The same caution applies to dependence between outcomes. Outcomes measured on the same participants may be correlated, and their estimates may therefore be statistically dependent. The pointwise checklist across outcomes does not require treating them as independent, but a formal joint inference procedure must account for the relevant dependence and its intended error control. Multiplicity procedures are a separate issue; here, the key task is to state clearly whether the conclusion uses marginal or simultaneous intervals.
Derivation and Practical Checks
The all-outcome condition follows from the meaning of “every outcome under every scenario.” For each fixed pair \((j,\eta)\), interval-based threshold-exceeding benefit requires \(L_j(\eta)>B_j\). Requiring this for all pairs gives the displayed criterion. If it fails, at least one outcome-scenario pair has \(L_j(\eta)\leq B_j\), so the specified all-outcome claim is not established. Conversely, if it holds for all pairs, each outcome’s interval lower limit is above its threshold at every scenario, so the criterion is satisfied. This is a direct logical justification, not a claim that the outcomes have been combined into a new estimand.
In practice, keep an outcome-by-scenario record and check the following:
Confirm the effect direction, scale, units, and clinical threshold for each outcome. A positive effect must mean the same favorable direction even if the original measurement is lower-is-better.
Compare the appropriate conditional interval limit with its outcome-specific threshold. Do not infer robustness from a selected row or an average across scenarios.
Label each outcome’s conclusion and then determine whether all required outcome criteria hold at every scenario. Preserve mixed results instead of hiding them in a single summary label.
Report whether intervals are conditional on scenarios and whether coverage is marginal or simultaneous. Do not turn assumption robustness into a claim of joint confidence coverage.
A common failure is a threshold, sign, or unit mismatch. For example, “fewer events” is favorable, but an un-reoriented treatment-minus-control contrast may be negative when events are reduced. In that case, applying a positive-benefit threshold to the unreversed contrast gives the wrong classification. Another failure is checking only point estimates: an estimate above a threshold does not establish that the conditional interval lies above it. A third is choosing the most favorable sensitivity scenario after seeing the results and presenting it as representative of the full plausible set.
How This Summary Differs From Alternatives
| Approach | What it answers | What it does not establish by itself |
|---|---|---|
| Outcome-by-outcome sensitivity results | Which outcomes retain a specified conclusion across scenarios | A single coherent conclusion for all outcomes |
| All-outcome robustness check | Whether every specified outcome meets its criterion throughout the same scenario set | A joint confidence level unless simultaneous inference is used |
| One selected scenario | What follows conditional on that assumption | Robustness over other plausible assumptions |
| Composite outcome | An effect on a prespecified combined scale, if defined and justified | That every component outcome benefits |
A composite can be appropriate when its definition, scale, and weighting are scientifically justified, but it answers a different question from whether each component outcome is robustly favorable. Conversely, separate outcome summaries retain clinical detail but may leave mixed findings. Neither presentation should be mistaken for the other.
Common Mistake and Limitations
Similarly, requiring \(\sup_{\eta\in S}U_j(\eta)<-H_j\) is a sufficient uniform-margin condition for robust harm, but it is not equivalent to \(U_j(\eta)<-H_j\) at every scenario when the supremum is not attained. For example, if \(S=(0,1]\) and \(U(\eta)=-H-\eta\), then every allowed value is below \(-H\), while the supremum equals \(-H\). State the pointwise claim directly unless attainment is justified.
Finally, the conclusions inherit the limitations of the plausible sensitivity set, the effect models, the interval procedures, and the chosen thresholds. Robustness over \(S\) does not prove that \(S\) contains every scientifically credible assumption, nor that missingness assumptions are true. It means only that the stated conclusion persists over the set that was examined.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- For two outcomes, the lower interval limits are above their benefit thresholds in every scenario for outcome 1, but one scenario’s limit is below the threshold for outcome 2. Is an all-outcome robust benefit established?
- If \(S=(0,1]\) and \(L(\eta)=B+2\eta\), is \(L(\eta)>B\) for every scenario? Is \(\inf_{\eta\in S}L(\eta)>B\)?
- An outcome is measured so that a negative treatment-minus-control contrast is favorable. What must be checked before applying a positive-benefit threshold?
- Do separate 95% confidence intervals for three outcomes automatically provide 95% simultaneous coverage?
Solutions. (1) No. Outcome 2 fails the criterion in at least one scenario, so the all-outcome claim is not established, although outcome 1 may be robust. (2) Yes, because \(\eta>0\) implies \(B+2\eta>B\). No: the infimum is \(B\), approached as \(\eta\) approaches zero but not attained. (3) Reorient the effect, or define the threshold consistently on the original scale; otherwise the sign comparison reverses the clinical meaning. (4) No. Marginal 95% intervals do not by themselves guarantee 95% simultaneous coverage; that requires an appropriate joint procedure.