Why Summarize Decision Stability?
Sensitivity analyses can produce many model outputs: one for each combination of structural assumptions, parameter scenarios, or both. For a development decision, the central question is often simpler: does the action selected under the reference analysis change under these alternatives? A decision-stability table answers this directly by displaying decisions side by side and counting changes relative to a stated reference.
The table is a summary of the consequences of the analyses already specified, not a test of which model is true. Its value depends on making the decision rule, reference case, alternatives, and counting denominator transparent. A count is meaningful only for the set of alternatives actually evaluated.
Specify the Reference and the Decision Rule
A decision is the action selected after applying a rule to analysis results. For example, a program may proceed if an estimated benefit–risk score is at least zero. Write the rule before inspecting the stability count, including the direction of the inequality and how a value exactly at the threshold is handled. Otherwise, small or borderline values may be classified inconsistently.
Choose a reference analysis that is substantively justified, such as the primary model with the central parameter assumptions. The reference is an anchor for comparison, not a claim that it is correct. List the alternatives and explain what changes in each: a model structure, one or more parameter assumptions, or a combination. When comparing model structures, keep the decision rule and the matched scenario definitions fixed where possible. Otherwise, a difference in decision may reflect a changed rule or scenario rather than the model structure alone.
For a binary decision, let \(d_0\) be the reference decision and \(d_i\) the decision under alternative \(i\), for \(i=1,\ldots,N\). Code the decisions consistently, for example Go and No-Go. Define an indicator \(I_i\) equal to 1 when the decisions differ and 0 when they agree. Then the number and rate of reversals are:
This calculation is a short but important justification for reporting both the count and denominator: the same \(R\) can mean different things for different numbers of alternatives. The rate is merely the fraction of the enumerated alternatives that reverse the reference decision. Unless the alternatives have a justified probability distribution and the analysis accounts for it, \(\widehat{r}\) is not the probability that the decision is wrong or will reverse in practice.
Worked Example: Counting Reversals Across Assumption Scenarios
Hypothetical example. A team uses a benefit–risk score, in score units, and applies the rule “Go if the score is at least 0; otherwise No-Go.” The reference analysis has a score of 1.2 and therefore selects Go. Six alternative scenarios produce the following scores.
| Analysis | Score (units) | Decision | Differs from reference? |
|---|---|---|---|
| Reference | 1.2 | Go | Not applicable |
| Lower absorption assumption | -0.4 | No-Go | Yes |
| Higher dropout assumption | 0.3 | Go | No |
| Alternative comparator assumption | -0.1 | No-Go | Yes |
| Joint lower absorption and higher dropout | -0.8 | No-Go | Yes |
| Alternative structure A | 0.5 | Go | No |
| Alternative structure B | -0.2 | No-Go | Yes |
The decision rule is applied consistently: scores at or above 0 are Go, and negative scores are No-Go. Four of the six alternatives differ from the reference, so \(R=4\), \(N=6\), and \(\widehat{r}=4/6=0.667\), or 66.7%. Two alternatives agree, giving an agreement rate of \(2/6=33.3\%\). The appropriate interpretation is that four of these six specified alternatives reverse the reference decision. It is not that there is a 66.7% probability the program should be No-Go.
Compare Model Structures With Matched Scenarios
When the purpose is to assess stability across model structures, a useful table pairs each model's decision under the same input scenario. This prevents unlike comparisons—for example, a conservative scenario under one model against a central scenario under another—from being counted as evidence about structural differences. If some scenarios cannot reasonably be matched, report that limitation rather than implying a controlled comparison.
For each matched scenario, first apply the decision rule separately to each model's output. Then count pairwise disagreements against the chosen reference model, keeping the scenario as the comparison unit. A model may disagree in one scenario and agree in another; both outcomes should remain visible rather than being concealed by a single overall label.
Worked Example: A Model-by-Scenario Stability Table
Hypothetical example. Three model structures are evaluated under the same three assumption scenarios. Each cell gives a benefit–risk score in units. Go is selected for a score of at least 0. Model M1 is the reference structure.
| Scenario | M1 score and decision | M2 score and decision | M3 score and decision | Alternative-model reversals versus M1 |
|---|---|---|---|---|
| S1 | 0.8, Go | 0.4, Go | -0.1, No-Go | 1 of 2 |
| S2 | -0.3, No-Go | 0.1, Go | -0.5, No-Go | 1 of 2 |
| S3 | 0.2, Go | -0.2, No-Go | 0.6, Go | 1 of 2 |
There are \(3 \times 2=6\) comparisons: two alternative models against M1 in each of three matched scenarios. M2 reverses M1 in S2 and S3, giving 2 reversals in 3 comparisons. M3 reverses M1 only in S1, giving 1 in 3. Overall, \(R=2+1=3\), \(N=6\), and the unweighted reversal rate is \(3/6=50.0\%\). This combined rate summarizes all six comparisons, while the row and model-specific counts show where the disagreements occur. It does not establish that either alternative model is more credible or that the probability of reversal is 50%.
The matched comparison assumes that the scenarios have the same substantive meaning under each model and that the decision rule is unchanged. If a model cannot represent a scenario, or if its output is not comparable on the score scale, do not force a cell into the table; explain the incompatibility and report the available comparisons separately.
More Than Two Possible Actions
Not every decision is Go versus No-Go. A dose-selection decision may choose among several candidate doses. In that case, a change from 20 mg to 30 mg is a decision change even though both actions might represent continued development. Report the selected actions, not just a binary “stable” label. A transition count can show how often each reference action becomes each alternative action.
Worked Example: Stability Across Three Candidate Doses
Hypothetical example. A prespecified rule selects the candidate dose with the highest decision score, with any ties resolved in favor of the lower dose. The reference analysis selects 20 mg. Eight defined alternatives select doses as follows: 10, 20, 30, 20, 30, 10, 20, and 30 mg.
| Alternative selected dose | Count | Interpretation relative to reference |
|---|---|---|
| 10 mg | 2 | Different action |
| 20 mg | 3 | Same action |
| 30 mg | 3 | Different action |
| Total | 8 |
Five of the eight alternatives select a dose other than 20 mg, so \(R=2+3=5\), \(N=8\), and \(\widehat{r}=5/8=0.625\), or 62.5%. The table also shows the direction of change: two alternatives move to 10 mg and three to 30 mg. Reporting only “five reversals” would omit this useful distinction. The calculation assumes the candidate set and tie-breaking rule are identical across analyses.
Reading the Table and Diagnosing Problems
A stability table is strongest when it preserves enough detail to audit the count. Include the reference decision, alternative identifier, changed model or assumption, decision output, reversal status, and the total number of alternatives. For model comparisons, identify the matched scenario and show model-specific results. If space requires a summary, retain a more detailed table or clearly defined scenario list so that the numerator and denominator can be reconstructed.
Several checks help detect a misleading or incorrect summary:
- Reapply the rule. Verify every decision from its underlying score or criterion, especially observations at the threshold. State rounding rules if reported precision could change classification.
- Check the reference. Confirm that it is the intended primary analysis and that every reversal is calculated against the same reference. Changing the reference changes the count.
- Check the denominator. Count only the alternatives actually evaluated, state exclusions, and avoid counting multiple outputs from one scenario as independent alternatives without saying so.
- Check scenario coverage. A low reversal rate across a narrow list says little about unexamined assumptions. Explain how alternatives were selected and whether they represent plausible conditions already defined for the analysis.
- Inspect borderline decisions. If small output changes around the threshold flip the action, show the scores and distance from the threshold. A binary table alone can hide how close a decision is to changing.
- Separate decision change from model quality. Disagreement is a signal to understand, not proof that one structure is wrong. The table does not rank models or resolve their credibility.
An unweighted table treats every listed alternative equally. That is appropriate for a transparent count over a deliberately selected scenario set, but not necessarily for an uncertainty-weighted summary. If alternatives have unequal probabilities, the unweighted fraction must not be described as a probability-weighted risk. Likewise, alternative scenarios are often constructed to probe particular assumptions and may not be independent draws from any distribution.
How Stability Tables Complement Other Summaries
A decision-stability table answers a categorical question: how often does the selected action change over this specified set? A scenario table with continuous outputs, a sensitivity plot, or a threshold display can show the magnitude and pattern behind those changes. These are complementary rather than competing summaries: the stability table makes action changes easy to count, while the underlying outputs help explain why a reversal occurred.
A single count can conceal important structure. For example, the same reversal rate could arise from many changes under one assumption or one change in each of several model structures. Model-specific counts, scenario-level entries, and the direction of transition help distinguish these patterns. The table is also limited by the decision rule: a different threshold or tie-breaking rule may lead to a different stability result. Report the rule alongside the count rather than presenting the count as an intrinsic property of the model.
The practical goal is not to make a decision appear stable. It is to make clear whether the action remains the same across the analyses that stakeholders consider relevant, which alternatives change it, and what has not been evaluated. That makes the summary useful for discussion without converting a descriptive count into a guarantee of future success or failure.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A reference analysis selects Go. Among 12 specified alternatives, 3 select No-Go. What are the reversal count and unweighted reversal rate?
- Why is it usually preferable to compare two model structures under matched assumption scenarios?
- A dose decision has three possible actions. Why is it insufficient to report only whether a binary Go/No-Go decision changed?
- When can an unweighted reversal rate be interpreted as a probability of reversal?
Solutions. (1) There are \(R=3\) reversals among \(N=12\) alternatives, so \(\widehat{r}=3/12=0.25\), or 25.0%; this describes the specified set. (2) Matching gives the models the same scenario conditions, making differences more interpretable as structural disagreement rather than a mismatch in assumptions. (3) A change between two development actions, such as 10 mg and 30 mg, matters even if both imply continued development; reporting the selected actions shows the direction and destination of changes. (4) Only when the alternatives are assigned justified probabilities and the weighting and sampling design support that interpretation; an ordinary count over a selected scenario list is not itself a probability.