Why Look Beyond a Single Robustness Score?
A design may perform well on average while performing poorly in a particular region of parameter space. That distinction matters in pharmacometrics: a sampling schedule can be informative near the nominal clearance and absorption values yet lose much of its value when those values differ. A summary score compresses performance across scenarios into one number; diagnostic plots show where that compression hides unevenness.
This tutorial focuses on three complementary views: a parameter-scenario performance curve, a worst-case profile, and a criterion distribution. Each asks a different question. Curves reveal how performance changes as a scenario changes; worst-case profiles expose the weakest scenario for each design; distributions show the spread and tails of performance over a specified uncertainty distribution.
The criterion and its direction must be stated before interpreting any plot. If the underlying quantity is a loss for which smaller is better, one can plot the loss directly and reverse the interpretation, or define a clearly described transformed score for which larger is better. Do not silently reverse the direction between panels. Whenever possible, label the vertical axis with the actual criterion and its units; if a score is scaled or dimensionless, say so.
The plots are diagnostic summaries, not guarantees about unrepresented scenarios. Their interpretation assumes that the criterion is calculated consistently for all designs and scenarios, that the scenario set or distribution is scientifically defensible, and that numerical or simulation error is small enough not to obscure the comparisons. If the criterion is estimated by simulation, uncertainty in that estimate should be assessed separately from variability across parameter scenarios.
Parameter-Scenario Performance Curves
A performance curve is useful when a particular parameter dimension has a meaningful range and a fixed or clearly specified background scenario. For example, plot a design’s information score against a clearance multiplier while holding other parameters at stated values. Put multiple candidate designs on the same axes so that both sensitivity and changes in ranking are visible. If several parameters vary together, describe the path through parameter space: a one-dimensional curve is not a complete map of a multidimensional region.
The curve’s slope and shape help locate regions that deserve closer inspection. A steep decline signals sensitivity to the parameter change shown; a crossing of two curves indicates that the designs’ ranking depends on the scenario. A flat curve is not proof of robust performance overall: the fixed values of the other parameters may have selected an unusually favorable slice of the scenario space.
Worked Example: A Ranking That Changes Across Clearance Scenarios
Suppose two hypothetical sampling designs are evaluated at four clearance-multiplier scenarios. Their criterion is a scaled information score, in dimensionless information units, with larger values better.
| Clearance multiplier | Design A score | Design B score |
|---|---|---|
| 0.5 | 8 | 7 |
| 1.0 | 10 | 9 |
| 2.0 | 12 | 10 |
| 4.0 | 5 | 9 |
Plot the multiplier on the horizontal axis and score on the vertical axis, joining the values for each design. Design A rises from 8 to 12 as the multiplier goes from 0.5 to 2.0, then falls to 5 at 4.0. Design B rises from 7 to 10 and then remains comparatively high at 9. The curves cross between multipliers 2.0 and 4.0: A is better at 2.0, while B is better at 4.0. Thus the design ranking is scenario-dependent.
For illustration, an equally weighted average over these four scenarios is \( (8+10+12+5)/4=8.75 \) for A and \( (7+9+10+9)/4=8.75 \) for B. The equal averages do not imply equal behavior: the curve reveals that A has a pronounced weak region at the high multiplier, whereas B is more even across this grid. The conclusion applies to the displayed scenarios, not automatically to every possible clearance value.
Worst-Case Profiles
A worst-case profile reduces a scenario set to its lowest performance for each design. It is particularly informative when the design question emphasizes protection against poor performance throughout a specified plausible set. Compute the minimum separately for every design; the scenario attaining that minimum may differ by design. A plot of \(W(d)\) across candidate designs makes the weakest-case comparison direct.
This minimum is sensitive to the scenario set. Adding an extreme but implausible scenario can determine the profile, while a sparse grid can miss a genuine weak region. Report the scenario definition, including any parameter bounds and any constraints on combinations of values. When parameter dependence matters, use plausible joint scenarios rather than assuming that independently selected marginal extremes form a realistic combination; the earlier material on correlated parameter uncertainty is relevant to constructing such scenarios.
Worked Example: Finding the Weakest Scenario for Each Design
Consider three hypothetical designs evaluated on the same four plausible scenarios. The scores are scaled information units, and higher is better.
| Scenario | Design A | Design B | Design C |
|---|---|---|---|
| 1 | 8 | 7 | 9 |
| 2 | 10 | 9 | 8 |
| 3 | 12 | 10 | 7 |
| 4 | 5 | 9 | 6 |
The worst-case profiles are \(W(A)=\min(8,10,12,5)=5\), \(W(B)=\min(7,9,10,9)=7\), and \(W(C)=\min(9,8,7,6)=6\) information units. The weakest scenarios are scenario 4 for A, scenario 1 for B, and scenario 4 for C. A plot with design on the horizontal axis and worst-case score on the vertical axis therefore has values 5, 7, and 6.
Under a criterion that prioritizes the highest minimum score over this scenario set, B ranks first. This does not mean B is best under every objective: A has a score of 12 in scenario 3, and C has the highest score in scenario 1. The worst-case profile answers a specific question—how high is each design’s lowest score on this grid?—rather than replacing all other performance summaries.
Criterion Distributions and Tail Behavior
A criterion distribution is formed by evaluating a design across parameter scenarios drawn from a stated distribution. It displays how performance is spread over that distribution, not just its average. A histogram, density estimate, or empirical cumulative distribution function (ECDF) can show concentration, skew, multiple regions, and lower-tail behavior. A boxplot is a compact alternative, but it can hide multimodality and the location of individual poor scenarios.
State what is varying. If \(\Theta\) represents parameter uncertainty, the spread of \(S(d,\Theta)\) reflects variation across the specified parameter distribution. It is not automatically a confidence interval, a sampling distribution for an estimator, or a measure of Monte Carlo error. If the score itself is estimated by simulation, repeated simulation error can add another source of spread; quantify or control it rather than interpreting the total spread as parameter sensitivity alone.
Worked Example: Comparing Lower Tails Despite Similar Means
Suppose hypothetical scores from ten equally weighted parameter draws are as follows, in scaled information units:
| Draw | Design A | Design B |
|---|---|---|
| 1 | 2 | 5 |
| 2 | 4 | 5 |
| 3 | 5 | 6 |
| 4 | 6 | 6 |
| 5 | 7 | 7 |
| 6 | 8 | 7 |
| 7 | 8 | 8 |
| 8 | 9 | 8 |
| 9 | 10 | 8 |
| 10 | 11 | 8 |
The means are \(70/10=7.00\) for A and \(68/10=6.80\) for B. Using the empirical distribution and the nearest-rank convention, the 10th percentile is the smallest value: 2 for A and 5 for B. The medians are the averages of the fifth and sixth sorted values: \((7+8)/2=7.50\) for A and \((7+7)/2=7.00\) for B. Thus A has a slightly higher mean and median, but B has the better observed lower tail in these draws.
An ECDF would show that B’s scores are more concentrated and that A includes poorer low-end values as well as a high value of 11. The result is conditional on these ten draws and their weighting. With more draws, tail estimates may be more stable; a small simulation should not be presented as a precise estimate of a rare-tail probability.
How to Read the Plots Together
No one plot answers every design question. The short derivation below explains why an average can conceal a weak scenario. For equally weighted scenarios, let \(\bar{S}(d)\) be the arithmetic mean of the scores. Because every scenario score is at least the minimum score, their mean must also be at least that minimum:
The inequality can be wide: a few high scores can lift the mean while one score remains low. A performance curve helps locate such a low region when scenarios have an ordered parameter axis; a worst-case profile summarizes the minimum; and a criterion distribution shows how common or isolated low values are under a specified scenario distribution. These are complementary views, not interchangeable calculations.
| Plot or summary | Main question | Important limitation |
|---|---|---|
| Parameter-scenario curve | How does performance change along this parameter path? | Only displays the chosen path and fixed background values. |
| Worst-case profile | What is the minimum score over the specified scenario set? | Can be dominated by one scenario and depends on the grid. |
| Criterion distribution | How does performance vary under the stated scenario distribution? | Tail conclusions depend on the distribution and draw count. |
| Single average | What is the weighted mean score? | Conceals where poor performance occurs. |
Diagnostics, Pitfalls, and Limitations
Before interpreting a plot, check that each design uses the same criterion definition, units, parameter scenarios, and scenario weights. Curves should identify which parameter varies and which values are held fixed. A worst-case profile should make the scenario set explicit. A criterion distribution should identify its weighting distribution and, if applicable, its number of draws. If simulation error is appreciable, use adequate replication or show uncertainty in the estimated scores so that Monte Carlo noise is not mistaken for a design effect.
- Comparing incompatible scales. Raw scores and transformed scores may tell different visual stories. Label any normalization and retain an interpretable scale where possible.
- Over-reading a sparse grid. Connecting a few evaluated scenarios with a smooth line does not establish the score between them. Add evaluations where curves change rapidly or where rankings cross.
- Treating a marginal sweep as a full robustness assessment. Varying one parameter while fixing all others cannot reveal every joint weak region.
- Equating a low quantile with a worst case. A quantile summarizes a specified distribution; a minimum summarizes a finite set. Neither is automatically a guarantee over all plausible values.
A common mistake is to conclude that a design is robust because its average criterion is high. The correction is to inspect the scenario-level performance as well: identify where the score falls, calculate the minimum for the defined scenario set when worst-case protection matters, and examine lower-tail summaries under a defensible uncertainty distribution. Conversely, a single low minimum should not be treated as decisive without checking whether its scenario is plausible and whether the score is numerically reliable.
These plots support comparison and diagnosis; they do not choose the relevant uncertainty set, determine acceptable trade-offs, or prove performance beyond evaluated scenarios. Their value is to make the consequences of those choices visible before relying on a compressed summary.
Check Your Understanding
Answer each question using the stated criterion direction and scenario definition.
- A score is 9, 11, and 4 across three equally weighted scenarios. What are its mean and worst-case score, and what does the difference reveal?
- Two designs’ performance curves cross as a parameter changes. What can and cannot be concluded from that crossing?
- Why is the spread of scores over parameter draws not automatically Monte Carlo error or a confidence interval?
- A worst-case score is very low at one corner of a parameter grid. Name two checks to make before treating it as a meaningful design weakness.
Solutions. (1) The mean is \((9+11+4)/3=8\), and the worst-case score is 4. The mean summarizes typical equally weighted performance, while the minimum reveals the weak scenario. (2) The ranking changes along the displayed parameter path: whichever curve is higher performs better there. The crossing does not establish the ranking elsewhere in the multidimensional parameter space. (3) Parameter-draw variation describes score variation across the specified uncertainty distribution; Monte Carlo error concerns numerical estimation from finite simulation, and a confidence interval concerns uncertainty in a statistical estimator under a sampling model. (4) Check whether the corner is a plausible joint parameter combination and whether the scenario grid or simulation calculation is sufficiently accurate; also verify consistent criterion scaling and implementation.