Diagnostic Plots for Intermediate-Based Selection

Why plot intermediate-based selection?

An intermediate outcome can be observed sooner than the final time-to-event outcome, so a MAMS design may use its estimated treatment effect to decide which arms continue. The central diagnostic question is not simply whether intermediate and final outcomes are associated. It is whether the intermediate information gives a dependable guide to the final treatment effect and to the arms that a prespecified selection rule will retain.

Three plots answer different parts of that question. A joint-effect scatterplot displays intermediate and final effects together, making discordance and ranking reversals visible. A calibration plot compares final effects with predictions based on intermediate effects. A retention curve shows how many arm-cases a selection threshold keeps, and how many of those have a prespecified level of final benefit. None of these plots alone establishes that a selection rule has acceptable error rates or power; they diagnose patterns that a design assessment should investigate.

Assumptions before interpreting the plots

The paired effects must refer to comparable treatment contrasts, populations, and effect definitions. For time-to-event outcomes, the usual log-hazard-ratio convention makes a more negative value favorable. For the plots here, reverse that orientation—for example, use minus the log hazard ratio—so that larger values are favorable for both outcomes. State the transformation and units on the axes. If intermediate and final effects are on different scales, a diagonal identity line is not a meaningful calibration reference unless the effects have first been put on a comparable scale.

The source of the points also matters. They may come from external evidence or from prespecified scenarios and data sets used for design assessment. Estimated effects have sampling error; points should not automatically be treated as known true effects. When estimates are plotted, their uncertainty and any covariance between \(\hat{\theta}_I\) and \(\hat{\theta}_F\) matter. If multiple points come from the same trial, scenario, or population, they are not independent observations simply because they appear as separate dots. A plot is descriptive, not a substitute for accounting for this dependence.

A threshold \(c\) must also be interpreted on the scale actually used by the selection rule. A cutoff on \(\hat{\theta}_I\) differs from a cutoff on a standardized statistic such as \(\hat{\theta}_I/SE\). If precision changes across arms or stages, the same raw effect estimate need not represent the same evidence. Plotting the rule's actual decision quantity, or clearly labeling a raw-effect diagnostic as such, prevents a misleading comparison.

Joint-effect scatterplots: association is not enough

Plot \(\hat{\theta}_I\) on the horizontal axis and \(\hat{\theta}_F\) on the vertical axis, or use corresponding scenario-level effects when those are the available quantities. Add a vertical line at the proposed selection cutoff and a horizontal reference at the final-benefit level of interest. The cloud's direction and spread show whether arms with stronger intermediate effects tend to have stronger final effects. Points in the upper-right region are favorable on both measures; points to the right but low on the vertical axis are potential false selections under the stated final-benefit criterion.

A positive association does not guarantee stable ranking. A plot can show an upward trend while still containing important rank reversals: an arm with a larger intermediate effect may have a smaller final effect than another arm. Conversely, a narrow range of intermediate effects can make a correlation appear weak even if the ordering within the range is practically useful. Therefore inspect the locations of the points relative to selection and benefit boundaries, not just a correlation coefficient.

Calibration plots: compare predictions with final effects

A calibration plot makes the prediction being assessed explicit. For each arm-case, first obtain a predicted final effect from a prespecified or fitted relationship, then plot the observed final effect against that prediction. The identity line \(y=x\) represents perfect calibration when prediction and observed effect use the same scale: points above the line have larger observed than predicted effects, and points below it have smaller observed than predicted effects. If predictions are systematically too large or too small, the plot can reveal that bias. A fitted calibration line can summarize this pattern, but it should not obscure individual errors or uncertainty.

For a simple linear calibration model, write the predicted final effect as \(a+b\hat{\theta}_I\), where \(a\) is an intercept and \(b\) is a slope. In a data set of \(m\) paired points, the least-squares slope and intercept are \(b=S_{IF}/S_{II}\) and \(a=\bar{\theta}_F-b\bar{\theta}_I\), where \(S_{IF}=\sum_{j=1}^{m}(\hat{\theta}_{Ij}-\bar{\theta}_I)(\hat{\theta}_{Fj}-\bar{\theta}_F)\) and \(S_{II}=\sum_{j=1}^{m}(\hat{\theta}_{Ij}-\bar{\theta}_I)^2\). This line summarizes the average linear relationship in the data; it does not assert that every arm-case follows the line.

Calibration should be checked on data not used to fit the prediction relationship whenever possible. Otherwise, the same data can make predictions look better calibrated than they will be in new settings. A calibration curve may also be nonlinear, especially if the intermediate-final relationship changes across the effect range. Binning points can aid visualization, but bin averages can hide variability and depend on bin boundaries; display the individual points as well where practical.

Retention curves: examine the cutoff, not only the cloud

For arm-cases \(j=1,\ldots,m\), define the empirical retention fraction at cutoff \(c\) as \(R(c)=m^{-1}\sum_{j=1}^{m}I(\hat{\theta}_{Ij}>c)\), where \(I(\cdot)\) equals one when its condition is true and zero otherwise. A useful companion curve is the fraction of retained arm-cases that meet a prespecified final-benefit criterion \(\theta_F>\delta\), where \(\delta\) is a chosen benefit threshold. This second curve is undefined when no arm-cases are retained. It is a descriptive quantity, not the probability that a newly considered treatment will benefit.

These empirical curves are step functions: as the cutoff rises, the number of retained arm-cases can decrease or stay the same. It stays constant between observed intermediate estimates and changes only when a cutoff crosses one of those values. A curve of the benefit fraction among retained points need not be monotone: dropping one arm-case can raise or lower that fraction. Show the number retained alongside the fraction, since a seemingly high benefit fraction based on one retained point is unstable.

Derivation: why retention is a step function

Each indicator \(I(\hat{\theta}_{Ij}>c)\) is one for cutoffs below that arm-case's intermediate estimate and zero once the cutoff reaches or exceeds it. Therefore the sum of indicators can only change when \(c\) passes an observed estimate. For any \(c_2>c_1\), every arm-case retained at \(c_2\) is also retained at \(c_1\), so \(R(c_2)\leq R(c_1)\). This proves the retention fraction is non-increasing, while allowing it to remain constant over intervals and at some increases in \(c\). The argument depends on the specified strict rule \(\hat{\theta}_I>c\); a rule using \(\geq\) changes what happens exactly at a point value, not the overall step-function property.

Diagnostics, alternatives, and limitations

Use the plots together. A joint scatterplot identifies discordant regions; calibration assesses the scale and bias of predictions; the retention curve shows what the actual cutoff does to candidate arms. When a proposed cutoff appears acceptable only because it retains very few cases, the retention count exposes the fragility. When a high intermediate estimate frequently pairs with an unhelpful final effect, the scatterplot and benefit fraction reveal a possible misleading threshold. Examine these patterns across plausible populations and assumptions rather than relying on a single favorable display.

A correlation coefficient is a compact alternative summary of linear association, but it does not show calibration, threshold consequences, or which cases drive the association. A regression line summarizes an average relationship, but can conceal nonlinearities and reversals. Formal design-specific operating-characteristic calculations address probabilities of correct or incorrect selection under specified assumptions; diagnostic plots complement that work rather than replace it. A plot cannot by itself establish acceptable familywise error, power, or clinical benefit.

Common failure modes include mixing effect directions, comparing different estimands, treating noisy estimates as true effects, ignoring the dependence between paired estimates, and changing the cutoff scale without explanation. Missing intermediate or final outcomes can also alter which arm-cases appear; a plot of complete pairs alone may be unrepresentative if missingness is informative. Finally, a relationship observed in one set of scenarios may not transport to the population or treatment class where the rule will be used. State the data source, units, selection rule, benefit threshold, and uncertainty clearly, and treat visual diagnostics as prompts for investigation rather than proof.