Why Display a Two-Dimensional Delta Grid?
A delta sensitivity analysis asks how conclusions change when the unobserved outcomes of participants with missing data are shifted from their values under a reference imputation model. In a two-arm trial, the shift may differ between treatment and control. A grid of values for both arms shows how the prespecified decision changes across combinations of assumptions, rather than reporting only a few one-at-a-time scenarios.
The grid is a display of sensitivity assumptions, not a probability distribution for those assumptions. Its usefulness depends on clear units, a meaningful range of values, a consistent analysis at each point, and a decision rule specified before inspecting the resulting boundary.
For the examples below, the outcome is continuous, measured in points, and higher values are better. A negative delta lowers the imputed missing outcomes relative to the reference imputation; a positive delta raises them. The treatment effect is the treatment-arm mean minus the control-arm mean, so a positive estimate favors treatment. These direction conventions must be stated because delta signs have no interpretation apart from the outcome scale and contrast definition.
Specify the Grid and the Decision Rule
Choose the delta values to reflect a defensible range of departures from the reference assumptions. The values may be equally spaced, but equal spacing is not required. Use the same units as the outcome when deltas are additive shifts. A grid with \(m\) treatment values and \(k\) control values contains \(m k\) combinations; each row-column pair must identify one combination unambiguously.
Before evaluating the grid, define the analysis and its decision criterion. For example, a superiority analysis may declare a positive treatment effect supported when the lower endpoint of a two-sided 95% confidence interval for the treatment-minus-control contrast exceeds zero. If the prespecified criterion instead uses a clinically important margin, that margin—not zero—must be used consistently. The criterion is a rule for classifying results, not a claim that the assumptions at any one grid point are correct.
At each grid point, use the same analysis specification, including the outcome definition, estimand, imputation procedure, and pooling method. If multiple imputation is used, the estimate and its uncertainty must be obtained by the chosen pooling procedure at that grid point. Do not substitute the estimate from one scenario into another scenario’s standard error unless that approximation is explicitly justified.
Worked Example: Building a Labeled Grid
Suppose a hypothetical trial measures a continuous symptom score in points, with higher scores indicating better health. The analyst defines \(\delta_T\) and \(\delta_C\) as additive point shifts to missing outcomes in the treatment and control arms. A negative value represents outcomes worse than the reference imputations. Consider treatment values \(0,-2,-4\) points and control values \(0,-2,-4\) points.
The Cartesian product contains \(3\times3=9\) grid points. A clear labeling scheme is to index each result by the ordered pair \((\delta_T,\delta_C)\), not by an unlabeled row number. For example, \((-2,-4)\) means a 2-point downward shift for treatment-arm missing outcomes and a 4-point downward shift for control-arm missing outcomes.
For an implementation, form all pairs, attach the units and the sign convention to the specification, and send each pair to the same analysis routine. The nine inputs are \((0,0)\), \((0,-2)\), \((0,-4)\), \((-2,0)\), \((-2,-2)\), \((-2,-4)\), \((-4,0)\), \((-4,-2)\), and \((-4,-4)\), all in points. This listing makes it possible to check that no combination is omitted or duplicated.
This step constructs the grid only. It does not imply that the nine scenarios are equally plausible, nor that a conclusion at one point is more likely than a conclusion at another.
Evaluate Each Point and Derive a Simple Check
The main calculation at each point is the sensitivity analysis itself. Save, at minimum, both delta values, the pooled treatment-effect estimate, its standard error or confidence interval, and the binary result of applying the prespecified criterion. Keeping the numerical results alongside the pass/fail label makes the display auditable and helps reveal mistakes such as reversed signs or accidentally repeated outputs.
A simple derivation helps explain why the two arm-specific deltas can move the treatment contrast in different directions. Suppose, only for this calculation, that the reference-imputed arm mean is a weighted average of observed and missing outcomes, that a fraction \(q_a\) of outcomes in arm \(a\) are missing, and that every missing value is shifted by the same additive delta. The observed outcomes remain unchanged. The arm mean therefore changes by the missing fraction times the shift:
Subtracting the control mean change from the treatment mean change gives the shift in the treatment-minus-control contrast:
Here \(\theta(0,0)\) is the reference contrast, and \(q_T\) and \(q_C\) are the missing fractions in the treatment and control arms. This identity is a transparent weighted-mean calculation under the stated simplifications; it is not a general substitute for rerunning a fitted imputation analysis. In an actual analysis, the pooled estimate and its standard error can both vary across grid points.
Worked Example: Evaluating the Decision Rule Across a Grid
Consider a hypothetical analysis on a continuous outcome measured in points. At the reference point \((0,0)\), suppose the pooled treatment-minus-control estimate is \(0.30\) points and its standard error is \(0.12\) points. For illustration, assume the pooled standard error remains \(0.12\) at the grid points shown. The prespecified criterion is that the lower endpoint of the two-sided 95% confidence interval must exceed zero. Using \(1.96\) as the normal critical value, the lower endpoint is \(\hat{\theta}-1.96SE\).
At the reference point, the lower endpoint is \(0.30-1.96(0.12)=0.0648\) points, so the criterion is met. The cutoff for the estimate is \(1.96(0.12)=0.2352\) points: with this fixed standard error, a point meets the criterion exactly when its estimate exceeds \(0.2352\) points.
For a compact illustration of the grid arithmetic, suppose the missing fractions are \(q_T=0.40\) and \(q_C=0.25\), and use \(\theta(0,0)=0.30\). The weighted-mean identity gives \(\theta(\delta_T,\delta_C)=0.30+0.40\delta_T-0.25\delta_C\), with deltas in points. At \((-0.20,0)\), the estimate is \(0.30+0.40(-0.20)-0.25(0)=0.220\) points. Its lower endpoint is \(0.220-1.96(0.12)=-0.0152\) points, so the criterion is not met.
At \((-0.20,-0.10)\), the estimate is \(0.30+0.40(-0.20)-0.25(-0.10)=0.245\) points. Its lower endpoint is \(0.245-0.2352=0.0098\) points, so the criterion is met. The control-arm shift is negative, but it raises the treatment-minus-control contrast here because the control mean is subtracted.
The following table applies the same criterion to all nine combinations. “Meets” means that the lower endpoint exceeds zero; “does not meet” means it does not. Values are pooled contrast estimates in points under the stated illustrative calculation.
| \(\delta_T\) (points) | \(\delta_C=0\) (points) | \(\delta_C=-0.10\) (points) | \(\delta_C=-0.20\) (points) |
|---|---|---|---|
| 0 | 0.300, meets | 0.325, meets | 0.350, meets |
| -0.10 | 0.260, meets | 0.285, meets | 0.310, meets |
| -0.20 | 0.220, does not meet | 0.245, meets | 0.270, meets |
This display shows a decision change between treatment delta values of \(-0.10\) and \(-0.20\) points when \(\delta_C=0\). It also shows why a two-dimensional display is useful: the same treatment delta can meet or fail the criterion depending on the control delta.
Find and Display the Tipping Boundary
A tipping-point table is useful when the grid is modest in size or the exact values need to be read directly. A contour plot can make the decision boundary easier to see on a larger grid. Put \(\delta_T\) on one axis and \(\delta_C\) on the other, use labeled axes with units, and distinguish the regions that meet and do not meet the criterion. The contour of interest is where the decision statistic reaches its threshold. For the confidence-interval rule above, that is where the lower confidence limit equals zero.
With the fixed standard error in the preceding illustration, the threshold is \(\hat{\theta}=0.2352\) points. When \(\delta_C=0\), the illustrative contrast reaches that threshold when \(0.30+0.40\delta_T=0.2352\), giving \(\delta_T=-0.162\) points. When \(\delta_C=-0.10\), it reaches the threshold when \(0.30+0.40\delta_T-0.25(-0.10)=0.2352\), giving \(\delta_T=-0.2245\) points. These are boundary calculations under the simplified fixed-SE formula, not universal tipping values.
For a real grid with scenario-specific standard errors, calculate the decision statistic at every point and draw or tabulate its threshold boundary. If the grid is coarse, a boundary drawn between neighboring points is an approximation. Do not imply that an interpolated contour is an exactly evaluated scenario; add grid points near the apparent boundary and evaluate them using the analysis procedure.
State the outcome scale, arm labels, delta sign convention, grid values, analysis method, and decision criterion before interpreting results.
For each \((\delta_T,\delta_C)\), apply the corresponding adjustments and obtain the pooled estimate and uncertainty using the same analysis procedure.
Apply the criterion consistently, then verify the number of results equals the number of grid pairs and that the displayed labels match the inputs.
Use a labeled table or contour plot to show where the criterion changes, and identify any interpolation as approximate.
Common Pitfalls, Diagnostics, and Limitations
A grid display is only as reliable as its inputs and the analysis at each point. Check that the outcome units and delta units agree; that treatment and control are not swapped; and that the contrast direction and delta signs are stated in the display. Check that every grid point has an estimate and uncertainty, and that the decision label follows from the same formula everywhere. A table whose labels change while its numeric estimates do not may signal a coding or indexing error.
Do not interpret a colored region as a probability statement. For example, a larger region of “meets” cells does not mean there is a greater probability that the treatment effect is positive. Grid points represent selected assumptions, and the shape of the display depends on the chosen ranges and spacing. Nor does a tipping point establish that outcomes under that scenario are plausible; plausibility requires substantive justification.
The simple weighted-mean derivation assumes an additive shift, a fixed missing fraction, and no other changes to the analysis. Real multiple-imputation procedures may have uncertainty that changes with the delta values, and nonlinear or bounded outcomes require extra care. For binary outcomes, adding a constant directly to a probability can produce values outside \([0,1]\); a suitable scale and model must be chosen rather than silently truncating invalid values. The grid also cannot compensate for a poorly specified reference model or an inadequately chosen range of assumptions.
The main alternative to a two-dimensional grid is to report a small set of named scenarios, such as changing one arm’s delta while holding the other at a reference value. Those scenarios can be easier to explain but may miss interactions between the two assumptions. A grid makes that joint dependence visible, at the cost of more computation and a greater need for careful labeling. A contour plot summarizes the boundary compactly, while a table preserves exact evaluated results; either can be accompanied by the other when both clarity and precision matter.
Worked Example: Interpreting a Boundary Without Overclaiming
Suppose a hypothetical contour plot has \(\delta_T\) on the horizontal axis and \(\delta_C\) on the vertical axis, both in points. The region with lower confidence limit above zero is shaded. On the line where the lower limit equals zero, the decision criterion is exactly at its boundary; points on either side may receive different classifications.
If a plotted boundary crosses approximately \((\delta_T,\delta_C)=(-0.16,0)\), the correct interpretation is that, near this grid location, the decision changes as the treatment-arm missing outcomes are shifted downward while the control delta is held at zero. It is not correct to conclude that the probability of the treatment effect being positive is 50% there, or that \(-0.16\) points is the true missing-outcome shift. If the location comes from interpolation between evaluated grid points, report it as approximate and evaluate a finer grid near that location before describing it as a tipping scenario.
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions.
- A grid has 4 treatment delta values and 6 control delta values. How many distinct combinations should be evaluated?
- For a higher-is-better outcome and a treatment-minus-control contrast, why can a negative control delta increase the contrast?
- Using \(\theta(0,0)=0.30\), \(q_T=0.40\), \(q_C=0.25\), and the illustrative mean-shift formula, calculate the contrast at \((\delta_T,\delta_C)=(-0.10,-0.20)\).
- With a pooled estimate of \(0.22\) points and \(SE=0.12\) points, does the two-sided 95% confidence-interval lower-limit criterion exceed zero? Use \(1.96\) as the critical value.
- Why should an interpolated contour boundary not automatically be reported as an exactly evaluated tipping point?
Solutions. (1) There are \(4\times6=24\) combinations. (2) The contrast subtracts the control mean; lowering that mean makes the treatment-minus-control difference larger, all else equal. (3) Substitute into the formula: \(0.30+0.40(-0.10)-0.25(-0.20)=0.30-0.04+0.05=0.310\) points. (4) The lower limit is \(0.22-1.96(0.12)=0.22-0.2352=-0.0152\) points, so it does not exceed zero and the criterion is not met. (5) A contour between evaluated grid points estimates where the threshold lies; unless the analysis was run at that location, it is an interpolation rather than a directly evaluated result.