Why Trace the Conclusion Across a Grid?
A sensitivity parameter represents an assumption that the observed data alone do not determine. In a delta-adjusted analysis, for example, \(\delta\) can specify how far an unobserved outcome is shifted from its reference imputation. The analysis is repeated at a sequence of plausible values of \(\delta\), and the treatment-effect estimate and its uncertainty may change at each value. The practical question is often not just how the estimate moves, but whether a prespecified decision changes anywhere in the examined range.
Tracing the conclusion means recording the estimate, the quantity that governs the decision, and the resulting decision at each grid value. If adjacent grid values have different decisions, they delimit an interval containing a conclusion-changing value, provided the underlying decision quantity behaves appropriately between those values. The table does not, by itself, establish that the change is unique or that it occurs at a particular point within the interval.
The decision quantity depends on the rule, not just on the estimate. For a rule requiring the lower endpoint of a two-sided 95% confidence interval to exceed zero, a convenient margin is \(Q(\delta)=\hat{\theta}(\delta)-1.96SE(\delta)\), and the favorable decision is made when \(Q(\delta)>0\). For a clinical rule requiring an estimated benefit to exceed a prespecified minimum, the margin could instead be \(Q(\delta)=\hat{\theta}(\delta)-\theta_{\min}\). These rules answer different questions and can change at different sensitivity values.
How to Build and Read the Trace
Before calculating a table, specify the estimand, the meaning and units of \(\delta\), the values to examine, and the decision rule. Keep those definitions fixed across the grid. At every value, apply the same analysis procedure, calculate the estimate and its relevant uncertainty if the decision rule uses uncertainty, and record the decision. A table should make clear which column is the estimate, which is the decision quantity, and which is the final decision.
For a delta adjustment that lowers imputed outcomes in the treatment group by \(\delta\) outcome units, suppose a proportion \(r\) of the treatment group has missing outcomes and the control group is unchanged. Under a simple mean-shift setup, the treatment mean decreases by \(r\delta\). The treatment-minus-control effect estimate therefore has the form \(\hat{\theta}(\delta)=\hat{\theta}(0)-r\delta\). This expression assumes the shift is applied to the specified missing treatment outcomes, the estimand and observed data remain fixed, and the control mean is not shifted. In a full analysis, uncertainty may also change with \(\delta\); it must be recalculated if the procedure requires it.
Worked Example: A Confidence-Interval Decision Across Delta Values
Consider hypothetical outcomes measured in points, where higher is better. The treatment-minus-control estimate at \(\delta=0\) is 2.80 points. Thirty percent of treatment outcomes are missing, and the delta convention lowers those imputed outcomes by \(\delta\) points; control outcomes are not shifted. For illustration, suppose \(SE=1.10\) points at every value, and the rule declares evidence of a positive effect only when the lower endpoint of the two-sided 95% confidence interval is above zero. These assumptions fix the standard error and use the normal critical value 1.96; they are simplifying assumptions, not a general property of delta adjustment.
The estimate is \(\hat{\theta}(\delta)=2.80-0.30\delta\), and the decision margin is \(Q(\delta)=\hat{\theta}(\delta)-1.96(1.10)=\hat{\theta}(\delta)-2.156\). A positive margin means the lower confidence limit is above zero.
| \(\delta\) (points) | \(\hat{\theta}(\delta)\) (points) | \(Q(\delta)\) (points) | Decision |
|---|---|---|---|
| 0 | 2.80 | 0.644 | Positive-effect rule met |
| 1 | 2.50 | 0.344 | Positive-effect rule met |
| 2 | 2.20 | 0.044 | Positive-effect rule met |
| 3 | 1.90 | -0.256 | Positive-effect rule not met |
| 4 | 1.60 | -0.556 | Positive-effect rule not met |
For example, at \(\delta=2\), the estimate is \(2.80-0.30(2)=2.20\) points and its lower confidence limit is \(2.20-1.96(1.10)=0.044\) points. At \(\delta=3\), the estimate is \(2.80-0.30(3)=1.90\) points and the lower limit is \(1.90-2.156=-0.256\) points. The decision differs at these adjacent grid values, so the table brackets a change between 2 and 3 points under the stated model and rule. It does not establish the change's exact location or, without the monotonicity evident in this particular fixed-SE linear setup, guarantee there is only one crossing.
What the Table Can Establish
A short derivation clarifies why the decision margin is more informative than the binary decision alone. In the preceding example, the lower confidence limit is \(\hat{\theta}(\delta)-1.96SE\). Substituting the mean-shift expression gives
If \(r>0\) and the standard error is fixed, \(Q(\delta)\) decreases linearly as \(\delta\) increases. Thus it is continuous and strictly decreasing: it cannot cross zero and then cross back. When the table contains a positive margin at one grid value and a negative margin at the next, continuity guarantees a zero between them, and strict decrease guarantees only one. This justification applies to the stated simplified model. If the standard error varies with \(\delta\), or the estimator is not linear in \(\delta\), the margin's continuity and direction must be assessed rather than assumed.
In more general analyses, estimates and standard errors may come from multiple imputation and be recalculated at each parameter value. The table should then report the actual analysis outputs at each grid value; one should not impose a fixed-SE formula merely because it is convenient. A sign change in the margin still supplies a bracket if the margin is continuous, but a single change requires an additional condition, such as monotonicity over that interval. Finite computation can also make an otherwise smooth profile look irregular, so small fluctuations should be checked against the imputation and analysis procedure.
Worked Example: Standard Error Changes Across the Grid
Suppose a hypothetical multiple-imputation analysis estimates a treatment benefit in score points. The prespecified rule is that the lower endpoint of a two-sided 95% normal confidence interval must exceed zero. The delta shift and analysis are held fixed in definition across the grid, but both the estimate and standard error are recalculated at each value. Assume the following outputs are obtained:
| \(\delta\) (points) | \(\hat{\theta}(\delta)\) (points) | \(SE(\delta)\) (points) | \(\hat{\theta}(\delta)-1.96SE(\delta)\) (points) | Decision |
|---|---|---|---|---|
| 0 | 1.60 | 0.50 | 0.620 | Rule met |
| 1 | 1.42 | 0.50 | 0.440 | Rule met |
| 2 | 1.24 | 0.51 | 0.240 | Rule met |
| 3 | 1.05 | 0.52 | 0.0308 | Rule met |
| 4 | 0.86 | 0.53 | -0.179 | Rule not met |
At \(\delta=3\), the margin is \(1.05-1.96(0.52)=1.05-1.0192=0.0308\) points, so the rule is met. At \(\delta=4\), it is \(0.86-1.96(0.53)=0.86-1.0388=-0.1788\) points, which rounds to \(-0.179\), so the rule is not met. The conclusion-changing value is bracketed by the sampled interval from 3 to 4 points, provided the underlying margin is continuous there. Because these grid outputs alone do not prove the margin is monotone between points, they do not prove uniqueness of a crossing.
Worked Example: A Clinical Decision Rule
In a separate hypothetical setting, suppose the treatment effect is measured in symptom-score points, with positive values representing improvement. A clinical decision rule, specified before examining the sensitivity results, requires the estimated benefit to exceed 1.00 point. Suppose the estimate follows \(\hat{\theta}(\delta)=1.80-0.25\delta\) points over the grid. This is a decision about the point estimate relative to a clinical threshold; it is not a confidence-interval or significance rule.
| \(\delta\) (points) | Estimate (points) | Estimate exceeds 1.00 point? |
|---|---|---|
| 0 | 1.80 | Yes |
| 1 | 1.55 | Yes |
| 2 | 1.30 | Yes |
| 3 | 1.05 | Yes |
| 4 | 0.80 | No |
For \(\delta=3\), the estimate is \(1.80-0.25(3)=1.05\) points, which exceeds 1.00. At \(\delta=4\), it is \(1.80-0.25(4)=0.80\) points, which does not. The table therefore places a change in this clinical decision between 3 and 4 points. It would be misleading to describe this as loss of statistical significance: the rule uses the estimate and a clinical threshold, not a standard error or test.
Diagnostics, Comparisons, and Limitations
A trace is only as trustworthy as the calculations and definitions behind it. Check that the same effect contrast, units, analysis population, delta convention and decision rule are used for every row. Verify that the sign of the shift matches its stated meaning: if increasing \(\delta\) makes missing treatment outcomes less favorable, the treatment benefit would ordinarily be expected to move downward when higher outcomes are better, other parts of the analysis held fixed. A trend in the opposite direction is a reason to inspect coding, treatment-minus-control order, and implementation; it is not automatically proof of an error, since the complete estimator may respond in a more complex way.
Also inspect the decision margin, not only the yes/no column. A table with estimates but no standard errors cannot support a confidence-interval decision. Conversely, a confidence interval should not be used to impose an inferential rule if the prespecified question is clinical relevance of the point estimate. Report the grid range and spacing, and state whether no switch was observed. If no adjacent rows differ in decision, the correct statement is that the examined grid contains no observed decision change—not that no tipping point exists outside it.
A table is more auditable than reporting only the first grid value at which a decision changes: it shows the estimates, uncertainty and decision at every examined value. A plot can help reveal patterns across a dense grid, but it does not replace the underlying numerical results or the rule that determines the decision. A single trace also differs from an analysis that examines multiple sensitivity parameters simultaneously; here, only one parameter varies while the other assumptions are held fixed.
The interval between adjacent grid values is limited by the grid resolution and by what is known about behavior between its points. A wide interval may be too imprecise for a useful report. A fine grid can clarify where a transition appears, but cannot repair a misspecified sensitivity model, an unsuitable decision rule, or unstable computation. Finally, the tipping point is conditional on the chosen estimand, delta interpretation, analysis method and rule. It is not a data-determined boundary that applies independently of those choices.
Common Mistake
A common error is to see different decisions at two adjacent grid values and claim that the binary decision function is continuous between them. It is not: a binary label jumps when its threshold is crossed. The relevant continuity assumption concerns the underlying margin, such as the estimate minus a clinical threshold or the lower confidence limit relative to zero. If that margin is continuous and has opposite signs at the endpoints, it must equal zero somewhere between them. To claim a unique change, one must also rule out a crossing and reversal inside the interval, for example by establishing monotonicity. Without that justification, report a bracket for at least one change rather than asserting a unique tipping value.
Check Your Understanding
For each question, decide what the table supports. Solutions follow the questions.
- A confidence-interval rule is based on \(Q(\delta)=\hat{\theta}(\delta)-1.96SE(\delta)\). The margin is 0.12 at \(\delta=5\) and \(-0.08\) at \(\delta=6\). What can be concluded if \(Q\) is continuous between these values?
- A clinical rule requires an estimate above 2.0 units. At adjacent grid values, the estimates are 2.3 and 1.8 units. What is bracketed, and what cannot be claimed from these two rows alone?
- At every examined grid value the decision is “rule met.” What is the appropriate conclusion about a tipping point?
- Why is it not enough to report estimates when the decision rule requires a lower confidence limit above zero?
Solutions. (1) Since the continuous margin has opposite signs at 5 and 6, it equals zero at least once between them, so the interval brackets at least one change. Uniqueness requires an additional condition, such as monotonicity. (2) The estimate-based clinical decision changes between the two sampled values, so they bracket at least one decision change if the estimate is continuous. The rows alone do not identify the exact location or establish uniqueness. (3) No decision change was observed on the examined grid. A change outside the grid, or between grid values where the decision changes and changes back, has not been ruled out without further assumptions. (4) The rule depends on uncertainty as well as the estimate: the lower limit is \(\hat{\theta}(\delta)-1.96SE(\delta)\). Without \(SE(\delta)\), the decision margin and hence the stated rule cannot be evaluated.