Scale Dependence Of Treatment-Effect Robustness

Why the Effect Scale Matters

A sensitivity analysis asks how conclusions depend on assumptions that cannot be verified from the observed data alone. But a conclusion also depends on how treatment effect is expressed. A five-unit difference and a 20% increase describe effects on different scales; one can remain stable across plausible assumptions while the other changes. Neither conclusion is automatically the uniquely correct one. The appropriate scale depends on the outcome and the clinical question.

The key is to keep the estimand, the sensitivity scenarios, and the decision threshold aligned. A change in scale does not change the assumptions used to obtain scenario-specific estimates. It changes the mathematical summary of those estimates and may change what counts as a clinically important effect.

Definition: Let \(s\in\mathcal S\) index a sensitivity scenario, and let \(\mu_1(s)\) and \(\mu_0(s)\) be the treatment and control means for the same outcome, population, and estimand under that scenario. The absolute effect is \(\Delta(s)=\mu_1(s)-\mu_0(s)\). When both means are positive, the relative effect is \(R(s)=\mu_1(s)/\mu_0(s)\), and the log-relative effect is \(\log R(s)\). For a binary success outcome, replace the means with success probabilities \(p_1(s)\) and \(p_0(s)\): the risk difference is \(p_1(s)-p_0(s)\), and the risk ratio is \(p_1(s)/p_0(s)\), when \(p_0(s)>0\).

Here, higher values are assumed to be better. If lower values are better, the direction of benefit must be defined consistently, for example by reversing the contrast or defining an appropriate ratio. Absolute effects retain the outcome's units; ratios are dimensionless. A ratio is meaningful only when the outcome has a meaningful zero and its reference value is positive. A ratio of temperatures in Celsius, for example, is not a useful relative treatment effect.

What Changes—and What Does Not—When the Scale Changes

For positive means, the scales are related by \(\Delta(s)=\mu_0(s)\{R(s)-1\}\). Thus the absolute difference depends on both the relative effect and the control mean. Conversely, \(R(s)=1+\Delta(s)/\mu_0(s)\), so a fixed absolute difference can correspond to different relative effects when the control mean changes.

A useful distinction follows. If the decision is simply whether the treatment effect is positive, then \(\Delta(s)>0\) if and only if \(R(s)>1\), provided \(\mu_0(s)>0\). Likewise, \(\log R(s)>0\) has the same direction. A consistent transformation cannot reverse that sign conclusion. But clinical robustness usually concerns a threshold above zero: for instance, a difference of at least five units or a risk ratio of at least 1.20. Such thresholds do not generally describe the same set of effects.

Key result: A relative threshold \(R(s)\ge r_*\) corresponds to the absolute threshold \(\Delta(s)\ge\mu_0(s)(r_*-1)\). An absolute threshold \(\Delta(s)\ge\delta_*\) corresponds to \(R(s)\ge 1+\delta_*/\mu_0(s)\). If \(\mu_0(s)\) varies across sensitivity scenarios, the equivalent threshold on the other scale varies too.

This result is a direct substitution, not a new assumption about missing outcomes. Starting from \(R(s)=\mu_1(s)/\mu_0(s)\), subtract one and multiply by \(\mu_0(s)>0\):

\mu_0(s)\{R(s)-1\} =\mu_0(s)\left\{\frac{\mu_1(s)}{\mu_0(s)}-1\right\} =\mu_1(s)-\mu_0(s) =\Delta(s).

Therefore, \(R(s)\ge r_*\) is equivalent to \(\Delta(s)\ge\mu_0(s)(r_*-1)\). If the control mean changes with \(s\), holding \(r_*\) fixed changes the absolute benefit required to meet it. Similarly, holding \(\delta_*\) fixed changes the relative benefit required.

Worked Example: A Stable Absolute Effect but a Changing Relative Effect

Suppose a hypothetical trial estimates mean weekly symptom-free days under two plausible sensitivity scenarios. Higher values are better, and both arm means are positive. A clinical panel has specified a minimum worthwhile difference of 4 days. For illustration, also consider a relative benchmark of at least 10% improvement, or \(R\ge1.10\).

Scenario	Control mean (days)	Treatment mean (days)	Absolute effect (days)	Relative effect
A	20	25	5	1.25
B	100	105	5	1.05

In scenario A, \(\Delta=25-20=5\) days and \(R=25/20=1.25\). In scenario B, \(\Delta=105-100=5\) days and \(R=105/100=1.05\). The absolute effect meets the 4-day threshold in both scenarios. The relative effect meets the 1.10 benchmark in A, since \(1.25\ge1.10\), but not in B, since \(1.05<1.10\).

The treatment means in this example are each 5 days above the corresponding control means, but the reference level differs greatly. These arithmetic results do not establish that either scenario is clinically plausible; that judgment belongs to the sensitivity analysis. They show that a stable absolute conclusion need not imply a stable conclusion against a fixed relative benchmark.

Binary Outcomes: Absolute and Relative Benefits

For a binary success outcome, an absolute effect is often directly interpretable as a difference in the probability of success. A relative effect compares the probabilities proportionally. These can support different clinical judgments when the baseline probability changes across assumptions. The same arithmetic applies to risks of an adverse event, but the direction of benefit must then be stated carefully.

Worked Example: A Stable Risk Ratio but Different Risk Differences

Consider a hypothetical binary outcome of recovery by a specified follow-up time. Suppose two sensitivity scenarios give the following success probabilities, and a risk ratio of at least 1.25 is the relative benchmark. Separately, suppose the clinical minimum worthwhile improvement is an absolute increase of 8 percentage points.

Scenario	Control probability	Treatment probability	Risk difference	Risk ratio
A	0.10	0.14	0.04	1.40
B	0.30	0.42	0.12	1.40

For A, the risk difference is \(0.14-0.10=0.04\), or 4 percentage points, and the risk ratio is \(0.14/0.10=1.40\). For B, the difference is \(0.42-0.30=0.12\), or 12 percentage points, and the ratio is \(0.42/0.30=1.40\). Thus both scenarios meet the relative benchmark, but only B meets the 8-percentage-point absolute threshold.

The scenarios imply the same proportional increase but different absolute numbers of additional successes per 100 people: 4 in A and 12 in B. Neither scale should be called robust or non-robust without naming the criterion. The risk ratio is defined here because both control probabilities are positive.

Changing the Scale Consistently

A third issue is whether a scale change itself creates a different conclusion, or whether the thresholds have simply been mismatched. A ratio threshold and its corresponding log-ratio threshold produce the same scenario classification because the logarithm is strictly increasing. The corresponding threshold must be transformed too.

Worked Example: Matching a Ratio Threshold to a Log-Ratio Threshold

Suppose a hypothetical analysis reports treatment-to-control ratios of 1.19, 1.21, and 1.24 across three prespecified scenarios. The criterion is at least a 20% increase, so the ratio threshold is \(R\ge1.20\). On the log scale, the equivalent threshold is \(\log(1.20)\approx0.1823\).

The log ratios are \(\log(1.19)\approx0.1740\), \(\log(1.21)\approx0.1906\), and \(\log(1.24)\approx0.2151\). The first is below 0.1823, while the second and third are above it. The ratio-scale comparison likewise gives \(1.19<1.20\), \(1.21>1.20\), and \(1.24>1.20\). Both scales therefore classify the same two scenarios as meeting the criterion.

It would be incorrect to compare the ratios with 1.20 but compare the log ratios with 0.20: \(\log(1.20)\) is about 0.1823, not 0.20. A threshold of 0.20 on the log scale instead corresponds to a ratio of \(\exp(0.20)\approx1.221\), or about a 22.1% increase.

Assumptions, Diagnostics, and Failure Modes

Scale comparisons do not repair a poorly specified estimand or an implausible sensitivity set. Before interpreting them, verify that each scenario refers to the same outcome, population, follow-up time, and treatment strategies. The treatment and control quantities being contrasted must be defined under the same scenario. The scenarios should be justified by the sensitivity assumptions, not selected because they produce a preferred scale-specific conclusion.

Check the following points when the scale-specific results appear unstable or surprising:

Reference values near zero. If \(\mu_0(s)\) or \(p_0(s)\) is zero or very small, the ratio is undefined or can change sharply with small changes in the estimates. Report the denominator and inspect the range across scenarios.
Outcome support and meaning. A ratio needs a meaningful zero and positive denominator. For a probability, estimates must remain between 0 and 1. A transformed estimate outside that range signals a problem with the model or calculation.
Threshold alignment. Record whether the benchmark is absolute, relative, or log-relative. If translating it to another scale, use the scenario-specific reference value where required.
Direction and units. State which direction is beneficial. Give absolute effects in outcome units or percentage points and ratios as dimensionless quantities. Do not call a 0.04 risk difference a 4% relative increase: it is a 4-percentage-point increase, while the relative increase depends on the reference risk.
Sampling uncertainty as well as assumption uncertainty. Scenario-specific point estimates do not show their sampling precision. When interval-based conclusions are needed, use intervals appropriate to the estimand and scenario; do not infer interval behavior from point estimates alone.

For intervals, a monotone transformation preserves ordering when the entire interval and its threshold are transformed consistently. But a confidence interval for a ratio is not generally obtained by dividing the endpoints of a difference interval by one estimated control mean. The reference mean is uncertain and may be correlated with the effect estimate. Use an interval method for the target parameter, and distinguish its sampling uncertainty from the variation across sensitivity scenarios.

Choosing What to Report

The main alternative is not to declare one scale universally preferable, but to choose the primary scale based on the outcome and decision. An absolute difference often answers how many additional successes, events, or outcome units are associated with treatment. A relative effect often answers how large the change is in proportion to the reference level. Each can be useful, but they answer different questions. Reporting both can make a sensitivity analysis more informative, provided the main clinical criterion is clear and is not changed after inspecting results.

A standardized effect, such as a difference divided by a standard deviation, is another possible summary, but it introduces dependence on the variability used for standardization and may be less directly interpretable clinically. It is not a substitute for stating the absolute outcome units or the relative comparison when those are the relevant decisions.

When reporting robustness, describe the set of scenarios, the effect on the chosen scale, and the threshold used. If conclusions differ by scale, report that directly. For example: “The estimated improvement exceeded the 4-day absolute threshold in every prespecified scenario, whereas the 10% relative benchmark was met in only two.” This is more precise than saying the result is simply robust or not robust.

Common pitfall: “The absolute effect stayed above five units, so the relative conclusion is also robust.” This does not follow: the control value may change across scenarios, and \(R(s)=1+\Delta(s)/\mu_0(s)\). The correction is to calculate the relative effect in each scenario and compare it with a prespecified, meaningful relative threshold. If translating a threshold between scales, use the scenario-specific reference value.

Check Your Understanding

Answer each question, then compare your reasoning with the solution immediately below it.

A hypothetical study has treatment and control success probabilities of 0.24 and 0.20 in one scenario. Find the risk difference and risk ratio. Does it meet an absolute threshold of 5 percentage points?
Solution: The risk difference is \(0.24-0.20=0.04\), or 4 percentage points. The risk ratio is \(0.24/0.20=1.20\). The difference is below 0.05, so it does not meet the specified absolute threshold.
If \(\mu_0(s)=40\) and the relative threshold is \(R(s)\ge1.15\), what absolute difference is equivalent in that scenario?
Solution: The equivalent difference is \(\mu_0(s)(1.15-1)=40(0.15)=6\) outcome units. An absolute effect of at least 6 units is equivalent to the relative threshold for this reference mean.
Why can a fixed 6-unit threshold fail to represent the same relative benefit in two scenarios with different control means?
Solution: The implied relative effect is \(1+6/\mu_0(s)\). If the control mean is 20, the ratio is \(1+6/20=1.30\); if it is 60, the ratio is \(1+6/60=1.10\). The same absolute difference therefore represents different relative increases.
A ratio is 1.30. What is the corresponding log ratio, approximately, and what relative increase does the ratio represent?
Solution: \(\log(1.30)\approx0.2624\). The ratio represents a 30% increase relative to the reference value, since \((1.30-1)\times100\%=30\%\). The log ratio is not itself a 30% increase; it is a different scale for expressing the same ratio.
Across all plausible scenarios, \(\Delta(s)>0\), but \(\Delta(s)\) exceeds the clinical minimum in only some scenarios. What can be concluded about the direction of effect and clinical robustness?
Solution: The estimated effect is positive in every scenario, so its direction is stable across the set. The effect is not robust to the stated clinical minimum, because that threshold is met in only some scenarios. Directional stability and clinical robustness are different conclusions.