Eliciting Delta Ranges from Subject-Matter Judgment

Why Elicit a Range?

A delta sensitivity analysis asks what conclusions would follow if the missing outcomes differed from their MAR-based imputations by a specified amount. The data generally cannot identify that departure: for people with an unobserved outcome, the outcome needed to compare with its imputed value is unavailable. Subject-matter judgment can therefore help define values of delta worth examining, but it does not turn those values into estimates from the observed data.

The task is not to ask experts to guess every missing outcome. Instead, experts judge a carefully stated contrast: for people with specified observed characteristics, how might their unobserved outcome differ from the outcome predicted under the chosen MAR reference? A defensible range makes that contrast explicit, uses anchors experts can reason about, and records why its endpoints are plausible.

Definition: Let \(X\) denote observed predictors, \(R=1\) indicate that an outcome \(Y\) is observed, and \(R=0\) indicate that it is missing. A conditional delta is a specified departure of the distribution of \(Y\) for \(R=0\) from the MAR reference distribution given \(X\). On an additive mean scale, one possible definition is \(\delta(x)=E(Y\mid X=x,R=0)-m_{\text{MAR}}(x)\), where \(m_{\text{MAR}}(x)\) is the MAR-reference conditional mean. The scale and the sign convention must be stated; delta is not meaningful without them.

For a continuous outcome, an additive delta is expressed in outcome units. For a binary outcome, a probability-scale delta is an absolute risk difference, not a relative percentage change. For a count outcome modeled with a log-rate shift, delta is a log rate ratio. These interpretations differ, so experts should not be asked for “a delta” before the outcome scale and model have been specified. The earlier material on Interpreting the Direction of a Delta Shift provides the relevant sign convention; here the emphasis is on eliciting plausible values and ranges.

State the Reference and Assumptions First

Before elicitation, specify the outcome, its time point, the population being discussed, and the predictors that define the conditional comparison. Also state exactly what the MAR reference predicts. For example, an expert might be asked whether patients with a given recorded baseline severity and treatment assignment could have a 30-day outcome risk above the MAR-predicted risk. “Missing patients do worse” is not yet a usable judgment: it leaves the outcome, comparison group, time period, and meaning of “worse” unspecified.

A constant delta assumes the same shift on the specified scale for every missing case to which it is applied. That may be a useful simplification, but it is an assumption, not something established by expert agreement. If experts expect different departures across treatment arms, patterns, or predictor levels, the elicitation should describe those distinctions explicitly rather than silently averaging them into one number. The elicited range must also be compatible with the imputation implementation and with valid outcome values.

Working assumptions for an elicited range: The MAR imputation model is the chosen reference; experts understand the target population, predictor information, outcome definition, and direction of the contrast; and a proposed delta has a consistent interpretation for the cases to which it will be applied. The resulting range represents plausible sensitivity scenarios conditional on these choices. It is not identified by the observed data, and it is not a confidence interval or a probability distribution for the true delta.

A useful elicitation is structured. Experts first consider a scenario in plain language, then reason from concrete anchors, and finally state a lower and upper plausible value on the named scale. Ask what would make each endpoint plausible and what evidence or clinical mechanism would make values outside it implausible. Separate “plausible” from “possible”: an imaginable extreme need not belong in the main range. If views differ, retain the disagreement or report a reasoned range that covers the distinct defensible judgments; do not conceal it by reporting an unexplained average.

Define the contrast.
Name the outcome, time point, population, predictors, MAR reference, and delta scale. State which direction corresponds to a positive delta.

Give experts a concrete scenario.
Describe the people with missing outcomes and the information available about them. Ask about their outcome relative to the stated MAR prediction, not relative to an unspecified overall average.

Use meaningful anchors.
Ask for a minimally important departure, a central plausible departure, and a departure that would be unusually large but still credible. Convert these judgments to the model’s delta scale.

Check coherence and record rationale.
Check units, direction, outcome bounds, and consistency across experts and subgroups. Record the judgments, assumptions, and reasons for the chosen endpoints.

Why the Conditional Contrast Matters

The basic justification is a conditional-mean decomposition. Let \(f(x\mid R=0)\) be the distribution of observed predictors among cases with missing outcomes. If the missing-outcome conditional mean differs from the MAR reference by \(\delta(x)\), then the mean among those cases is obtained by averaging the reference mean plus that departure over their predictor distribution:

E(Y\mid R=0) =\int E(Y\mid X=x,R=0)f(x\mid R=0)\,dx =\int \{m_{\text{MAR}}(x)+\delta(x)\}f(x\mid R=0)\,dx.

The second equality follows from the definition \(E(Y\mid X=x,R=0)=m_{\text{MAR}}(x)+\delta(x)\), substituted inside the integral. Thus an expert’s judgment should concern the difference from the conditional MAR prediction at relevant values of \(X\), not necessarily the difference between an unadjusted mean for observed cases and an imagined mean for missing cases. Those overall means can differ just because the groups have different observed characteristics.

Worked Example: Eliciting an Additive Range for a Symptom Score

Suppose a hypothetical study records a fatigue score from 0 to 100 at a fixed follow-up visit, with higher values indicating worse fatigue. For a defined group of patients with missing follow-up scores, the MAR model predicts a mean of 48 for 60% of the group and 62 for the remaining 40%. After reviewing a vignette describing this group and the score, a panel judges that missing patients could plausibly average 5 to 11 points worse than their MAR predictions, conditional on the recorded predictors.

The scale is additive, so the elicited delta range is \([5,11]\) score points. The MAR-reference average for the missing group is:

0.60(48)+0.40(62)=28.8+24.8=53.6\text{ points}.

Under a constant additive shift, the range of shifted group means is found by adding each endpoint to 53.6:

53.6+5=58.6,\qquad 53.6+11=64.6\text{ points}.

Thus the elicited scenarios imply means from 58.6 to 64.6 points for this group, subject to the stated model and constant-shift assumption. The range is not a 95% interval and does not assert that the actual mean must lie between these endpoints. The panel’s rationale might be that a 5-point difference is a modest but meaningful worsening in the vignette, while 11 points represents a substantial worsening that remains credible. Because the score is bounded, the analyst should also check that the shifted distribution—not only its mean—can be implemented without generating invalid values.

Worked Examples on Other Scales

Worked Example: Eliciting a Probability-Scale Range

In a hypothetical study, the outcome is 30-day readmission, coded as a binary event. Experts consider patients with missing outcomes in two observed-risk strata. The MAR model predicts risks of 0.10 for 70% of the missing group and 0.30 for 30%. Based on a stated clinical scenario, the experts judge that the missing patients’ readmission risk could be 0.02 to 0.06 higher in absolute probability than the MAR prediction in each stratum.

The probability-scale delta range is \([0.02,0.06]\), or 2 to 6 percentage points. The MAR-reference average risk is:

$$ 0.70(0.10)+0.30(0.30)=0.07+0.09=0.16. $$

Applying the lower endpoint to each stratum gives risks \(0.10+0.02=0.12\) and \(0.30+0.02=0.32\). Their weighted average is \(0.70(0.12)+0.30(0.32)=0.084+0.096=0.180\). At the upper endpoint the stratum risks are \(0.16\) and \(0.36\), and the weighted average is \(0.70(0.16)+0.30(0.36)=0.112+0.108=0.220\). The implied average-risk range is therefore 0.180 to 0.220, or 18.0% to 22.0%.

All these probabilities are between 0 and 1, so the stated shifts are numerically valid for these strata. That check does not guarantee validity for every possible predictor profile: if some MAR-predicted risks were already near 1, adding 0.06 could exceed the probability range. The analyst would need to address that incompatibility in the model rather than treat invalid probabilities as acceptable.

Worked Example: Eliciting a Log-Rate Range

Consider a hypothetical count outcome modeled through its event rate. Experts judge that, for the specified missing-outcome group, a plausible rate could be 0.80 to 1.25 times the MAR-predicted rate. This judgment is naturally expressed as a log-rate delta. If the MAR rate is 2.00 events per person-year, the endpoint rates are \(2.00(0.80)=1.60\) and \(2.00(1.25)=2.50\) events per person-year.

The corresponding log-rate shifts are:

\delta_{\mathrm{low}}=\log(0.80)=-0.2231,\qquad \delta_{\mathrm{high}}=\log(1.25)=0.2231.

This follows because a log-rate shift \(\delta\) multiplies the reference rate by \(\exp(\delta)\). Here \(\exp(-0.2231)\approx0.80\) and \(\exp(0.2231)\approx1.25\). The elicited range is therefore approximately \([-0.2231,0.2231]\) on the log-rate scale, corresponding to rates from 1.60 to 2.50 events per person-year for the reference rate of 2.00. The rate units are unchanged by the multiplicative shift. The experts’ judgment concerns a plausible relative departure; the logarithms simply express that departure on the model scale.

Evaluating the Range and Avoiding False Precision

Elicitation should be treated as a disciplined sensitivity-analysis input, not as a substitute for information the data do not contain. A narrow range may reflect strong, well-grounded expert judgment, but it may also reflect overconfidence or a poorly specified vignette. A wide range may be honest about uncertainty, or it may combine incompatible populations and assumptions. Ask experts to explain why the endpoints are credible and whether their judgments apply to all missing cases or only a defined pattern or subgroup.

Several checks help identify a range that is going wrong:

Check the reference. Confirm that experts are comparing against the MAR prediction conditional on the stated predictors, rather than against the raw mean among observed cases.
Check direction and units. A positive shift should mean the same thing throughout the elicitation. Record whether the shift is in score points, absolute probability, or log rate.
Check boundaries. For bounded scores and probabilities, verify that the implied imputed values remain valid. For rates, verify that the multiplicative interpretation is understood.
Check heterogeneity. Ask whether the same range is credible across treatment arms, missingness patterns, or predictor levels. If not, state where the judgments differ rather than assuming one universal shift.
Check disagreement and rationale. Preserve meaningful expert disagreement and document how endpoint judgments were combined. A point estimate that hides disagreement is not more objective.

A common mistake is to ask experts for the difference between all observed outcomes and all missing outcomes. The missing outcomes are unobserved, and the crude observed-group mean may also reflect differences in predictors. The correction is to provide a defined conditional MAR prediction and ask for a plausible departure from that reference among the specified cases. Another mistake is to describe the selected range as “the likely interval for delta.” Unless a probability model for expert uncertainty has been specified, the endpoints are judgments about plausible scenarios, not probabilistic coverage limits.

Subject-matter elicitation is one way to define sensitivity scenarios. A single delta is simpler to communicate but can conceal uncertainty; a range exposes how much the conclusions may depend on the judgment. A mechanistic scenario can make the direction and scale more concrete, while a delta range makes the departure directly usable in the imputation model. If suitable external information exists, it may inform the judgment, but converting such information into delta values is a separate calibration task. In all cases, the elicited range remains conditional on the reference model, target population, and assumptions stated at the outset.

Key takeaway: A defensible elicited delta range begins with an explicit conditional contrast on a named scale. Its endpoints need concrete anchors, a documented rationale, and checks for heterogeneity and valid outcomes. Expert judgment defines plausible scenarios; it does not identify the unobserved departures from the data.

Check Your Understanding

Answer each question, then compare with the short solutions.

A hypothetical continuous outcome has an MAR-predicted mean of 40 units for a group with missing outcomes. Experts judge a plausible additive delta range of 3 to 8 units. What range of shifted means follows under a constant shift?
For a binary outcome, experts describe a delta of 0.05 on the probability scale. State its interpretation, and identify one necessary validity check.
Why is a crude difference between the observed-outcome mean and an imagined missing-outcome mean generally not the conditional delta experts should be asked to judge?
Is an expert-elicited range automatically a confidence interval for the true delta? Explain.

Solutions. (1) Add each endpoint to 40: \(40+3=43\) and \(40+8=48\), so the shifted means range from 43 to 48 units. (2) It is an absolute increase of 0.05, or 5 percentage points, in event probability relative to the MAR prediction; check that shifted probabilities remain between 0 and 1 for relevant predictor profiles. (3) The groups can have different distributions of observed predictors, so their raw means need not represent a conditional departure from the MAR prediction. Experts should judge the contrast for a defined group or predictor profile. (4) No. It is a range of plausible scenarios based on judgment, not a confidence interval with a stated coverage probability.