Eliciting Delta Values From Clinical Experts

Why Elicit Delta Values?

A delta adjustment represents an assumption about outcomes that are missing, relative to a specified reference model. Observed data can inform that reference model, but they cannot by themselves determine how unobserved outcomes differ from the model’s predictions. Clinicians can contribute informed judgments about those outcomes, provided the judgments are elicited as explicit comparisons rather than as unexplained parameter values.

The task is not to ask, “What delta should we use?” without context. A clinician may reasonably answer that question differently depending on the outcome scale, the patient’s observed history, the reason follow-up ended, the treatment received, and the time point. A useful elicitation therefore describes a concrete patient scenario, states what the reference model predicts for a comparable case, and asks how the missing outcome is expected to differ.

Definition: For this tutorial, use an additive delta on a continuous outcome scale. Let $Y$ be the outcome, with higher values representing better status. Let $m_{\mathrm{ref}}(H)$ be the reference-model conditional mean for observed history $H$. The delta for a specified missing-outcome scenario is $$ \delta(H)=E(Y\mid R=0,H)-m_{\mathrm{ref}}(H), $$ where $R=0$ indicates that the outcome is missing. Thus, $\delta(H)$ is expressed in the same units as $Y$. A negative value means the expert judges the missing outcomes to be worse, on average, than the reference prediction for that scenario.

This definition depends on the chosen reference model and scenario. It does not mean that the expert knows an individual’s unobserved outcome, nor that the judgment has been estimated from the observed data. The expert supplies a substantive assumption about a conditional mean difference. The reference model, missingness pattern, outcome scale, and target population must all be stated before the judgment can be interpreted.

Structure the Question Before Asking It

A well-posed elicitation describes the comparison in clinical terms and in statistical terms. For example, ask about patients with a specified treatment, observed symptom history, and reason for missing a later assessment. Then state the reference prediction for those patients and ask for the expected average outcome difference if their outcomes were observed. Avoid inviting a comparison with an unspecified “typical patient.”

A practical interview can proceed in stages. First, ask the clinician to explain the likely clinical course without showing a proposed delta. Next, provide a concrete reference prediction and ask for a comparative judgment. Finally, ask for a plausible range and the conditions that would move the judgment toward either end. The numerical value should be recorded alongside the scenario and rationale, not separated from them.

Specify the target.
Name the outcome, its units and direction, the assessment time, treatment arm, missingness pattern, and observed history to which the judgment applies.

State the reference comparison.
Give the reference model’s predicted mean for a patient with that same observed history. Clarify that this is a model-based comparator, not a known outcome for the missing patient.

Ask for a concrete difference.
Ask what average outcome the clinician expects among the described patients if their outcomes could be obtained, and how that compares with the reference prediction.

Probe uncertainty and reasons.
Ask for a plausible lower and upper judgment, what clinical features would change the answer, and how confident the expert is. Record the reasoning in the expert’s own terms.

Translate and verify.
Subtract the reference mean from the judged missing-outcome mean. Confirm the sign, units, scenario, and whether the value is conditional or intended as an overall summary.

The questions should distinguish an outcome judgment from a judgment about why data are missing. For instance, “patients stopped attending because they felt worse” is a possible clinical explanation, but it does not yet specify how much worse their outcomes are expected to be than the reference model predicts. Eliciting the expected difference makes the assumption usable and open to review.

Worked Example: Translating a Symptom-Score Judgment

Hypothetical scenario. A trial measures a symptom score from 0 to 100 at week 12, with higher scores indicating better status. Among patients in a specified treatment arm with the stated observed history, the reference model predicts a mean week-12 score of 62. A clinician is asked to judge the average score for comparable patients whose week-12 scores are missing.

The clinician says that, on average, these patients would score about 54 if assessed. The elicited delta is the judged missing-outcome mean minus the reference prediction:

\delta=54-62=-8\text{ score points}.

The negative sign means the clinician expects the missing outcomes to be, on average, 8 points lower than the reference model predicts for this scenario. This is not a claim that every patient would score exactly 8 points below a prediction. It is a conditional mean-shift assumption, stated in symptom-score points.

The comparison is interpretable because the outcome direction, reference prediction, treatment context, and missing-outcome group were specified. If higher scores instead represented worse symptoms, the same clinical judgment would need to be translated with that scale direction in mind; the sign cannot safely be chosen from the phrase “worse outcome” alone.

From Case-Specific Judgments to an Overall Assumption

Experts may find it easier to reason about a small set of clinically distinct scenarios than about one average across all missing outcomes. This is especially important when patients with different observed histories, treatment arms, or reasons for missingness are likely to have different outcomes. In that setting, elicit scenario-specific values first. An overall mean difference can then be derived using the target group’s scenario proportions, if those proportions and the intended estimand are defined.

Suppose scenario $j$ has reference mean $m_{\mathrm{ref},j}$, expert-judged missing-outcome mean $m_{\mathrm{expert},j}$, and delta $\delta_j=m_{\mathrm{expert},j}-m_{\mathrm{ref},j}$. If scenario $j$ has proportion $w_j$ among the target missing outcomes, with $w_j\geq 0$ and $\sum_j w_j=1$, the weighted mean missing outcome is:

\sum_j w_jm_{\mathrm{expert},j} =\sum_j w_j(m_{\mathrm{ref},j}+\delta_j) =\sum_j w_jm_{\mathrm{ref},j}+\sum_j w_j\delta_j.

Therefore, the overall delta relative to the same weighted reference mean is $\sum_j w_j\delta_j$. This calculation is valid only when the scenario definitions and weights correspond to the target group being summarized. It does not justify averaging experts’ answers across incompatible scenarios.

Worked Example: Combining Two Clinical Scenarios

Hypothetical scenario. In a specified treatment arm, an expert considers two groups with missing week-12 function scores. The score is measured in points, with higher values indicating better function. Among the missing outcomes, 60% are expected to have the first observed-history profile and 40% the second.

For the first profile, the reference mean is 70 points and the expert judges the missing-outcome mean to be 64 points. Thus,

\delta_1=64-70=-6\text{ points}.

For the second profile, the reference mean is 50 points and the expert judges the missing-outcome mean to be 47 points. Thus,

\delta_2=47-50=-3\text{ points}.

The weighted reference mean is $0.60(70)+0.40(50)=42+20=62$ points. The weighted expert-judged missing-outcome mean is $0.60(64)+0.40(47)=38.4+18.8=57.2$ points. Hence the overall delta is

57.2-62=-4.8\text{ points},

which agrees with the weighted scenario deltas: $0.60(-6)+0.40(-3)=-3.6-1.2=-4.8$ points. The overall value is meaningful here because the scenario proportions refer to the target missing-outcome group. Applying it to a group with different proportions would generally imply a different overall delta.

Representing Uncertainty and Expert Disagreement

A single elicited value can conceal substantial uncertainty. Ask for a central judgment and a plausible range, then record what the endpoints mean. For example, a range might describe values the expert regards as clinically plausible for the specified scenario, not a confidence interval calculated from a sample. Do not label a judgment range as a statistical confidence interval unless it was constructed using an appropriate statistical procedure.

When experts disagree, retain the individual judgments and their rationales before deciding how to summarize them. One expert may be thinking about patients with early deterioration, while another imagines patients who miss visits for logistical reasons. Their answers may differ because they are answering different conditional questions. Clarifying the scenario can reveal this; averaging the original answers may obscure it. If a combined value is required, state the aggregation rule and its purpose rather than presenting it as an objective consequence of the elicitation.

Worked Example: Recording a Judgment Range

Hypothetical scenario. For patients in one defined missing-outcome scenario, the reference-model mean is 40 points on a scale where higher values are better. A clinician gives a central expected mean of 35 points and says that 32 to 39 points is a plausible range.

The central delta is $35-40=-5$ points. Applying the same subtraction to the endpoints gives a judgment range from $32-40=-8$ to $39-40=-1$ points. Thus, the elicited central value is $-5$ points, with a plausible delta range of $[-8,-1]$ points under this expert’s stated interpretation.

This range records the expert’s uncertainty about the assumption. It is not an interval estimated from observed outcomes, and it does not imply that all values within it are equally likely. The record should also note why the clinician considered the endpoints plausible and whether the range applies to a conditional scenario or a broader patient group.

Assumptions, Diagnostics, and Alternatives

The elicitation is most useful when several conditions are met. The expert understands the outcome scale and the reference comparison; the scenario is clinically recognizable; the expert has relevant knowledge of the patients and care setting; and the question concerns a quantity the expert can reason about. The analyst must also keep the reference model fixed while interpreting the elicited difference. If the model or scenario changes, the same numerical delta may no longer represent the same assumption.

A brief review can expose problems before a value is used. Check whether the expert interpreted the direction of the scale correctly, whether the stated mean is for the missing group rather than for all patients, and whether the reference prediction corresponds to the same history and treatment context. Ask the expert to explain the clinical pathway underlying the judgment. A value with no coherent explanation, a sign that conflicts with the expert’s verbal account, or a very narrow range despite acknowledged uncertainty deserves clarification.

Elicitation complements, rather than replaces, other sources of assumptions. Calibration with external evidence can ground a delta in relevant observed data; expert judgment can help interpret whether those data apply to the present missing-outcome scenario. Neither source makes the unobserved outcomes empirically identified by the study’s observed outcomes. The elicited delta remains a sensitivity assumption to be reported transparently. It is also distinct from asking for a value that marks a clinically important threshold: here the question is how missing outcomes compare with a model-based reference.

The method has limits. Clinicians may be more comfortable describing individual trajectories than averages, may interpret “missing” patients differently, or may be influenced by how the reference prediction is presented. A single expert may not represent the range of clinical knowledge. Scenario detail can improve relevance but make the elicitation burdensome, while overly broad scenarios can hide important differences. The purpose is not to eliminate judgment, but to make its target, reasoning, and uncertainty inspectable.

Common pitfall: “The clinician thinks missing patients do worse, so set delta to $-5$.” This skips the outcome scale, reference mean, patient scenario, and meaning of the number. Instead, ask for an expected missing-outcome mean in a fully described scenario, subtract the matching reference mean, and state the resulting delta with its units and uncertainty.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A clinician judges the missing-outcome mean to be 48 points when the matching reference mean is 55 points, and higher values are better. What is the delta and its interpretation?
Why is “missing patients have worse outcomes” insufficient to specify an additive delta?
Two clinical scenarios have deltas of $-4$ and $-10$ points. Their proportions among the target missing outcomes are 25% and 75%. Find the overall delta relative to the same weighted reference mean.
Is an expert’s plausible range of delta values a statistical confidence interval? Explain.

Solutions. (1) $\delta=48-55=-7$ points. For the specified scenario, the expert expects missing outcomes to average 7 points below the reference prediction. (2) The statement gives neither a numerical difference nor the outcome units, reference prediction, assessment context, or group to which the judgment applies. (3) The weighted delta is $0.25(-4)+0.75(-10)=-1-7.5=-8.5$ points. This uses the stated proportions for the target missing-outcome group. (4) No. It is an expert judgment range unless a statistical procedure has constructed an interval with a defined sampling-based interpretation; its meaning and rationale should be recorded.