Eliciting Expert Judgments With Probability Questions

Why the Wording of a Probability Question Matters

An expert’s probability judgment is useful only when it is clear what the probability refers to. Asking, “How likely is a high event rate?” leaves several questions unanswered: high compared with what threshold, in which patients, for which event, and over what period? If experts answer different versions of that question, their numerical answers are not directly comparable.

In Bayesian analysis, an expert may be asked to express uncertainty about an unknown clinical parameter. The parameter represents a feature of a defined clinical setting, such as a 30-day event risk or a mean change on a specified scale. The expert’s answer is a probability assigned to a precisely defined event involving that parameter. It is not necessarily a probability that a particular patient will experience an outcome.

The target parameter is treated as a fixed but unknown quantity for the specified setting. The expert’s probability represents uncertainty about that quantity, given the information available to the expert. In contrast, a patient-level probability describes the chance of an outcome for a patient or group under a specified model. Keeping these two meanings separate prevents a common interpretation error.

Specify the Parameter Before Asking About It

A well-formed question starts by defining the target, rather than by choosing a convenient threshold. In a clinical trial context, that usually means stating who is in the target population, what outcome is being summarized, how it is measured, and when it is assessed. If the parameter depends on a treatment group or a comparison, those conditions must also be stated. The clinical definition should be specific enough that two readers would identify the same parameter.

For a probability parameter, such as \(p\), a question might ask for \(P(p>c)\), where \(c\) is a clinically interpretable threshold. For a continuous parameter, such as a mean \(\mu\), the question might ask for \(P(\mu<c)\) or for the probability that \(\mu\) lies in a specified interval. In each case, the expert is assigning probability to a statement about the parameter—not estimating the proportion of individual patients whose outcomes exceed \(c\).

A useful specification typically makes the following points explicit:

• Population: Which patients does the parameter describe? State eligibility or other clinically relevant defining characteristics.
• Outcome: What event, measurement, or summary is being considered? For an event risk, define what counts as an event.
• Time: Give a time horizon or assessment time when the outcome changes over time.
• Conditions: State the treatment, care setting, or other conditions to which the parameter refers.
• Scale and threshold: Give units and define any cutoff, including whether equality is included.

These details are not administrative decoration. For example, “risk by day 30” differs from “risk during an average 30-day period” if follow-up, censoring, or the endpoint definition differs. Similarly, a mean change needs a stated scale, direction of improvement, and assessment time. A question that omits these choices may elicit an answer, but the meaning of that answer remains uncertain.

Separate Parameter Uncertainty From Patient Variability

Clinical experts often speak naturally in terms of patient outcomes: “I think about one in ten patients will have the event.” That statement may concern the event risk itself, a forecast of observed trial data, or a probability about the unknown value of a population parameter. These are not interchangeable.

Let \(Y\) denote a patient’s binary outcome and \(p\) the population risk in a defined setting. A statement about \(P(Y=1)\) describes the event probability for a patient drawn from that setting, under the model and conditions specified. An elicitation question such as \(P(p>0.08)\), by contrast, concerns uncertainty about the value of \(p\). It asks whether the underlying risk is above a threshold, not whether an individual patient will experience the event.

The distinction matters when the expert’s knowledge is limited. Even if the expert’s best estimate of a risk is 8%, the expert may be quite uncertain whether the true risk is above or below 8%. A probability question makes the uncertainty explicit. The question should therefore use wording such as “the probability that the population risk is greater than 8%,” not simply “the probability of stroke is greater than 8%.”

Use Thresholds and Related Questions Coherently

One probability question can be useful, but a small set of related questions often reveals whether the expert has understood the target and whether the answers fit together. Probability answers must obey basic probability rules. In particular, an event and its complement have probabilities that sum to one. Also, probabilities assigned to disjoint events that exhaust all possibilities must sum to one.

For a parameter \(p\) and cutoff \(c\), the events \(p\leq c\) and \(p>c\) are complements. Therefore:

For ordered cutoffs \(a<b\), the events \(p<a\), \(a\leq p\leq b\), and \(p>b\) are mutually exclusive and exhaustive. Consequently:

This is a short but important coherence check. If an expert assigns probabilities to all three events, their sum should be 1, apart from any small discrepancy caused by rounding. The boundary conventions matter: specifying which interval includes equality makes the events genuinely disjoint. If a parameter is continuous, equality at a fixed cutoff often has probability zero under a continuous distribution, but that should not be used to excuse unclear wording.

A Practical Elicitation Sequence

A disciplined sequence helps keep questions clear without leading the expert toward a particular answer. First, describe the target population and clinical setting in neutral terms. Next, define the parameter and its units. Then state the event involving that parameter, including the cutoff and boundary convention. Finally, ask the expert to report a probability on a 0-to-1 scale or as a percentage, and invite them to explain what information or interpretation informed the judgment.

The restatement is especially useful when the question contains technical terms. If an expert interprets a question about a population mean as a question about the percentage of patients who improve, the elicitation should pause. Rephrase the parameter and event before recording a numerical answer.

Diagnostics, Alternatives, and Limitations

Several warning signs indicate that a probability question may not be working as intended. An expert may ask what population is meant, use different time horizons in successive answers, switch between patient-level and parameter-level language, or give answers to related events that fail the complement or partition checks. A response such as “about 10%” is also ambiguous unless it is clear whether 10% is a risk estimate, a probability assigned to an event about a parameter, or a proportion of patients.

When a response is unclear, do not infer the intended meaning from the number alone. Repeat the definition in plain language, identify the distinction the expert is making, and ask for a revised probability if needed. Record the exact question and any clarification so that the judgment can be interpreted later. If the expert cannot assign a probability to a precisely stated event, that is useful information: it may indicate that the parameter or target is unfamiliar or that the available evidence does not support a confident judgment.

Probability questions are one way to elicit beliefs. They are particularly useful when an expert can judge whether a parameter is above or below a clinically understandable threshold. A direct point estimate is simpler to ask for, but it does not by itself reveal uncertainty. Asking for quantiles or ranges is another alternative, but it requires the expert to interpret different kinds of statements. These approaches can be complementary; none removes the need to define the target parameter carefully.

Probability questions also have limits. A highly precise question can still be difficult if the expert lacks relevant knowledge or if the target population is poorly defined. A coherent set of answers is necessary but does not establish that the judgments are well calibrated or supported by adequate evidence. Nor does a number become meaningful merely because it is recorded precisely. The clinical definitions and the expert’s interpretation remain essential.

Common Mistakes

• Leaving the event vague. “The risk is high” has no defined cutoff. State the threshold and whether equality is included.
• Omitting the time horizon. A risk without a follow-up period may refer to different quantities for different experts. Give the assessment time or period.
• Confusing patient outcomes with parameter uncertainty. \(P(p>c)\) is a probability about the unknown population risk; it is not the probability that an individual patient experiences the event.
• Changing the target mid-question. The population, treatment condition, outcome definition, and scale must remain fixed across related questions.
• Treating a coherence check as a validation check. Probabilities that sum correctly are mathematically consistent, but that alone does not show that the expert’s judgments are accurate.