Why Elicitation Needs Diagnostics
An expert may have deep clinical knowledge and still give a judgment affected by how a question is asked, what number was mentioned first, or how uncertainty was discussed. In prior elicitation, the goal is not to label an expert as biased. It is to identify judgments that may depend on irrelevant cues, express more certainty than the evidence supports, or fail to fit together logically. Structured follow-up questions and independent elicitation rounds help reveal these problems while preserving legitimate differences of opinion.
The diagnostics in this tutorial apply to elicited probabilities, quantiles, ranges, and treatment-effect judgments. A warning sign is a reason to clarify and investigate, not an automatic reason to discard a judgment. In particular, a narrow range alone does not prove overconfidence, and disagreement between experts alone does not prove anchoring.
Use Independent Judgments Before Discussion
A useful first safeguard is to collect each expert’s initial judgment privately, before sharing another person’s estimate or a group summary. Record the question wording, any values shown, the order of questions, and the expert’s reasoning. Then use a separate round of structured follow-up questions. Experts can revise their judgments, but both the initial and revised responses should remain documented.
The sequence matters. A group discussion can supply relevant information, but it can also make a confident or senior participant’s estimate an anchor for everyone else. Private estimates before discussion preserve a record of independent starting points. In a later round, experts can be asked what evidence changed their judgment and whether the change reflects new information, a clarified interpretation, or reconsideration of their own uncertainty.
Ask the target question without displaying a suggested numerical answer or other experts’ responses. Record the response and the information available to the expert.
Ask for a central estimate and uncertainty judgments, then ask what evidence or assumptions determine them. Use neutral prompts rather than proposing a preferred answer.
Compare related answers for logical compatibility. Where feasible, vary the order or reference information across experts to see whether judgments shift with presentation.
Re-ask the target question privately, without showing a group estimate. Ask experts to explain any revision and retain both rounds for review.
The questions should be neutral and specific. For example: “What evidence would make a risk above 20% plausible?” is more informative than “Are you sure the risk is not too high?” A follow-up can also ask the expert to consider a different plausible scenario, explain the endpoints of a range, or state what new evidence would change the estimate. These prompts diagnose reasoning; they should not steer the expert toward the elicitor’s preferred answer.
Detecting Possible Anchoring
An anchoring check is strongest when the reference value is deliberately varied across otherwise comparable elicitation conditions. If experts are randomly assigned different initial values, and their later judgments systematically differ by assignment, that pattern supports concern that the reference influenced responses. It does not by itself establish that any particular expert’s judgment is wrong. The reference could also have changed how experts understood the question, so the wording and information provided must be held constant.
In clinical work, it is generally inappropriate to present an arbitrary number merely to test its influence and then treat the resulting estimate as an unaltered judgment. A practical alternative is to avoid suggested values in the first private round, document any unavoidable numerical cues, and compare responses across question orders or elicitation conditions when doing so is justified. If an expert’s estimate moves toward a number introduced during discussion, ask whether that number conveyed relevant evidence or merely became a reference point.
Worked Example: Comparing Judgments After Different Numerical Cues
In a hypothetical elicitation exercise, 16 clinicians estimate the control-arm event risk for a specified trial population. Before giving an estimate, eight are shown a low numerical reference of 10% and eight are shown a high reference of 30%. Assignment is random; the clinical description and question are otherwise identical. The initial estimates, in percentage points, are:
Low-reference group: 14, 16, 17, 18, 19, 20, 21, 23. Their mean is \(148/8=18.5\%\). High-reference group: 20, 22, 23, 24, 25, 27, 28, 31. Their mean is \(200/8=25.0\%\).
The difference in group means is \(25.0-18.5=6.5\) percentage points. Because assignment was randomized and the question was held constant, this difference is a warning that the displayed reference may have influenced the initial judgments. It is not proof that all 6.5 points are an anchoring effect, particularly with only eight clinicians in each group.
In a later private round, no group estimates are displayed and the reference values are not repeated. The low-reference group’s estimates have mean \(172/8=21.5\%\), and the high-reference group’s estimates have mean \(176/8=22.0\%\). The smaller difference is compatible with reduced influence of the original cue, but it does not prove that influence has disappeared: experts may remember the first round, and other explanations for the change remain possible. The initial and follow-up estimates should both be reported, with the elicitation conditions described.
Assessing Excessive Precision
Precision describes the spread of an expert’s stated uncertainty; overconfidence is a claim about whether that uncertainty is too small. A narrow uncertainty range may be defensible when relevant evidence is strong and the question is well defined. Conversely, a broad range can still be poorly calibrated. Therefore, do not diagnose overconfidence solely by comparing an expert’s range with another expert’s range or with a group average.
A more direct diagnostic uses calibration questions with outcomes that can later be resolved. An expert might provide a central 90% interval for each of several comparable, objectively verifiable questions. If the intervals are calibrated for that class of questions, approximately 90% of the resolved values should fall inside them over many such questions. One outcome outside an interval is expected sometimes; a pattern of frequent misses may indicate that the stated intervals are too narrow. Calibration depends on questions being relevant to the target judgment and on recording intervals before outcomes are known.
Worked Example: Checking Coverage of 90% Intervals
Suppose an expert gives a central 90% interval for each of 10 hypothetical historical calibration questions. After resolution, 4 of the 10 values fall outside their stated intervals. If the intervals were calibrated and the questions had independent outcomes with 90% coverage, the number \(X\) of misses would follow \(X\sim\mathrm{Binomial}(10,0.10)\). The probability of four or more misses under that model is:
For example, the \(k=4\) term is \({10\choose4}(0.10)^4(0.90)^6=0.01116\); adding the terms for \(k=5,\ldots,10\) gives approximately 0.0128. This is a relatively unusual outcome if the calibration model holds, so it warrants review of the expert’s interval practice. It does not establish that the expert is generally overconfident: ten questions provide limited evidence, and independence, comparable difficulty, and the 90% coverage assumption may not hold. The appropriate conclusion is to investigate and, if possible, gather more relevant calibration evidence.
When objective calibration questions are unavailable, use careful follow-up rather than claiming a calibration result. Ask what evidence supports the endpoints, which assumptions would move them, and whether the expert has considered plausible contrary scenarios. A range that changes dramatically after a neutral prompt may signal that the original precision was not well grounded; it may also mean that the prompt supplied genuinely useful structure. Record the explanation instead of treating the change alone as proof.
Checking Internal Consistency
Some coherence checks follow directly from probability rules. For a quantity \(R\), its cumulative distribution function \(F(r)=P(R\leq r)\) cannot decrease as \(r\) increases. Likewise, probabilities for mutually exclusive and exhaustive categories must be nonnegative and sum to 1. A median and stated quantiles must be ordered consistently. These checks do not require experts to agree with one another; they ask whether one expert’s answers can describe a single distribution.
Worked Example: Resolving Contradictory Risk Probabilities
An expert gives judgments about the control-arm event risk \(R\): \(P(R<0.15)=0.70\) and \(P(R<0.25)=0.60\). These answers cannot both be correct. Since the event \(\{R<0.15\}\) is contained in \(\{R<0.25\}\), probability monotonicity requires:
The reported values instead imply \(0.70>0.60\). A neutral follow-up asks the expert to restate each probability and explain the event meant by “below,” checking that the thresholds and question wording were not reversed. The expert then confirms a median of 0.20 and revises the two probabilities to \(P(R<0.15)=0.35\) and \(P(R<0.25)=0.70\). These values are ordered correctly and are compatible with a median of 0.20: probability below 0.15 is less than one half, while probability below 0.25 is greater than one half. They do not uniquely determine the distribution, and the elicitor should retain the reason for the correction rather than infer more than the answers establish.
Not every apparent conflict is a true contradiction. Experts may be answering different interpretations of an ambiguous endpoint, population, time horizon, or treatment contrast. Before changing a judgment, clarify the target quantity and the conditions to which it refers. Also distinguish a probability statement from a point estimate: a median of 0.20 does not imply that the expert assigns probability 1 to risk being exactly 0.20.
Comparison, Failure Modes, and Limitations
| Approach | What it helps reveal | Main limitation |
|---|---|---|
| Private first round followed by a private follow-up | Whether judgments change after reflection or discussion | Change can reflect learning, memory, or a changed interpretation, not only bias |
| Randomized reference or question-order comparison | Whether estimates vary systematically with presentation | Small groups and differences in interpretation can make patterns unstable |
| Calibration questions with resolved outcomes | Whether stated uncertainty ranges achieve their claimed coverage | Results may not transfer to a different subject or question type |
| Logical coherence checks | Whether judgments obey basic probability and ordering rules | Coherence alone does not show that the judgment is accurate |
Several failures can make these diagnostics misleading. Repeatedly asking leading questions can replace one anchor with another. Showing experts a group average before recording private estimates removes the ability to observe an independent starting point. Treating every revision as correction can erase legitimate learning. And forcing inconsistent answers into a single distribution without clarifying the intended events can conceal an ambiguity rather than resolve it.
The methods also have limits. Anchoring is difficult to separate from relevant information carried by a reference, especially when the reference is part of normal clinical discussion. Calibration requires enough suitable questions and resolved outcomes; coverage from a small set is noisy. Coherence checks identify logical conflicts, not the “right” clinical judgment. Expert estimates may reasonably remain different after careful elicitation. The introductory material on eliciting expert judgments with probability questions, clinical scenarios, quantiles, and ranges provides the basic elicitation forms; the diagnostics here concern how to scrutinize and document the resulting judgments.
A Short Justification for Randomized Anchor Checks
The logic of a randomized comparison is that, before the reference is shown, assignment makes the two groups comparable in expectation. Let \(Y_i(L)\) and \(Y_i(H)\) denote clinician \(i\)’s estimate under low- and high-reference conditions. Random assignment allows the difference between group means to estimate the average effect of the displayed reference, provided the question and other conditions are the same and one clinician’s assignment does not affect another’s response. The estimate can still be imprecise in a small exercise.
In the worked example, \(\widehat{\Delta}=25.0-18.5=6.5\) percentage points. Random assignment supports interpreting this contrast as evidence about the effect of presentation in that exercise, not as proof that a particular expert’s true belief should equal either group mean. If assignment is not randomized, group differences may also reflect who received each condition, so the anchor interpretation is less secure.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A group receives a suggested event risk before estimating it; another group estimates privately without a number. What additional design feature would make a difference in their estimates more informative about anchoring?
- An expert’s central 90% intervals include 18 of 20 resolved values. Does this alone show overconfidence? Explain.
- An expert reports \(P(R<0.10)=0.40\) and \(P(R<0.20)=0.65\). Is this pair coherent with probability monotonicity?
- Why should an elicitation team retain both initial and revised estimates rather than recording only the final value?
Solutions. (1) Randomly assign comparable experts to the two conditions while holding the question and other information constant; this reduces confounding by group composition. (2) No. The observed coverage is \(18/20=0.90\), but 20 questions provide limited evidence, and the questions’ comparability and dependence matter. (3) Yes. Because \(0.10<0.20\), the first event is contained in the second, and \(0.40\leq0.65\). (4) The record shows how judgments changed and allows reviewers to distinguish an independent starting point from a revision prompted by new evidence, discussion, or question clarification.