Documenting Expert Prior Elicitation

Why Document the Elicitation Process?

A prior intended for regulatory use is not fully explained by naming its distribution and parameters. A reviewer also needs to understand how the judgments behind it were obtained: who contributed, what quantity they judged, what information they saw, how their opinions were combined, and how uncertainty was represented. Without that record, a prior may be difficult to reproduce or assess, even when its mathematical form is clearly stated.

This tutorial focuses on documenting that process. It complements the earlier material on justifying prior distributions: the aim here is to make the expert-elicitation pathway transparent and auditable, not to repeat the general argument for using a particular prior family.

A defensible record makes the target quantity precise. For example, “the response rate” is not enough if it is unclear whether the target is the rate among all eligible patients, a subgroup, or patients treated under a particular regimen and follow-up definition. The record should state the population, treatment, endpoint definition, time point, handling of missing outcomes, and any other features needed to interpret the parameter. If these are not fixed, experts may answer different questions while appearing to disagree about one.

What a Reproducible Record Contains

Documentation should preserve both the planned procedure and what actually happened. A useful record includes the items below, with dates, responsible personnel, and version identifiers where applicable.

• Purpose and target. State the regulatory decision context and define the parameter or uncertain quantity being elicited, including its population and endpoint.
• Expert selection. Record each expert’s relevant experience, how the expert was identified, the basis for inclusion, and any relevant financial, professional, or intellectual conflicts. Explain how conflicts were managed.
• Question wording and materials. Preserve the exact questions, response scales, definitions, time horizons, background material, and supporting data shown to experts. Record the order in which questions and materials were presented.
• Elicitation procedure. Describe whether judgments were obtained individually or in a group, the facilitator’s role, the elicitation method, opportunities for clarification, and whether experts could revise their answers.
• Individual judgments and uncertainty. Retain each expert’s responses, including probability judgments, stated ranges or quantiles, explanations, and uncertainty. Do not report only the final pooled prior.
• Aggregation and transformation. Specify any weights, pooling rule, distributional family, fitting method, rounding, and software or code used to translate judgments into a prior.
• Safeguards and deviations. Record measures used to limit bias, any deviations from the plan, and how those deviations were handled.

The record should distinguish observed facts from judgments. For example, a historical dataset may inform an expert’s opinion, but the elicited probability is still a judgment about the defined target, not a direct measurement of the target parameter. State what evidence was supplied and whether experts were asked to incorporate it. This helps reviewers understand what information is already reflected in the prior and reduces the risk of counting the same evidence twice.

Representing and Combining Uncertainty

Experts should be asked for uncertainty, not just a best guess. Depending on the parameter and elicitation design, this may involve a probability distribution, selected quantiles, probabilities of events, or a stated range with a clear interpretation. A range alone is ambiguous unless the record explains whether it is a plausible range, a confidence-like interval, or a range intended to contain a specified fraction of the expert’s uncertainty.

For a probability parameter \(p\), one possible representation is a Beta prior, \(p\sim\operatorname{Beta}(\alpha,\beta)\), where \(\alpha>0\) and \(\beta>0\). Its mean is \(\alpha/(\alpha+\beta)\), and a common measure of prior concentration is \(\alpha+\beta\). If an elicitation protocol uses an elicited mean \(m\) and an agreed effective sample size \(s_0\), the corresponding parameters are \(\alpha=ms_0\) and \(\beta=(1-m)s_0\). This is a modeling translation, not evidence that the expert literally observed \(s_0\) patients. Document why this representation was chosen and whether its shape adequately captures the elicited uncertainty.

Aggregation: Preserve the Rule and the Disagreement

When several experts contribute, the record must state how their judgments become a prior. A consensus meeting, an unweighted average, a weighted average, and a mixture distribution are different procedures and can produce different uncertainty. State the aggregation rule before applying it when feasible, preserve the individual contributions, and explain the rationale for any weights. A numerical weight is not self-justifying: document its basis and whether it reflects expertise, calibration, or another prespecified criterion.

For a mixture of expert distributions, let expert \(j\) contribute distribution \(F_j\) with weight \(w_j\), where \(w_j\geq 0\) and \(\sum_j w_j=1\). The pooled distribution assigns probability \(w_j\) to drawing from expert \(j\)’s distribution. If the expert-specific means and variances are \(\mu_j\) and \(\sigma_j^2\), the pooled mean is \(\mu=\sum_jw_j\mu_j\). Its variance is \[ \operatorname{Var}(\theta)=\sum_jw_j\{\sigma_j^2+(\mu_j-\mu)^2\}. \] The first component captures within-expert uncertainty; the second captures disagreement between expert means. Omitting the second component understates uncertainty when experts disagree.

To see the decomposition, write the deviation from the pooled mean as \(\theta-\mu=(\theta-\mu_j)+(\mu_j-\mu)\) conditional on expert \(j\). Squaring and taking the conditional expectation gives the within-expert variance plus \((\mu_j-\mu)^2\), because \(E(\theta-\mu_j\mid j)=0\) makes the cross-product expectation zero. Averaging these conditional quantities with weights \(w_j\) gives the stated formula.

Bias Safeguards, Diagnostics, and Limitations

Elicitation can be affected by ambiguous questions, anchoring on a proposed value, overconfidence, group influence, and selective choice of experts. A protocol should use neutral wording, define terms in advance, and avoid presenting a desired answer as the natural starting point. Independent elicitation before group discussion can reduce the chance that an early speaker anchors later judgments. If experts discuss their answers, preserve initial and revised judgments and the reasons for changes. Facilitators should clarify the question without steering experts toward a particular response.

Calibration questions with outcomes known to the elicitation team can provide a limited diagnostic of how experts express uncertainty. The questions should be relevant enough to assess the elicitation process, but they do not prove that an expert’s judgment about the trial parameter is correct. A scoring measure can be reported as a diagnostic; it should not be used mechanically to rank or weight experts without a prespecified rationale and adequate calibration evidence.

Other useful diagnostics include checking whether every expert interpreted the target consistently, whether elicited quantiles are ordered, and whether the fitted distribution reproduces the judgments it was intended to represent. Investigate implausible parameter values, unexplained revisions, or a pooled result that conceals substantial disagreement. Record how discrepancies were resolved rather than silently editing responses.

Elicitation has limitations even when carefully conducted. Experts may share assumptions or evidence, so their judgments are not necessarily independent. A weighted pool can imply precision that the evidence does not warrant, while a consensus value can hide genuine disagreement. Elicited distributions also depend on question wording, selected experts, and the chosen parametric family. A complete record makes these dependencies visible; it cannot eliminate them.

Building the Audit Trail

Store the protocol, question versions, expert responses, meeting records, calculation files, code, and final prior in a controlled location with version history. The final report should identify which version generated the analysis and link each transformation from raw judgments to prior parameters. If the elicitation changes after seeing trial data, document the timing, rationale, authorization, and resulting analyses; otherwise it may be impossible to distinguish planned elicitation from a data-driven revision.