Why Elicitation Needs a Precise Target
In a pediatric trial, clinicians may have relevant experience but little directly applicable randomized evidence. Their judgments can help express what effect sizes are plausible before the trial begins. Turning those judgments into a prior distribution requires more than asking whether treatment is “likely to work”: the effect must be defined, the direction of benefit stated, and uncertainty represented on the same scale as the trial result.
Suppose the endpoint is a symptom score measured after treatment, with lower scores indicating fewer symptoms. Define \(\theta\) as the mean score in the control group minus the mean score in the treatment group. Then positive values of \(\theta\), measured in score points, favor treatment. The estimand might be the treatment-group contrast in mean change from baseline at a specified follow-up time, under a stated analysis population. The clinical team should elicit beliefs about that estimand, not about an undefined notion of “improvement.”
Elicitation is most useful when clinicians are asked for interpretable quantities, such as a median and lower and upper quantiles, in the units of the estimand. They should be told what evidence is relevant, what population and endpoint are under consideration, and whether the question concerns an average effect or a particular subgroup. Their answers then need to be translated into a distribution whose assumptions can be checked and communicated.
From Quantiles to a Normal Prior
A normal prior is a convenient choice when a continuous treatment effect can plausibly take values on either side of zero and the elicited distribution is approximately symmetric. Write the prior as \(\theta\sim N(\mu_0,\sigma_0^2)\), where \(\mu_0\) is the prior mean and \(\sigma_0\) is the prior standard deviation, in treatment-effect units. For a standard normal quantile \(z_a\), the \(a\) quantile of this prior is \(\mu_0+\sigma_0z_a\).
If clinicians supply a lower quantile \(q_L\) at probability \(\gamma\) and an upper quantile \(q_U\) at probability \(1-\gamma\), then fitting those two quantiles gives:
For symmetric probabilities, \(z_\gamma=-z_{1-\gamma}\). The fitted mean is then the midpoint of the two quantiles. However, this does not automatically make the elicited median the fitted median. A normal distribution has its mean and median at the same value, so a supplied median \(m\) is matched along with the two quantiles only if \(q_L+q_U=2m\). When the summaries are not symmetric around \(m\), a normal fit must prioritize some summaries over others, or a distribution that can represent asymmetry must be considered.
Worked Example: Fitting a Symmetric Elicitation
Suppose a clinical panel considers a mean treatment benefit of 4 symptom-score points most plausible. The panel also judges that the 5th and 95th percentiles are approximately 0.7103 and 7.2897 points, respectively. These quantiles are symmetric around 4. Use a normal prior.
For the 5th and 95th percentiles, \(z_{0.05}\approx-1.6449\) and \(z_{0.95}\approx1.6449\). The interval width is \(7.2897-0.7103=6.5794\) points, so:
The midpoint is \((0.7103+7.2897)/2=4.000\) points. Thus the fitted prior is \(\theta\sim N(4,4)\), where 4 is the variance in squared score points. Its median is 4 points, so it matches the panel’s stated median as well as the two quantiles, up to rounding. The prior says that substantial benefit is plausible, but it also assigns probability to harmful values and to values near zero.
Worked Example: Detecting Incompatible Elicited Summaries
A different panel reports a median benefit of 2 points, a 5th percentile of \(-1\) point, and a 95th percentile of 7 points. A normal distribution cannot match all three summaries, because the two quantiles are centered at \((-1+7)/2=3\), not at the reported median of 2.
If the two tail quantiles are prioritized, their implied normal mean is 3 points and the standard deviation is:
The resulting normal prior has median 3 points, not 2. It therefore fits the tail quantiles but not the median. Alternatively, if the median is prioritized as the normal mean and the same standard deviation is used, the fitted 5th and 95th percentiles are approximately \(-2.000\) and \(6.000\) points, not \(-1\) and 7. The analyst should report this mismatch rather than implying that one normal prior exactly represents all the elicited summaries. The panel can revisit the judgments, the analysis can prioritize specified summaries, or an asymmetric distribution can be fitted.
Updating the Prior With Trial Data
For a transparent case study, suppose the trial provides an estimated treatment benefit \(y\) with standard error \(SE\). Use the approximate sampling model \(y\mid\theta\sim N(\theta,SE^2)\). This likelihood is appropriate when the estimate is approximately unbiased for the defined treatment effect and its sampling distribution is adequately approximated by a normal distribution with known standard error. The prior and the trial estimate must refer to the same estimand and units. The analysis also treats the elicited prior as specified before observing these trial data and assumes that the likelihood adequately represents how the estimate arose.
With a normal prior \(\theta\sim N(\mu_0,\sigma_0^2)\) and this normal likelihood, the posterior is normal. Its variance is the reciprocal of the sum of prior and data precisions; its mean is the precision-weighted average of the prior mean and trial estimate.
Here, \(\sigma_1^2\) is posterior variance and \(\mu_1\) is posterior mean. These formulas do not mean that the prior is extra trial data; they describe how the specified prior distribution and likelihood combine under the model.
Why the Normal-Normal Update Works
The posterior density is proportional to the product of the prior density and the likelihood. Ignoring factors that do not depend on \(\theta\), its exponent is:
Expanding the terms that involve \(\theta\), the coefficient of \(\theta^2\) is \(1/\sigma_0^2+1/SE^2\), and the coefficient of \(\theta\) is \(-2(\mu_0/\sigma_0^2+y/SE^2)\). Completing the square therefore gives a normal density with precision \(1/\sigma_0^2+1/SE^2\) and mean equal to that precision’s reciprocal multiplied by \(\mu_0/\sigma_0^2+y/SE^2\). These are exactly \(\sigma_1^2\) and \(\mu_1\) above. This derivation also shows why greater prior or data precision gives greater weight to its corresponding estimate.
Worked Example: Updating With a Pediatric Trial Estimate
Use the symmetric elicitation from the first example, \(\theta\sim N(4,4)\), and suppose a hypothetical randomized trial estimates a treatment benefit of \(y=3.000\) score points with \(SE=2.000\) points. Assume the endpoint, follow-up time, and analysis population match the elicited estimand, and that the normal sampling approximation is suitable.
The prior precision is \(1/4=0.2500\) per squared point, and the data precision is \(1/2^2=0.2500\) per squared point. Thus:
The posterior is \(\theta\mid y\sim N(3.500,2.000)\), with posterior standard deviation \(\sqrt{2}=1.414\) points. Because benefit is defined as positive, the posterior probability of any benefit is:
Thus, under this prior and likelihood, the posterior probability that the mean benefit is greater than zero is about 99.3%. This is a probability conditional on the model, prior, and observed trial estimate; it is not the probability that the model is correct. The posterior mean is 3.500 points, while the posterior standard deviation communicates substantial remaining uncertainty about the effect’s size.
Any Benefit Versus Clinically Meaningful Benefit
A positive effect need not be large enough to matter to patients. Before interpreting posterior probabilities, specify a clinically meaningful threshold \(\delta\) in the same units and beneficial direction as \(\theta\). The relevant probability is then \(P(\theta>\delta\mid y)\). Under the normal posterior, it is calculated by standardizing the threshold:
Worked Example: Probability of Exceeding a Clinical Threshold
Continue with the posterior \(N(3.500,2.000)\) from the hypothetical trial. Suppose clinicians had specified in advance that a mean benefit greater than 1.000 score point would be clinically meaningful. The posterior standard deviation is 1.414 points, so:
The posterior probability of exceeding the stated threshold is about 96.2%, compared with about 99.3% for any benefit. These probabilities answer different questions. A decision rule could use a threshold probability, but the threshold and the consequences of acting or not acting require clinical and regulatory justification; the probability alone does not establish that a treatment should be adopted.
Diagnostics, Alternatives, and Limitations
The arithmetic of a conjugate update is simple; the quality of the answer depends on the elicitation and model. Check that experts understand the endpoint, units, direction of benefit, population, and time point. Compare their stated median and quantiles for symmetry before fitting a normal prior. Ask whether judgments reflect evidence that is already represented elsewhere in the analysis, and document the elicitation process and the summaries the fit prioritizes.
The normal likelihood also deserves scrutiny. A reported standard error may be unreliable with very small samples, sparse outcomes, substantial missingness, or an estimator whose sampling distribution is strongly non-normal. A single common \(\theta\) may not describe variation across ages, disease severity, or settings. If the estimand is bounded or the elicited distribution is markedly asymmetric, a normal prior can assign implausible probability to extreme effects or fail to represent the experts’ judgments. These are reasons to revisit the model and examine how conclusions change under reasonable alternatives, not reasons to hide the assumptions.
Quantile matching is direct and easy to explain when the elicited distribution is approximately symmetric. Other approaches include fitting an asymmetric parametric distribution to elicited quantiles or eliciting probabilities for ranges and constructing a distribution from them. These approaches can represent asymmetry more faithfully but require additional choices and may be harder to communicate. Whatever the approach, describe which judgments determine the distribution and assess whether the treatment conclusion is sensitive to plausible alternatives.
A posterior probability should be reported with its event, direction, units, and conditioning clear. “There is a 96.2% posterior probability that the mean benefit exceeds 1 point under the specified model” is precise. It does not say that 96.2% of children benefit, that the treatment works for every child, or that the assumed prior and likelihood are beyond question.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- A panel gives a median benefit of 5 points and 10th and 90th percentiles of 2 and 8 points. Can a normal prior match all three? If so, find its standard deviation.
- A panel gives 5th and 95th percentiles of \(-2\) and 6 points, and a median of 1 point. What normal mean and standard deviation fit the two quantiles, and what mismatch remains?
- For prior \(\theta\sim N(2,9)\) and trial estimate \(y=4\) with \(SE=3\), calculate the posterior mean, variance, and probability of any benefit.
- For the posterior in question 3, calculate the probability that benefit exceeds 2 points. Explain why this differs from the probability of any benefit.
Solutions. (1) The quantiles are centered at 5, so all three summaries are compatible. With \(z_{0.90}\approx1.2816\), \(\sigma_0=(8-2)/(2\times1.2816)\approx2.340\) points. (2) The quantile midpoint is 2, so the fitted normal mean is 2, not the stated median of 1. Its standard deviation is \(8/3.2898\approx2.432\) points; the fitted median is 2 points, one point above the elicited median. (3) The prior and data precisions are both \(1/9\), giving posterior variance \(1/(2/9)=4.500\) and mean \(4.500(2/9+4/9)=3.000\) points. The posterior standard deviation is \(\sqrt{4.5}\) points, so \(P(\theta>0\mid y)=\Phi(3/\sqrt{4.5})\approx0.9214\). (4) \(P(\theta>2\mid y)=\Phi((3-2)/2.121)=\Phi(0.4714)\approx0.6813\). The threshold of 2 points is more demanding than merely having a positive effect, so its posterior probability is lower.