Prior Probability of Clinically Meaningful Benefit

From a Clinical Threshold to a Prior Probability

A prior for a treatment effect is easier to discuss when its implications are expressed in clinical terms. Rather than asking an expert to name a mean and standard deviation, one can ask how plausible it is that the treatment effect exceeds a prespecified threshold for a clinically meaningful benefit. The calibration task is to translate that probability judgment into parameters of a chosen prior distribution.

The direction and units of the effect must be fixed first. In this tutorial, let \(\theta\) denote the true treatment effect on a scale where larger values indicate greater benefit. For example, \(\theta\) could be the reduction in symptom score, in points, relative to control; then positive values favor treatment. Let \(\delta>0\) be the threshold for a clinically meaningful benefit, expressed in the same units. The target judgment is \(P(\theta>\delta)=q\).

Definition: The prior probability of clinically meaningful benefit at threshold \(\delta\) is \(P(\theta>\delta)\), where \(\theta\) is the treatment effect before observing the trial data and larger values of \(\theta\) favor treatment. It is a probability under the prior model, not a probability calculated from the trial results.

This probability depends on the effect scale, the sign convention, and the clinical threshold. If a lower outcome value is better, define \(\theta\) so that benefit still corresponds to larger \(\theta\), or reverse the inequality consistently. A threshold stated in score points cannot be used directly with a treatment effect measured in standardized units.

Calibration Within a Normal Family

For a transparent calibration, suppose the prior for the treatment effect is normal, \(\theta\sim N(\mu,\sigma^2)\), with \(\sigma>0\). Here \(\mu\) is the prior mean effect and \(\sigma\) is the prior standard deviation, both in the units of \(\theta\). Let \(\Phi\) be the cumulative distribution function of a standard normal random variable. Standardization gives the relationship between the target probability and the prior parameters.

P(\theta>\delta) =1-\Phi\!\left(\frac{\delta-\mu}{\sigma}\right) =\Phi\!\left(\frac{\mu-\delta}{\sigma}\right).

If \(\sigma\) is specified independently, the equation can be inverted to obtain \(\mu\). Write \(\Phi^{-1}(q)\) for the \(q\) quantile of the standard normal distribution. Then the target probability is attained by setting the prior mean as follows.

\mu=\delta+\sigma\Phi^{-1}(q).

This is a one-equation calibration: a single probability target does not determine both \(\mu\) and \(\sigma\). The spread must be specified from other information, or a second probability judgment at another threshold must be elicited. Treating one target probability as if it uniquely identified both parameters is a common calibration error.

Worked Example: Setting the Mean for a Target Benefit Probability

Suppose a hypothetical trial studies a symptom score, and define \(\theta\) as the treatment-related reduction in score, in points, so that positive values favor treatment. A reduction of 5 points is considered clinically meaningful. The elicited target is \(P(\theta>5)=0.80\). For this illustration, suppose the prior standard deviation is set to 4 points based on separate judgments about the plausible range of treatment effects. Find the prior mean under a normal prior.

Use \(\delta=5\) points, \(q=0.80\), and \(\sigma=4\) points. The standard normal quantile is \(\Phi^{-1}(0.80)\approx0.841621\). Substitution gives

\mu=5+4(0.841621)=8.366484\text{ points}\approx8.3665\text{ points}.

The implied probability can be checked using the rounded mean: \((5-8.3665)/4\approx-0.841625\), so \(P(\theta>5)=1-\Phi(-0.841625)\approx0.8000\). Thus the prior places about 80% probability on a benefit exceeding 5 points. The prior mean is about 8.37 points, not 5 points; the threshold is below the mean because the target probability is greater than one half.

Using Two Threshold Probabilities

If both the prior mean and spread are unknown, two probability judgments can identify them within the normal family. Let \(\delta_1<\delta_2\), and suppose the target probabilities are \(q_1=P(\theta>\delta_1)\) and \(q_2=P(\theta>\delta_2)\). Define \(z_1=\Phi^{-1}(q_1)\) and \(z_2=\Phi^{-1}(q_2)\). Each target implies \(\mu-\delta_j=\sigma z_j\). Subtracting the equations eliminates \(\mu\), giving the spread; substituting back gives the mean.

\sigma=\frac{\delta_2-\delta_1}{z_1-z_2}, \qquad \mu=\delta_1+\sigma z_1.

A valid solution with \(\sigma>0\) requires \(q_1>q_2\): exceeding the lower threshold must be at least as probable as exceeding the higher one. When the probabilities are strictly between zero and one and \(q_1>q_2\), the normal quantiles satisfy \(z_1>z_2\), so the fitted spread is positive. If the stated probabilities reverse this ordering, they cannot describe any distribution for which “exceeding” is nested in this way, normal or otherwise.

Calibration check: For thresholds \(\delta_1<\delta_2\), any coherent prior must satisfy \(P(\theta>\delta_1)\ge P(\theta>\delta_2)\). Under the normal calibration, the two fitted probabilities should reproduce the original targets, up to rounding.

Worked Example: Fitting Two Target Probabilities

In a hypothetical study, let \(\theta\) be treatment benefit in score points, with larger values indicating greater benefit. Suppose clinicians judge that \(P(\theta>2)=0.80\) and \(P(\theta>3)=0.10\). Fit a normal prior that matches both judgments.

The standard normal quantiles are \(z_1=\Phi^{-1}(0.80)\approx0.841621\) and \(z_2=\Phi^{-1}(0.10)\approx-1.281552\). The higher threshold has the lower exceedance probability, as required. With \(\delta_1=2\) and \(\delta_2=3\), the prior standard deviation is

\sigma=\frac{3-2}{0.841621-(-1.281552)} =\frac{1}{2.123173} \approx0.470993\text{ points}.

The mean is

\mu=2+(0.470993)(0.841621) \approx2.3964\text{ points}.

Check the first target using the displayed rounded parameters: \((2-2.3964)/0.470993\approx-0.8416\), so \(P(\theta>2)\approx0.8000\). For the second, \((3-2.3964)/0.470993\approx1.2816\), so \(P(\theta>3)=1-\Phi(1.2816)\approx0.1000\). The prior matches both stated probabilities to the displayed precision.

Interpreting and Assessing the Calibration

The parameter values are a mathematical encoding of the elicited judgments, not an independent source of clinical evidence. In the two-threshold example, the fitted mean is near 2.40 points and the spread is about 0.471 points because the probability falls from 0.80 to 0.10 over a one-point increase in threshold. A rapid drop across nearby thresholds implies a relatively concentrated prior. If the same probabilities were asserted across thresholds far apart, the fitted spread would be larger.

A third probability judgment is useful as a diagnostic even though two targets determine the two normal parameters. Calculate the probability at that additional threshold from the fitted prior and compare it with the elicited value. A substantial discrepancy signals that the normal shape may not represent the full set of beliefs, that one judgment may have been misunderstood, or that the judgments need further discussion. Do not silently alter the fitted parameters to force a third target: a two-parameter normal prior generally cannot match three arbitrary probability constraints exactly.

Worked Example: Checking a Second Clinical Probability

Consider a hypothetical trial of a treatment that reduces a symptom score. Let \(\theta\) be the reduction in points, with positive values favoring treatment. Experts judge \(P(\theta>0)=0.8413\) and \(P(\theta>4)=0.50\). Find the normal prior parameters and interpret the two targets.

The corresponding standard normal quantiles are approximately \(z_1=1\) and \(z_2=0\). Thus, for thresholds \(\delta_1=0\) and \(\delta_2=4\),

\sigma=\frac{4-0}{1-0}=4\text{ points}, \qquad \mu=0+(4)(1)=4\text{ points}.

The prior is \(\theta\sim N(4,4^2)\), with mean 4 points and standard deviation 4 points (variance 16 points squared). Checking, \(P(\theta>0)=\Phi((4-0)/4)=\Phi(1)\approx0.8413\), while \(P(\theta>4)=\Phi((4-4)/4)=\Phi(0)=0.50\). In context, the prior assigns about 84.13% probability to any positive score reduction and 50% probability to a reduction greater than 4 points. These are prior probabilities, not conclusions about a completed trial.

The assumptions behind this calibration should be made explicit. The effect is represented on a meaningful, fixed scale; the sign convention is consistent; the clinical threshold is agreed before calibration; and a normal prior is considered an adequate family for the beliefs being represented. In addition, the elicited probabilities must refer to the same parameter and information state. A clinician answering about an effect in a selected patient subgroup is not necessarily answering about the overall treatment effect.

The normal family is convenient, but it may be unsuitable when beliefs are markedly asymmetric, have heavy tails, or put special weight on a point null. The earlier material on skeptical priors, enthusiastic priors, and spike-and-slab priors addresses different prior structures. For the present task, the relevant question is whether a normal distribution can faithfully represent the elicited exceedance probabilities, not whether it is always the best prior family.

Common Mistakes and Alternatives

Confusing prior probability with posterior probability. \(P(\theta>\delta)\) describes belief before observing trial data under the prior. It is not the updated probability of benefit after seeing trial results.
Ignoring effect direction or units. If larger values mean worse outcomes, then \(P(\theta>\delta)\) may describe harm rather than benefit. Define the effect so the inequality matches the clinical question, and keep the threshold in the same units.
Trying to fit two parameters from one target. One exceedance probability supplies one equation. Fix \(\sigma\) using an independently justified judgment or obtain another threshold probability.
Using incompatible targets. If a higher threshold is assigned a higher exceedance probability, the targets violate the nesting of events. Clarify the judgments rather than forcing a negative standard deviation or changing the inequality.
Assuming a fitted normal prior matches all beliefs. Two targets determine a normal prior, but other elicited probabilities may not agree with it. Check additional thresholds and reconsider the distributional family if discrepancies are important.

An alternative is to elicit several quantiles or a plausible range and assess a distribution against those statements; another is to consider a different prior family when the normal shape is inadequate. These approaches can reveal more about tail behavior, but they do not remove the need to define the treatment-effect scale and clinical meaning of the threshold. A calibrated probability is useful only when the judgment it encodes is clear and defensible.

Key takeaway: A target probability \(P(\theta>\delta)=q\) determines the normal-prior mean only after the spread is fixed: \(\mu=\delta+\sigma\Phi^{-1}(q)\). Two coherent threshold-probability judgments can determine both parameters, but the fitted prior should be checked against clinical context and additional judgments.

Check Your Understanding

Answer each question, then compare with the solutions below.

A prior assigns 70% probability to \(\theta>6\), where larger values indicate greater benefit. If \(\sigma=2\), what mean gives this target? Use \(\Phi^{-1}(0.70)\approx0.5244\).
Why can one target probability \(P(\theta>\delta)=q\) not identify both the mean and standard deviation of a normal prior?
For thresholds \(\delta_1=1\) and \(\delta_2=5\), suppose \(P(\theta>\delta_1)=0.90\) and \(P(\theta>\delta_2)=0.50\). Find the normal prior parameters, using \(\Phi^{-1}(0.90)\approx1.2816\) and \(\Phi^{-1}(0.50)=0\).
Can a prior assign \(P(\theta>2)=0.30\) and \(P(\theta>4)=0.60\)? Explain.
What does a prior probability of 0.80 that the treatment effect exceeds the meaningful threshold say, and what does it not say?

Solutions. (1) \(\mu=6+2(0.5244)=7.0488\), or about 7.049 effect units. (2) The target supplies one equation involving two unknowns, \(\mu\) and \(\sigma\), so an additional specification is needed. (3) Here \(z_1=1.2816\), \(z_2=0\), so \(\sigma=(5-1)/(1.2816-0)\approx3.121\) units and \(\mu=1+(3.121)(1.2816)\approx5.000\) units. (4) No. Since \(2<4\), the event \(\theta>4\) is contained within \(\theta>2\), so its probability cannot be larger. (5) It says that, under the specified prior, 80% of probability is above the threshold; it does not say that the posterior probability after trial data is 0.80 or that treatment has been shown effective.