Calibrating Skepticism to a Negligible Effect Region

Why Calibrate Skepticism to a Clinical Region?

A skeptical prior for a treatment effect is centered at no difference and gives relatively strong prior support to effects near zero. But a standard deviation by itself can be hard to interpret clinically. Saying that a prior standard deviation is 0.8 points does not explain whether the prior mostly favors effects too small to matter, or whether it gives substantial probability to effects that clinicians would regard as important.

An alternative is to begin with a clinical judgment: effects between \(-\delta\) and \(+\delta\) are negligible for the treatment decision, and the prior probability of an effect lying in that range should be \(q\). Here \(\delta\) is a positive threshold in the treatment effect’s units, and \(q\) is a probability between 0 and 1. This tutorial shows how those two choices determine the scale of a zero-centered normal prior.

Definition: Let \(\theta\) denote a signed treatment effect, with its direction and units specified. A symmetric clinically negligible region with threshold \(\delta>0\) is the interval \([-\delta,\delta]\). A calibrated skeptical normal prior has \(\theta\sim N(0,\sigma^2)\), where \(\sigma>0\) is chosen so that \(P(-\delta\leq\theta\leq\delta)=q\). The quantity \(q\) is the prior probability assigned to the negligible region under this model.

The definition presumes that effects on either side of zero are treated symmetrically for the purpose of this calibration. For example, if \(\theta\) is the treatment-minus-control difference in a score where lower values are better, a negative value favors treatment and a positive value favors control; the interval still treats equal-magnitude effects in either direction as negligible. If the clinical interpretation is not symmetric around zero, this particular calibration does not represent it without modification.

Assumptions and Calibration Formula

The calculation requires a continuous effect parameter whose prior is being modeled as normal on the chosen effect scale. The prior mean is fixed at zero, representing a skeptical center at no treatment difference. The negligible threshold \(\delta\) must be expressed in the same units as \(\theta\), and the elicited probability must satisfy \(0<q<1\). The calibration also assumes that the symmetric interval is the clinical region of interest; it does not determine whether that region is clinically appropriate.

Let \(\Phi\) denote the cumulative distribution function of a standard normal random variable. Because \(\theta/\sigma\) has a standard normal distribution, the probability in the negligible region is

\begin{aligned} P(-\delta\leq\theta\leq\delta) &=P\left(-\frac{\delta}{\sigma}\leq\frac{\theta}{\sigma}\leq\frac{\delta}{\sigma}\right)\\ &=\Phi\left(\frac{\delta}{\sigma}\right)-\Phi\left(-\frac{\delta}{\sigma}\right)\\ &=2\Phi\left(\frac{\delta}{\sigma}\right)-1. \end{aligned}

The final equality uses the symmetry of the standard normal distribution, \(\Phi(-z)=1-\Phi(z)\). Setting this probability equal to \(q\), then isolating \(\sigma\), gives the calibration rule.

\begin{aligned} 2\Phi\left(\frac{\delta}{\sigma}\right)-1&=q,\\ \Phi\left(\frac{\delta}{\sigma}\right)&=\frac{1+q}{2},\\ \frac{\delta}{\sigma}&=\Phi^{-1}\left(\frac{1+q}{2}\right),\\ \sigma&=\frac{\delta}{\Phi^{-1}\left(\frac{1+q}{2}\right)}. \end{aligned}

Calibration result: For a zero-centered normal prior and a symmetric negligible region \([-\delta,\delta]\), the standard deviation that assigns prior probability \(q\) to that region is \(\sigma=\delta/\Phi^{-1}((1+q)/2)\). The inverse normal quantile is positive for \(0<q<1\), so this gives a positive finite standard deviation.

For fixed \(\delta\), a larger \(q\) produces a smaller \(\sigma\): more prior probability concentrated in the same central region requires a narrower prior. For fixed \(q\), doubling \(\delta\) doubles \(\sigma\). These relationships provide immediate checks on a calculation. The formula calibrates probability inside the interval; it does not put a point mass at zero or state that the effect is exactly zero.

Worked Calibrations

Worked Example: Pain-Score Difference

Suppose, hypothetically, that \(\theta\) is the difference in average pain score, measured in points, for a new treatment minus control. Lower scores are better, and clinicians regard differences between \(-1.0\) and \(+1.0\) points as negligible. A prior elicitation chooses \(q=0.80\): before observing trial data, 80% of prior probability should lie in that range. Assume the zero-centered normal prior is suitable on this score-difference scale.

Here \(\delta=1.0\) point. The required standard normal quantile is \(\Phi^{-1}((1+0.80)/2)=\Phi^{-1}(0.90)=1.2816\). Therefore,

\sigma=\frac{1.0\ \text{point}}{1.2816}=0.7803\ \text{points}.

The resulting prior is \(\theta\sim N(0,(0.7803\ \text{points})^2)\). Its central 80% probability lies between \(-1.0\) and \(+1.0\) points. This is a statement about the prior distribution, not about the probability after trial data have been observed. Negative values favor treatment under the stated coding, but the calibration places the same prior probability in equally wide ranges on either side of zero.

Worked Example: Blood-Pressure Difference

In a hypothetical trial, let \(\theta\) be the treatment-minus-control difference in mean systolic blood pressure at a specified follow-up, measured in mmHg. Suppose differences from \(-3\) to \(+3\) mmHg are considered negligible, and prior experts want 90% probability inside that region. Assume a symmetric zero-centered normal prior is a reasonable approximation for this effect.

Set \(\delta=3\) mmHg and \(q=0.90\). The required quantile is \(\Phi^{-1}(0.95)=1.6449\). Substitution gives

\sigma=\frac{3\ \text{mmHg}}{1.6449}=1.824\ \text{mmHg}.

Thus the calibrated prior is \(\theta\sim N(0,(1.824\ \text{mmHg})^2)\). To check the result, \(3/1.824\) is approximately 1.645, and the standard normal probability between \(-1.645\) and \(+1.645\) is approximately 0.90. A mistaken calculation using \(\Phi^{-1}(0.90)\) would use the wrong tail probability: the central 90% interval leaves 5% in each tail, so its positive endpoint corresponds to the 95th percentile.

Worked Example: A Tighter Central Probability

Suppose, hypothetically, that \(\theta\) is a treatment effect measured in days to recovery, with a negative difference favoring treatment. Clinicians define differences between \(-0.5\) and \(+0.5\) days as negligible. They want 95% of prior probability in that interval. Assume the effect can be adequately represented by an unbounded normal distribution for this calibration.

Here \(\delta=0.5\) days and \(q=0.95\). The central 95% interval uses the standard normal quantile \(\Phi^{-1}(0.975)=1.9600\). Hence

\sigma=\frac{0.5\ \text{days}}{1.9600}=0.2551\ \text{days}.

The calibrated prior has standard deviation 0.2551 days, and its central 95% probability is the interval from \(-0.5\) to \(+0.5\) days. Compared with choosing \(q=0.80\) for the same threshold, the 95% choice implies a smaller standard deviation and therefore concentrates more probability near zero.

Checking and Interpreting the Calibration

The arithmetic is short, but the calibration is only as defensible as its inputs and model. Before using the prior in a trial analysis, check the following:

Effect direction: State exactly how \(\theta\) is coded. Reversing treatment and control reverses the sign; the symmetric interval is unchanged, but explanations of which side favors treatment are not.
Units: Use the same unit for \(\delta\), \(\theta\), and \(\sigma\). If \(\delta\) is entered in points while the reported \(\sigma\) is interpreted as a standardized effect, the numerical result will be misinterpreted.
Central probability: Confirm that \((1+q)/2\) is the upper endpoint quantile. A central probability of \(q\) leaves \((1-q)/2\) in each tail.
Clinical threshold: Check that \(\delta\) reflects a defensible judgment about negligible effects for the endpoint and context. The algebra cannot establish that a clinical threshold is appropriate.
Distributional shape and support: Inspect whether a symmetric normal prior is credible on the selected effect scale. A normal distribution has unbounded support, which can be problematic for parameters with strict bounds or strongly asymmetric plausible values.

A useful numerical diagnostic is to substitute the calculated \(\sigma\) back into \(2\Phi(\delta/\sigma)-1\). The result should agree with the chosen \(q\), apart from rounding. A second diagnostic is to compare prior-implied effects with clinically plausible values, including values in the tails. Matching the probability in one interval does not guarantee plausible probabilities everywhere else.

Sensitivity analysis is especially useful when the negligible threshold or elicited probability is uncertain. For example, calculate scales for several defensible values of \(q\), keeping \(\delta\) fixed, and compare the resulting prior distributions. Alternatively, hold \(q\) fixed and assess whether reasonable changes to \(\delta\) alter the prior enough to affect the analysis. These comparisons reveal which elicited judgment is driving the prior’s concentration.

Alternatives and Limits

One alternative is to specify \(\sigma\) directly, perhaps using prior knowledge from earlier studies. That approach can be appropriate when the scale is already interpretable and supported by relevant evidence. Interval calibration instead asks for a probability that clinicians may find easier to discuss: how much prior probability belongs to a specified negligible range? The two approaches are mathematically equivalent when the same normal model is used; each can be converted into the other.

Another equivalent approach specifies a central prior quantile or interval endpoint. For example, saying that the central 90% prior interval should extend to \(\pm\delta\) is exactly the same calibration as setting \(q=0.90\). Interval-based elicitation is not a separate source of information: it organizes a judgment about the spread of the prior in clinically meaningful terms.

The normal form imposes symmetry and a particular tail shape. If plausible effects are asymmetric, if the parameter is constrained, or if a single scale cannot reflect both near-zero and large effects, the calibrated normal prior may be a poor model even when the interval probability is matched exactly. The calculated \(\sigma\) should then be treated as a description of the chosen normal approximation, not as proof that the prior is suitable.

Common Mistakes

Using the wrong normal quantile. The wrong reasoning is to use \(\Phi^{-1}(q)\) for a central interval. The correction is to use \(\Phi^{-1}((1+q)/2)\), because the desired probability is split between two tails.
Calling \(q\) the probability of no effect. The wrong reasoning is that 80% inside \([-\delta,\delta]\) means an 80% probability of exactly zero effect. A continuous normal prior assigns probability zero to the single value \(\theta=0\); \(q\) is the probability of an interval.
Ignoring units or effect coding. A numerical \(\sigma\) has the units of \(\theta\), and the sign convention determines which direction favors treatment. State both before interpreting the prior.
Assuming a matched interval validates the whole prior. Matching \(q\) controls one central probability only. Check the normal shape, support, and remaining tail behavior separately.

Key takeaway: A clinically negligible interval makes the spread of a skeptical normal prior interpretable. For a zero-centered normal prior, choose \(\delta\) and \(q\), then use \(\sigma=\delta/\Phi^{-1}((1+q)/2)\). The clinical judgment and the distributional assumptions still require separate scrutiny.

Check Your Understanding

For each question, identify the central-interval quantile before calculating or interpreting the scale. Solutions follow the questions.

A treatment effect is measured in symptom-score points. If \(\delta=2\) points and \(q=0.80\), what normal-prior standard deviation gives the requested calibration?
For a negligible interval of \([-4,4]\) mmHg, what positive standard normal quantile is used when \(q=0.90\), and why?
If \(\delta\) stays fixed while \(q\) increases from 0.80 to 0.95, does the calibrated \(\sigma\) increase or decrease? Explain.
What does \(q=0.90\) mean for a continuous normal prior, and what does it not mean?
A parameter is constrained to values from 0 to 1. Name one concern with using the calibrated normal prior directly for that parameter.

Solutions. (1) The quantile is \(\Phi^{-1}(0.90)=1.2816\), so \(\sigma=2/1.2816=1.561\) points. (2) Use \(\Phi^{-1}(0.95)=1.6449\), because 90% in the center leaves 5% above the upper endpoint and 5% below the lower endpoint. (3) It decreases: the denominator \(\Phi^{-1}((1+q)/2)\) increases with \(q\), while \(\delta\) is fixed. (4) It means 90% of prior probability lies in the specified interval; it does not mean there is a 90% probability of an exact zero effect. (5) A normal prior has unbounded support and can assign probability to values below 0 or above 1, outside the parameter’s allowed range.