Prior Sensitivity Analysis for Bayesian Designs

Why Examine Prior Sensitivity?

A Bayesian design ranks candidate sampling schedules by averaging their performance over a prior distribution for the unknown model parameters. The resulting ranking can be useful, but it is conditional on the prior used. If plausible prior assumptions place more probability on different parameter values, the preferred schedule may change. Prior sensitivity analysis makes this dependence visible before a schedule is selected.

The question is not whether a prior is “correct” in an absolute sense. Rather, it is whether the design recommendation remains similar across a defensible range of prior locations, scales, and dependence structures. A design that wins under all such alternatives has a different evidential basis from one that wins only under a narrow prior specification.

Let \(d\) denote a candidate sampling design, \(\theta\) the unknown model parameters, \(U(d,\theta)\) the utility of design \(d\) at parameter value \(\theta\), and \(\pi(\theta)\) a prior density. The Bayesian expected utility is

For a discrete prior with parameter scenarios \(\theta_j\) and probabilities \(w_j\), the same quantity is \(\operatorname{EU}(d;\pi)=\sum_j w_j U(d,\theta_j)\), where \(w_j\geq 0\) and \(\sum_j w_j=1\). The preferred design maximizes expected utility when larger utility is better. In a sensitivity analysis, the candidate designs, utility definition, model, and constraints should be held fixed while the prior is varied, unless the purpose is explicitly to examine several changes together.

A Controlled Comparison

A useful analysis changes one prior feature at a time where possible. To examine location, shift the prior center while retaining its scale and dependence structure. To examine scale, widen or narrow the prior while keeping its center fixed. To examine correlation, vary the joint dependence while preserving the marginal distributions. These controlled comparisons help identify which assumption is responsible for a change in ranking.

Prior alternatives should be scientifically defensible, not selected merely to make a preferred design win. Plausible parameter ranges, previous knowledge, and elicited uncertainty can inform the alternatives. The earlier material on selecting plausible parameter ranges provides a basis for choosing such alternatives; the sensitivity analysis then asks how the design behaves across them.

For two designs \(d_A\) and \(d_B\), define their expected-utility difference under prior \(\pi_k\) as \(\Delta_k=\operatorname{EU}(d_A;\pi_k)-\operatorname{EU}(d_B;\pi_k)\). A positive value favors \(d_A\), a negative value favors \(d_B\), and a value near zero indicates little separation under that prior. The sign and magnitude of this difference are more informative than a rank label alone.

Scale and Correlation Sensitivity

A prior’s scale represents the degree of uncertainty about parameter values, not simply the location of its center. If a design’s utility rises or falls sharply as a parameter moves away from the center, widening the prior can change its average utility substantially. Correlation matters when joint parameter combinations affect utility: priors with identical marginal distributions can assign different probabilities to combinations such as high clearance with high volume.

Calculation, Diagnostics, and Failure Modes

For complex models, expected utilities are often computed by numerical integration or Monte Carlo sampling. If \(\theta^{(r)}\) are draws from prior \(\pi_k\), an estimator is \(\widehat{\operatorname{EU}}(d;\pi_k)=R^{-1}\sum_{r=1}^R U(d,\theta^{(r)})\). To compare designs reliably, use the same parameter draws for every candidate schedule under a given prior. This paired calculation makes the estimated utility difference less noisy because variation shared across designs can cancel.

When comparing several priors, record their location, scale, and correlation assumptions explicitly. Recompute the expected utility for every candidate design under every prior, then report the utility differences and rankings. A table is often useful: it shows both ranking changes and near-ties that a rank-only summary would conceal.

Several diagnostics help identify a misleading result. First, check that prior weights sum to one, that covariance matrices are valid, and that parameter scales use the intended units or parameterization. Second, inspect whether a small utility difference is being treated as a decisive ranking when numerical uncertainty is of comparable size. Third, confirm that the same utility definition and candidate set were used throughout. If a comparison changes both prior scale and utility criterion, it cannot isolate the effect of prior scale.

A particularly common mistake is to vary prior correlation while accidentally changing the marginal distributions as well. The resulting ranking change cannot then be attributed to correlation alone. The correction is to construct alternatives with the same marginal distributions and different valid dependence structures, or to state clearly that the sensitivity comparison changes more than one feature.

How to Interpret the Results

Prior sensitivity analysis complements rather than replaces other design comparisons. A single-prior Bayesian expected-utility analysis answers which candidate has the highest average utility under that specified prior. Scenario-based robust design compares performance across selected parameter scenarios, while prior sensitivity asks whether the Bayesian ranking persists when the prior itself is changed. These approaches answer related but distinct questions.

A stable ranking does not prove that the prior assumptions are correct, nor does it guarantee good performance under every possible parameter value. It indicates only that the ranking was stable over the alternatives examined and for the chosen utility. Conversely, a ranking reversal does not automatically invalidate Bayesian design: it signals that the recommendation depends on assumptions that should be made transparent and, where possible, investigated further.

Report the actual prior alternatives and utility differences, not just “robust” or “sensitive.” Mention the parameterization, units, and what was held fixed. For example, a conclusion might state that schedule \(A\) ranked first under prior centers from one range, while schedule \(B\) ranked first under centers from another, with the differences measured in a specified information-score unit. Avoid implying that the probability a design is best is given by its rank under a prior; that is a different quantity and requires a separate probabilistic definition.

The main limitation is that sensitivity analysis explores only the alternatives selected. It cannot protect against implausible or omitted prior structures, misspecified utility functions, or errors in the model used to calculate utility. It can also be computationally expensive when each utility evaluation involves a complex model. These limitations make transparent prior specification and numerical checks essential.