Sensitivity to Prior Effective Sample Size

Why Prior Effective Sample Size Matters

In a posterior-driven adaptive trial, observed outcomes update beliefs about treatment performance, and those beliefs influence later assignments. The prior therefore matters most when little trial information has accumulated. Its effective sample size controls how strongly early observations can move the posterior and, through the allocation rule, how quickly assignment probabilities can become unequal.

This tutorial studies that sensitivity while holding the prior location and the allocation rule fixed. The aim is not to identify one universally correct prior size. It is to understand the consequences of plausible choices for early allocation, expected patient response, and inferential precision. All numerical data below are hypothetical.

A Conjugate Model and an Allocation Rule

Suppose outcomes on treatment \(i\), for \(i=0,1\), are independent conditional on their treatment means and follow \(Y_{ij}\sim N(\theta_i,\sigma^2)\), with the same known variance on both treatments. Before the trial, use independent priors \(\theta_i\sim N(m_i,\sigma^2/\nu)\). For the sensitivity comparison, hold \(m_0\), \(m_1\), and \(\sigma^2\) fixed as \(\nu\) changes. After \(n_i\) outcomes on treatment \(i\), with sample mean \(\bar{x}_i\), conjugate updating gives

For independent treatment posteriors, the contrast \(\theta_1-\theta_0\) is normal. Its posterior mean is the difference between the two posterior means, and its posterior variance is the sum of their variances. A common posterior-probability allocation rule assigns treatment 1 with probability

where \(0\leq\varepsilon\leq1\) is a fixed exploration fraction. The rule mixes posterior-guided allocation with equal randomization: each treatment has probability at least \(\varepsilon/2\). In the examples, \(\varepsilon=0.10\), so assignments remain between 0.05 and 0.95. This rule is stated explicitly because conclusions about prior size depend on how posterior beliefs are translated into randomization probabilities.

Deriving the Posterior Comparison

For the balanced case \(n_0=n_1=n\), let \(d_0=m_1-m_0\) be the prior mean contrast and \(d=\bar{x}_1-\bar{x}_0\) the observed sample-mean contrast. Subtracting the posterior means gives the posterior contrast mean \(\delta_\nu\); adding the independent posterior variances gives its variance \(V_\nu\):

If \(Z\sim N(0,1)\), then standardizing the posterior contrast yields

where \(\Phi\) is the standard normal cumulative distribution function. Thus changing \(\nu\) changes both the posterior contrast and its standard deviation. Looking only at posterior variance, or only at the posterior mean, is not enough to determine the change in allocation.

Consequences for Expected Patient Response

Allocation sensitivity has a direct one-step patient-benefit interpretation when the true treatment means are specified. Conditional on the current history, if the next patient’s expected outcomes are \(\theta_1\) and \(\theta_0\), the expected response under the allocation rule is \(q_1\theta_1+q_0\theta_0\). This is an expectation under the stated true means, not a guarantee about an individual patient or a complete-trial benefit measure.

Posterior Precision Is Not the Whole Design

In the conjugate model, increasing \(\nu\) reduces posterior variance because prior precision is added to data precision. That reduction is conditional on the assumed prior and likelihood. It should not be confused with a guaranteed improvement in frequentist precision or coverage for the treatment contrast. A posterior can be narrow because the prior is strong, even when the trial itself has collected little information.

How to Assess a Prior-Size Sensitivity Analysis

A useful analysis varies \(\nu\) over a scientifically defensible range, keeps the prior location, outcome model, allocation rule, and other design settings unchanged, and compares the resulting operating characteristics. The update above explains why early assignments may be especially sensitive: when \(n\) is small, the relative weight \(\nu/(\nu+n)\) on the prior is substantial. As data accumulate, the data weight \(n/(\nu+n)\) increases, though the full adaptive path can still differ because early allocations determine which outcomes are observed.

A simulation can summarize that path dependence. For each candidate \(\nu\), simulate many trials under the same specified treatment means and outcome model, applying the corresponding posterior update and allocation rule at every assignment. Compare allocation proportions over time, expected total response, final estimator precision, and inferential error or coverage as relevant to the design. Monte Carlo uncertainty should be reported: apparent differences that are small relative to simulation error are not persuasive evidence of sensitivity.

Useful diagnostics include the following:

• Plot treatment assignment probabilities against time for several \(\nu\) values, not just the final allocation ratio.
• Check whether early observed outcomes repeatedly push probabilities toward the boundary of the rule; early noise can then shape many later assignments.
• Separate posterior interval width from the empirical sampling variability and coverage of the trial’s treatment-effect estimator.
• Verify that the assumed outcome variance and prior calibration are plausible. A sensitivity grid cannot compensate for a seriously misspecified likelihood.

Comparisons, Pitfalls, and Limitations

A fixed equal-allocation design does not use posterior probabilities to change assignments, so it avoids this source of prior-size sensitivity but also forgoes posterior-guided adaptation. A posterior-driven design with a larger \(\nu\) is not simply “more conservative” in every respect: it may reduce early allocation imbalance while also making posterior intervals narrower through prior precision. Comparing only one of these features can give an incomplete account. The appropriate comparison depends on the design’s goals and should include patient outcomes and inferential performance.

The calculations here assume independent normal outcomes with known, common variance; independent normal priors; equal numbers observed in the two arms at the comparison point; and a fixed exploration fraction. If these assumptions do not describe the trial, the displayed formulas do not apply unchanged. In particular, an unknown or unequal outcome variance requires a model that represents that uncertainty. The effective sample size is also a modeling calibration, not a universally objective quantity: different prior families or outcome models can attach different meanings to a chosen numerical value.

• Pitfall: treating a large prior as extra randomized patients. The prior contributes modeled information, but those observations were not enrolled, randomized, or observed in this trial. Report the prior contribution separately from actual sample size.
• Pitfall: inferring benefit from allocation alone. More assignments to a treatment do not prove that treatment is better. State the true means or clearly describe the expectation being computed.
• Pitfall: rounding intermediate probabilities too early. In the allocation rule, retain precision in \(\Phi(\cdot)\) and round the final \(q_1\) and expected response only at the end. Here \(q_1\) for \(\nu=8\) is about 0.655376, not 0.656.
• Pitfall: calling a narrower posterior interval better frequentist precision. Posterior width is conditional on the model and prior; assess repeated-sampling variability and coverage separately when those are design objectives.