Sensitivity of Neyman Targets to Variance Misspecification

Why Variance Misspecification Matters

A Neyman allocation assigns more participants to the arm with the larger outcome standard deviation, because that arm needs more observations to contribute comparable precision to a difference in means. In practice, the standard deviations used to set the target may come from a pilot study or other preliminary information. If their relative sizes are wrong, the target shifts. The key question is not only how far the target moves, but how much precision that shift costs when the true variances differ from the assumed ones.

This tutorial studies that sensitivity for a two-arm contrast, with equal per-participant costs and a fixed total sample size. The calculations describe the variance of a difference in sample means under a fixed allocation. They are a benchmark for a target; they do not by themselves specify a response-adaptive randomization procedure or guarantee that a trial will realize the target exactly.

The variance formula assumes independent observations within and between arms, finite arm-specific variances, and positive sample sizes. The contrast is a difference of sample means, so its variance is the sum of the two sample-mean variances. For the target derived here, costs per participant are equal and the total \(N\) is fixed. If the variances are not known, replacing them with estimates creates a plug-in target; the earlier material on Plug-In Neyman Targets From Pilot Variances introduces that construction. Here the focus is what happens when those estimates are inaccurate.

How the Target Responds to Assumed Variances

Write \(\hat{\sigma}_1^2\) and \(\hat{\sigma}_0^2\) for the variances assumed when choosing the target. These might be pilot sample variances \(s_1^2\) and \(s_0^2\), or other planning values. The plug-in share is \( \hat{q}=\sqrt{\hat{\sigma}_1^2}/(\sqrt{\hat{\sigma}_1^2}+\sqrt{\hat{\sigma}_0^2}) \). It is the ratio of assumed standard deviations, not the ratio of assumed variances, that directly determines the allocation ratio: \( \hat{q}/(1-\hat{q})=\sqrt{\hat{\sigma}_1^2/\hat{\sigma}_0^2} \).

Consequently, multiplying both assumed variances by the same positive constant leaves the target unchanged. What matters is their relative misspecification. If the assumed variance in arm 1 is too large relative to the assumed variance in arm 0, the target assigns an excessive share to arm 1. If the relative error goes in the other direction, the target shifts toward arm 0.

Deriving the Variance Penalty

Let \(q=n_1/N\), so \(n_0=N(1-q)\), with \(0<q<1\). Write \(a=\sigma_1\) and \(b=\sigma_0\) for the true standard deviations. The true contrast variance at share \(q\) is \(V(q)=N^{-1}(a^2/q+b^2/(1-q))\). The correctly specified continuous target is \(q^*=a/(a+b)\), at which the minimum variance is \((a+b)^2/N\).

The following algebra gives the excess variance directly. It also shows why small target errors are often less costly than large ones, but does not make a poor allocation costless.

This identity is exact for the continuous fixed-allocation variance model. The relative excess depends on the discrepancy between the chosen and optimal shares, scaled by \(q(1-q)\). It is zero at the correct target and positive for every other interior share. It is not a probability statement and does not include other sources of uncertainty, such as uncertainty in estimating the treatment effect.

Worked Examples and Sensitivity Calculations

Diagnostics, Comparisons, and Limitations

A practical sensitivity analysis should vary the relative assumed variances, not just add or subtract the same amount from both. One simple approach is to choose a set of plausible assumed standard-deviation ratios, convert each to a target share, and calculate the true or scenario-specific variance at each share. If the plausible ratios produce widely separated shares or appreciably different variances, the allocation target is sensitive to the variance inputs. When the true variances are unknown, this is a scenario analysis, not proof that any particular scenario is correct.

Pilot estimates can be unstable, especially with few observations, and the uncertainty in \(s_1^2\) and \(s_0^2\) can make their ratio unstable as well. Inspect the arm-specific sample sizes and variability, check that variance estimates use comparable outcome definitions and scales, and report the assumed values alongside the resulting target. A suspiciously extreme ratio, or a target that changes substantially under modest plausible perturbations, is a warning to avoid presenting the plug-in allocation as precisely determined.

These comparisons apply to an unweighted difference in two arm means under equal costs and a fixed total sample size. They do not account for unequal costs, additional constraints, changing variances over time, or other objectives for allocation. Nor does the continuous formula eliminate integer rounding: when the total sample size is small, calculate the variance for feasible integer allocations before interpreting a continuous-target comparison. The calculations also presume that the variance relevant to the planned contrast is represented by the stated arm variances.

Common Pitfall