Distinguishing Bias From Variance In Design Comparisons

Why Separate Bias From Variance?

A design comparison can sound straightforward: one procedure produces estimates that vary less from trial to trial, so it appears more precise. But precision is not the only feature that matters. If an estimator systematically misses the treatment effect, its low variability may come at the cost of bias. A fair comparison therefore asks how far estimates tend to be from the same target, taking account of both systematic error and random variation.

This distinction is especially relevant when treatment assignments depend on accumulating outcomes. As the earlier material on outcome-dependent sampling and estimator bias explains, a difference in observed group means need not estimate the intended treatment contrast without bias under such sampling. The goal here is to compare properties of estimators, not to assume that any one design is always more or less accurate.

Definition: Let \(\theta\) be the treatment-effect quantity a trial aims to estimate, and let \(\hat{\theta}\) be an estimator produced by a specified trial procedure. Its bias is \(E(\hat{\theta})-\theta\). Its variance is \(E\{[\hat{\theta}-E(\hat{\theta})]^2\}\). Its mean squared error (MSE) is \(E\{(\hat{\theta}-\theta)^2\}\). These expectations are taken over repeated trials generated by the stated model and design.

Bias and variance are properties of an estimator under a particular model and trial procedure, not labels attached permanently to an estimator formula. If the outcomes, assignment mechanism, or target changes, the properties can change too. Bias has the same units as \(\theta\); variance and MSE have squared units.

The Bias–Variance–MSE Identity

The key relationship follows by writing the estimation error as a random deviation around its mean plus the estimator's average offset from the target. Let \(b=E(\hat{\theta})-\theta\). Then

\hat{\theta}-\theta = \{\hat{\theta}-E(\hat{\theta})\}+b.

Squaring and taking expectations gives

\begin{aligned} E\{(\hat{\theta}-\theta)^2\} &=E\{[\hat{\theta}-E(\hat{\theta})]^2\} +2bE\{\hat{\theta}-E(\hat{\theta})\}+b^2\\ &=\operatorname{Var}(\hat{\theta})+b^2, \end{aligned}

because \(E\{\hat{\theta}-E(\hat{\theta})\}=0\). Thus MSE is the estimator's variance plus its squared bias. The identity requires a finite second moment; it does not require normal outcomes.

Key result: \(\operatorname{MSE}(\hat{\theta})=\operatorname{Var}(\hat{\theta})+\{\operatorname{Bias}(\hat{\theta})\}^2\). For estimators of the same target under the same trial conditions, MSE summarizes average squared estimation error. A smaller variance alone does not imply a smaller MSE.

A Fixed-Allocation Benchmark

Consider a two-treatment trial in which each participant receives treatment A or B, and the target is the difference in average outcomes, \(\theta=\mu_A-\mu_B\). A simple fixed-allocation estimator is the difference in sample means, \(\hat{\theta}=\bar{x}_A-\bar{x}_B\). For the benchmark calculation below, assume participants' outcomes are independent, outcomes within each treatment group have a common finite variance \(\sigma^2\), and the numbers \(n_A\) and \(n_B\) assigned to the two treatments are fixed. The group sample means then have expectations \(\mu_A\) and \(\mu_B\), respectively, and are independent. Consequently,

E(\hat{\theta})=\mu_A-\mu_B=\theta,\qquad \operatorname{Var}(\hat{\theta})=\sigma^2\left(\frac{1}{n_A}+\frac{1}{n_B}\right).

The estimator is unbiased under these assumptions, so its MSE equals its variance. The formula is a model-based benchmark, not a universal variance formula for every trial. In particular, a response-adaptive procedure requires expectations under its own allocation mechanism; the fixed-allocation calculation cannot simply be carried over.

Worked Example: Fixed Allocation and the Three Quantities

Hypothetical setup. Suppose the true mean outcome is 12 points under A and 8 points under B, so the target difference is \(\theta=12-8=4\) points. A fixed-allocation trial assigns \(n_A=n_B=20\) participants. Assume independent outcomes with common standard deviation \(\sigma=5\) points in each group. The assumptions match the benchmark above.

The difference in sample means has expected value \(E(\hat{\theta})=4\) points. Its bias is therefore \(4-4=0\) points. Its variance is

\operatorname{Var}(\hat{\theta}) =25\left(\frac{1}{20}+\frac{1}{20}\right) =25(0.10) =2.50\ \text{points}^2.

Because the bias is zero, \(\operatorname{MSE}(\hat{\theta})=2.50+0^2=2.50\ \text{points}^2\). The repeated-trial standard deviation of the estimator is \(\sqrt{2.50}=1.581\) points. This standard deviation describes spread around the estimator's expectation; it is not the bias and is not itself an estimate of the treatment effect.

Comparing Repeated-Trial Distributions

For a response-adaptive design, the full repeated-trial distribution can reflect both outcome variation and the fact that later assignments depend on earlier outcomes. One way to compare procedures is to calculate each estimator's expectation, variance, and MSE under the same specified trial conditions. The following discrete distributions are deliberately hypothetical: each row gives an estimator value and its probability over repetitions of the entire stated procedure. They are not observations from a real study.

Worked Example: A Biased Adaptive-Procedure Estimator

Hypothetical setup. Let the target remain \(\theta=4\) points. Suppose repeated fixed-randomization trials produce estimates 0, 4, and 8 points with probabilities 0.25, 0.50, and 0.25. Suppose an outcome-dependent allocation procedure, paired with its unadjusted difference-in-means estimator, produces estimates 2, 4, and 6 points with probabilities 0.20, 0.50, and 0.30. Treat these probabilities as the stipulated repeated-trial laws for this example.

For the fixed procedure, the expected estimate is \(0(0.25)+4(0.50)+8(0.25)=4\) points. Its bias is 0 points, and its variance is

(0-4)^2(0.25)+(4-4)^2(0.50)+(8-4)^2(0.25) =8.00\ \text{points}^2.

Its MSE is \(8.00+0^2=8.00\ \text{points}^2\). For the adaptive-procedure estimator, the expectation is \(2(0.20)+4(0.50)+6(0.30)=4.20\) points, so its bias is \(4.20-4=0.20\) points. Its variance is

(2-4.20)^2(0.20)+(4-4.20)^2(0.50)+(6-4.20)^2(0.30) =1.96\ \text{points}^2.

The adaptive estimator's MSE is \(1.96+(0.20)^2=2.00\ \text{points}^2\), lower than the fixed estimator's MSE of \(8.00\ \text{points}^2\). In this stipulated example, the adaptive estimator is biased but has lower MSE because its variance is much lower. This calculation does not establish that response-adaptive randomization generally lowers MSE; another outcome model or allocation rule could give a different result.

Worked Example: An Unbiased Estimator Can Have Higher MSE

Hypothetical setup. Keep the target at 4 points and compare the adaptive-procedure estimator above with a second candidate estimator whose repeated-trial values are 1, 4, and 7 points, each with probability \(1/3\). Assume this distribution is the estimator's full repeated-trial distribution under the same hypothetical trial conditions.

Its expected value is \((1+4+7)/3=4\) points, so its bias is 0. Its variance, and hence its MSE, is

\frac{(1-4)^2+(4-4)^2+(7-4)^2}{3} =\frac{9+0+9}{3} =6.00\ \text{points}^2.

The adaptive estimator in the preceding example has MSE \(2.00\ \text{points}^2\), despite its nonzero bias of \(0.20\) points. The unbiased candidate has MSE \(6.00\ \text{points}^2\). Thus, unbiasedness is valuable but does not by itself guarantee the smallest average squared error. Any real comparison would need a justified distribution for each estimator, rather than an assumed table of values.

How to Make a Design Comparison

A meaningful comparison starts by fixing the estimand and the trial conditions. If one procedure estimates a treatment difference while another estimates a different population quantity, their biases and MSEs are not directly comparable. Likewise, comparison requires a clearly specified estimator for each design: a randomization procedure alone does not determine which estimator is used.

For response-adaptive assignment, expectations should account for the full procedure, including outcome-dependent assignment. A variance calculated as though the final group sizes had been fixed in advance may not describe that procedure. If properties are evaluated by simulation, each simulated trial should reproduce the assignment rule, outcome generation, and estimator being compared. The simulated average estimate approximates \(E(\hat{\theta})\); the average squared deviation from \(\theta\) approximates MSE. These are estimates of operating characteristics, and finite simulation runs introduce Monte Carlo error.

Specify the target.
Write down the treatment-effect quantity \(\theta\) and its units. Use the same target for every design in the comparison.

Specify the repeated-trial setup.
State the outcome assumptions, the allocation rule, and how outcomes and assignments vary across repetitions.

Specify each estimator.
Do not attribute properties to a design without naming the estimate computed from its trial data.

Calculate and interpret.
Find bias, variance, and MSE on their appropriate scales. Report both the bias in the effect's units and the MSE in squared units.

Diagnostics, Pitfalls, and Limitations

A useful diagnostic is to inspect the average estimate as well as its spread across repeated trials. A narrow distribution centered away from the target signals bias; a broad distribution centered at the target signals variance. A single realized trial cannot reveal either property on its own. When using simulation, compare the simulated mean estimate with \(\theta\), and calculate squared errors relative to \(\theta\), not relative to the simulated mean.

The main alternatives answer different questions. Bias alone describes systematic offset; variance alone describes spread around the estimator's own mean; MSE combines both as average squared error for the chosen target. If unbiasedness is required for a scientific or regulatory reason, it remains an important constraint even when a biased estimator has smaller MSE. MSE also assigns a squared-error loss: a different loss function or a different target could lead to a different design ranking.

Mistake: “The adaptive estimator has less variance, so it is more accurate.” Lower variance only means less spread around its own expectation. Calculate the bias and MSE before making an accuracy comparison.
Mistake: measuring spread around the target and calling it variance. Variance is centered at \(E(\hat{\theta})\); MSE is centered at \(\theta\). Their difference is the squared bias.
Mistake: comparing estimates of different targets. State a common estimand before comparing properties. Otherwise, an apparent bias difference may reflect a change in what is being estimated.

These calculations are conditional on the model and design assumptions used to obtain them. A convenient benchmark with equal outcome variances, for example, need not apply if that assumption fails. Nor does a low MSE guarantee that every scientific aim is met: an estimator's average squared error does not describe all features of patient outcomes or all consequences of the allocation rule.

Key takeaway: Compare a treatment-effect estimator with respect to a common target and a fully specified repeated-trial procedure. Report bias, variance, and MSE separately: a design may reduce variability while introducing bias, and the resulting MSE determines the average squared error under the chosen assumptions.

Check Your Understanding

Answer each question, then compare with the solutions below.

An estimator targets 5 points, has expectation 5.3 points, and variance \(2.0\ \text{points}^2\). Find its bias and MSE.
Why is the variance of an estimator not generally the same as its MSE?
In the fixed-allocation benchmark, let \(n_A=n_B=10\) and \(\sigma=4\) points. Calculate the variance of the difference in sample means.
A procedure has smaller variance than a competitor but a larger squared bias. What additional calculation is needed to compare average squared error?

Solutions. (1) Bias is \(5.3-5=0.30\) points. MSE is \(2.0+(0.30)^2=2.09\ \text{points}^2\). (2) Variance is centered at \(E(\hat{\theta})\), whereas MSE is centered at the target \(\theta\); their difference is squared bias. (3) \(\operatorname{Var}(\hat{\theta})=16(1/10+1/10)=3.20\ \text{points}^2\). Under the stated benchmark assumptions the estimator is unbiased, so its MSE is also \(3.20\ \text{points}^2\). (4) Add each estimator's variance to its squared bias, then compare the resulting MSEs under the same target and conditions.