Selection Bias With Unequal Standard Errors

Why Unequal Standard Errors Matter

When several treatment arms are compared, the arm with the largest observed effect may be selected for further study or emphasis. If estimates have different standard errors, their chances of producing an unusually high value are not the same. In particular, an estimate with greater variability can be more likely to produce an extreme value even when its true treatment effect is no larger. The observed winner therefore reflects both the underlying effects and the precision of the estimates.

The key question here is how unequal standard errors affect selection by the largest raw treatment-effect estimate. This rule differs from selecting the largest standardized statistic, such as an estimate divided by its standard error. The distinction matters: the two rules can select different arms and produce different expected reported effects.

Definition: Let $X_i$ be the estimated treatment effect for arm $i$, with true effect $\theta_i$, standard error $s_i$, and $X_i\sim N(\theta_i,s_i^2)$. Under selection by the largest raw estimate, the selected arm is $J=\arg\max_i X_i$, and the selected estimate is $X_J=\max_i X_i$. Its expected exaggeration is $E[X_J-\theta_J]$, where $\theta_J$ is the true effect of the arm selected in that repetition.

For the formulas below, assume that the estimates are jointly normal and that their standard errors and correlations are known or treated as fixed. The model concerns repeated hypothetical trials under the same true effects and design. In practice, a reported standard error may itself be estimated; then these calculations are an approximation unless that additional uncertainty is modeled. With shared controls, estimates can be correlated, so independence must not be assumed automatically.

Two Arms: Selection Probability and Expected Exaggeration

For two arms, write $X_1=\theta_1+\varepsilon_1$ and $X_2=\theta_2+\varepsilon_2$, where the errors have mean zero. Let their variances be $v_1=s_1^2$ and $v_2=s_2^2$, and their covariance be $c$. The difference $D=X_1-X_2$ is normal with mean $\delta=\theta_1-\theta_2$ and variance $\tau^2=v_1+v_2-2c$. Assume $\tau>0$. Arm 1 is selected exactly when $D>0$, so

P(J=1)=\Phi\left(\frac{\delta}{\tau}\right), \qquad P(J=2)=\Phi\left(-\frac{\delta}{\tau}\right),

where $\Phi$ is the standard normal cumulative distribution function. Thus unequal standard errors enter through the variability $\tau$ of the contrast between estimates. They affect how reliably selection favors the arm with the larger true effect.

Result: For two jointly normal estimates selected by the larger raw estimate, $$ E[X_J-\theta_J]=\tau\,\phi\left(\frac{\delta}{\tau}\right), $$ where $\phi$ is the standard normal density. This is nonnegative and is positive for finite $\delta/\tau$. The result includes correlated estimates, provided $c$ is used in $\tau^2=v_1+v_2-2c$.

This is an expectation over repeated trials, not a claim that every selected estimate exceeds its own arm’s true effect. The amount of expected exaggeration depends on how hard the arms are to distinguish: if the true effects are close relative to the variability of their estimated difference, the chance of selecting on noise is substantial.

Deriving the Two-Arm Result

The maximum of two numbers can be written as their average plus half their absolute difference. With $D=X_1-X_2$, this gives $X_J=(X_1+X_2+|D|)/2$. For a normal variable $D\sim N(\delta,\tau^2)$, integration of its density over the positive and negative regions gives

E|D|=\tau\sqrt{\frac{2}{\pi}}\exp\left(-\frac{\delta^2}{2\tau^2}\right) +\delta\left\{2\Phi\left(\frac{\delta}{\tau}\right)-1\right\}.

Also, $E[X_1+X_2]=\theta_1+\theta_2$. Substituting these expectations into the maximum identity yields

E[X_J] =\theta_2+\tau\phi\left(\frac{\delta}{\tau}\right) +\delta\Phi\left(\frac{\delta}{\tau}\right).

Arm 1 is selected with probability $\Phi(\delta/\tau)$, so the expected true effect of the selected arm is

E[\theta_J] =\theta_1\Phi\left(\frac{\delta}{\tau}\right) +\theta_2\Phi\left(-\frac{\delta}{\tau}\right) =\theta_2+\delta\Phi\left(\frac{\delta}{\tau}\right).

Subtracting the expected selected true effect from the expected selected estimate gives $E[X_J-\theta_J]=\tau\phi(\delta/\tau)$, as stated. This derivation shows why the bias is not simply the standard error of the arm that wins most often: it is governed by the variability of the difference between the two estimates.

Worked Example: Equal True Effects, Unequal Standard Errors

Suppose two hypothetical treatment-effect estimates are independent and normally distributed, with true effects of 0.30 response units for both arms. Their standard errors are 0.20 and 0.10 response units. The larger raw estimate is selected. Find the selection probabilities and expected exaggeration.

Here $\delta=0$, $v_1=0.20^2=0.0400$, $v_2=0.10^2=0.0100$, and $c=0$. Therefore,

\tau=\sqrt{0.0400+0.0100}=\sqrt{0.0500}=0.2236 \quad\text{response units}.

Because the true effects are equal, $P(J=1)=\Phi(0)=0.500$, and $P(J=2)=0.500$. Equal selection probabilities do not mean that the two selected estimates have the same distribution: the less precise arm has more variable estimates.

The expected exaggeration is

E[X_J-\theta_J] =0.2236\,\phi(0) =0.2236(0.3989) =0.0892 \quad\text{response units}.

Thus the expected selected estimate is $0.30+0.0892=0.3892$ response units. This is an average over repeated trials; it does not imply that the selected estimate in any particular trial is biased upward by exactly 0.0892 response units.

Worked Example: Unequal True Effects and Unequal Standard Errors

Consider two hypothetical independent estimates. Arm 1 has true effect $\theta_1=0.30$ response units and standard error $0.20$; arm 2 has true effect $\theta_2=0.20$ response units and standard error $0.10$. Select the arm with the larger raw estimate.

The contrast has $\delta=0.10$ response units and $\tau=\sqrt{0.20^2+0.10^2}=0.2236$ response units. Hence $\delta/\tau=0.4472$, giving $P(J=1)=\Phi(0.4472)\approx0.6726$ and $P(J=2)\approx0.3274$.

The expected true effect of the selected arm is

E[\theta_J] =0.30(0.6726)+0.20(0.3274) =0.2673 \quad\text{response units}.

Using $\phi(0.4472)\approx0.3605$, the expected exaggeration is

E[X_J-\theta_J] =0.2236(0.3605) =0.0806 \quad\text{response units}.

Therefore $E[X_J]\approx0.2673+0.0806=0.3479$ response units. The better-performing arm is selected more often, but the selected estimate still exaggerates the true effect of the selected arm on average.

More Than Two Arms and the Role of the Selection Rule

With $K$ independent normal estimates, unequal standard errors affect selection through each arm’s full distribution, not just its mean. The probability arm $i$ is selected can be written as

P(J=i)=\int_{-\infty}^{\infty} f_i(x)\prod_{j\ne i}F_j(x)\,dx,

where $f_i$ and $F_i$ are the density and cumulative distribution function of $X_i$. The product is the probability that every other estimate is at most $x$. Expected exaggeration is then $E[\max_i X_i]-\sum_i\theta_iP(J=i)$. In general there is no two-arm formula that reduces this calculation to one standard normal density; numerical integration or simulation may be needed. For correlated estimates, the joint distribution must replace the independence product.

Selection by a standardized statistic changes the event defining the winner. For example, selecting the largest $X_i/s_i$ compares estimates relative to their standard errors, rather than comparing the estimated effects in their original units. It is not a general correction for selection bias: it changes which arm is chosen, and the raw estimate reported for that arm can still be exaggerated.

Worked Example: Raw-Effect Selection Versus Standardized Selection

Suppose two hypothetical independent estimates have true effects of zero response units and standard errors $s_1=0.20$ and $s_2=0.10$ response units. Compare the expected selected raw estimate under two rules: choose the larger raw estimate, or choose the larger standardized statistic $X_i/s_i$ and report that arm’s raw estimate.

Under raw-effect selection, the estimates have equal means and the difference has standard deviation $\tau=\sqrt{0.20^2+0.10^2}=0.2236$ response units. The expected selected raw estimate is the expected exaggeration, $0.2236\phi(0)=0.0892$ response units.

For standardized selection, write $Z_i=X_i/s_i$. Under the stated zero-effect model, $Z_1$ and $Z_2$ are independent standard normal variables. The selected raw estimate is $s_JZ_J$, where $J$ indexes the larger $Z_i$. Symmetry gives $E[Z_1I(Z_1>Z_2)]=E[\max(Z_1,Z_2)]/2$, and $E[\max(Z_1,Z_2)]=1/\sqrt{\pi}$. Therefore,

E[X_J] =\frac{s_1+s_2}{2\sqrt{\pi}} =\frac{0.20+0.10}{2\sqrt{\pi}} =0.0846 \quad\text{response units}.

The two rules produce different expected selected raw estimates, and neither expectation is zero despite both true effects being zero. The calculations depend on the specified selection and reporting rules; one should not transfer a bias formula from one rule to the other.

Assumptions, Diagnostics, and Limitations

Before applying these results, verify that the modeled estimates correspond to the effects being ranked and that the selection rule really is the one represented in the calculation. In a multi-arm multi-stage design, an estimate may be selected at an interim stage using one data set and later reported using additional data. That process is not the single-stage maximum considered here; its selection event and estimate distribution must be represented separately.

Useful checks include:

Check the scale. Raw effect estimates must share a meaningful common scale and direction. If estimates use different scales, a raw maximum may not represent the intended selection rule.
Check the standard errors and covariance. Unequal arm sizes can produce unequal standard errors. Shared controls can induce covariance; omitting it changes $\tau$ and therefore the calculated selection probability and exaggeration.
Check the ranking rule. Ranking by raw estimates, standardized statistics, or a thresholded rule defines different selection events.
Check the normal approximation. Strong skewness, small samples, boundary estimates, or unstable standard-error estimates can make the normal model inaccurate. Simulation under the planned design can assess the impact.

A common mistake is to conclude that, with equal true effects, the less precise arm must win more often when there are only two arms. Under the stated bivariate normal model, equal means imply $P(J=1)=P(J=2)=0.5$, even when the standard errors differ. The correction is to distinguish the probability of winning from the size of the estimate when an arm wins: the higher-variance estimate can contribute more to the expected maximum even though it is not more likely to win in this two-arm equal-mean case.

These formulas quantify expected exaggeration under a specified model; they do not adjust an observed estimate, guarantee a particular amount of bias in a realized trial, or provide a confidence interval. They also do not establish that the selected treatment is truly best. Their practical value is to show why precision and the selection rule belong in any assessment of selected treatment-effect estimates.

Check Your Understanding

Answer each question using the model and selection rule stated. Brief solutions follow.

Two independent estimates have equal true effects and standard errors 0.15 and 0.15 response units. What is the expected exaggeration when the larger raw estimate is selected?
For two estimates with $\theta_1-\theta_2=0.08$, $s_1=0.12$, $s_2=0.16$, and covariance zero, write the expression for $P(J=1)$.
Why can the arm with the larger standard error have the same probability of selection as the more precise arm when there are two arms with equal true effects?
What additional issue arises if the estimates share a control group?

Solutions. (1) Here $\tau=\sqrt{0.15^2+0.15^2}=0.2121$ response units, so the expected exaggeration is $0.2121\phi(0)=0.0846$ response units. (2) $\tau=\sqrt{0.12^2+0.16^2}=0.20$ response units, so $P(J=1)=\Phi(0.08/0.20)=\Phi(0.40)\approx0.6554$. (3) The difference of the estimates has mean zero, so its symmetric normal distribution is positive with probability $0.5$; unequal variances change the difference’s spread, not this symmetry. (4) The estimates may be correlated, so the covariance must be included in $\tau^2=s_1^2+s_2^2-2c$; treating them as independent can misstate both selection probability and expected exaggeration.