Small-Sample Variance Updates And Degrees Of Freedom

Why Small-Sample Variance Updates Need Care

A sample-size re-estimation rule for a continuous outcome often uses an interim estimate of the outcome variance. In a small trial, that estimate can be imprecise, and the final analysis may use a $t$-test rather than a normal approximation. The variance estimate affects the number of participants needed; the degrees of freedom affect the critical value and the test's power. Both parts matter when translating interim data into a revised target.

The calculations below use a two-arm, equal-allocation trial with a continuous outcome. The basic model assumes independent observations, normally distributed outcomes within each arm, and a common variance $\sigma^2$ in both arms. The target treatment difference is denoted by $\delta$, in the outcome's units. A variance update does not change that target difference: it revises the sample size needed to detect the prespecified difference.

Definition: For independent normal outcomes with a common variance in two arms, the pooled sample variance is $$ s_p^2=\frac{(n_1-1)s_1^2+(n_0-1)s_0^2}{n_1+n_0-2}. $$ Here $n_1,n_0$ and $s_1,s_0$ are the sample sizes and sample standard deviations in the two arms. Under the common-variance model, $s_p^2$ estimates $\sigma^2$, and the pooled two-sample $t$-test has $df=n_1+n_0-2$.

How Variance and Degrees of Freedom Enter the Update

For a planned final sample size of $n$ participants per arm, the standard error of the difference in sample means under the common-variance model is $s_p\sqrt{2/n}$. The pooled $t$-statistic divides the observed difference in means by this standard error and uses $2n-2$ degrees of freedom. Under a true difference $\delta$, the corresponding noncentrality parameter is $\lambda=\delta/(\sigma\sqrt{2/n})$. Thus, even if the target $\delta$ is fixed, increasing the variance lowers $\lambda$ and generally requires more participants to attain the same power.

For a two-sided test at level $\alpha$, exact model-based power at a candidate $n$ is the probability that a noncentral $t$ random variable with $2n-2$ degrees of freedom and noncentrality parameter $\lambda$ falls below $-t_{1-\alpha/2,\,2n-2}$ or above $t_{1-\alpha/2,\,2n-2}$. Here $t_{q,df}$ is the $q$-quantile of a central $t$ distribution with $df$ degrees of freedom. A t-based re-estimation can find the smallest integer $n$ meeting the target power by checking candidate sample sizes and their corresponding degrees of freedom.

In practice, the interim value $s_p$ is substituted for the unknown $\sigma$. This produces estimated power under the planning model, not a guarantee of the achieved power. A simpler planning approximation replaces the relevant normal critical value with a $t$ critical value and uses a normal quantile for the target power:

n \mathrel{\approx} \frac{2s_p^2\left(t_{1-\alpha/2,\,2n-2}+z_{1-\beta}\right)^2}{\delta^2},

where $1-\beta$ is the target power and $z_{1-\beta}$ is a standard normal quantile. Because $n$ appears in its own degrees of freedom, calculate iteratively: choose a candidate $n$, evaluate its $t$ critical value, recalculate the required $n$, and round up. This approximation is useful for transparent planning calculations, but exact noncentral-$t$ power is preferable when implementing a design.

Why the Pooled Estimate Has These Degrees of Freedom

The degrees of freedom are not an arbitrary small-sample correction. With normal observations in arm $j$, the within-arm sum of squared deviations, divided by $\sigma^2$, has a chi-square distribution with $n_j-1$ degrees of freedom. The two arms' sums of squares are independent. Adding them therefore gives a chi-square quantity with $(n_1-1)+(n_0-1)=n_1+n_0-2$ degrees of freedom. Dividing that combined sum of squares by its degrees of freedom gives $s_p^2$.

Under the same normal, common-variance assumptions, the difference in arm means is independent of the pooled variance estimate. Standardizing the difference by its estimated standard error consequently gives a central $t$ distribution under $H_0$, with $n_1+n_0-2$ degrees of freedom. This justification depends on the model; a small sample does not make the assumptions optional.

Worked Example: Updating a Two-Arm Target From an Interim Variance

Suppose a hypothetical trial has 12 participants in each arm at an interim analysis. The observed standard deviations are 9 units in the treatment arm and 11 units in the control arm. The planned final test is two-sided with $\alpha=0.05$; the target difference is 5 outcome units and the desired power is 80%. Assume independent normal outcomes and a common variance, and suppose the re-estimation rule uses the pooled interim variance.

The pooled variance and its degrees of freedom are $$ s_p^2=\frac{(12-1)(9^2)+(12-1)(11^2)}{12+12-2} =\frac{11(81)+11(121)}{22} =\frac{2222}{22}=101, \qquad df=22. $$ Thus $s_p=\sqrt{101}=10.05$ units. The estimate is based on only 22 degrees of freedom, so it can vary substantially from trial to trial.

Use the stated t-based planning approximation and iterate over $n$. Near the solution, $n=65$ per arm gives $df=128$, $t_{0.975,128}\approx1.979$, and $z_{0.80}\approx0.842$. The required sample size from the approximation is $$ \frac{2(101)(1.979+0.842)^2}{5^2} =\frac{202(7.958)}{25} \approx64.30. $$ Rounding up gives 65 participants per arm. This is an approximate plug-in planning result, not an exact-power calculation. It implies an increase from the interim total of 24 to a planned final total of 130, subject to the prespecified rule's limits and any separate inferential requirements.

For comparison, if the planning standard deviation had remained 8 units, using a nearby $t$ critical value of about 1.99 gives $$ \frac{2(8^2)(1.99+0.842)^2}{5^2} \approx41.1, $$ or about 42 per arm after rounding up. The larger interim variance estimate is what drives the substantial increase; the critical-value adjustment is present but is not the main source of it.

Degrees of Freedom Can Matter Even When the Difference Is Fixed

At small sample sizes, a $t$ critical value is larger than the corresponding normal critical value. That increases the sample size required by a t-based planning approximation. Degrees of freedom also depend on the analysis chosen. The pooled test uses $n_1+n_0-2$ degrees of freedom when the common-variance model is used. If unequal variances are allowed, the Welch test uses a different standard error and an approximate degrees-of-freedom calculation.

Worked Example: Comparing a Small-Sample t Adjustment With a Normal Approximation

Consider a hypothetical equal-allocation design targeting a difference of 10 units, with a planning standard deviation of 10 units in each arm, two-sided $\alpha=0.05$, and 80% power. For illustration, use the t-critical-value planning approximation and compare it with the normal-critical-value approximation. Assume the common-variance model is suitable.

With $n=17$ per arm, $df=2(17)-2=32$, so $t_{0.975,32}\approx2.037$. The approximate requirement is $$ \frac{2(10^2)(2.037+0.842)^2}{10^2} =2(2.879)^2 \approx16.58. $$ Rounding up gives 17 per arm. Checking $n=16$ gives $df=30$, $t_{0.975,30}\approx2.042$, and a requirement of $$ 2(2.042+0.842)^2\approx16.63, $$ which exceeds 16; hence 16 per arm does not meet this approximation's criterion.

If the normal critical value 1.96 were used instead, the corresponding calculation would be $$ 2(1.96+0.842)^2\approx15.70, $$ suggesting 16 per arm. In this illustration the normal approximation misses one participant per arm. Exact power calculations can differ from these approximations, but the example shows why using a normal critical value by habit can matter in a small design.

Worked Example: Welch Degrees of Freedom With Unequal Variances

Suppose a hypothetical small trial has $n_1=8$, $s_1=12$ units and $n_0=10$, $s_0=8$ units. If the analysis uses Welch's two-sample $t$-test, its estimated standard error is based on $s_1^2/n_1+s_0^2/n_0$, and its approximate degrees of freedom are $$ df_W= \frac{(s_1^2/n_1+s_0^2/n_0)^2} {(s_1^2/n_1)^2/(n_1-1)+(s_0^2/n_0)^2/(n_0-1)}. $$

Substituting gives $$ df_W= \frac{(144/8+64/10)^2}{(144/8)^2/7+(64/10)^2/9} =\frac{(18+6.4)^2}{324/7+40.96/9} =\frac{595.36}{50.84} \approx11.71. $$ The pooled-test degrees of freedom would be $8+10-2=16$, but that number does not apply to this Welch calculation. A two-sided 5% critical value for roughly 11.7 degrees of freedom is about 2.18, compared with about 2.12 at 16 degrees of freedom. The lower Welch degrees of freedom make the cutoff somewhat more conservative here. A sample-size update should use a planning method consistent with the intended final test, rather than combining a Welch variance estimate with pooled-test degrees of freedom.

Interim Variance, Final Degrees of Freedom, and Design Validity

The interim degrees of freedom describe the precision of the interim variance estimate. They are not the degrees of freedom for the final test. For example, with $m_1,m_0$ interim observations, the pooled interim variance has $m_1+m_0-2$ degrees of freedom under the common-variance model. If the final allocation is equal with $n$ participants per arm, the conventional final pooled $t$-test uses $2n-2$ degrees of freedom. The interim estimate informs the target; the final data determine the final test statistic and its nominal degrees of freedom.

This distinction does not, by itself, settle whether the final test has its advertised type I error. If the total sample size is selected using interim data, the final statistic and the sample-size decision are part of one adaptive procedure. A textbook fixed-sample $t$-test is not automatically an exact unconditional test after such a data-dependent change. In particular, replacing $\sigma$ with an interim estimate and then using the usual final $t$ critical value does not prove that the overall rejection probability under $H_0$ equals $\alpha$. The re-estimation rule and inferential procedure must be specified and justified for the design; the plug-in power calculation alone addresses neither that validity question nor the operating characteristics of the full rule.

For a variance update that retains interim participants in the final analysis, the variance estimate used for planning should also be distinguished from the final variance estimate. Under the model, a final pooled variance is calculated from the within-arm residual sums of squares across all final participants, with $2n-2$ degrees of freedom for equal allocation. It is not obtained by simply treating the interim variance as if it were an independent new observation. Any combination of interim and later information must respect the sums of squares and their degrees of freedom.

Assumptions, Diagnostics, and Alternatives

Before applying the pooled update, check that the outcome scale and measurement process are consistent across interim and later participants, that observations are independent at the analysis-unit level, and that a common within-arm variance is defensible. The small-sample $t$ result relies on normal outcomes; severe skewness, influential outliers, or heavy tails can make a variance estimate unstable and the nominal small-sample calibration unreliable. Inspect arm-specific standard deviations and distributions, and examine whether the interim estimate is being driven by a few extreme values. These checks diagnose concerns; they do not justify changing the prespecified endpoint or target difference in response to the interim results.

A pooled-variance t-based update is a natural choice when the common-variance normal model is credible. A normal approximation is simpler, but can understate the required sample size when degrees of freedom are limited. Welch's test is an alternative when arm variances may differ; it avoids pooling the variances, but its degrees of freedom and power calculations must be handled consistently with that test. No method removes the uncertainty in a small interim variance estimate. A sample-size cap, floor, or other restriction may be operationally necessary, but a cap can leave estimated power below target and should not be presented as if it guarantees the original power.

Common pitfall: “The interim estimate is $s_p$, so I can substitute it into a normal formula, use $n_1+n_0-2$ as the final degrees of freedom, and assume the final test still has level $\alpha$.” This mixes an interim variance estimate with final-test degrees of freedom and treats a plug-in planning calculation as an inferential guarantee. Instead, use the interim degrees of freedom to describe the variance estimate's information, calculate planning power using the intended final test and its candidate final degrees of freedom, and separately justify the adaptive inferential procedure.

Check Your Understanding

Answer each question, then compare with the concise solutions below.

With 9 participants per arm, what are the pooled-variance degrees of freedom under the common-variance model?
Why can an interim variance estimate with few degrees of freedom lead to an unstable sample-size update?
In an equal-allocation trial with 30 participants per arm at final analysis, what degrees of freedom does the conventional pooled $t$-test use?
Why is it incorrect to use pooled-test degrees of freedom for the Welch calculation in the third worked example?
Does meeting a plug-in t-based power target establish that the adaptive procedure has type I error $\alpha$?

Solutions: (1) $9+9-2=16$. (2) A variance estimate based on few residual degrees of freedom has substantial sampling variability; because the required sample size scales approximately with the estimated variance, unusually high or low interim estimates can cause large changes. (3) $30+30-2=58$. (4) Welch's test uses separate arm variance contributions and the Welch-Satterthwaite approximation, not the pooled common-variance model's degrees of freedom. (5) No. Plug-in power is a planning calculation; the type I error of the complete adaptive procedure requires separate justification.