Why Compare Site-Specific Variance Summaries?
A pooled variance estimate can look stable while being driven largely by a few unusually variable centers. That matters when a blinded nuisance estimate is used for sample-size planning: if the overall estimate is elevated by a small number of sites, treating it as a typical variance for every future participant may be misleading. Conversely, a high estimate at one small site may be mostly sampling noise rather than evidence of a different underlying variance.
The diagnostic question is therefore not simply whether site estimates differ. It is whether the overall summary is being dominated by particular sites, and whether the observed pattern is plausibly about underlying dispersion rather than site size, random variation, or the way masked outcomes were pooled. These comparisons are diagnostic; by themselves, they do not establish that sites have different population variances.
This definition assumes each site estimate is based on a comparable outcome scale and a clearly specified variance calculation. If site estimates come from a model with residual degrees of freedom rather than the ordinary sample variance, use the appropriate residual degrees of freedom in place of \(d_j\). Pooling by degrees of freedom is natural when the estimates are independent estimates of a common variance under the model. When true variances differ, it instead summarizes them with weights proportional to their degrees of freedom; it does not make them equal.
Measures of Site Dominance
Two quantities help distinguish a large estimate from a large contribution. Define site \(j\)'s share of total degrees of freedom as \(w_j=d_j/\sum_k d_k\), and its share of the pooled variance numerator as \(q_j=d_jS_j^2/\sum_k d_kS_k^2\). The \(q_j\) values sum to one. A site's \(q_j\) can be large because its variance estimate is high, because it has many degrees of freedom, or both.
Thus, a site contributes a greater share of the variance numerator than of the degrees of freedom precisely when its variance estimate exceeds the pooled estimate. A useful companion diagnostic is the leave-one-site-out pooled estimate:
A large change between \(S_{\mathrm{pool}}^2\) and \(S_{\mathrm{pool},(-j)}^2\) shows sensitivity to site \(j\). It does not, on its own, tell us why that site differs or which estimate is preferable for planning.
Worked Example: Finding a Site With a Large Pooled Contribution
Suppose three hypothetical sites report these masked variance estimates: Site A has \(d_A=19\) and \(S_A^2=100\) square units; Site B has \(d_B=39\) and \(S_B^2=121\) square units; Site C has \(d_C=9\) and \(S_C^2=400\) square units. Calculate the overall estimate and assess whether one site dominates it.
The pooled estimate is $$ S_{\mathrm{pool}}^2= \frac{19(100)+39(121)+9(400)}{19+39+9} =\frac{1900+4719+3600}{67} =\frac{10219}{67} \approx152.5\text{ square units}. $$
Site C supplies \(9/67\approx13.43\%\) of the degrees of freedom, but contributes \(3600/10219\approx35.23\%\) of the variance numerator. Its variance estimate is well above the pooled estimate, so its contribution is disproportionate to its degrees of freedom.
Without Site C, the estimate is $$ S_{\mathrm{pool},(-C)}^2= \frac{1900+4719}{19+39} =\frac{6619}{58} \approx114.1\text{ square units}. $$ The pooled estimate falls from about 152.5 to 114.1 square units when Site C is removed. This is a meaningful sensitivity signal to investigate, not proof that Site C is erroneous or that its population variance is different.
Why a Masked Site Variance Can Include Treatment Separation
A site-specific variance calculated by pooling all masked outcomes around one site mean may reflect more than within-arm outcome variability. If treatment and control have different means, their mixture is more dispersed than either arm alone. The effect on the sample variance depends on the treatment allocation counts at that site. Consequently, a site comparison can show apparent heterogeneity even when the within-arm variance is common across sites.
For a precise illustration, suppose site \(j\) has \(n_{0j}\) control participants and \(n_{1j}\) treatment participants, where \(n_j=n_{0j}+n_{1j}\), and both arms have variance \(\sigma_j^2\). Let the arm means differ by \(\Delta_j\), and let \(S_j^2\) be the sample variance of all \(n_j\) outcomes pooled together. Assume independent observations, both arm counts are positive, and the arm means and variances are well-defined. A short decomposition gives the expected masked variance.
The sum of squared deviations from the site-wide mean is the sum of the within-arm sums of squares plus the between-arm sum of squares. The expected within-arm total is \((n_j-2)\sigma_j^2\), since the two arm means are each estimated from their samples.
The between-arm sum of squares is \(\frac{n_{0j}n_{1j}}{n_j}(\bar{Y}_{1j}-\bar{Y}_{0j})^2\). The squared difference has expectation \(\Delta_j^2+\sigma_j^2(1/n_{0j}+1/n_{1j})\), so the expected between-arm sum of squares is \(\frac{n_{0j}n_{1j}}{n_j}\Delta_j^2+\sigma_j^2\).
Adding the two expected components gives \((n_j-1)\sigma_j^2+\frac{n_{0j}n_{1j}}{n_j}\Delta_j^2\). Dividing by \(n_j-1\) yields the expected pooled masked variance.
The extra term is nonnegative and is larger when the treatment groups are more balanced, all else equal. The derivation includes the sampling variation in the observed arm means: the expected between-arm sum of squares contains an additional \(\sigma_j^2\), not just the squared difference between the population means. Omitting that component would understate the expected total sum of squares.
Worked Example: Checking for Allocation-Related Inflation
Consider two hypothetical sites with the same within-arm variance, \(\sigma_j^2=100\) square units, and the same treatment mean difference, \(\Delta_j=4\) outcome units. At Site A, \(n_{0A}=20\) and \(n_{1A}=20\), so \(n_A=40\). At Site B, \(n_{0B}=30\) and \(n_{1B}=10\), also with \(n_B=40\). Find each expected masked variance.
For Site A, $$ E(S_A^2)=100+\frac{20(20)(4^2)}{40(39)} =100+\frac{6400}{1560} \approx104.1\text{ square units}. $$ For Site B, $$ E(S_B^2)=100+\frac{30(10)(4^2)}{40(39)} =100+\frac{4800}{1560} \approx103.1\text{ square units}. $$
The underlying within-arm variances are identical, yet the expected all-participant masked variance is slightly larger at Site A because its allocation is more balanced. This difference is a consequence of mixing groups with different means, not evidence that the site-specific within-arm variances differ. In practice, the treatment effect is masked, so this calculation explains a possible source of distortion rather than supplying a correction that can be applied using an unknown \(\Delta_j\).
Reading the Diagnostics in Context
A sound review uses several summaries together. The pooled estimate describes the weighted overall level; the site table shows the distribution of estimates; contribution shares identify which sites drive the weighted numerator; and leave-one-out values show sensitivity. A median across sites can provide a robust description of a typical site estimate, but it answers a different question from the degrees-of-freedom-weighted pooled estimate. It gives each site equal weight and may be unstable as a planning estimate when the number of sites is small.
Worked Example: Separating a High Estimate From High Influence
Five hypothetical sites each have 19 degrees of freedom. Their variance estimates, in square units, are 64, 81, 100, 121, and 400. Calculate the pooled estimate, identify the most influential site, and compare the pooled estimate with the median.
Because the degrees of freedom are equal, the pooled estimate is the arithmetic mean: $$ S_{\mathrm{pool}}^2= \frac{64+81+100+121+400}{5} =\frac{766}{5}=153.2\text{ square units}. $$ The median site estimate is 100 square units. The site with estimate 400 contributes \(400/766\approx52.22\%\) of the variance numerator while contributing \(19/95=20.00\%\) of the degrees of freedom.
Removing that site gives $$ S_{\mathrm{pool},(-j)}^2= \frac{64+81+100+121}{4} =\frac{366}{4}=91.5\text{ square units}. $$ The pooled value is therefore highly sensitive to the site with estimate 400. The median and leave-one-out value both indicate that the weighted mean is not representative of most of these five estimates. They do not establish whether the high value reflects a real site difference, random variation, data quality, or treatment-mixture effects.
Assumptions, Comparisons, and Failure Modes
The arithmetic diagnostics require correctly computed, comparable site estimates and their correct degrees of freedom. Interpreting them as evidence about underlying site variances additionally requires care about the sampling process. For independent, identically distributed normal observations (including a common mean and variance), the sample variance's relative variability decreases as its degrees of freedom increase. Under independent normal outcomes with a common within-arm variance \(\sigma_j^2\) and fixed arm means, \((n_j-1)S_j^2/\sigma_j^2\) has a noncentral chi-square distribution with \(n_j-1\) degrees of freedom and noncentrality \(\lambda_j=n_{0j}n_{1j}\Delta_j^2/(n_j\sigma_j^2)\); thus \(S_j^2\) is a scaled noncentral chi-square, and its relative variability depends on noncentrality, and hence on mean separation and allocation pattern, as well as degrees of freedom. Even without normality, small samples generally make site estimates less stable. Therefore, a site with a striking estimate but very few observations should not automatically be treated as an outlier or as a new variance level.
| Summary | What it describes | Main caution |
|---|---|---|
| Degrees-of-freedom-weighted pooled variance | Overall weighted dispersion, with larger sites generally receiving more weight | A few high-variance sites may dominate it |
| Unweighted mean across sites | Average of the observed site estimates, giving each site equal weight | Small and large sites count equally despite different precision |
| Median site variance | A robust description of the middle site estimate | Does not estimate the same weighted target as the pooled variance |
| Leave-one-site-out pooled estimates | Sensitivity of the pooled estimate to each site | Sensitivity is not evidence that a site should be excluded |
A useful workflow is to verify the site-specific calculation and degrees of freedom, compare the site's variance-numerator share with its degrees-of-freedom share, and inspect leave-one-site-out changes. Then consider whether site size, allocation balance, outcome measurement, missingness, or data quality could explain the pattern. Investigate unusual values using prespecified data checks; do not remove a site merely because doing so lowers the variance estimate.
A common mistake is to see one high site estimate and conclude that the site's true variance is high, or to replace the overall estimate with the median without acknowledging the changed weighting. The correction is to distinguish the observed estimate, its weight, and its influence; treat the pattern as a diagnostic signal; and interpret alternatives according to the planning target and prespecified procedure. These summaries cannot identify a treatment effect or determine whether the trial should be unblinded. They are also limited when centers are very small, estimates are not comparable, or masking and allocation patterns affect the pooled variance.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Two sites have the same variance estimate, but one has four times the degrees of freedom. Which contributes more to the pooled variance numerator, and why?
- A site supplies 10% of the total degrees of freedom and 30% of the pooled variance numerator. Is its variance estimate above or below the pooled estimate?
- Why can a site with the same within-arm variance as other sites have a higher all-participant masked variance estimate?
- A leave-one-site-out estimate changes substantially when a high-variance site is omitted. What does this establish, and what does it not establish?
Solutions. (1) The site with four times the degrees of freedom contributes four times as much to the numerator because its contribution is \(d_jS_j^2\), and the estimates are equal. (2) Its contribution share exceeds its degrees-of-freedom share, so \(S_j^2>S_{\mathrm{pool}}^2\), as shown by \(q_j/w_j=S_j^2/S_{\mathrm{pool}}^2\). (3) If treatment means differ, pooling the masked arms adds a between-arm component; its expected size depends on the allocation counts as well as the treatment mean difference. (4) It establishes that the pooled estimate is sensitive to that site's data. It does not establish that the site is erroneous, has a different population variance, or should be excluded.