Uncertainty in Estimated Correlations

Why an Estimated Correlation Has Its Own Uncertainty

Two treatment-versus-control contrasts share the control-group mean, so their sampling errors are generally correlated. Earlier material on shared-control covariance establishes how the variances and covariance of those contrasts determine their model-based correlation. In practice, the group variances are usually unknown. Replacing them with sample variances therefore produces an estimated correlation, which can differ from the model-based value and itself vary from sample to sample.

It is useful to keep two sources of uncertainty distinct. The contrast estimates vary because new participants would produce different group means. Separately, the estimated correlation varies because new participants would also produce different sample variances. The correlation is a parameter of the assumed model; its plug-in estimate is a statistic calculated from observed data.

Model-Based and Plug-In Correlations

Suppose independent groups \(0,i,j\) have sample sizes \(n_0,n_i,n_j\). Within group \(g\), assume the outcomes are independent and normally distributed with mean \(\mu_g\) and variance \(\sigma_g^2\). Assume also that observations in different groups are independent and that all three variances are positive. Define the treatment-versus-control contrasts \(D_i=\bar{X}_i-\bar{X}_0\) and \(D_j=\bar{X}_j-\bar{X}_0\).

Under the stated model, the group means are independent across groups, and the only random component shared by the two contrasts is the control mean. Thus the covariance is \(\sigma_0^2/n_0\); the two contrast variances are \(\sigma_i^2/n_i+\sigma_0^2/n_0\) and \(\sigma_j^2/n_j+\sigma_0^2/n_0\). Consequently,

The corresponding estimate is

For these two contrasts, oriented in the same treatment-minus-control direction, the correlation is positive and less than one when all group variances are positive. The means \(\mu_g\) do not enter the formula: changing treatment effects does not change this model-based correlation. The sample estimate, however, depends on the observed sample variances and can differ from \(\rho_{ij}\).

Approximating the Uncertainty of the Estimate

A standard way to quantify the sampling uncertainty in \(\hat{\rho}_{ij}\) is the delta method. It approximates the variance of a smooth function of estimated quantities by combining the function’s derivatives with the variances of those quantities. Under independent normal sampling, each group sample variance is unbiased for its population variance and has variance \(2\sigma_g^4/(n_g-1)\). Sample variances from different groups are independent. These facts give the ingredients for an approximate standard error.

For the derivation, write \(q_g=\sigma_g^2/n_g\), \(A=q_i+q_0\), \(B=q_j+q_0\), and \(\rho=q_0/\sqrt{AB}\). Differentiating with respect to \(q_i,q_j,q_0\) gives

Since \(q_g=\sigma_g^2/n_g\), the derivative with respect to \(\sigma_g^2\) is the derivative with respect to \(q_g\), divided by \(n_g\). Substituting the corresponding estimated quantities into the delta-method variance yields

where, with \(\hat{q}_g=s_g^2/n_g\), \(\hat{A}=\hat{q}_i+\hat{q}_0\), \(\hat{B}=\hat{q}_j+\hat{q}_0\), and \(\hat{\rho}_{ij}=\hat{q}_0/\sqrt{\hat{A}\hat{B}}\),

The approximation uses the first-order change in the correlation as each estimated variance changes. The estimated standard error is the square root of the displayed estimated variance. A normal-based interval, \(\hat{\rho}_{ij}\pm 1.96\,SE\), can be used as a rough large-sample interval when the approximation is reasonable. It is not an exact interval, and it can extend outside the possible correlation range.

When a Common-Variance Assumption Changes the Answer

If all groups share a common variance, \(\sigma_g^2=\sigma^2\), the factor \(\sigma^2\) cancels from the correlation formula. With fixed group sizes, the model-based correlation then depends only on allocation. Estimating the common variance does not add uncertainty to this correlation: a pooled variance estimate appears in both numerator and denominator and cancels exactly. This is different from estimating group-specific variances, where their relative sizes affect the correlation and their estimation contributes uncertainty.

Diagnostics, Alternatives, and Limitations

Before reporting a plug-in correlation or its standard error, check whether the calculations match the design and model:

• Check group independence and allocation. The formula assumes independent groups and uses each group’s actual sample size. Dependence between observations within a group, or a different sampling structure, can invalidate the variance calculations.
• Inspect the group variances and degrees of freedom. Very small groups give noisy variance estimates. A zero sample variance also makes the plug-in formula degenerate and is a warning that the usual approximation is not usable as written.
• Check sensitivity to the variance model. If the groups have different spreads, imposing a common variance can change the estimated correlation. Conversely, estimating three variances separately may be unstable when sample sizes are small.
• Check the approximation near a boundary. The delta method relies on a local, approximately linear representation. A standard-error interval may exceed the correlation range, particularly near zero or one, or when variance estimates are unstable.

The delta method is convenient and transparent when the normal model is plausible and group sizes are adequate, but it is an approximation. A parametric bootstrap is an alternative: repeatedly simulate independent normal samples using fitted group means and variances, recompute the plug-in correlation, and use the simulated distribution to assess its variability. That procedure still relies on the fitted model and can be unreliable if the model is poor or the sample is small. A Fisher transformation is not automatically an appropriate substitute: this is a correlation between two specified contrasts, not the usual sample Pearson correlation calculated from many independent pairs.

A common mistake is to treat one observed pair of contrast estimates as though it were a sample of paired observations and calculate their empirical correlation. One pair cannot estimate a correlation. The target here is the model-based dependence implied by the shared control and group variances; its plug-in estimate is computed from the group variance estimates. Another mistake is to report \(\hat{\rho}_{ij}\) as if it were known exactly. When variances are estimated separately, a standard error or other uncertainty assessment is needed, subject to the model and approximation checks above.