Information When Variance Is Estimated

Why Estimating the Variance Changes Precision

For a normal mean, precision depends on the variance: with \(n\) independent observations, the variance of the sample mean is \(\sigma^2/n\). If \(\sigma\) is known, its standard error can be calculated directly. In most applications, however, \(\sigma\) is unknown and must be estimated from the data. Replacing \(\sigma\) by the sample standard deviation \(s\) gives a useful estimate of precision, but it also means the standard error itself is estimated and variable.

This distinction matters in information-based planning. A design calculation usually starts with a planning value for the standard deviation and uses it to project how much information a proposed sample size will provide. After data are collected, the observed \(s\) gives a data-based estimate of precision. The two quantities answer related but different questions: one is a planning calculation, and the other reflects the variability observed in the sample.

For a mean measured in milligrams, for example, information has units of inverse milligrams squared, and a standard error has units of milligrams. Larger information corresponds to a smaller standard error when the model and estimand are fixed. But because \(s^2\) varies across samples, \(\hat{I}\) and \(SE\) vary too.

Planning Information and Observed Information

A planning calculation generally specifies an assumed standard deviation \(\sigma_{\text{plan}}\). The planned information at sample size \(n\) is \(n/\sigma_{\text{plan}}^2\). This is a model-based projection used to compare sample sizes or set a target; it does not claim that the future sample variance will equal the planning variance. Once outcomes are observed, \(n/s^2\) summarizes the information estimated from that sample.

The usual normal-theory formulas require independent observations with a common variance and a normal outcome distribution when exact small-sample inference is intended. The sample size is fixed for the calculation, and there are no missing observations or other complications in this simple model. If observations are dependent, variances differ across observations, or the outcome model is not appropriate, the formulas may not describe the actual precision. Those issues cannot be repaired merely by substituting a different value for \(s\).

Why the t Statistic Accounts for Variance Estimation

If \(\sigma\) is known and the normal model holds, standardizing \(\bar{x}\) by \(\sigma/\sqrt{n}\) gives a standard normal statistic. When \(\sigma\) is unknown, the natural statistic uses \(s/\sqrt{n}\) instead. The extra variation from estimating \(\sigma\) changes the reference distribution.

For independent observations \(X_1,\ldots,X_n\) from \(N(\mu,\sigma^2)\), with \(n\geq 2\), the sample mean is independent of the sample variance. In particular, \(Z=(\bar{x}-\mu)/(\sigma/\sqrt{n})\) has a standard normal distribution, and \(U=(n-1)s^2/\sigma^2\) has a chi-square distribution with \(n-1\) degrees of freedom. Their independence gives the following result.

The denominator includes the random sample standard deviation, so \(T\) has heavier tails than a standard normal variable for finite \(n\). As \(n\) grows, the \(t\) distribution approaches the standard normal distribution. The exact result depends on the normal model; for small samples, substantial skewness or outliers can make that model questionable.

For a two-sided test of \(H_0:\mu=\mu_0\), the statistic is \(t=(\bar{x}-\mu_0)/(s/\sqrt{n})\), and its p-value under the null is \(P(|T|\geq |t|)\) for \(T\sim t_{n-1}\). A \(100(1-\alpha)\%\) confidence interval is \(\bar{x}\pm t_{1-\alpha/2,n-1}s/\sqrt{n}\). The interpretation of the interval is about the procedure's long-run coverage under its assumptions, not a probability assigned to the fixed parameter after observing the interval.

Using Estimated Variance in Precision Planning

For known \(\sigma\), a normal-approximation calculation targets a half-width \(h\) by choosing \(n\) so that \(z_{1-\alpha/2}\sigma/\sqrt{n}\leq h\), or equivalently \(n\geq (z_{1-\alpha/2}\sigma/h)^2\). When variance must be estimated, a common planning adjustment uses a t critical value:

Because the t critical value depends on \(n\), this is checked over integer sample sizes rather than solved by a single rearrangement. The calculation is still a planning approximation: it inserts an assumed \(\sigma_{\text{plan}}\) in place of the future random \(s\). It does not guarantee that the realized confidence interval will have half-width at most \(h\). The future sample standard deviation may be larger or smaller than the planning value.

Choosing Between Normal and t Calculations

In group sequential settings, estimating a variance can also affect the joint behavior of statistics across analyses. The fixed-sample t result above does not, by itself, justify using ordinary fixed-sample t critical values at repeated interim looks. Sequential error control requires a procedure calibrated for the planned repeated analyses; the point here is the distinction between known-variance information and precision estimated from data.

Diagnostics, Limitations, and a Common Mistake

Before trusting a t-based calculation, check whether its model is plausible. Look for unusual observations, marked skewness in a small sample, inconsistent measurement procedures, or dependence among observations. In a small sample, a single extreme value can substantially change \(s\), hence both the estimated information \(n/s^2\) and the t statistic. A zero sample standard deviation makes the usual t statistic undefined and is a warning to inspect the data and measurement process, not evidence of infinite precision.

• Do not treat a planning standard deviation as known. It is an assumption used to plan; the realized standard deviation can differ.
• Do not use a normal critical value automatically when \(s\) replaces \(\sigma\). For small samples under the normal model, use the t distribution with \(n-1\) degrees of freedom.
• Do not interpret a larger observed \(s\) as a change in enrollment. At fixed \(n\), a larger \(s\) lowers estimated information and increases the estimated standard error.
• Do not assume the t adjustment solves every design problem. It addresses variance estimation in the stated one-sample normal model, not unequal variances, clustering, missing outcomes, or repeated-look calibration.

The practical value of the distinction is that planning can be transparent about its assumed variability, while inference can reflect the uncertainty introduced by estimating that variability. Reporting \(n\), \(s\), the estimated standard error, and the inferential reference distribution makes the precision calculation easier to evaluate.