Why Estimate the Standard Error?
An asymptotic normal approximation is useful for inference only if its scale can be estimated from the data. Often an estimator \(\hat{\theta}\) satisfies a limit theorem of the form \(\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N(0,v(\theta))\), where \(v(\theta)\) is an asymptotic variance that depends on an unknown parameter. Replacing that unknown variance with a consistent estimate makes it possible to form a statistic whose limiting distribution is standard normal.
The distinction between variance and standard error matters. The asymptotic variance \(v(\theta)\) describes the scale of \(\sqrt{n}(\hat{\theta}-\theta)\); consequently, the approximate variance of \(\hat{\theta}\) is \(v(\theta)/n\), and its standard error is \(\sqrt{v(\theta)/n}\). If \(\hat v\) estimates \(v(\theta)\), the plug-in standard error is \(\sqrt{\hat v/n}\). This tutorial develops the conditions that justify using it and works through three common models.
Studentization and Its Conditions
The conditions say more than “substitute an estimate into the formula.” The estimator must have the stated asymptotic normal distribution, the target variance must be strictly positive, and the variance estimate must converge in probability to that same variance. In particular, \(\hat v\) estimates the variance of the limiting distribution of \(\sqrt{n}(\hat{\theta}-\theta)\), not the variance of \(\hat{\theta}\) itself.
If \(\hat v\) can be zero or negative in finite samples, the displayed ratio may fail to be a real-valued statistic on those samples. It is enough for the usual asymptotic argument that the probability of this problem tends to zero; one can define the statistic arbitrarily on that exceptional event without changing its limiting distribution. If a proposed variance estimate is negative with nonvanishing probability, the usual real-valued studentized statistic is undefined on a set that is not asymptotically negligible, and the result cannot be justified by this theorem.
Why Plug-In Studentization Works
Proof. Since \(v>0\) and \(\hat v\xrightarrow{p}v\), the continuous mapping theorem applied to \(x\mapsto 1/\sqrt{x}\), which is continuous at \(v\), gives \[ \frac{1}{\sqrt{\hat v}}\xrightarrow{p}\frac{1}{\sqrt v}. \] Slutsky’s theorem then gives \[ \frac{\sqrt{n}(\hat{\theta}-\theta)}{\sqrt{\hat v}} = \bigl[\sqrt{n}(\hat{\theta}-\theta)\bigr]\frac{1}{\sqrt{\hat v}} \xrightarrow{d} Z\frac{1}{\sqrt v}, \] where \(Z\sim N(0,v)\). The random variable \(Z/\sqrt v\) has distribution \(N(0,1)\). Changing a statistic on events whose probabilities tend to zero does not change its limiting distribution: for any real \(x\), the difference between the probabilities that the original and modified statistics are at most \(x\) is bounded by the probability of the event on which they differ. That bound tends to zero. \(\square\)
The consistency assumption for \(\hat v\) is often verified by substituting a consistent estimator into a continuous variance function. The elementary result below makes that step explicit.
Proof. Fix \(\varepsilon>0\). By continuity at \(\theta\), there is a \(\delta>0\) such that \(|x-\theta|<\delta\) implies \(|g(x)-g(\theta)|<\varepsilon\). Therefore \[ P\bigl(|g(\tilde{\theta})-g(\theta)|\geq\varepsilon\bigr) \leq P\bigl(|\tilde{\theta}-\theta|\geq\delta\bigr)\longrightarrow 0, \] by convergence in probability of \(\tilde{\theta}\). This is precisely \(g(\tilde{\theta})\xrightarrow{p}g(\theta)\). \(\square\)
From the Studentized Statistic to an Interval
Let \(z_{1-\alpha/2}\) denote the \(1-\alpha/2\) quantile of the standard normal distribution. The symmetric, two-sided interval obtained from the studentized approximation is
Its center is the estimate, and its half-width is the chosen normal critical value multiplied by the estimated standard error. The interval is an approximation, not an exact finite-sample statement. The studentized limit supplies the calibration: it says that the estimation error measured in estimated standard-error units is approximately standard normal in large samples. Whether the resulting endpoints satisfy constraints such as nonnegativity is a separate issue; the formula itself does not enforce parameter-space restrictions.
Worked Example: A Poisson Rate
Suppose \(X_1,\ldots,X_n\) are independent and identically distributed as \(\operatorname{Pois}(\lambda)\), where \(\lambda>0\). For a Poisson variable, \(E[X_i]=\operatorname{Var}(X_i)=\lambda\). The sample mean \(\bar X\) estimates \(\lambda\), and the central limit theorem gives \[ \sqrt{n}(\bar X-\lambda)\xrightarrow{d}N(0,\lambda). \] Thus \(v(\lambda)=\lambda\).
The plug-in estimate is \(\hat v=\bar X\). Since \(E[\bar X]=\lambda\) and \(\operatorname{Var}(\bar X)=\lambda/n\), Chebyshev’s inequality gives, for every \(\varepsilon>0\), \[ P(|\bar X-\lambda|\geq\varepsilon) \leq \frac{\lambda}{n\varepsilon^2}\longrightarrow 0. \] Therefore \(\bar X\xrightarrow{p}\lambda\), so the variance estimate is consistent. The studentized statistic is \[ \frac{\sqrt{n}(\bar X-\lambda)}{\sqrt{\bar X}}, \] and it converges in distribution to \(N(0,1)\) wherever the denominator is positive.
For a numerical interval, suppose \(n=100\) observations have total count \(120\). Then \(\bar X=120/100=1.2\), and the estimated standard error is \[ \sqrt{\frac{\hat v}{n}}=\sqrt{\frac{1.2}{100}}=\sqrt{0.012}\approx 0.10954. \] Using \(z_{0.975}=1.96\), the margin of error is \(1.96(0.10954)\approx0.2147\). The approximate 95% interval is \[ [1.2-0.2147,\ 1.2+0.2147]=[0.9853,\ 1.4147]. \] If all observed counts are zero, then \(\bar X=0\) and this studentized ratio is undefined. For a fixed \(\lambda>0\), however, that event has probability \(P(X_1=\cdots=X_n=0)=e^{-n\lambda}\), which tends to zero. It does not obstruct the asymptotic result, though it can matter in small samples.
Worked Example: A Bernoulli Success Probability
Let \(X_1,\ldots,X_n\) be independent Bernoulli variables with success probability \(p\), where \(0<p<1\). The estimator \(\hat p=\bar X\) has asymptotic variance \(v(p)=p(1-p)\): \[ \sqrt{n}(\hat p-p)\xrightarrow{d}N(0,p(1-p)). \] The plug-in estimate \(\hat v=\hat p(1-\hat p)\) is consistent. Indeed, \(\hat p\xrightarrow{p}p\) by the law of large numbers, and \(g(x)=x(1-x)\) is continuous, so the plug-in consistency result yields \(\hat v\xrightarrow{p}p(1-p)\).
Suppose \(n=100\) trials produce 36 successes. Then \(\hat p=36/100=0.36\) and \[ \hat v=0.36(1-0.36)=0.36(0.64)=0.2304. \] The estimated standard error is \[ \sqrt{\frac{0.2304}{100}}=\sqrt{0.002304}=0.048. \] The approximate 95% interval is \[ 0.36\pm1.96(0.048)=0.36\pm0.09408, \] or approximately \([0.2659,0.4541]\). At a sample proportion of zero or one the estimated variance is zero, but under a fixed interior value \(0<p<1\), the probabilities of these outcomes are \((1-p)^n\) and \(p^n\), respectively, and both tend to zero.
Worked Example: A Mean with an Estimated Variance
Suppose \(X_1,\ldots,X_n\) are independent and identically distributed with finite mean \(\mu\) and finite, strictly positive variance \(\sigma^2\). The central limit theorem gives \[ \sqrt{n}(\bar X-\mu)\xrightarrow{d}N(0,\sigma^2). \] Use the sample variance \(S^2=(n-1)^{-1}\sum_{i=1}^n(X_i-\bar X)^2\) to estimate \(\sigma^2\). Under the stated finite second-moment condition, \(S^2\xrightarrow{p}\sigma^2\): write \[ S^2=\frac{n}{n-1}\left(\frac{1}{n}\sum_{i=1}^nX_i^2-\bar X^2\right). \] The law of large numbers gives \(n^{-1}\sum X_i^2\xrightarrow{p}E[X_1^2]\) and \(\bar X\xrightarrow{p}\mu\); hence the expression in parentheses converges in probability to \(E[X_1^2]-\mu^2=\sigma^2\), while \(n/(n-1)\to1\). The studentized statistic therefore converges to \(N(0,1)\).
For example, if \(n=25\), \(\bar X=10.4\), and \(S=2.5\), then the estimated standard error is \(S/\sqrt n=2.5/5=0.5\). With \(z_{0.975}=1.96\), the margin is \(1.96(0.5)=0.98\), giving the approximate 95% interval \[ [10.4-0.98,\ 10.4+0.98]=[9.42,\ 11.38]. \] The calculation uses the sample standard deviation \(S\), not the sample variance \(S^2\), in the standard error.
Common Pitfalls and Practical Meaning
- Using the wrong scale. If \(\hat v\) estimates the asymptotic variance of \(\sqrt n(\hat{\theta}-\theta)\), the standard error is \(\sqrt{\hat v/n}\), not \(\sqrt{\hat v}\).
- Assuming any plug-in estimate is consistent. A variance estimate must converge to the same positive limiting variance in the normal approximation. A formula that looks plausible is not enough; establish consistency under the model’s conditions.
- Ignoring an undefined denominator. A zero or negative variance estimate may prevent the statistic from being defined. The theorem handles this only when the probability of that event tends to zero. If a proposed variance estimate is negative with nonvanishing probability, the usual real-valued studentized statistic is undefined on a set that is not asymptotically negligible.
- Treating the interval as exact. The normal critical value yields an asymptotic approximation. Its accuracy in a particular finite sample depends on the model and the sample size.
The central practical advantage of studentization is that it replaces an unknown asymptotic scale with a data-based estimate while preserving the standard normal limit, provided the estimate is consistent and the limiting variance is positive. This is what allows the same normal critical values to be used across models with different variance formulas.
Check Your Understanding
Use the conditions and calculations in this tutorial to answer the following questions.
- If \(\sqrt n(\hat{\theta}-\theta)\xrightarrow{d}N(0,v)\), what does \(\hat v\) need to converge to for studentization?
- Why does \(v>0\) matter when applying the continuous mapping theorem to \(1/\sqrt{\hat v}\)?
- For a Poisson sample with rate \(\lambda>0\), what is the plug-in estimate of the asymptotic variance of \(\sqrt n(\bar X-\lambda)\)?
- In the Bernoulli example, explain why \(\hat p(1-\hat p)\) consistently estimates \(p(1-p)\).
- When can a zero estimated variance be ignored in an asymptotic argument, and what changes if the statistic is undefined with nonvanishing probability?