Deriving the Standard Error of a Sample Proportion

Why Derive the Standard Error?

A sample proportion is not fixed across repeated samples. If a clinic surveys a different set of patients each time, the fraction reporting a particular symptom will generally vary, even when the underlying chance of reporting it stays the same. The standard error describes the standard deviation of that repeated-sample variation. Deriving it shows exactly which assumptions make the familiar formula valid.

The result in this tutorial applies when the sample size is fixed and the observations are independent, with the same probability $p$ of success for every observation. “Success” simply means the outcome being counted, such as reporting a symptom or meeting a specified criterion. The outcome for each observation has only two possible values: success or failure.

Definition: For a fixed sample size $n$, let $Y_i=1$ if observation $i$ is a success and $Y_i=0$ otherwise. The sample proportion is the number of successes divided by the sample size: $$ \hat{p}=\frac{Y_1+Y_2+\cdots+Y_n}{n}. $$ If the $Y_i$ are independent and each has success probability $p$, the standard error of $\hat{p}$ is the standard deviation of its sampling distribution under this model.

This notation makes the count explicit: each $Y_i$ contributes one to the total if it is a success and zero if it is a failure. Thus, the sum in the numerator is exactly the number of successes. The parameter $p$ describes the success probability in the model; $\hat{p}$ is the proportion observed in one particular sample. They are not interchangeable.

Start With One Observation

First find the variance of a single success-or-failure outcome $Y_i$. Since $Y_i$ equals one with probability $p$ and zero with probability $1-p$, its expected value is

E[Y_i]=1\cdot p+0\cdot(1-p)=p.

For either possible value, $Y_i^2=Y_i$, so $E[Y_i^2]=p$ as well. Using the variance identity $\operatorname{Var}(Y_i)=E[Y_i^2]-(E[Y_i])^2$, we obtain

\operatorname{Var}(Y_i)=p-p^2=p(1-p).

The variance is largest when $p=0.5$, when success and failure are equally likely. It is zero at $p=0$ or $p=1$, because then the outcome never varies. This is the variance of one binary outcome, not yet the variance of the sample proportion.

Worked Example: Variance of One Binary Outcome

Suppose, hypothetically, that each patient independently has probability $p=0.30$ of reporting a specified symptom. Let $Y$ be one patient's outcome, coded as 1 for reporting the symptom and 0 otherwise. Find its expected value and variance.

The expected value is $E[Y]=p=0.30$. Since $Y^2=Y$, its second moment is also $E[Y^2]=0.30$. Therefore,

\operatorname{Var}(Y)=E[Y^2]-(E[Y])^2 =0.30-(0.30)^2 =0.30-0.09 =0.21.

The outcome is measured in success-or-failure units, so its variance is $0.21$ in squared outcome units. This calculation concerns one patient; combining $n$ independent patients and dividing by $n$ changes the variance.

From Individual Outcomes to the Sample Proportion

The sample proportion is the sum of the $n$ binary outcomes divided by $n$. Two variance facts give the derivation: multiplying a random variable by a constant $c$ multiplies its variance by $c^2$; and the variance of a sum of independent random variables is the sum of their variances. Independence matters here because, in general, the variance of a sum also includes covariance terms.

Theorem: If $Y_1,\ldots,Y_n$ are independent binary outcomes, each with probability $p$ of success, then $$ \operatorname{Var}(\hat{p})=\frac{p(1-p)}{n}, \qquad SE(\hat{p})=\sqrt{\frac{p(1-p)}{n}}. $$ The first expression is the variance of the sampling distribution; the second is its standard deviation.

Here is the variance calculation in full. Write $\hat{p}$ as the sum of the binary outcomes divided by $n$. Scaling the sum by $1/n$ scales its variance by $1/n^2$. Because the observations are independent, the variance of the sum is the sum of their variances. Each of those variances is $p(1-p)$, as derived above.

\begin{aligned} \operatorname{Var}(\hat{p}) &=\operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^{n}Y_i\right)\\ &=\frac{1}{n^2}\operatorname{Var}\left(\sum_{i=1}^{n}Y_i\right)\\ &=\frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(Y_i)\\ &=\frac{1}{n^2}\bigl(np(1-p)\bigr)\\ &=\frac{p(1-p)}{n}. \end{aligned}

Taking the nonnegative square root of this variance gives the standard error:

SE(\hat{p})=\sqrt{\operatorname{Var}(\hat{p})} =\sqrt{\frac{p(1-p)}{n}}.

The independence step can also be seen by expanding the variance of a sum. For two distinct observations $i$ and $j$, independence gives $E[Y_iY_j]=E[Y_i]E[Y_j]$. Consequently, their covariance, $E[Y_iY_j]-E[Y_i]E[Y_j]$, is zero. Every cross-term in the expanded variance therefore vanishes, leaving only the $n$ individual variances. If observations are dependent, those cross-terms need not vanish, and this derivation does not justify the usual formula.

Model one outcome.
Represent each observation as 1 for success and 0 for failure, with success probability $p$.

Find its variance.
A single binary outcome has variance $p(1-p)$.

Combine independent outcomes.
The variance of their sum is $np(1-p)$, because the $n$ variances add.

Divide by the sample size.
The sum is divided by $n$ to form $\hat{p}$, so its variance is divided by $n^2$.

Take the square root.
The standard error is $\sqrt{p(1-p)/n}$.

Using the Formula in Context

The formula uses the model's probability $p$, which may be specified in a hypothetical calculation or known under a stated model. In many applications $p$ is unknown. Then the formula still describes the theoretical standard error, but its numerical value cannot be calculated exactly from the model without knowing $p$. A common estimate substitutes the observed sample proportion $\hat{p}$ for $p$, giving $\sqrt{\hat{p}(1-\hat{p})/n}$. This is an estimated standard error, not the exact theoretical standard error.

Worked Example: Standard Error for a Hypothetical Symptom Survey

Suppose a hypothetical model says that each of $n=100$ independently surveyed patients has probability $p=0.30$ of reporting a symptom. Find the variance and standard error of the sample proportion under this model.

The assumptions match the formula: the sample size is fixed at 100, each outcome is success or failure, each patient has the same probability $0.30$ of success, and the outcomes are independent. Substitute $p=0.30$ and $n=100$:

\operatorname{Var}(\hat{p}) =\frac{0.30(1-0.30)}{100} =\frac{0.30(0.70)}{100} =\frac{0.21}{100} =0.002100.

The standard error is the square root of this variance:

SE(\hat{p})=\sqrt{0.002100}=0.04583.

Thus the standard deviation of the sample proportion's repeated-sample distribution is $0.04583$ in proportion units, or about $4.583$ percentage points. This describes the model-based spread; it does not claim that every sample proportion will be exactly that far from $p$.

Worked Example: A Less Common Success

In another hypothetical setting, each of $n=50$ independent observations has probability $p=0.08$ of meeting a specified criterion. Calculate the variance and standard error of the sample proportion.

The model has a fixed sample size, a common success probability, and independent binary outcomes. The variance is

\operatorname{Var}(\hat{p}) =\frac{0.08(1-0.08)}{50} =\frac{0.08(0.92)}{50} =\frac{0.0736}{50} =0.001472.

Therefore,

SE(\hat{p})=\sqrt{0.001472}=0.03837.

The standard error is about $0.03837$, or $3.837$ percentage points. The value of $p$ being relatively small does not make the standard error zero: both success and failure remain possible, so individual outcomes still vary.

Worked Example: Estimating the Standard Error From a Sample

Suppose a hypothetical survey of $n=80$ independent people finds that 22 meet a stated criterion. The underlying success probability is not supplied. Calculate the sample proportion and the estimated standard error.

The observed proportion is

\hat{p}=\frac{22}{80}=0.275.

Because the true $p$ is unknown, use $\hat{p}=0.275$ in the standard-error formula to estimate the model-based standard error:

\widehat{SE}(\hat{p}) =\sqrt{\frac{0.275(1-0.275)}{80}} =\sqrt{\frac{0.275(0.725)}{80}} =\sqrt{0.002492} =0.04992.

The estimated standard error is $0.04992$ in proportion units, or about $4.992$ percentage points. It is an estimate because it uses the sample to stand in for the unknown success probability. The exact theoretical standard error would require the actual value of $p$.

Assumptions and a Common Mistake

The formula is not a property of every proportion calculated from data. Its derivation depends on the model: a fixed number $n$ of observations, binary outcomes, a common success probability $p$, and independence across observations. If an observation's success probability differs from the others, or if outcomes are related, the individual variances or covariance terms can differ from those used in the derivation. One should check the sampling or data-generating setup before applying the formula.

Common mistake: using $p(1-p)/n$ as the standard error. This expression is the variance, not the standard error. For example, when $p=0.30$ and $n=100$, the variance is $0.002100$, while the standard error is its square root, $0.04583$. Reporting $0.002100$ as the standard error confuses squared proportion units with proportion units.

Another related error is to use the formula without accounting for dependence. If outcomes tend to occur together, the covariance terms in the variance of the sum may be positive; if they tend to offset one another, they may be negative. Independence is what allows those terms to be set to zero. The formula is therefore a result of both the binary-outcome model and the independence assumption.

Key takeaway: For independent binary observations with a common success probability $p$, each outcome has variance $p(1-p)$. Averaging $n$ such outcomes gives a sample-proportion variance of $p(1-p)/n$, and its standard error is $\sqrt{p(1-p)/n}$. When $p$ is unknown, replacing it with $\hat{p}$ gives an estimated standard error.

Check Your Understanding

Use the assumptions and derivation above to answer these questions.

For one binary outcome with success probability $p$, explain why $E[Y^2]=p$, and use this to obtain $\operatorname{Var}(Y)$.
For independent observations with $p=0.40$ and $n=25$, calculate the variance and standard error of the sample proportion. Include the square-root step.
Which assumptions allow the variance of the sum of the outcomes to equal the sum of their variances?
A sample of 60 independent observations contains 18 successes, and the model's $p$ is unknown. Find the estimated standard error and explain why it is an estimate.
Why is $p(1-p)/n$ not itself the standard error? State the units distinction in words.