Ordinary t-Tests and Their Small-Sample Instability

Why Ordinary Gene-Wise Tests Can Be Unstable

A gene-wise linear model turns expression measurements into an estimate of a gene-specific effect and its uncertainty. The ordinary t-test uses a residual variance estimate from that gene alone. When the fitted model leaves only a few residual degrees of freedom, that estimate can be highly variable: a modest change in the observed residuals can substantially change the standard error, the t-statistic, and the resulting p-value.

The earlier material on residual variance and residual degrees of freedom defines the residual sum of squares and the degrees of freedom available to estimate variance. Here the focus is how those quantities enter the ordinary t-test, what assumptions justify its reference distribution, and why few residual degrees of freedom make inference fragile.

Model and Ordinary t-Statistic

For gene $g$, let $y_g$ be the $n$-vector of observed expression values, and let $X$ be the $n$ by $p$ design matrix. The gene-wise linear model is

y_g = X\beta_g + \varepsilon_g.

Here $\beta_g$ is the vector of gene-specific coefficients. The usual ordinary t-test assumes that $X$ has full column rank, the errors are independent and normally distributed with mean zero and a common variance for this gene, and the design is treated as fixed. In notation, $\varepsilon_g \sim N(0,\sigma_g^2 I)$. The contrast $c^\mathsf{T}\beta_g$ must be estimable; for a full-rank $X$, this holds for any specified $p$-vector $c$. The null hypothesis tests whether that contrast equals a specified value $\delta_0$, commonly zero.

Definition: The ordinary gene-wise t-statistic for testing $H_0:c^\mathsf{T}\beta_g=\delta_0$ is $$ t_g = \frac{c^\mathsf{T}\hat{\beta}_g-\delta_0}{s_g\sqrt{c^\mathsf{T}(X^\mathsf{T}X)^{-1}c}}, \qquad s_g^2=\frac{\mathrm{RSS}_g}{df}, \qquad df=n-p. $$ Here $\hat{\beta}_g$ is the least-squares estimate, $\mathrm{RSS}_g$ is the residual sum of squares, and $s_g^2$ estimates the gene’s residual variance. Under the stated model, with $n>p$, $c\ne 0$, and $H_0$, $t_g$ has a t distribution with $df$ degrees of freedom.

The square-root factor describes how the design scales the uncertainty in the contrast. Multiplying it by $s_g$ gives the estimated standard error, or $SE$, of $c^\mathsf{T}\hat{\beta}_g$. The numerator is the estimated contrast minus its null value. Thus the statistic measures the observed departure from the null in estimated standard-error units.

Why the Reference Distribution Is a t Distribution

A short derivation shows where the t distribution comes from. Write $H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$ for the projection onto the columns of $X$, and $M=I-H$ for the residual projection. Least squares gives $\hat{\beta}_g=(X^\mathsf{T}X)^{-1}X^\mathsf{T}y_g$ and $\mathrm{RSS}_g=y_g^\mathsf{T}My_g$. Under the normal model, the contrast estimate is normal:

c^\mathsf{T}\hat{\beta}_g \sim N\left(c^\mathsf{T}\beta_g,\ \sigma_g^2 c^\mathsf{T}(X^\mathsf{T}X)^{-1}c\right).

The residual projection has rank $df=n-p$. Normality and the orthogonality of the fitted and residual projections imply $\mathrm{RSS}_g/\sigma_g^2\sim\chi^2_{df}$, independently of the coefficient estimate. Therefore $s_g^2/\sigma_g^2\sim\chi^2_{df}/df$. Under $H_0$, the contrast estimate minus $\delta_0$, divided by its true standard deviation, is standard normal. Dividing that normal quantity by the square root of an independent chi-squared quantity divided by its degrees of freedom gives a t random variable. This establishes the stated reference distribution, rather than merely treating it as a large-sample approximation.

If normality or independence fails, this exact finite-sample result need not hold. A t-shaped reference distribution can still be a useful approximation in some settings, but it is not guaranteed by the formula alone. The ordinary test also relies on the model’s variance structure being appropriate: the same $\sigma_g^2$ applies to all observations for that gene.

Worked Example: A Two-Group Gene-Wise Contrast

Suppose a hypothetical microarray experiment measures one gene in three control samples and three treated samples. The expression values, in log-expression units, are 7.8, 8.2, 8.6 for controls and 6.4, 6.8, 7.2 for treated samples. Use a model with an intercept and a treatment indicator, and test the treated-minus-control contrast against zero.

The group means are $\bar{x}_{C}=8.2$ and $\bar{x}_{T}=6.8$, so the estimated treated-minus-control contrast is $6.8-8.2=-1.4$ log-expression units. The residual sum of squares is the sum of within-group squared deviations: $0.4^2+0^2+0.4^2+0.4^2+0^2+0.4^2=0.64$. There are $n=6$ observations and $p=2$ fitted coefficients, so $df=6-2=4$. Thus

s^2=\frac{0.64}{4}=0.16, \qquad SE(\bar{x}_{T}-\bar{x}_{C}) =\sqrt{0.16\left(\frac{1}{3}+\frac{1}{3}\right)} =0.3266.

The ordinary statistic is $t=(-1.4-0)/0.3266=-4.287$, with 4 degrees of freedom. Its two-sided p-value is approximately 0.0128. Under the assumed model and the null of no mean difference, this is the probability of a t statistic at least as extreme in absolute value as 4.287. It is not the probability that the null hypothesis is true. The estimated difference is in log-expression units; this test alone does not establish a causal treatment effect.

For this example, exact inference requires independent, normally distributed errors with a common variance across the six samples for this gene, plus a correctly specified group model. These assumptions are not established merely because the arithmetic can be completed. With only three samples in each group, residual diagnostics have limited ability to detect departures from them.

Why Few Residual Degrees of Freedom Matter

The ordinary variance estimate is unbiased under the model, but unbiasedness does not mean precision. Since $df\,s_g^2/\sigma_g^2\sim\chi^2_{df}$, the variance of the estimate is

\operatorname{Var}(s_g^2)=\frac{2\sigma_g^4}{df}, \qquad \frac{\operatorname{SD}(s_g^2)}{\sigma_g^2}=\sqrt{\frac{2}{df}}.

The second expression is the relative standard deviation of the variance estimate. It is about 0.7071 when $df=4$, compared with about 0.3162 when $df=20$. These values do not say that a particular estimate must be inaccurate; they quantify the substantial sampling variability expected with few degrees of freedom.

Worked Example: How Often a Variance Estimate Can Be Far From Its Target

Consider a hypothetical gene whose model satisfies the normal assumptions and has $df=4$. The scaled variance estimate $4s_g^2/\sigma_g^2$ follows a chi-squared distribution with 4 degrees of freedom. The probability that the estimate exceeds twice the true variance is

P(s_g^2>2\sigma_g^2) =P(\chi^2_4>8) =e^{-8/2}\left(1+\frac{8}{2}\right) =5e^{-4} \approx 0.09158.

The probability that it is below one quarter of the true variance is

P(s_g^2<0.25\sigma_g^2) =P(\chi^2_4<1) =1-e^{-1/2}\left(1+\frac{1}{2}\right) \approx 0.09020.

So even under the model, either a more-than-twofold high estimate or a less-than-quarter-sized estimate has probability near 9% in this example. The resulting standard error is proportional to $s_g$, not $s_g^2$, but a large or small variance estimate still changes the t-statistic and can strongly affect the p-value. This is sampling variability, not evidence that every gene’s variance is wrong in the same direction.

Degrees of Freedom Change the p-Value

A t statistic is interpreted relative to its degrees of freedom. Small $df$ gives a heavier-tailed reference distribution than large $df$, reflecting the uncertainty from estimating the variance. Thus the same numerical t statistic need not provide the same evidence against its null hypothesis when degrees of freedom differ.

Worked Example: The Same Statistic With Different Degrees of Freedom

Suppose a hypothetical gene has an estimated contrast of 0.30 log-expression units and an estimated standard error of 0.1225 log-expression units. Testing a zero contrast gives $t=0.30/0.1225\approx2.449$. If this test has $df=20$, its two-sided p-value is approximately 0.02367. If the same statistic is instead evaluated with $df=4$, the two-sided p-value is approximately 0.0704.

The arithmetic ratio is the same, but the reference distributions differ. At $df=20$, the observed statistic is less compatible with the null t distribution than it is at $df=4$, so the p-value is smaller. The p-value is a tail probability under a specified null model; it is not a direct measure of effect size. Here the effect estimate remains 0.30 log-expression units in both comparisons.

In practice, the degrees of freedom arise from the design and fitted model; one does not choose whichever value gives the preferred result. This comparison isolates why it is not enough to report a t-statistic without its degrees of freedom or p-value.

Diagnostics, Alternatives, and Limits

A small residual degrees of freedom count is a warning about uncertainty, not a diagnosis that the model is false. Check the design and residuals for issues that can make the ordinary t-test unreliable: incorrectly encoded groups or covariates, dependence among samples treated as independent, influential outliers, and patterns of residual spread inconsistent with a common variance. Normality is especially difficult to assess from a handful of residuals, so a plot that looks unremarkable with very few observations cannot verify the assumption.

Ordinary gene-wise tests are simple and interpretable: each gene’s residual variance is estimated from its own residuals, and the coefficient contrast is tested using the corresponding t distribution. A larger, well-designed sample can provide more residual degrees of freedom and more precise variance estimates. In a two-group setting with unequal group variances, a Welch test is a familiar alternative, but it is not the same as the common-variance linear model above. In complex designs, the model and its contrast should represent the sampling structure rather than forcing a simple two-sample test onto the data.

The ordinary procedure has an important limitation in microarray analysis: it estimates a separate variance for each gene, often with little information per gene. A very small estimated variance can make a t-statistic unusually large, while a large one can make it unusually small. A nominal gene-wise p-value also does not, by itself, control error across the many genes tested. These issues motivate careful variance assessment and appropriate genome-wide interpretation; they do not make the ordinary statistic’s derivation incorrect when its assumptions hold.

Common pitfall: “The estimated variance is unbiased, so with only a few residual degrees of freedom it is reliable.” This conclusion confuses unbiasedness with precision. Under the model, $E(s_g^2)=\sigma_g^2$, but $\operatorname{Var}(s_g^2)=2\sigma_g^4/df$; a small $df$ means substantial sampling variability. Report and interpret the degrees of freedom along with the statistic, and examine whether the model assumptions are plausible.

Key takeaway: An ordinary gene-wise t-test divides a gene’s estimated contrast by its gene-specific residual standard error. Its exact t reference distribution depends on a correctly specified normal linear model, and few residual degrees of freedom make the estimated variance—and therefore the inference—unstable.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A gene model has $n=12$ observations and 3 full-rank fitted coefficients. What are its residual degrees of freedom?
For a test of $c^\mathsf{T}\beta_g=\delta_0$, identify the numerator and denominator of the ordinary t-statistic.
Under the normal linear model, what distribution does $df\,s_g^2/\sigma_g^2$ follow?
Why can the same t-statistic have a larger two-sided p-value with fewer degrees of freedom?
Does an unbiased residual variance estimate necessarily have low variability when $df=2$? Explain using the relative standard deviation.

Solutions. (1) $df=n-p=12-3=9$. (2) The numerator is $c^\mathsf{T}\hat{\beta}_g-\delta_0$; the denominator is $s_g\sqrt{c^\mathsf{T}(X^\mathsf{T}X)^{-1}c}$, the estimated standard error of the contrast. (3) It follows a chi-squared distribution with $df$ degrees of freedom. (4) The t distribution has heavier tails at smaller degrees of freedom, so the same absolute statistic has more probability in its two-sided tail. (5) No. The relative standard deviation is $\sqrt{2/df}=\sqrt{2/2}=1$, so the standard deviation of $s_g^2$ equals its mean under the model; unbiasedness does not imply low variability.