Penalty Calibration With Unknown Noise Variance

Why Unknown Variance Changes the Penalty

In Gaussian linear regression, a dimension penalty is measured in units of the error variance. If that variance is unknown, replacing it with an estimate is not just a numerical convenience: the penalty must still be large enough when the estimate is too small. That matters especially when the estimate is computed from the same data used to select a model.

A safe construction is available when the candidate models lie inside a known containing linear model. Residuals orthogonal to that containing model estimate the noise variance without contamination from its mean. We use a one-sided upper confidence bound from those residuals, then prove a selection guarantee for a finite collection of candidate subspaces. The guarantee is deliberately conservative; it illustrates how variance uncertainty enters calibration rather than prescribing a uniquely optimal constant.

Setup and Definitions

Let $y=\mu+\varepsilon\in\mathbb{R}^n$, where the design is fixed and $\varepsilon\sim N(0,\sigma^2 I_n)$ for an unknown $\sigma^2>0$. The mean vector $\mu$ is fixed. For each candidate model $m$ in a finite collection $\mathcal M$, let $S_m\subseteq\mathbb{R}^n$ be its fitted-value subspace, $P_m$ its orthogonal projection, and $d_m=\dim(S_m)$. Its residual sum of squares is $\operatorname{RSS}_m=\|(I-P_m)y\|^2$. Write $a_m=\|(I-P_m)\mu\|^2$ for its squared approximation error.

Definition: A containing model is a known subspace $S_0$ such that $S_m\subseteq S_0$ for every candidate $m$. Its dimension is $d_0$, and its residual degrees of freedom are $\nu=n-d_0$. For variance calibration below, assume $\nu>0$ and $\mu\in S_0$.

The condition $\mu\in S_0$ makes the residual in the containing model pure noise: $(I-P_0)y=(I-P_0)\varepsilon$. If the condition fails, that residual also contains a mean component, and the confidence statement below need not hold.

An Upper Confidence Bound for the Variance

Let $R_0=\|(I-P_0)y\|^2$. For $0<\alpha<1$, put $x_\alpha=\log(1/\alpha)$ and define

\widehat{\sigma}^{\,2}_{U} = \frac{R_0}{\nu-2\sqrt{\nu x_\alpha}}, \qquad \text{provided }\nu-2\sqrt{\nu x_\alpha}>0.

The denominator is positive when $\nu>4x_\alpha$. This choice uses a convenient concentration bound rather than an exact chi-square quantile.

Result: Under the Gaussian model and $\mu\in S_0$, $P(\sigma^2\leq\widehat{\sigma}^{\,2}_{U})\geq1-\alpha$, whenever the denominator above is positive.

To verify the claim, choose an orthonormal basis for the $\nu$-dimensional orthogonal complement of $S_0$. The coordinates of $(I-P_0)\varepsilon/\sigma$ in that basis are independent standard normal variables. Therefore $R_0/\sigma^2\sim\chi^2_\nu$. The lower-tail chi-square inequality is $P(\chi^2_\nu\leq \nu-2\sqrt{\nu x})\leq e^{-x}$ for $x>0$ when the displayed threshold is positive. With $x=x_\alpha$, it follows that $P(R_0/\sigma^2\geq\nu-2\sqrt{\nu x_\alpha})\geq1-\alpha$. Rearranging this event gives the stated upper bound.

Worked Example: Computing a Residual-Based Upper Bound

Suppose $\nu=1000$, $R_0=4000$, and $\alpha=0.05$. Then $x_\alpha=\log(20)\approx2.995732$. The denominator is

1000-2\sqrt{1000(2.995732)} =1000-2\sqrt{2995.732} \approx 890.533439 \approx 890.533.

Thus

\widehat{\sigma}^{\,2}_{U} =\frac{4000}{890.533439} \approx4.491.

The denominator is positive, as required. Under the containing-model assumptions, this construction gives at least 95% coverage for $\sigma^2$; it does not assert that the unknown variance equals 4.491.

Dimension Penalties and a Selection Guarantee

Assign each candidate a weight $\pi_m>0$, with $\sum_{m\in\mathcal M}\pi_m\leq1$. The weights encode relative complexity: a model given a smaller weight pays a larger penalty. For a chosen $0<\delta<1$, set

u_m=\log\!\left(\frac{4}{\delta\pi_m}\right), \qquad D_m=d_m+u_m.

Consider the selector $\hat m$ minimizing the penalized residual sum of squares

\operatorname{RSS}_m+10\widehat{\sigma}^{\,2}_{U}D_m, \qquad m\in\mathcal M.

The numerical constant 10 is a convenient conservative choice for the proof below. Ties can be broken in any fixed way.

Theorem: Assume the Gaussian model, $\mu\in S_0$, $\nu>0$, and $\nu-2\sqrt{\nu\log(1/\alpha)}>0$. With probability at least $1-\alpha-\delta$, the selector above satisfies, simultaneously for every $m\in\mathcal M$, $$ a_{\hat m}+\sigma^2d_{\hat m} \leq 3a_m+20\widehat{\sigma}^{\,2}_{U}(d_m+u_m). $$

The left side combines the selected model's approximation error and a dimension-scaled noise cost. The right side compares it with any candidate's approximation error and a penalty that accounts for both dimension and the finite collection's complexity.

Proof of the Selection Guarantee

We first establish simultaneous noise bounds. For $Z\sim N(0,I_d)$, the chi-square concentration inequalities give $P(|\|Z\|^2-d|>2\sqrt{du}+2u)\leq2e^{-u}$. Since $2\sqrt{du}\leq d+u$, the deviation is at most $d+3u$ except on an event of probability at most $2e^{-u}$. Also, for a fixed vector $r$, $\langle r,\varepsilon\rangle\sim N(0,\sigma^2\|r\|^2)$, so the Gaussian tail bound implies $P(|2\langle r,\varepsilon\rangle|>\|r\|^2/2+4\sigma^2u)\leq2e^{-u}$. The latter follows because the threshold is at least $\sqrt{8\sigma^2\|r\|^2u}$, by $b/2+4c\geq\sqrt{8bc}$ for $b,c\geq0$.

Apply these inequalities to $P_m\varepsilon$, whose squared norm divided by $\sigma^2$ has a chi-square distribution with $d_m$ degrees of freedom, and to $r_m=(I-P_m)\mu$. For each $m$, the probability either bound fails is at most $4e^{-u_m}=\delta\pi_m$. A union bound and $\sum_m\pi_m\leq1$ show that both bounds hold for every candidate with probability at least $1-\delta$. Intersect this event with $\{\sigma^2\leq\widehat{\sigma}^{\,2}_{U}\}$, whose probability is at least $1-\alpha$. The intersection has probability at least $1-\alpha-\delta$; independence is not needed.

On this intersection, expand $\operatorname{RSS}_m-\|\varepsilon\|^2$. Since $y=\mu+\varepsilon$ and $P_m$ is an orthogonal projection,

\operatorname{RSS}_m-\|\varepsilon\|^2 =a_m-\|P_m\varepsilon\|^2-2\langle r_m,\varepsilon\rangle.

The simultaneous bounds therefore give

\frac{a_m}{2}-2\sigma^2d_m-7\sigma^2u_m \leq \operatorname{RSS}_m-\|\varepsilon\|^2 \leq \frac{3a_m}{2}+7\sigma^2u_m.

For the lower bound, use $\|P_m\varepsilon\|^2\leq\sigma^2d_m+\sigma^2(d_m+3u_m)$ and $2|\langle r_m,\varepsilon\rangle|\leq a_m/2+4\sigma^2u_m$. For the upper bound, use $\|P_m\varepsilon\|^2\geq\sigma^2d_m-\sigma^2(d_m+3u_m)$ and the same cross-term bound. These substitutions yield the displayed inequalities directly.

By the definition of $\hat m$, its penalized criterion is no greater than that of any fixed $m$. Subtract the common $\|\varepsilon\|^2$, use the lower bound for $\hat m$ and the upper bound for $m$, and then use $\widehat{\sigma}^{\,2}_{U}\geq\sigma^2$. This gives

\frac{a_{\hat m}}2+8\sigma^2d_{\hat m}+3\sigma^2u_{\hat m} \leq \frac{3a_m}{2}+7\sigma^2u_m+ 10\widehat{\sigma}^{\,2}_{U}(d_m+u_m).

Multiply by 2 and drop nonnegative terms on the left. Since $14\sigma^2u_m\leq20\widehat{\sigma}^{\,2}_{U}(d_m+u_m)$, the result is $a_{\hat m}+\sigma^2d_{\hat m}\leq3a_m+20\widehat{\sigma}^{\,2}_{U}(d_m+u_m)$. This holds for every comparator $m$ on the same event, proving the theorem. $\square$

Worked Example: Evaluating the Complexity Part of the Penalty

Suppose there are four equally weighted candidates, $\delta=0.05$, and the upper variance estimate in the previous example is $\widehat{\sigma}^{\,2}_{U}=4.491$. Each weight is $\pi_m=1/4$, so

u_m=\log\!\left(\frac{4}{0.05(1/4)}\right) =\log(320)\approx5.768.

For a candidate of dimension $d_m=2$, $D_m=2+5.768=7.768$. Its penalty is

10(4.491)(7.768)\approx348.9.

This is a penalty on the residual-sum-of-squares scale. It is large because the theorem uses a conservative constant and includes a high-probability complexity term; it should not be mistaken for a claim that every practical implementation needs this exact penalty.

Worked Example: How the Variance Estimate Can Change the Selected Dimension

Consider two candidates with dimensions 2 and 5 and residual sums of squares 420 and 390, respectively. Suppose they have equal weights and the same confidence level as above, so $u_m=\log(320)$ for both. The common $u_m$ contribution cancels when comparing their criteria. The dimension-5 criterion minus the dimension-2 criterion is

$$ (390-420)+10(4.491)(5-2) =-30+134.73 =104.73. $$

The penalized criterion therefore selects the dimension-2 candidate, despite its larger residual sum of squares. With a smaller variance scale, the dimension increment would cost less. This calculation illustrates why a plug-in variance estimate that is too small can weaken the complexity penalty and change the selection.

What the Guarantee Does—and Does Not—Say

The theorem compares the selected model with every candidate in a finite family, on one event whose probability is controlled by $\alpha+\delta$. Its benchmark $a_m+\sigma^2d_m$ is an approximation-plus-complexity quantity for fixed design. The result is not a guarantee of recovering a uniquely correct model, and it does not make a claim about a collection whose size grows without adjusting its weights or confidence level.

The containing-model assumption is essential to this particular variance calibration. If $\mu\notin S_0$, then $R_0/\nu$ includes squared residual mean as well as noise and is not the chi-square variance estimate used in the proof. Likewise, estimating variance from the residuals of the selected model can produce a downward-biased scale: the selection has already favored models with unusually small residuals. Substituting that estimate into the penalty and treating it as an upper bound has no justification from the argument above.

Safe interpretation: on the stated coverage event, the calibrated penalty uses a scale at least as large as the true variance.
Important limitation: if no suitable containing model is available, a different variance-estimation argument is required.
Practical distinction: the theorem establishes a transparent sufficient calibration; sharper penalties may be possible under additional analysis.

Check Your Understanding

Use the assumptions and proof above to answer these questions.

Why does $\mu\in S_0$ make the containing-model residual sum of squares a variance estimator?
What must be true of $\nu-2\sqrt{\nu\log(1/\alpha)}$ for the stated upper bound to be defined?
How does decreasing a model weight $\pi_m$ affect its complexity penalty?
Why does the simultaneous selection guarantee not require the variance estimate and candidate-model noise terms to be independent?
What part of the variance-calibration argument fails if the containing model does not contain $\mu$?