Scaling And Parameterization In Quadratic Loss

Why Units and Parameterization Matter

A squared error is measured in the square of the units used for the quantity being estimated. An error of $0.1$ metres and an error of $10$ centimetres describe the same discrepancy, but their numerical squares are $0.01$ and $100$, respectively. Risk is the expectation of squared error, so its numerical value changes in the same way. A risk comparison is meaningful only when the losses use compatible units and the same parameterization.

For a positive parameter, one can instead measure error relative to the parameter itself. This produces a dimensionless loss that is unchanged by a conversion between units. It is important, however, to distinguish relative squared error from ordinary squared error after a nonlinear transformation such as taking logarithms. Those are different losses and can rank estimators differently.

Squared-Error Risk Under a Change of Units

Definition: In a model indexed by $\theta$, an estimator $\hat{\theta}=\delta(X)$ has squared-error loss $L(\theta,\hat{\theta})=(\hat{\theta}-\theta)^2$ and risk $$ R(\theta,\delta)=E_{\theta}\!\left[(\delta(X)-\theta)^2\right], $$ provided the expectation exists. Here $E_{\theta}$ denotes expectation when the data have the distribution specified by $\theta$.

Suppose a quantity is expressed in new units by the affine conversion $\eta=a\theta+b$, where $a\ne 0$. The corresponding estimator is $\hat{\eta}=a\hat{\theta}+b$. The offset cancels when taking an estimation error:

\hat{\eta}-\eta =(a\hat{\theta}+b)-(a\theta+b) =a(\hat{\theta}-\theta).

Thus the error is multiplied by $a$, and its square by $a^2$. For example, changing from Celsius to Fahrenheit multiplies squared errors by $(9/5)^2=81/25$; the Fahrenheit offset does not affect the error. The same argument applies to a pure change of units, for which $b=0$.

Theorem: Let the model for $X$ be indexed by $\theta$, and let $\hat{\theta}=\delta(X)$ be a measurable estimator with finite squared-error risk. Under the corresponding parameterization $\eta=a\theta+b$, with $a\ne0$, use the estimator $\hat{\eta}=a\delta(X)+b$ and squared-error loss in $\eta$-units. Then $$ R_{\eta}(\eta,\hat{\eta})=a^2R_{\theta}(\theta,\delta). $$

Proof. At each fixed $\theta$, the transformed estimator has error $a(\delta(X)-\theta)$. Squaring and taking expectation under the same data distribution gives

R_{\eta}(\eta,\hat{\eta}) =E_{\theta}\!\left[\{a\delta(X)+b-(a\theta+b)\}^2\right] =E_{\theta}\!\left[a^2\{\delta(X)-\theta\}^2\right] =a^2R_{\theta}(\theta,\delta).

The finite-risk assumption ensures these expectations are defined and finite. Since $a^2$ is a positive constant shared by all estimators being compared under this conversion, their risk ordering at corresponding parameter values is unchanged. The numerical risks differ because they are expressed in different squared units. $\square$

Worked Example: Converting Risk From Metres to Centimetres

A true length is $\theta=1.5$ metres. An estimator reports $1.4$ metres with probability $1/2$ and $1.7$ metres with probability $1/2$. Calculate its squared-error risk in metres squared and then in centimetres squared.

In metres, the two errors are $1.4-1.5=-0.1$ and $1.7-1.5=0.2$. Therefore,

R_{\mathrm{m}}=\frac12(-0.1)^2+\frac12(0.2)^2 =\frac12(0.01)+\frac12(0.04)=0.025\ \mathrm{m}^2.

There are $100$ centimetres in a metre, so $a=100$. The risk in the new units must be multiplied by $100^2$:

R_{\mathrm{cm}}=100^2(0.025)=250\ \mathrm{cm}^2.

Checking directly, the true length is $150$ centimetres, and the estimates are $140$ and $170$ centimetres. The errors are $-10$ and $20$, giving $\frac12(100)+\frac12(400)=250\ \mathrm{cm}^2$, as required.

Relative Squared Error for Positive Parameters

When $\theta>0$, the error can be divided by the true parameter value before it is squared. This compares the discrepancy with the size of the quantity being estimated. The resulting loss has no physical units.

Definition: For a positive parameter $\theta$, the relative squared-error loss of an estimate $d$ is $$ L_{\mathrm{rel}}(\theta,d)=\left(\frac{d-\theta}{\theta}\right)^2, $$ and the relative risk of an estimator $\delta(X)$ is $$ R_{\mathrm{rel}}(\theta,\delta) =E_{\theta}\!\left[\left(\frac{\delta(X)-\theta}{\theta}\right)^2\right], $$ when this expectation exists.

The parameter must be positive so the denominator is nonzero. The estimate itself need not be positive for this loss to be defined. If positive estimates are required by the application, that is an additional restriction on the allowed estimators, not a requirement of the formula.

At any fixed positive $\theta$, the denominator $\theta^2$ is a positive constant. Consequently, relative risk is ordinary squared-error risk divided by $\theta^2$. This gives a simple but useful result.

Theorem: If $\theta>0$ and $R(\theta,\delta)$ is finite, then $$ R_{\mathrm{rel}}(\theta,\delta)=\frac{R(\theta,\delta)}{\theta^2}. $$ For any two estimators with finite risks, their relative-risk ordering at that same $\theta$ is therefore identical to their squared-error-risk ordering.

Proof. Because $\theta$ is fixed when taking expectation under $E_{\theta}$, it can be taken outside the expectation:

R_{\mathrm{rel}}(\theta,\delta) =E_{\theta}\!\left[\frac{\{\delta(X)-\theta\}^2}{\theta^2}\right] =\frac{1}{\theta^2}E_{\theta}\!\left[\{\delta(X)-\theta\}^2\right] =\frac{R(\theta,\delta)}{\theta^2}.

For estimators $\delta_1,\delta_2$, division by the same positive $\theta^2$ preserves the inequality: $R(\theta,\delta_1)\le R(\theta,\delta_2)$ if and only if $R_{\mathrm{rel}}(\theta,\delta_1)\le R_{\mathrm{rel}}(\theta,\delta_2)$. $\square$

This pointwise agreement does not mean the two loss functions answer the same question in every comparison. Across different values of $\theta$, ordinary risk and relative risk weight errors differently: relative risk divides each risk by $\theta^2$. Thus a summary that combines performance across parameter values can depend on which loss is chosen.

Worked Example: Comparing Ordinary and Relative Risk

Suppose the true parameter is $\theta=10$. Estimator A returns $8$ or $12$, each with probability $1/2$. Estimator B always returns $11$. Calculate both risks and compare the estimators under each loss.

For A, the errors are $-2$ and $2$. Its ordinary squared-error risk is

R(10,\mathrm{A})=\frac12(-2)^2+\frac12(2)^2=4.

Its relative errors are $-2/10=-0.2$ and $2/10=0.2$, so

R_{\mathrm{rel}}(10,\mathrm{A}) =\frac12(-0.2)^2+\frac12(0.2)^2 =0.04.

B has error $11-10=1$, giving $R(10,\mathrm{B})=1^2=1$. Its relative error is $1/10=0.1$, giving $R_{\mathrm{rel}}(10,\mathrm{B})=(0.1)^2=0.01$. B has lower risk under both losses, as the theorem predicts: the relative risks are the ordinary risks divided by $10^2=100$.

A change of units leaves relative squared error unchanged. Under $\eta=a\theta$ and the corresponding estimate $\hat{\eta}=a\hat{\theta}$, where $a>0$,

\left(\frac{\hat{\eta}-\eta}{\eta}\right)^2 =\left(\frac{a\hat{\theta}-a\theta}{a\theta}\right)^2 =\left(\frac{\hat{\theta}-\theta}{\theta}\right)^2.

For instance, converting a positive length from metres to centimetres changes ordinary squared-error risk by a factor of $10{,}000$, but does not change relative squared-error loss or risk. This is why relative error is useful when the scale of the quantity varies substantially or when only proportional accuracy is meaningful.

Nonlinear Parameterizations Define Different Losses

A nonlinear change of parameter is not merely a change of units. Suppose $\psi=h(\theta)$ is a one-to-one transformation and the transformed estimate is $h(\hat{\theta})$. Squared error on the transformed scale is

\{h(\hat{\theta})-h(\theta)\}^2.

In general, this is not a constant multiple of $(\hat{\theta}-\theta)^2$. Therefore its risk can change the ranking of estimators. For a positive parameter, taking logarithms is a common example: squared error in $\log\theta$ compares $(\log\hat{\theta}-\log\theta)^2$, which is not the same as relative squared error.

Worked Example: A Logarithmic Parameterization Reverses a Ranking

Let the true positive parameter be $\theta=1$. Consider two deterministic estimates, A with $\hat{\theta}_{\mathrm{A}}=2$ and B with $\hat{\theta}_{\mathrm{B}}=0.1$. Under ordinary squared-error loss, A has loss $(2-1)^2=1$, while B has loss $(0.1-1)^2=0.81$. Thus B is preferred.

Now use the parameterization $\psi=\log\theta$ and compare squared errors on the log scale. The true value is $\log 1=0$. A has transformed loss

(\log 2-\log 1)^2=(\log 2)^2\approx 0.4805.

B has transformed loss

(\log 0.1-\log 1)^2=(\log 0.1)^2\approx 5.3019.

A is now preferred. Both estimates are positive, so both log estimates are defined. The reversal occurs because logarithmic squared error penalizes these discrepancies differently; it is not a unit conversion of ordinary squared error.

For positive $\hat{\theta}$, write its relative error as $u=(\hat{\theta}-\theta)/\theta$. Then $\hat{\theta}/\theta=1+u$, and the log-scale error has the exact form

\log\hat{\theta}-\log\theta=\log(1+u), \qquad u>-1.

When $u$ is close to zero, $\log(1+u)$ is approximately $u$, so log-scale squared error and relative squared error can be similar for small relative discrepancies. They are not identical in general: the logarithm is nonlinear, and large overestimates and underestimates need not receive the same penalties under the two losses.

Common Mistakes and Practical Interpretation

Comparing numerical risks across units without conversion. A risk of $4\ \mathrm{m}^2$ is not numerically comparable to a risk of $100{,}000\ \mathrm{cm}^2$ until one is converted. Since $1\ \mathrm{m}^2=10{,}000\ \mathrm{cm}^2$, these are $4\ \mathrm{m}^2$ and $10\ \mathrm{m}^2$, respectively.
Assuming every reparameterization preserves risk rankings. Affine unit conversions multiply squared error by a common positive constant. A nonlinear transformation generally does not, as the logarithmic example shows.
Claiming relative loss always ranks estimators differently. At a fixed $\theta>0$, relative risk is exactly ordinary risk divided by $\theta^2$, so the rankings agree. The weighting differs when comparing risk across parameter values or when the loss itself is changed to another parameterization.
Calling log-scale squared error relative squared error. The former squares $\log(\hat{\theta}/\theta)$; the latter squares $\hat{\theta}/\theta-1$. They are close only when the relative error is small.

The loss function expresses what counts as an important estimation error. Use ordinary squared error when absolute discrepancies in a specified unit are relevant. Use relative squared error when proportional discrepancies are the target and the parameter is positive. In either case, state the parameterization and units before interpreting a numerical risk.

Check Your Understanding

Use the definitions and results above to answer these questions.

If an estimator has squared-error risk $0.03\ \mathrm{m}^2$, what is its risk in $\mathrm{cm}^2$?
Under an affine unit conversion $\eta=a\theta+b$, why does the offset $b$ not appear in the transformed squared-error risk?
At a fixed positive $\theta$, how is relative risk obtained from ordinary squared-error risk, and why does this preserve estimator rankings?
Explain why squared error for $\log\theta$ is not generally a constant multiple of squared error for $\theta$.
For what kind of relative error is log-scale squared error approximately similar to relative squared-error loss?