Uncertainty in Averaged Marginal Means

Why an Average Needs Its Own Uncertainty Calculation

An averaged marginal mean summarizes predictions across a set of specified profiles. Its point estimate is a weighted average of the profile predictions, but its uncertainty is not generally the weighted average of their standard errors. Predictions from the same fitted regression model are related: they are calculated using shared estimated coefficients. Their covariance is part of the uncertainty calculation.

The target average depends on the profiles and weights chosen for the question. For example, equal weights might describe an average over a deliberately balanced set of profiles, while other weights might represent a different target population. Once the target is set, the main task here is to obtain the uncertainty of that weighted average from the model’s covariance information.

Model and Definition

Use the standard linear regression model with fixed predictor values:

\mathbf{Y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}, \qquad \boldsymbol{\varepsilon}\sim N(\mathbf{0},\sigma^2\mathbf{I}).

This model assumes a linear mean structure, independent errors with mean zero and common variance $\sigma^2$, and normally distributed errors for exact small-sample $t$-based intervals. The design matrix $\mathbf{X}$ is treated as fixed and has full column rank. Under these assumptions, the least-squares coefficient estimate has estimated covariance matrix $s^2(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}$, where $s^2$ estimates the error variance.

Let $\mathbf{x}_i$ denote the design row for profile $i$. Its fitted conditional mean is $\hat{\mu}_i=\mathbf{x}_i^{\mathsf T}\hat{\boldsymbol{\beta}}$. Choose $m$ profiles and fixed weights $w_1,\ldots,w_m$ that sum to one. The corresponding target marginal mean and its estimate are:

\theta=\sum_{i=1}^{m} w_i\mu_i, \qquad \hat{\theta}=\sum_{i=1}^{m}w_i\hat{\mu}_i.

Definition: An averaged marginal mean is a weighted average of conditional means for a specified set of profiles. Its uncertainty is the sampling uncertainty of that weighted estimate, calculated using the joint covariance among the profile predictions.

Weights are treated as fixed in the formulas below. If weights are themselves estimated from data, their uncertainty may also need to be included; the calculation here does not account for that additional source.

How the Covariances Enter

A weighted sum has a variance determined by every pair of terms in the sum. Write $\widehat{\boldsymbol{\mu}}$ for the vector of profile predictions and $\mathbf{w}$ for the vector of weights. Then $\hat{\theta}=\mathbf{w}^{\mathsf T}\widehat{\boldsymbol{\mu}}$. The variance formula is:

\operatorname{Var}(\hat{\theta}) = \mathbf{w}^{\mathsf T} \operatorname{Cov}(\widehat{\boldsymbol{\mu}}) \mathbf{w} = \sum_{i=1}^{m}w_i^2\operatorname{Var}(\hat{\mu}_i) + 2\sum_{i<j}w_iw_j\operatorname{Cov}(\hat{\mu}_i,\hat{\mu}_j).

The standard error is the square root of this estimated variance. Positive covariance increases the variance of an average when the weights are positive; negative covariance decreases it. The covariance is not an optional adjustment: it is part of the variance of a sum.

For a regression model, each profile prediction is a linear combination of the same estimated coefficients. If $\mathbf{X}_*$ contains the design rows for the profiles, then $\widehat{\boldsymbol{\mu}}=\mathbf{X}_*\hat{\boldsymbol{\beta}}$. The coefficient covariance matrix therefore gives the prediction covariance matrix:

\operatorname{Cov}(\widehat{\boldsymbol{\mu}}) = \mathbf{X}_* \operatorname{Cov}(\hat{\boldsymbol{\beta}}) \mathbf{X}_*^{\mathsf T}.

Equivalently, form the weighted design row $\bar{\mathbf{x}}^{\mathsf T}=\sum_i w_i\mathbf{x}_i^{\mathsf T}$. Then $\hat{\theta}=\bar{\mathbf{x}}^{\mathsf T}\hat{\boldsymbol{\beta}}$, so its estimated variance is $\bar{\mathbf{x}}^{\mathsf T}\operatorname{Cov}(\hat{\boldsymbol{\beta}})\bar{\mathbf{x}}$. Both calculations produce the same result.

Justification and Confidence Interval

The variance rule follows directly from the definition of covariance. For two random estimates $A$ and $B$, expanding the squared deviation of $w_1A+w_2B$ gives a variance containing their covariance:

\begin{aligned} \operatorname{Var}(w_1A+w_2B) &=E\left[\left\{w_1(A-E[A])+w_2(B-E[B])\right\}^2\right]\\ &=w_1^2\operatorname{Var}(A)+w_2^2\operatorname{Var}(B) +2w_1w_2\operatorname{Cov}(A,B). \end{aligned}

Applying this expansion repeatedly to all profile predictions yields the matrix formula above. In a normal linear model with fixed weights and the stated assumptions, the estimated average is a linear combination of the coefficient estimates. Dividing its centered value by its estimated standard error gives a $t$ statistic with the model’s residual degrees of freedom under the null value of the target.

Result: If the model has residual degrees of freedom $df$, a two-sided $100(1-\alpha)\%$ confidence interval for the averaged marginal mean is $$ \hat{\theta}\ \pm\ t_{1-\alpha/2,df}\,SE(\hat{\theta}), \qquad SE(\hat{\theta})=\sqrt{\mathbf{w}^{\mathsf T}\widehat{\operatorname{Cov}}(\widehat{\boldsymbol{\mu}})\mathbf{w}}. $$ Here $t_{1-\alpha/2,df}$ is the appropriate quantile of the $t$ distribution.

The interval describes uncertainty in the model-estimated average conditional mean. It does not describe the spread of individual outcomes around that mean. That distinction matters because the residual variation of individual outcomes is not added to this mean-estimate standard error.

Worked Example: Averaging Three Profile Predictions

Suppose a hypothetical regression analysis reports predicted mean scores of 12, 15, and 18 points for three specified profiles. The target average gives these profiles weights 0.20, 0.30, and 0.50. The model’s covariance matrix for the three predictions, in squared points, is shown below. Assume the model’s conditions are suitable for a $t$-based interval, with $df=30$.

\widehat{\operatorname{Cov}}(\widehat{\boldsymbol{\mu}}) = \begin{pmatrix} 4 & 1 & 0.5\\ 1 & 9 & 2\\ 0.5 & 2 & 16 \end{pmatrix}.

First calculate the estimated average:

\hat{\theta} =0.20(12)+0.30(15)+0.50(18) =2.40+4.50+9.00 =15.90\text{ points}.

Next use the full covariance matrix. The diagonal terms contribute $0.20^2(4)+0.30^2(9)+0.50^2(16)=0.16+0.81+4.00=4.97$. The covariance terms contribute $2\{(0.20)(0.30)(1)+(0.20)(0.50)(0.5)+(0.30)(0.50)(2)\}=2(0.41)=0.82$. Thus:

\widehat{\operatorname{Var}}(\hat{\theta})=4.97+0.82=5.79, \qquad SE(\hat{\theta})=\sqrt{5.79}=2.406\text{ points}.

For a 95% confidence interval, $t_{0.975,30}\approx2.042$. Therefore:

15.90\pm2.042(2.406) \approx15.90\pm4.914 =(10.986,\ 20.814) \approx(10.99,\ 20.81)\text{ points}.

The interval estimates the target weighted average conditional mean score. Its width reflects both the uncertainty in each profile prediction and their covariances.

Using the Coefficient Covariance Matrix

The same calculation can be performed without first constructing a covariance matrix for the profile predictions. This is useful when software reports coefficient estimates and their covariance matrix. The weighted design row represents the average profile for the specified weights; applying the coefficient covariance matrix to that row gives the average’s variance directly.

Worked Example: Averaging Predictions From a Linear Model

Consider a hypothetical fitted mean model $\mu(x)=\beta_0+\beta_1x$, with estimates $\hat{\beta}_0=10$ points and $\hat{\beta}_1=2$ points per unit of $x$. The estimated coefficient covariance matrix is:

\widehat{\operatorname{Cov}}(\hat{\boldsymbol{\beta}}) = \begin{pmatrix} 4 & -0.6\\ -0.6 & 0.25 \end{pmatrix}.

Average the predictions at $x=2$ and $x=6$ with equal weights. The weighted design row is $(1,4)$, because the average of the two predictor values is 4. The estimated average is $10+2(4)=18$ points. Its estimated variance is:

\widehat{\operatorname{Var}}(\hat{\theta}) = \begin{pmatrix}1&4\end{pmatrix} \begin{pmatrix} 4 & -0.6\\ -0.6 & 0.25 \end{pmatrix} \begin{pmatrix}1\\4\end{pmatrix} =4-4.8+4=3.2\text{ points}^2.

Thus $SE(\hat{\theta})=\sqrt{3.2}=1.789$ points. If the model has $df=20$, then $t_{0.975,20}\approx2.086$, giving:

18\pm2.086(1.789) \approx18\pm3.732 =(14.268,\ 21.732) \approx(14.27,\ 21.73)\text{ points}.

The negative covariance between the coefficient estimates enters the calculation through the $-4.8$ term. It would be incorrect to discard that term merely because the intercept and slope are different coefficients.

Weights, Software Output, and a Common Pitfall

A software package may provide the covariance matrix of the profile predictions, or it may provide the standard error for a specified linear combination. Either can be used, provided the reported result corresponds to the intended profiles and weights. If only the separate standard errors are available, they are not enough to calculate the average’s standard error unless the covariances are also known or justified to be zero.

The choice of weights defines the target, not a computational detail. For example, equal weighting of two groups targets their equally weighted average even if the groups have different sample sizes. A different set of weights answers a different averaging question. State the profiles and weights so the estimated mean has a clear interpretation.

Worked Example: Unequal Weights and Positive Covariance

Suppose a hypothetical model gives profile mean estimates of 20 and 24 points. The target gives the first profile weight 0.75 and the second profile weight 0.25. Their estimated covariance matrix, in squared points, is:

\begin{pmatrix} 4 & 1.5\\ 1.5 & 9 \end{pmatrix}.

The estimated weighted mean is $0.75(20)+0.25(24)=21$ points. Its variance includes the covariance term:

\widehat{\operatorname{Var}}(\hat{\theta}) =(0.75)^2(4)+(0.25)^2(9)+2(0.75)(0.25)(1.5) =2.25+0.5625+0.5625 =3.375\text{ points}^2.

The standard error is $\sqrt{3.375}=1.837$ points. With $df=18$, $t_{0.975,18}\approx2.101$, so the 95% confidence interval is:

21\pm2.101(1.837) \approx21\pm3.859 =(17.141,\ 24.859) \approx(17.14,\ 24.86)\text{ points}.

Because the covariance is positive and both weights are positive, the covariance contribution increases the variance compared with a calculation that incorrectly assumes independence.

Common mistake: Treating the profile predictions as independent and calculating the variance using only the squared weights and individual variances. That omits $2\sum_{i<j}w_iw_j\operatorname{Cov}(\hat{\mu}_i,\hat{\mu}_j)$, which need not be zero. The correction is to use the full covariance matrix or a software standard error for the exact weighted linear combination. Also avoid adding residual outcome variation: the interval here concerns an average conditional mean, not a future individual outcome.

Key takeaway: An averaged marginal mean is a linear combination of fitted means. Its standard error comes from applying the same weights to the full covariance information, including covariances between profile predictions.

Check Your Understanding

Use the model’s covariance information as well as its profile estimates when answering.

Why is the standard error of an averaged marginal mean not generally the weighted average of the profile standard errors?
For two estimates with positive covariance and positive weights, what does the covariance term do to the variance of their weighted average?
What model information is needed to calculate an average’s variance using a weighted design row?
Explain why changing the weights can change the target mean even when the profile predictions stay the same.
Does a confidence interval for an averaged conditional mean include the residual variation of a future individual outcome? Explain.