Pitfall: Averaging Probabilities Over The Wrong Population

Why the Population Being Averaged Matters

A GLMM can provide predicted outcomes for people or clusters with specified characteristics. But a single population-averaged estimate requires more than those predictions: it also requires a decision about which population’s characteristics to average over. The same fitted model can produce different standardized risks and contrasts when the target population changes.

This is not merely a technical choice of computation. Averaging over the covariate distribution in the study sample answers a question about that sample. Averaging over a specified clinic, region, or future population answers a different question. Likewise, giving every cluster equal weight describes a different target from giving every person equal weight when cluster sizes vary.

The calculations below start from fitted predictions on the response scale. For a binary outcome, these are probabilities. As in the earlier material on marginalization, a population-averaged prediction may require integrating over the random-effects distribution; here, assume that the required predictions have already been obtained using the random-effects distribution appropriate to the target. The focus is how to average those predictions over covariates and clusters.

Define the Target Before Averaging

Definition: Let $A=a$ denote a specified exposure or treatment level, let $X$ denote covariates, and let $m_a(x)$ be the GLMM-predicted mean response on the response scale for covariates $X=x$, with random effects treated according to the intended marginal prediction. For a target population with covariate distribution $F_T$, the standardized mean is $$ \mu_T(a)=E_T\{m_a(X)\}. $$ For discrete covariate strata $g=1,\ldots,G$, with target proportions $q_g$, this is $$ \mu_T(a)=\sum_{g=1}^{G}q_g m_a(g), \qquad \sum_{g=1}^{G}q_g=1. $$ A standardized contrast, such as a risk difference, is formed from standardized means: $\Delta_T=\mu_T(1)-\mu_T(0)$.

The weights $q_g$ describe the target population, not necessarily the distribution of covariates in the model-fitting sample. If individual records represent the target population, an empirical version is $\hat{\mu}_T(a)=n_T^{-1}\sum_{i=1}^{n_T}\hat{m}_a(X_i)$, where the $X_i$ are the target individuals’ covariates. If the target is specified by strata, use the stated stratum proportions instead.

Before calculating, specify the unit being averaged over. A “randomly selected person” gives more influence to large clusters because more people belong to them. A “randomly selected cluster, followed by a randomly selected person within that cluster” gives each cluster equal initial weight, even when cluster sizes differ. Neither target is universally correct; the scientific question determines which applies.

Why Incorrect Weights Change the Estimate

Suppose the target proportions are $q_g$, but an analyst uses proportions $r_g$ from another population. The resulting difference in standardized means for level $a$ is

\sum_g r_gm_a(g)-\sum_g q_gm_a(g) =\sum_g(r_g-q_g)m_a(g).

Because both sets of weights sum to one, this difference is zero only in special cases—for example, if the predictions are constant across strata or if the particular weight changes cancel. A contrast can also change, because the predictions can vary across strata differently at the two treatment levels. Thus, weights are part of the estimand, not an optional adjustment that can be chosen for convenience.

The same logic applies when covariates are combined into joint strata. If two covariates are associated in the target population, replacing their joint distribution with a product of their separate marginal distributions creates a different population—one in which the covariates are effectively treated as independent.

Worked Example: Study-Sample Weights Versus Target Weights

Suppose a hypothetical binary-outcome GLMM gives the following marginal predicted risks in two age strata. Assume the predictions correctly integrate over the random-effects distribution and are supported for both strata.

Age stratum	Predicted risk at $A=1$	Predicted risk at $A=0$
Younger	0.30	0.20
Older	0.80	0.50

The target population is 80% younger and 20% older. The study sample, however, is 30% younger and 70% older. Calculate the standardized risks and risk difference for the target, then compare them with the results obtained using the study-sample weights.

Correct target calculation. For $A=1$, $$ \hat{\mu}_T(1)=(0.80)(0.30)+(0.20)(0.80)=0.24+0.16=0.40. $$ For $A=0$, $$ \hat{\mu}_T(0)=(0.80)(0.20)+(0.20)(0.50)=0.16+0.10=0.26. $$ Therefore the target-standardized risk difference is $0.40-0.26=0.14$, or 14 percentage points.

Using the wrong weights. Applying the study-sample proportions instead gives $$ \hat{\mu}_S(1)=(0.30)(0.30)+(0.70)(0.80)=0.09+0.56=0.65, $$ $$ \hat{\mu}_S(0)=(0.30)(0.20)+(0.70)(0.50)=0.06+0.35=0.41. $$ The resulting risk difference is $0.65-0.41=0.24$, or 24 percentage points. These calculations are internally consistent, but the latter contrast targets the study-sample age distribution, not the specified target population. In this example the choice of weights changes the contrast by 10 percentage points.

Cluster Weights Define a Second Population Choice

When the target involves clusters, the weight assigned to a cluster must match the sampling or policy question. Suppose cluster $j$ has size $n_j$ and a predicted mean response $\hat{m}_j$. An equal-cluster average is $J^{-1}\sum_{j=1}^{J}\hat{m}_j$. An average over all individuals is $(\sum_j n_j)^{-1}\sum_{j=1}^{J}n_j\hat{m}_j$. These are generally unequal.

This distinction can arise even if the predictions themselves are perfectly calculated. For instance, a health system might care about the outcome for the average patient, while a regulator comparing facilities might care about the average facility. Choosing equal-cluster weights for the first question, or patient-count weights for the second, answers a different question. If covariates also vary across clusters, both the cluster weighting and the covariate distribution within clusters need to reflect the target.

Worked Example: Equal-Cluster and Equal-Person Averages

Consider three hypothetical clinics with sizes $2$, $8$, and $20$ patients. Their already-integrated predicted risks for the outcome are $0.10$, $0.30$, and $0.60$, respectively. Assume the target is explicitly defined in one of two ways: a randomly selected clinic, or a randomly selected patient across these clinics.

Randomly selected clinic. Each clinic receives weight $1/3$: $$ \hat{\mu}_{\text{clinic}}=\frac{0.10+0.30+0.60}{3} =\frac{1.00}{3}=0.3333. $$ The estimated average clinic risk is about 33.3%.

Randomly selected patient. Each clinic receives weight proportional to its patient count: $$ \hat{\mu}_{\text{person}} =\frac{(2)(0.10)+(8)(0.30)+(20)(0.60)}{2+8+20} =\frac{0.20+2.40+12.00}{30} =\frac{14.60}{30}=0.4867. $$ The estimated risk for a randomly selected patient is about 48.7%. The larger clinic has the highest predicted risk and contributes most patients, so it has much more influence in the person-weighted average. The two answers are not competing estimates of one undefined quantity: they correspond to different target populations.

Preserve the Target’s Joint Covariate Distribution

When standardizing over multiple covariates, weights should generally describe their joint distribution in the target population. Marginal proportions for each covariate are not enough to reconstruct that joint distribution unless an additional assumption—such as independence—is justified. Multiplying separate marginal proportions without justification can assign the wrong weights to covariate combinations.

Worked Example: Why Separate Margins Can Give the Wrong Answer

Suppose a hypothetical target population is defined by two binary covariates: site (Site 1 or Site 2) and severity (mild or severe). Its joint proportions and GLMM-predicted risks are as follows. All risks are marginal predictions on the response scale.

Stratum	Target proportion	Risk at $A=1$	Risk at $A=0$
Site 1, mild	0.40	0.20	0.10
Site 1, severe	0.10	0.80	0.60
Site 2, mild	0.10	0.40	0.30
Site 2, severe	0.40	0.90	0.50

The target has 50% at each site and 50% in each severity category, but site and severity are associated: severe cases are concentrated at Site 2. Using the correct joint weights, $$ \hat{\mu}_T(1)=(0.40)(0.20)+(0.10)(0.80)+(0.10)(0.40)+(0.40)(0.90) =0.08+0.08+0.04+0.36=0.56, $$ $$ \hat{\mu}_T(0)=(0.40)(0.10)+(0.10)(0.60)+(0.10)(0.30)+(0.40)(0.50) =0.04+0.06+0.03+0.20=0.33. $$ The standardized risk difference is $0.56-0.33=0.23$, or 23 percentage points.

If the analyst multiplies the marginal proportions as though site and severity were independent, every joint cell gets weight $(0.50)(0.50)=0.25$. That artificial distribution gives $$ \hat{\mu}_{\text{ind}}(1)=0.25(0.20+0.80+0.40+0.90)=0.575, $$ $$ \hat{\mu}_{\text{ind}}(0)=0.25(0.10+0.60+0.30+0.50)=0.375. $$ Its risk difference is $0.575-0.375=0.200$, or 20 percentage points. The separate margins are correct, but their product does not represent the target’s joint distribution. The resulting estimate answers a question about a different covariate mix.

Assumptions, Diagnostics, and Alternatives

A standardized estimate is only as credible as the predictions and weights used to construct it. Check the following before interpreting the result:

Target definition: State whether the target is a particular set of people, a distribution of covariate strata, an average clinic, or an average person. Clarify which covariates and joint distributions the target includes.
Prediction scale: Average predicted responses, such as probabilities or means, when the estimand is an average response. Do not substitute averages of linear predictors for response-scale averages.
Random-effects target: Use predictions integrated over a random-effects distribution when the target is a population of new or randomly selected clusters. A prediction conditional on a particular cluster effect answers a cluster-specific question instead.
Support and overlap: The model must provide credible predictions for the covariate combinations receiving target weight. If a target stratum is absent or nearly absent in the fitting data, standardization may rely heavily on extrapolation. Inspect the stratum weights and predictions, and report this limitation.
Weight validity: Check that weights are nonnegative, sum to one after normalization, and correspond to the stated target. For cluster targets, verify whether cluster size should or should not affect the weights.

Direct standardization, used in the examples, applies target weights directly to fitted predictions. An alternative is to use weighting methods to make a sample represent the target population and then average outcomes using those weights. These approaches rely on different modeling and sampling assumptions, but neither removes the need to define the target. Using study-sample weights may be appropriate when the study sample itself is the intended target; it is not an automatic default for every population-average question.

A useful diagnostic is to calculate the estimate under plausible alternative target distributions and display the weights alongside the predictions. Large changes show that the estimate is sensitive to population composition; they do not, by themselves, indicate which target is scientifically correct. When target weights are estimated from data, their uncertainty may also contribute to uncertainty in the standardized estimate.

Common Mistake

Common pitfall: “The model was fitted to this sample, so the sample’s covariate and cluster-size weights must be the right weights for the reported population-average effect.” This reasoning confuses the population used to fit a model with the population the estimate is meant to describe. The correction is to define the target first, then use its covariate distribution and its person-versus-cluster weighting rule. Sample weights are appropriate only when they represent that target or when the sample itself is the target.

Check Your Understanding

For each question, identify the target and show how its definition determines the weights. Solutions follow each question.

A target is 25% low-risk and 75% high-risk. Predicted risks under $A=1$ are 0.20 and 0.60, respectively. What is the standardized risk? Solution: Use the target proportions: $(0.25)(0.20)+(0.75)(0.60)=0.05+0.45=0.50$. The standardized risk is 50%.
Two clinics have sizes 5 and 15 and predicted risks 0.20 and 0.40. Calculate the equal-clinic average and the average for a randomly selected patient. Solution: Equal-clinic average $=(0.20+0.40)/2=0.30$. Patient average $=[(5)(0.20)+(15)(0.40)]/20=(1+6)/20=0.35$. The two targets produce 30% and 35%.
A target has two covariates with known marginal proportions, but their association is unknown. Can their product always be used as the joint standardization weights? Solution: No. Multiplying the margins imposes an independence distribution. Use the target joint proportions, or justify independence as an assumption.
A standardization uses target weights 0.70 and 0.30 for two strata, but reports weights 0.50 and 0.50. What should be checked? Solution: Check whether the reported calculation instead targets an equal-stratum population, whether the weights were copied or normalized incorrectly, and whether the target was changed. For the stated target, use 0.70 and 0.30; the weights must sum to one and match the target definition.
A target gives positive weight to a covariate stratum absent from the model-fitting data. What is the main concern? Solution: The prediction for that stratum requires extrapolation unsupported by observed data. The analyst should examine model plausibility, consider whether the target can be restricted or redefined, and communicate the limitation rather than treating the estimate as securely data-supported.