Start With the Target, Then Choose the Weights
In an unbalanced factorial design, the combinations of factor levels may have different sample sizes. A comparison averaged over the levels of another factor therefore needs a weighting rule. Equal weights, observed proportions, and known target-population proportions are all possible choices, but they do not generally estimate the same quantity. The useful question is not simply which weights are available: it is which population or reference distribution should the comparison represent?
Consider comparing two plans across several clinics. An equally weighted comparison represents an average across clinics that gives each clinic the same importance. A comparison using the clinics’ observed proportions gives more weight to clinics that contributed more observations. A comparison using specified population proportions represents a target population with that distribution of clinics. These targets can produce different numerical results even when based on the same cell means.
The same weights are used for each level being compared. This makes the comparison refer to a shared distribution across the levels being averaged over. In practice, the cell means \(\mu_{ij}\) are not known, so their estimates are weighted to obtain estimated marginal means. The weight choice defines the target; it is not merely a computational adjustment.
A Model for the Cell Means
For the calculations below, suppose each outcome \(Y_{ijk}\) is recorded for an individual \(k\) in factor-level combination \((i,j)\). Use the cell-means model
Assume the errors are independent, have mean zero, and have a common variance \(\sigma^2\). Under these assumptions, the sample mean in each observed cell estimates its cell mean. Normality is not needed to calculate the weighted means; it is commonly added when using this model for exact small-sample inference. The examples use complete tables, so every cell mean required for the comparisons is observed. The calculations describe weighted mean comparisons; by themselves, they do not establish a causal effect.
For a cell with \(n_{ij}\) observations, its sample mean is \(\bar{x}_{ij}\). The estimated weighted marginal mean and comparison are
These formulas clarify why weights should be selected to suit the estimand: each weight determines how much a particular level contributes to the result.
Worked Example: Equal Weights Across Clinics
In a hypothetical comparison, Plan A and Plan B are studied at three clinics. The outcome is days to recovery. The table gives the sample size and sample mean in each cell.
| Plan | Clinic | Sample size | Mean recovery time (days) |
|---|---|---|---|
| A | 1 | 30 | 10 |
| A | 2 | 10 | 14 |
| A | 3 | 10 | 18 |
| B | 1 | 10 | 12 |
| B | 2 | 30 | 15 |
| B | 3 | 10 | 24 |
All six cells are observed, so their sample means can be used. Suppose the target gives the three clinics equal importance. Each weight is \(1/3\), and the estimated marginal means are
Thus the equal-weight comparison, Plan A minus Plan B, is \(14-17=-3\) days. Under this target, Plan A has an estimated mean recovery time 3 days shorter than Plan B, averaged equally across the three clinics. The sample sizes do not determine these weights: the target is a distribution that gives each clinic weight \(1/3\).
Three Weighting Choices
The principal choices differ in what distribution over the levels being averaged over they represent. In each case, the chosen weights are applied to the cell means for every level of the factor being compared.
| Weighting choice | Weights across clinics | Target interpretation |
|---|---|---|
| Equal | \(w_j=1/J\) | An average that gives each clinic the same weight |
| Proportional | Observed pooled clinic proportions | An average using the clinic distribution in the analyzed sample |
| Population-based | Specified target-population proportions | An average using a stated distribution for the population of interest |
For proportional weighting with one factor being averaged over, the pooled sample proportion at clinic \(j\) is \(w_j=n_{\cdot j}/n_{\cdot\cdot}\), where \(n_{\cdot j}\) is the total number of observations from that clinic across the compared factor levels, and \(n_{\cdot\cdot}\) is the overall sample size. These are common weights for all compared levels. They are not the separate within-plan clinic proportions, which can differ between plans and would give each plan a different distribution.
Worked Example: Proportional Weights From the Sample
Use the hypothetical clinic data above. The pooled sample sizes are \(40\), \(40\), and \(20\) for Clinics 1, 2, and 3, respectively. There are \(100\) observations in total, so the proportional weights are
Applying the same weights to each plan gives
The proportional-weight comparison is \(13.2-15.6=-2.4\) days. It represents a comparison averaged over the observed pooled distribution of clinics. It differs from the equal-weight comparison of \(-3\) days because the two weighting rules place different emphasis on Clinic 3, where the estimated plan difference is largest in magnitude.
Worked Example: Weights for a Target Population
Suppose a health system wants its comparison to represent a target population in which 20% of patients attend Clinic 1, 30% attend Clinic 2, and 50% attend Clinic 3. These proportions are specified for the target, rather than inferred from the sample sizes. The weights are \(0.2\), \(0.3\), and \(0.5\), which are nonnegative and sum to one.
Using the same hypothetical cell means, the estimated marginal mean for Plan A is
For Plan B, it is
The target-population comparison is \(15.2-18.9=-3.7\) days. This result describes the weighted comparison for the stated clinic distribution. It is not the observed-sample comparison unless the sample’s pooled clinic proportions happen to match those target proportions.
Why the Weighting Choice Changes the Estimand
The key fact can be seen by rearranging the comparison:
Thus the weighted marginal comparison is a weighted average of the level-specific differences. If those differences vary across clinics, changing the weights can change the comparison’s magnitude and, in some cases, its direction. This algebra also gives a direct interpretation: a clinic with a larger weight contributes more to the overall comparison. If the difference is identical at every clinic, all valid common weight choices yield that same difference.
A weighting choice is defensible when it matches the question being asked. Equal weights suit a target that treats each level as equally important, whether or not that distribution occurs in the sample. Proportional weights describe the observed pooled composition and may be useful when that composition is itself the target. Population-based weights suit a specified target distribution, provided the cell means can reasonably be applied to the levels in that target. The number of observations in a cell affects how precisely its mean is estimated, but it does not, on its own, determine which target is scientifically relevant.
Worked Example: A Weighting Choice Can Reverse a Comparison
Consider another hypothetical two-clinic comparison of Plans A and B, again measured in recovery days. The cell means are 10 and 14 days at North for Plans A and B, and 18 and 16 days at South. All four cells are observed. Suppose the intended target places 20% weight on North and 80% on South.
The estimated clinic-specific differences, Plan A minus Plan B, are \(10-14=-4\) days at North and \(18-16=2\) days at South. The target-weighted comparison is
Equivalently, the weighted means are \((0.2)(10)+(0.8)(18)=16.4\) days for Plan A and \((0.2)(14)+(0.8)(16)=15.6\) days for Plan B, giving \(16.4-15.6=0.8\) days. Under this target distribution, Plan A has an estimated mean recovery time 0.8 days longer than Plan B. With equal clinic weights, the comparison would instead be \(\{(-4)+2\}/2=-1\) day. Neither calculation is inherently the universally correct answer: they describe different targets.
Common Mistake: Letting the Sample Sizes Choose the Question
A common mistake is to assume that the largest cells should automatically receive the largest weights because they contain more information. Cell sizes matter for the precision of cell-mean estimates, but choosing weights for a marginal mean answers a separate question: what distribution of the averaged-over factor should the comparison represent?
In the first hypothetical table, the ordinary sample means within each plan are \(12.4\) days for Plan A and \(16.2\) days for Plan B, giving a raw difference of \(-3.8\) days. These use each plan’s own clinic distribution: Plan A has more observations from Clinic 1, while Plan B has more from Clinic 2. That raw comparison is not the proportional estimated marginal comparison using common pooled clinic weights, which was \(-2.4\) days. It also is not the equal-weight comparison or the target-population comparison.
Check Your Understanding
For each question, identify what the weights make the comparison represent.
- A researcher gives each of four clinics weight \(1/4\). What clinic distribution does the resulting marginal mean represent?
- Across two plans, the pooled sample sizes at three sites are 20, 30, and 50. What are the proportional weights?
- Why should proportional weights for a comparison use the same clinic weights for both plan levels?
- Suppose the clinic-specific differences are 2, 2, and 2 outcome units. Explain why any common nonnegative weights summing to one produce the same weighted comparison.
- A target population has clinic proportions that differ from the sample’s pooled proportions. Which weighting choice directly represents that target, and what must be available to calculate its weighted means?