Choose the Model for the Question
Grouped or repeated observations do not automatically require a multilevel model. If the goal is to estimate an average regression relationship and obtain uncertainty that accounts for dependence within clusters, a regression with cluster-robust standard errors may be suitable, provided its assumptions are reasonable. But some questions are about more than the average relationship: they ask how clusters differ, how a person’s outcome changes over time, or what to expect in a cluster not yet observed. Those questions call for a model that represents the relevant variation or trajectory.
The distinction is about the target of the analysis, not a rule that one method is always superior. Cluster-robust standard errors adjust the estimated uncertainty for a fitted regression. They do not, by themselves, estimate a distribution of cluster effects, describe person-specific trajectories, or provide a model for predicting a new cluster.
What the Different Approaches Answer
Suppose observation \(i\) belongs to cluster \(j\). A regression with cluster-robust standard errors can estimate a common regression relationship and adjust its standard errors for within-cluster dependence. With the same regressors, changing from ordinary to cluster-robust standard errors does not change the fitted coefficient estimates; it changes the estimated uncertainty and therefore can change confidence intervals and tests. This approach does not specify how much of the outcome variation lies between clusters or provide a distribution for cluster-specific effects.
A simple multilevel model makes such variation explicit. In a random-intercept model, the outcome for observation \(i\) in cluster \(j\) is
Here \(\beta_0+\beta_1x_{ij}\) describes the overall mean relationship, \(b_j\) is the deviation for cluster \(j\), and \(\varepsilon_{ij}\) is the individual deviation within that cluster. A standard version assumes \(b_j\) has mean zero and variance \(\tau^2\); the errors \(\varepsilon_{ij}\) have mean zero and variance \(\sigma^2\); and cluster effects and errors are independent, with distinct clusters independent of one another. The model also assumes the conditional mean has been specified appropriately. Normal distributions for the random effects and errors are often used for likelihood-based estimation and prediction, but are not required for every possible multilevel analysis.
A random-intercept model is only one option. If the relationship between \(x\) and the outcome may vary across clusters, a model can allow cluster-specific slopes as well. For longitudinal data, time can be included in the mean model, and the model can represent how a person’s level or rate of change varies. A model for repeated measurements should reflect the actual time structure and make a defensible assumption about dependence within a person.
| Question of interest | What the analysis needs to represent |
|---|---|
| What is the average association, with uncertainty allowing for clustered observations? | A suitable mean model and cluster-aware inference may be enough. |
| How much do clusters differ in their outcome levels or slopes? | A model that estimates between-cluster variation, such as a multilevel model. |
| How does an outcome change over time, and how do individual trajectories differ? | A longitudinal model that represents time and relevant within-person variation. |
| What outcome should be predicted for a new cluster? | A predictive model that includes uncertainty about between-cluster variation. |
Worked Example: Variation Among Work Sites
Worked Example: Variation Among Work Sites
A hypothetical company records an annual performance score for employees at 20 work sites. The question is whether employee training hours are associated with scores on average, or whether sites also differ in their typical scores. Suppose a fitted random-intercept model estimates between-site variance \(\tau^2=25\) score-points squared and within-site variance \(\sigma^2=75\) score-points squared.
Under this model, the intraclass correlation is the proportion of the total variance attributed to differences between sites:
The fitted model attributes 25% of the modeled outcome variance to differences between work sites. That is a direct description of between-site variation; a regression with cluster-robust standard errors alone would not estimate this variance component. If the sole target were the average training-hours association, a cluster-robust analysis might address its uncertainty, assuming the mean model and independent-cluster requirements are reasonable. If the target includes site differences or site-specific predictions, a multilevel model is more aligned with the question.
The calculation assumes the random-intercept representation is suitable: observations are grouped correctly, distinct sites can reasonably be treated as independent, and the shared site component is a meaningful description of within-site similarity. The ICC is a model-based summary, not proof that sites cause score differences. The number of sites also matters for inference; having many employees in a small number of sites does not create many independent clusters.
Repeated Time Measurements Need a Time Model When Trajectories Matter
For repeated measurements, time is not merely another label for a cluster. The order and spacing of measurements may matter, and the scientific question may concern change, curvature, or differences in individual rates of change. A cluster-robust standard error can be used for inference about a correctly specified average time pattern, but it does not itself describe individual trajectories or the temporal covariance structure.
Worked Example: Average Change and Individual Trajectories
In a hypothetical follow-up, four adults have a symptom score recorded at baseline, three months, and six months. The observed scores are:
| Participant | Baseline | Three months | Six months |
|---|---|---|---|
| A | 8 | 11 | 10 |
| B | 10 | 12 | 11 |
| C | 12 | 13 | 12 |
| D | 10 | 12 | 11 |
The time-specific averages are \(10\), \(12\), and \(11\) score points. For example, the baseline mean is
The average score is one point higher at six months than at baseline, but the average rises by two points at three months and then falls by one point. Individual paths also differ: participant A’s score changes by \(+3\) and then \(-1\), while participant C’s changes by \(+1\) and then \(-1\). A single linear time slope could obscure the intermediate rise and the variation among people.
If the question is only about an average difference at the scheduled times, a model with time indicators and participant-clustered uncertainty may be appropriate when there are enough independent participants and its other assumptions are credible. If the question concerns individual change rates, a nonlinear time pattern, or prediction of a participant’s future trajectory, a longitudinal model is needed to represent those features. A reasonable model would treat participants as independent units, use the actual measurement times, and allow the within-person dependence and trajectory variation it claims to estimate. These four records illustrate the issue; they are far too few participants to support reliable cluster-robust inference.
Predictions for Existing and New Clusters
A multilevel model distinguishes prediction for a cluster with observed data from prediction for a new cluster. For an observed cluster, its measurements provide information about its deviation \(b_j\). Under a random-effects model, the estimate of that deviation is generally pulled toward zero, the overall mean deviation, especially when the cluster has little data. For a new cluster, no cluster-specific observations are available, so its expected deviation under the model is zero—but its actual deviation remains uncertain.
This difference matters for prediction intervals. For a new individual in a new cluster under a random-intercept model, uncertainty includes both within-cluster variation and between-cluster variation. Ignoring the latter would make predictions too certain when \(\tau^2\) is nonzero. The following calculation assumes known model parameters and a correctly specified normal random-intercept model; in practice, uncertainty from estimating the parameters should also be considered.
Worked Example: Predicting an Outcome in a New Clinic
Suppose a hypothetical clinics model has \(\tau^2=9\) outcome-units squared between clinics and \(\sigma^2=36\) outcome-units squared within clinics. The fixed part of the model predicts 50 units for a specified patient profile. For a new patient in a new clinic, the model’s prediction variance, treating the fixed prediction and variance components as known, is
The corresponding prediction standard deviation is \(\sqrt{45}=6.708\) units. Under the additional normality assumption, an approximate 95% prediction interval is
This interval is for an individual outcome in a new clinic under the stated model, not for the average outcome of all new clinics. Its width includes both clinic-to-clinic and patient-to-patient variation. Estimation uncertainty would generally add uncertainty beyond this calculation.
For comparison, suppose an existing clinic has four observed outcomes whose residuals from the fixed part of the model average \(6\) units. Under the normal random-intercept model, the conditional mean of its random intercept is a shrunken version of that residual average:
The cluster’s estimated deviation is \(3\) units rather than the raw average residual of \(6\). The shrinkage follows from combining the noisy cluster average with the model’s assumption that cluster effects vary around zero. It is useful for estimating cluster-specific effects, but it relies on the random-effects assumptions; it is not a generic adjustment supplied by cluster-robust standard errors.
A Short Justification for Shrinkage
The preceding shrinkage formula follows from the normal model. Write the residual for observation \(i\) in a cluster as \(r_{ij}=b_j+\varepsilon_{ij}\), with \(b_j\sim N(0,\tau^2)\) and independent errors \(\varepsilon_{ij}\sim N(0,\sigma^2)\). The cluster residual mean then satisfies
Combining this normal likelihood with the normal distribution for \(b_j\), the conditional distribution of \(b_j\) is normal. Its mean is the conditional variance multiplied by the likelihood precision times \(\bar r_j\):
The multiplier is between zero and one when both variance components are positive. As \(n\) increases, \(\sigma^2/n\) decreases and the cluster average becomes more informative, so the estimate is pulled less strongly toward zero. This result depends on the stated random-intercept and normality assumptions; it is not a property of cluster-robust regression.
A Practical Decision
Before choosing an approach, state what quantity the analysis must deliver. If it is an average coefficient with dependence-aware uncertainty, a regression with cluster-robust standard errors may suffice when the mean model is appropriate and there are enough independent clusters. If the target is a distribution of cluster effects, cluster-specific predictions, variation in slopes, or repeated-time trajectories, choose a model that represents those features. The methods can also be combined in some settings, but adding robust standard errors does not repair an inappropriate mean model or create a model for cluster-level predictions.
Decide whether the goal is average association, between-cluster variation, a time trajectory, or prediction for a particular or new cluster.
Ask whether the target needs cluster-specific effects, varying slopes, or an explicit representation of time and within-person dependence.
Consider the number of independent clusters, the cluster sizes, the measurement schedule, and whether the proposed model’s mean and variation structure are credible.
Common Mistake
A common mistake is to conclude that using cluster-robust standard errors has “accounted for clustering,” so the analysis can answer any question involving groups. This confuses uncertainty adjustment with modeling. Cluster-robust standard errors can support inference for coefficients in a specified regression while allowing within-cluster dependence under suitable conditions. They do not estimate cluster variance, reveal individual trajectories, or provide a distribution for predictions in new clusters. Choose the analysis according to the quantity you need, not just the form of the standard error.
Check Your Understanding
For each question, decide whether cluster-robust standard errors alone could address the target or whether a multilevel or longitudinal model is needed, and explain why.
- A company wants an average association between training hours and performance scores, with employees grouped by work site. What would determine whether cluster-robust inference is suitable?
- A researcher wants to estimate how much typical outcomes vary across clinics. Why are cluster-robust standard errors alone insufficient?
- A study records each participant’s outcome at five ordered visits. What additional questions would make a longitudinal model especially important?
- Why does a prediction for one person in a new cluster include more uncertainty than a prediction based only on within-cluster variation?
- In a random-intercept model, why is a cluster effect estimated from a small number of observations pulled more toward zero?