Pooled Versus Time-Specific Treatment Model Diagnostics

Why Compare Pooled and Time-Specific Models?

In longitudinal treatment models, a pooled specification uses observations from multiple periods while imposing some common parameters. That can be efficient when the assignment mechanism is sufficiently stable over time. But a common effect can also average over changing relationships: the fitted model may look acceptable overall even though it systematically overpredicts treatment in one period and underpredicts it in another, or makes opposing errors across covariate groups within periods.

The diagnostic question is therefore not simply whether a pooled model fits. It is whether its restrictions are credible in light of evidence separated by time. This tutorial focuses on comparing treatment-probability diagnostics from pooled and time-specific models. It does not treat a better-fitting model as automatically causally adequate: measured histories, positivity, and the intended weighting specification still matter.

Definition: Let $A_t$ denote treatment at period $t$, and let $H_t$ denote the observed history available before that treatment. A pooled treatment model fits the conditional assignment probability $P(A_t=1\mid H_t)$ using data from multiple periods while imposing at least one shared parameter across periods. A common version has period-specific intercepts but common covariate effects: $$ \operatorname{logit} P(A_t=1\mid H_t) =\alpha_t+\beta^\top g(H_t). $$ A time-specific model allows the relevant parameters to vary by period, for example: $$ \operatorname{logit} P(A_t=1\mid H_t) =\alpha_t+\beta_t^\top g(H_t). $$ Here $g(H_t)$ is a specified set of history features. The comparison concerns the restrictions on the assignment model, not a change in the observed treatment or history data.

The pooled model need not impose common intercepts. Including period indicators, or otherwise allowing period-specific intercepts, is often essential because baseline treatment frequency can change over follow-up even when covariate effects are stable. Conversely, a pooled model with one intercept and one slope imposes both kinds of stability. Always state which parameters are shared: “pooled” alone does not specify the restriction.

For a fair diagnostic comparison, use the same eligible observations, treatment definition, and history variables in both fits. Then ask whether allowing time-specific parameters changes predictions in a consequential and coherent way. In sparse periods, a more flexible model can produce unstable estimates; differences between fits must be read alongside uncertainty and the amount of information available.

What Aggregate Fit Can Conceal

A pooled model’s score equations can make its aggregate residuals appear reassuring. To see why, suppose the model includes a separate intercept for each period. Let $\hat p_{it}$ be the fitted probability for individual $i$ at period $t$, and let $A_{it}$ be the observed treatment. For the log-likelihood contribution $A_{it}\log(\hat p_{it})+(1-A_{it})\log(1-\hat p_{it})$, the derivative with respect to the period-$t$ intercept is $A_{it}-\hat p_{it}$. Thus, at an interior maximum, the intercept score equation gives:

\sum_{i\in \mathcal I_t}(A_{it}-\hat p_{it})=0,

where $\mathcal I_t$ is the set of observations contributing at period $t$. This is a statement about the total number treated in that period, not about calibration within every history subgroup. If overprediction in one subgroup cancels underprediction in another, the total can be zero. If a model has only one common intercept, even cancellation across periods can make its overall residual sum zero.

The distinction matters because assignment probabilities are conditional on history. A model can reproduce the marginal treatment frequency yet fail to represent how treatment depends on that history. Therefore, compare observed and predicted treatment patterns separately by period and across relevant ranges or categories of history. Calibration plots and residual summaries should be stratified finely enough to expose plausible time-specific departures, but not so finely that nearly empty cells dominate the interpretation.

Worked Example: Opposing Covariate Effects Hidden by Pooling

Suppose there are two periods and a binary history feature $X$. Each period has 100 observations with $X=0$ and 100 with $X=1$. The observed treatment proportions are:

Period	$X=0$	$X=1$
1	10/100 = 0.10	80/100 = 0.80
2	80/100 = 0.80	10/100 = 0.10

Consider a pooled logistic model with period-specific intercepts and one common coefficient for $X$. At a coefficient of zero, the fitted probability in period 1 is 0.45 for both values of $X$, since the period’s overall treatment fraction is $(10+80)/200=0.45$. Period 2 also has overall fraction $(80+10)/200=0.45$, so its fitted probability is likewise 0.45.

This is not merely a plausible fit: it satisfies the score equations. In each period, there are 200 observations and 90 treated, so the intercept score is $90-200(0.45)=0$. Across both periods, there are 200 observations with $X=1$ and $80+10=90$ treated among them. Their predicted treated count is $200(0.45)=90$, so the common-$X$ score is also zero. The logistic log-likelihood is concave, so these zero scores identify a maximum (with the stated data, the fitted probabilities are interior).

Yet the pooled model’s conditional predictions are wrong: in period 1 it predicts 0.45 instead of 0.10 for $X=0$, and 0.45 instead of 0.80 for $X=1$; period 2 has the errors reversed. The time-specific model fits the four observed proportions directly. Thus, pooled aggregate calibration at 0.45 in both periods does not establish conditional adequacy. The period-by-$X$ comparison reveals the problem.

A Structured Comparison

Compare models in a way that makes the hypothesized time variation visible. Start with a pooled model that allows period-specific baseline treatment probabilities, then consider a version that also allows the covariate effects to vary by period. If the scientific question concerns a particular covariate, a pooled model with a period-by-covariate interaction can be an interpretable intermediate specification. Use the same covariate transformations in each model unless the purpose is explicitly to compare those transformations.

For nested models fitted to the same observations by maximum likelihood, the more flexible model cannot have a lower maximized log-likelihood. This fact is useful, but a small increase is not automatically persuasive: adding parameters almost always improves in-sample fit to some extent.

Result: If a restricted pooled model is nested in a time-specific model, and both maximize the same likelihood over the same data, then $$ \ell_{\mathrm{time}} \geq \ell_{\mathrm{pooled}}. $$ The likelihood-ratio statistic is $2(\ell_{\mathrm{time}}-\ell_{\mathrm{pooled}})$. Under the null restriction, its usual chi-square reference requires regular maximum-likelihood conditions, including independent sampling units, an interior true parameter, identifiable parameters, nonsingular information, and a fixed-dimensional nested model. In longitudinal data, the independent sampling units are typically people, not repeated person-period records treated as unrelated. Without those conditions, do not rely uncritically on that reference distribution.

The inequality follows directly from the parameter spaces. Every pooled parameter value corresponds to a permitted time-specific value satisfying the pooled equality restrictions. The time-specific model maximizes over a parameter space containing that restricted space; its maximum therefore cannot be smaller. The likelihood-ratio test asks whether the improvement is large relative to sampling variation under its assumptions. It does not identify which model is better calibrated in every clinically relevant subgroup.

In practice, examine several kinds of evidence together:

Period-stratified calibration: Compare observed treatment frequencies with predicted probabilities within each period and across meaningful history strata.
Temporal coefficient patterns: Check whether the time-specific estimates differ in direction or magnitude in ways consistent with the observed prediction errors. Large standard errors may make apparent changes inconclusive.
Prediction contrasts: For the same history profiles, compare pooled and time-specific predicted probabilities period by period. Large differences near the probability boundaries deserve particular scrutiny.
Fit improvement: Use a nested likelihood comparison when its assumptions are appropriate, but interpret it alongside calibration patterns and the number of added parameters.

Worked Example: Separating Baseline Change From Effect Change

Suppose the fitted pooled model is $\operatorname{logit}(p_t)=\alpha_t+\beta X$, with $\beta=1.2$. The baseline probability for $X=0$ is estimated as 0.10 in period 1 and 0.40 in period 2. These imply intercepts $\alpha_1=\log(0.10/0.90)\approx-2.197$ and $\alpha_2=\log(0.40/0.60)\approx-0.405$.

For $X=1$, the pooled model adds 1.2 to the log-odds. In period 1, the predicted probability is $1/[1+\exp(-(-2.197+1.2))]\approx0.270$. In period 2 it is $1/[1+\exp(-(-0.405+1.2))]\approx0.689$. Therefore the model permits the baseline to change substantially over time while requiring the covariate’s log-odds effect to stay fixed.

Suppose period-stratified diagnostics instead show observed probabilities near 0.35 for $X=1$ in period 1 and 0.55 in period 2. The pooled model overpredicts the second value by about $0.689-0.55=0.139$, while it underpredicts the first by about $0.35-0.270=0.080$. This pattern motivates checking time-varying covariate effects; it is not evidence that a common intercept was the sole problem, because the baseline intercepts already vary by period. The comparison should also report uncertainty, particularly if either $X=1$ subgroup is small.

Interpreting the Evidence and Avoiding Overreaction

A useful diagnostic conclusion is specific: for example, “the pooled model matches overall treatment frequencies, but underpredicts treatment among high-risk histories in the first period and overpredicts it in the last.” That observation gives a reason to inspect a time interaction or a separate-period fit. It is stronger than saying only that one model has a smaller deviance.

At the same time, a time-specific fit is not automatically preferable. Separate models use less information per period, can estimate probabilities erratically, and may yield extreme predictions where support is thin. A fitted time interaction can also reflect random variation, a change in measurement, or an omitted feature of history rather than a genuine change in the assignment mechanism. Inspect period-specific sample sizes, treatment counts, uncertainty, and whether the fitted probabilities are supported by observed histories.

Finally, treatment-model fit is not the same as post-weighting balance or a valid causal analysis. A model can predict treatment well yet leave imbalance relevant to the weighting goal, and an apparently acceptable aggregate diagnostic cannot certify that every conditional assignment probability is correctly modeled. Interpret period-specific fit evidence as one part of diagnosing the assignment model, not as a substitute for other checks.

Common pitfall: “The pooled model’s predicted and observed treatment totals agree, so its assignment probabilities are adequate.” The total can agree because residuals cancel across histories or periods; with period intercepts, the fitted period totals may agree by construction. Check calibration within periods and across relevant histories before treating aggregate agreement as reassuring.

Worked Diagnostic Comparison

Worked Example: Reading Period-Specific Calibration Gaps

A pooled model with period-specific intercepts predicts treatment probability 0.30 for a particular history group in period 1 and 0.60 in period 2. In a diagnostic table, 40 of 100 people in that group are treated in period 1, while 50 of 100 are treated in period 2. Compute the observed-minus-predicted gaps.

In period 1, the observed proportion is $40/100=0.40$, so the gap is $0.40-0.30=0.10$: the model underpredicts by 10 percentage points. In period 2, the observed proportion is $50/100=0.50$, so the gap is $0.50-0.60=-0.10$: the model overpredicts by 10 percentage points.

The opposite signs show why a pooled summary can be misleading. With equal group sizes, the average observed proportion is $(0.40+0.50)/2=0.45$, and the average predicted probability is $(0.30+0.60)/2=0.45$. Their aggregate difference is zero even though each period has a 10-point discrepancy. The appropriate next diagnostic is to examine whether the pattern persists across related history groups and whether uncertainty is compatible with a real temporal departure—not to declare the pooled model adequate from the aggregate alone.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

A pooled logistic model includes period indicators and common covariate effects. What does it allow to change over time, and what does it constrain?
Why does a zero period-specific intercept score not establish calibration within every history subgroup?
A pooled model has maximized log-likelihood $-240$, and its nested time-specific alternative has maximized log-likelihood $-232$. Calculate the likelihood-ratio statistic and state what the calculation alone does not establish.
Observed and predicted treatment proportions in two equally sized periods are 0.25 and 0.55, and 0.35 and 0.45, respectively. Show that the aggregate proportions agree while period-specific gaps remain.

Solutions. (1) The period indicators allow baseline log-odds to vary; the common covariate coefficients constrain the covariate associations to be the same. (2) The score sets the sum of residuals over the period to zero, but positive and negative residuals can cancel across history groups. (3) The statistic is $2[-232-(-240)]=16$. By itself, it does not show that the time-specific model is well calibrated or causally valid; inferential use also depends on the likelihood and sampling assumptions. (4) The mean observed proportion is $(0.25+0.55)/2=0.40$, and the mean predicted proportion is $(0.35+0.45)/2=0.40$. The period gaps are $0.25-0.35=-0.10$ and $0.55-0.45=0.10$, so aggregate agreement conceals equal, opposite errors.