Time Structure Misspecification in Treatment Models

Why Time Structure Needs Its Own Checks

In a longitudinal study, treatment assignment can change as follow-up proceeds. Clinical guidelines may change, patients may become eligible for treatment, and clinicians may respond differently to evolving covariate histories. A treatment model that represents covariates adequately but represents time inadequately can therefore produce probabilities that are wrong in a systematic, period-dependent way.

The problem is not necessarily visible in a pooled fit summary. Overprediction in one period can cancel underprediction in another, and an overall balance statistic can conceal a failure at a particular treatment decision. The useful diagnostic question is therefore: does the model represent assignment adequately at each relevant time, for the histories observed at that time?

For a binary treatment, the period-specific inverse-probability factor is \(1/\hat p_{it}\) for a treated person and \(1/(1-\hat p_{it})\) for an untreated person, where \(\hat p_{it}\) is the fitted probability. The denominator part of an unstabilized longitudinal weight through period \(t\) is the product of these factors over periods \(1,\ldots,t\). Stabilized weights also have numerator factors; for diagnosing the treatment model, it is useful to inspect the denominator probabilities and incremental factors as well as the resulting weights.

Three Complementary Views of Time Misfit

A period-specific calibration check compares fitted probabilities with observed treatment frequencies separately by period. It can be done in bins of predicted probability or within prespecified history groups, provided each group has enough observations for a meaningful comparison. A balance check asks whether covariates at a treatment decision remain associated with current treatment after applying the appropriate weights, ideally within relevant prior-treatment histories. A weight check examines the distribution of the period’s inverse-probability factors and of the cumulative product.

These checks answer different questions. Calibration assesses whether predicted probabilities agree with observed frequencies; balance assesses a consequence of weighting for measured covariates; weight distributions reveal how strongly the fitted probabilities amplify particular observations. Agreement on one view does not guarantee agreement on the others. In particular, a model can have an acceptable overall calibration summary and still have period-specific errors.

Why Correct Period-Specific Weights Balance Histories

The following population identity explains why balance is a useful diagnostic. It concerns one treatment decision and the history \(H_t\) used to model it. Assume binary treatment, a correctly specified conditional probability \(p_t(H_t)\), and positivity: \(0<p_t(H_t)<1\) for histories under consideration. Assume the relevant expectations exist. Weight treated observations by \(1/p_t(H_t)\) and untreated observations by \(1/\{1-p_t(H_t)\}\).

To prove the result, condition on \(H_t\). For treated observations,

Here \(E\{I(A_t=1)\mid H_t\}=p_t(H_t)\), so the conditional probability cancels. Setting \(g(H_t)=1\) shows that the expected treated weight is 1. Dividing the displayed identity by this normalizing expectation gives the normalized weighted mean \(E[g(H_t)]\).

For untreated observations, the same calculation gives

because \(E\{I(A_t=0)\mid H_t\}=1-p_t(H_t)\). The normalizing expectation is again 1. Thus both weighted groups have the same distribution of \(H_t\), as asserted. In finite samples with estimated probabilities, balance is not guaranteed to be exact; residual imbalance may reflect sampling variability, estimation error, or misspecification. \(\square\)

This identity motivates checking balance in the histories that enter the assignment model, rather than relying only on an overall balance summary. In a longitudinal analysis, prior treatment can define distinct risk sets or history strata, and the period’s treatment decision must be interpreted in that context. A failed balance check is evidence that the chosen weighting procedure has not removed the measured association being assessed; it is not, by itself, proof of which model component is wrong.

Inspecting Incremental and Cumulative Weights

A large cumulative weight can arise from several moderately large factors, one near-violated positivity condition, or both. Conversely, a cumulative distribution can make it difficult to locate the period that contributes most to the problem. For that reason, inspect the distribution of each period’s factor by period and treatment status, and then inspect cumulative weights as a separate summary. Relevant features include quantiles, maxima, skewness, and the proportion of factors or weights above a prespecified threshold. These are descriptive warnings, not universal pass-or-fail rules.

Interpreting Diagnostics Without Overclaiming

A second useful identity explains how pooled calibration can hide time structure. Let \(\bar p_t\) be the mean fitted probability in period \(t\), and let \(\bar A_t\) be the observed treatment frequency there. If periods have equal sample sizes, the pooled difference is the average of the period differences:

This follows because the pooled mean is the sum of the period-specific totals divided by the combined sample size \(Tn\), which equals the average of the \(T\) period means when each period has \(n\) observations. Positive and negative discrepancies can therefore cancel. For unequal period sizes, the same cancellation occurs through a sample-size-weighted average. Pooled agreement is not evidence that each period is calibrated.

In practice, plot observed treatment frequency against mean predicted probability separately by period, and where possible within clinically meaningful history groups. Look for systematic period patterns in calibration residuals, covariate imbalance after weighting, and factor or weight tails. Sparse groups make these diagnostics noisy, while empty or nearly empty treatment groups can indicate a positivity problem rather than merely a functional time trend. Check counts and uncertainty alongside plots; do not interpret a few observations as precise evidence.

The main alternatives are complementary, not interchangeable. Period-specific calibration is closest to checking probability accuracy but may miss covariate imbalance hidden within broad bins. Weighted balance directly checks measured covariates but cannot verify the probabilities for every aspect of the history. Weight-distribution summaries flag instability but do not identify the mechanism that created it. Comparing these views helps distinguish a temporal calibration issue from residual measured imbalance or extreme weighting, without turning any one diagnostic into a causal validity test.

These checks have important limits. They cannot detect unmeasured confounding, establish that all confounders were measured, or prove the causal assumptions required for a marginal structural model. A model may be calibrated in observed groups yet fail to represent treatment probabilities within finer histories. Conversely, sampling variation can produce apparent period discrepancies even when the model is correct. Diagnostics should therefore guide model revision and transparent sensitivity analysis, not certify a model as true.