Why a Weight Diagnosis Needs More Than a Maximum
An extreme inverse-probability weight can arise because a treatment was genuinely rare for a history, because the fitted treatment model assigned that history an implausibly small probability, or because both problems are present. Those explanations are not interchangeable. A model revision may address misspecification, but it cannot create observations for treatment choices absent from a history. Conversely, labeling every large weight a positivity failure can conceal a poor model.
The aim here is a worked diagnosis: connect each large weight to the history and treatment probability that produced it, compare fitted probabilities with observed frequencies where possible, and examine whether weighting has reduced covariate imbalance. These are diagnostics, not proofs that a causal effect is identified.
A useful diagnosis keeps three questions separate. Support: are both treatment options represented for the relevant observed histories? Calibration and specification: do fitted probabilities agree reasonably with observed treatment frequencies, including across important history strata? Weight consequences: are weights concentrated in a few people, and does weighted balance improve? The earlier material on integrated weight diagnostics gives the broad workflow; the calculations below focus on interpreting competing explanations.
A Worked Longitudinal Diagnosis
Consider a two-decision study. At the second decision, suppose the records below all have the same prior treatment \(A_0\), and the displayed history strata are defined by two binary covariates \(X\) and \(L_1\). The fitted model for the second treatment uses these histories. The table gives the observed counts and the candidate model's fitted treatment probability. Treat those fitted values as the outputs of the model under investigation.
| History stratum | People | Treated at time 1 | Observed treatment fraction | Fitted \(\hat p_1(H_1)\) |
|---|---|---|---|---|
| \(X=0,L_1=0\) | 40 | 32 | 0.800 | 0.800 |
| \(X=0,L_1=1\) | 30 | 6 | 0.200 | 0.200 |
| \(X=1,L_1=0\) | 26 | 20 | 0.769 | 0.800 |
| \(X=1,L_1=1\) | 4 | 1 | 0.250 | 0.050 |
Worked Example: Tracing an Extreme Weight to Its History
Use a stabilized numerator probability of 0.5 at the second decision. For a treated person in the stratum \(X=1,L_1=1\), the second-time weight factor is
If that person also has a prior stabilized weight of 2, the cumulative weight is \(2\times10=20\). This calculation identifies the immediate source of the extreme value: the model assigned a small denominator probability to the observed treated history. The history contains only four people, with one treated, so the fitted value 0.05 is sharply lower than the observed fraction \(1/4=0.25\). That discrepancy is a calibration warning, but the stratum is too small to treat its observed fraction as a precise estimate of the population probability.
For comparison, in the stratum \(X=0,L_1=1\), an untreated person's second-time factor is \(0.5/(1-0.2)=0.625\). A treated person's factor there is \(0.5/0.2=2.5\). Thus a model's very small fitted probability, rather than the mere fact that some histories are uncommon, is what creates the factor of 10 in the first calculation.
The table also illustrates why a frequency of zero or near zero deserves a different interpretation from a model prediction of zero. Suppose an additional observed history has seven people and none treated. This is clear empirical sparsity: the data provide no treated examples in that stratum. It is not, by itself, evidence that the population probability of treatment is exactly zero. In a genuinely structural-zero setting, the treatment option is unavailable by design or clinical rule; in a small sample, it may simply not have occurred.
Worked Example: Calculating Residual Weighted Imbalance
Ignore prior weights for this calculation and apply the second-time stabilized factors from the fitted probabilities in the table. The total treated weight in the four strata is
The treated weight attached to \(X=1\) is \(12.5+10=22.5\), so the weighted treated mean of \(X\) is \(22.5/57.5\approx0.391\). For untreated people, the total weight is
The untreated weight attached to \(X=1\) is \(15+1.579=16.579\), giving weighted mean \(16.579/51.579\approx0.321\). The residual difference is therefore about \(0.391-0.321=0.070\). Before weighting, the treated mean is \(21/59\approx0.356\), and the untreated mean is \(9/41\approx0.220\), a difference of about 0.136. This candidate weighting reduces the imbalance in \(X\), but it does not eliminate it. The calculations do not establish why residual imbalance remains: model misspecification, sparse-cell noise, and other features of the histories may all contribute.
Two Results That Clarify the Diagnosis
The next identity explains why correct inverse probabilities balance observed histories in expectation. It is a one-decision statement; it does not assert that a finite weighted sample will balance exactly.
Here \(I(A=1)\) equals 1 when \(A=1\) and 0 otherwise. To prove the identity, condition on \(H\). Since \(f(H)\) and \(p(H)\) are then fixed,
Taking expectations of both sides gives the result. The same argument for \(A=0\) uses \(1-p(H)\). This expectation identity explains why imbalance is a useful model diagnostic: systematic failure to balance measured histories can signal inaccurate treatment probabilities. It does not require exact sample balance, and it says nothing by itself about unmeasured confounding. \(\square\)
A second calculation explains why tiny probabilities create unstable weights even when the probability model is correct. Conditional on \(H\), define the unstabilized treated weight \(W=I(A=1)/p(H)\). Then
Indeed, \(W\) is \(1/p(H)\) with conditional probability \(p(H)\), and zero otherwise. Thus \(E[W\mid H]=p(H)/p(H)=1\), while \(E[W^2\mid H]=p(H)/p(H)^2=1/p(H)\). Subtracting the squared mean yields the displayed variance. As \(p(H)\) approaches zero, this variance grows without bound. A large weight can therefore be a consequence of limited treatment probability even under a correctly specified model; re-fitting alone cannot remove the underlying instability without changing the target or imposing additional assumptions. \(\square\)
There is also a fundamental limit to what observed data can tell us about a genuinely unsupported treatment option. If \(P(A=1\mid H=h)=0\), no person with history \(h\) reveals the outcome under \(A=1\). Consider two data-generating descriptions that agree on the distribution of all observed histories, treatments, and observed outcomes, but assign different values to the unobserved outcome under \(A=1,H=h\). They produce the same observed-data distribution, because that treatment-history combination never occurs, yet they imply different counterfactual outcomes under treatment. Therefore the observed data alone cannot distinguish those values. This is why a zero or near-zero observed frequency is a support warning, not a data-only test of structural positivity failure.
How to Distinguish the Competing Explanations
Use the evidence jointly rather than treating any single diagnostic as decisive. A useful distinction is:
| Pattern in the diagnosis | What it suggests | What it does not establish |
|---|---|---|
| One treatment value is absent in a history stratum | Observed support is lacking there; investigate design rules and history definition. | That the population probability is exactly zero. |
| Observed treatment fractions vary across histories but fitted probabilities do not track them | Calibration or functional-form problems are plausible. | That misspecification is the only source of the discrepancy, especially in tiny cells. |
| Extreme weights arise repeatedly in well-populated histories with poor fitted-versus-observed agreement | Model specification deserves close scrutiny. | That a more complex model will be stable or correct. |
| Weights are extreme in sparse histories and observed treatment fractions are uncertain | Support limitation and sampling variability are both plausible. | That truncating weights resolves the underlying problem. |
For each extreme observation, record its treatment time, history, treatment value, fitted denominator probability, numerator probability, and contribution to the cumulative weight. Then compare model predictions with observed frequencies within meaningful history strata, taking stratum size seriously. Examine conditional balance as well as overall balance: aggregation can hide imbalance in a subgroup. The earlier material on probability calibration and conditional balance supplies these complementary views; neither replaces the other.
Compare candidate models using the same diagnostic targets. If a revised time structure or history specification improves calibration and balance without creating more extreme weights, that is evidence in its favor. If a more flexible model produces near-zero estimates in small strata, apparent fit may come at the cost of instability. Weight truncation can reduce the numerical influence of extremes, but it changes the weighting procedure and does not repair a false treatment model or restore absent treatment histories. A more stable-looking summary is not, on its own, evidence that the problem is solved.
Worked Example: Choosing a Diagnostic Follow-Up
Suppose an analyst finds that the only treated person in a four-person history stratum receives a weight of 10 at the second time, while a seven-person stratum contains no treated person. The fitted probabilities are substantially below the observed fraction in the four-person stratum and positive in the zero-treated stratum.
First, inspect the history construction and treatment rules. Confirm that the history uses information recorded before the treatment decision and that no coding error made unlike histories appear identical. If a clinical rule forbids treatment in the seven-person stratum, that is substantive evidence of structural lack of support. If treatment is possible, zero observed treated people remains a sparse-data finding, not proof of impossibility.
Next, inspect calibration with uncertainty in mind. In the four-person stratum, the observed fraction is \(1/4=0.25\). Its contrast with the fitted probability is concerning, but one observation can shift that fraction greatly. Compare additional strata, model forms, and time-specific diagnostics before concluding that the model is misspecified.
Finally, report the consequence rather than hiding it. State that the model produces a large contribution from a sparse history, show the balance before and after weighting, and describe any sensitivity to reasonable alternatives. Do not claim that truncation establishes positivity or that a revised model creates information where the data contain none.
Common Mistake and Practical Limits
Wrong reasoning: “The largest weight is 20, so positivity has failed; truncating it will fix the problem.” The weight's size alone does not diagnose its cause. It may reflect an incorrect denominator model, genuine rarity, sparse-sample variation, or a combination. Truncation may change the effect estimate and does not reveal which explanation is correct.
Correction: trace the weight to its history-specific fitted probability, check observed frequencies and relevant design knowledge, then assess conditional and overall balance. Report uncertainty where histories are sparse. Positivity is a property of the population treatment mechanism; fitted probabilities and finite-sample counts are imperfect evidence about it. These outcome-blind diagnostics also cannot verify exchangeability, choose an estimand, or guarantee unbiased effect estimation.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A treated person has fitted probability 0.10 and stabilized numerator 0.5. What is the treatment-time weight factor?
- In a history stratum, 0 of 8 people are treated. Does this prove the population probability of treatment is zero? Give one reason.
- For a treated inverse-probability weight \(W=I(A=1)/p(H)\), state its conditional mean and variance.
- A candidate model has moderate weights but leaves substantial covariate imbalance within several history strata. Is moderate weight size sufficient to endorse it?
Solutions. 1. The factor is \(0.5/0.10=5\). 2. No. The observed zero can occur in a finite sample even when the population probability is positive; design information may be needed to assess whether treatment was truly unavailable. 3. \(E[W\mid H]=1\) and \(\operatorname{Var}(W\mid H)=1/p(H)-1\), as derived above. 4. No. Moderate weights do not guarantee adequate treatment-model specification or balance; inspect calibration and conditional imbalance alongside weight summaries.