Observed Treatment Frequencies and Probability Calibration

Why Compare Predicted and Observed Treatment Frequencies?

A treatment model assigns each person a predicted probability of receiving treatment, conditional on the information available just before that treatment decision. A useful diagnostic asks whether people assigned similar predicted probabilities actually receive treatment at approximately those rates. In longitudinal studies, the comparison should also be made at each follow-up period: a model can match the overall treatment frequency while misrepresenting when treatment is given or which clinical risk groups receive it.

This is a diagnostic of probability calibration, not a test that establishes causal validity. A model might predict treatment frequencies well yet omit a confounder that matters for the outcome. Conversely, a discrepancy can indicate misspecification, but it can also reflect sampling variation, sparse data, or overfitting. The aim is to make these distinctions explicit.

Definition: Let $A_{it}\in\{0,1\}$ denote whether person $i$ receives treatment at period $t$, and let $H_{it}$ be that person's observed history immediately before the treatment decision. The true treatment probability is $e_t(H_{it})=P(A_{it}=1\mid H_{it})$; a fitted treatment model supplies a predicted probability $q_{it}$. For a group $G$ of observations at period $t$, the observed treatment frequency is $\hat r_{Gt}=\sum_{i\in G}A_{it}/n_{Gt}$, and the mean predicted probability is $\bar q_{Gt}=\sum_{i\in G}q_{it}/n_{Gt}$. The calibration discrepancy is $D_{Gt}=\hat r_{Gt}-\bar q_{Gt}$.

Groups can be defined by clinically meaningful characteristics known before treatment, such as baseline disease severity, or by ranges of predicted probability. These choices answer related but different questions. Clinical groups can reveal miscalibration in a population of direct clinical interest; predicted-risk groups summarize whether probabilities are reliable across the score range. In either case, make comparisons separately by follow-up period before considering any pooled summary.

What Correct Probabilities Imply

The central justification follows from conditional expectation. If the true probability is $e_t(H_{it})$, then among observations selected by any group defined from the observed pre-treatment history, the expected treatment frequency equals the expected true probability. This is a population statement; it does not say the two finite-sample quantities must be identical.

Theorem: Fix a period $t$. Suppose $A_{it}$ is binary and $e_t(H_{it})=P(A_{it}=1\mid H_{it})$ exists. For any group indicator $I_{it}$ that is a measurable function of $H_{it}$, provided $P(I_{it}=1)>0$, $$ E[A_{it}\mid I_{it}=1]=E[e_t(H_{it})\mid I_{it}=1]. $$ Consequently, the expected treatment rate in that group agrees with its mean true treatment probability.

Proof. Since $I_{it}$ is determined by $H_{it}$, it is known when conditioning on $H_{it}$. By iterated expectation,

\begin{aligned} E[A_{it}I_{it}] &=E\{E[A_{it}I_{it}\mid H_{it}]\}\\ &=E\{I_{it}E[A_{it}\mid H_{it}]\}\\ &=E[I_{it}e_t(H_{it})]. \end{aligned}

Dividing both sides by $P(I_{it}=1)>0$ gives the stated conditional means. This result also applies to a group defined by the true probability itself, because $e_t(H_{it})$ is a function of $H_{it}$. It does not guarantee equality when $e_t$ is replaced by an arbitrary fitted prediction $q_{it}$; checking that replacement is the purpose of the diagnostic. $\square$

A second implication is that an overall comparison can conceal group-specific errors. If groups $G_1,\ldots,G_J$ partition the observations at a period, then the overall discrepancy is a size-weighted average of group discrepancies:

\hat r_t-\bar q_t =\sum_{j=1}^{J}\frac{n_{jt}}{n_t}D_{jt}.

The identity follows by writing the total observed treatment count and total predicted count as sums over the groups, then dividing each by $n_t=\sum_j n_{jt}$. Positive and negative discrepancies can cancel. Thus, overall agreement is not evidence that each clinically important group is well calibrated.

Worked Comparisons by Risk Group and Time

Worked Example: Comparing Frequencies Across Clinical Risk Groups

At one follow-up period, a study divides 250 people into three pre-treatment severity groups. The fitted model's mean predicted probabilities and observed treatment counts are as follows. Calculate each discrepancy and interpret it.

Severity group	Number	Mean predicted probability	Observed treated
Lower	100	0.10	12
Moderate	100	0.25	30
Higher	50	0.50	30

For the lower-severity group, the observed frequency is $12/100=0.12$, so $D=0.12-0.10=0.02$. The model predicts 10 treated people on average and 12 were observed.

For the moderate group, the observed frequency is $30/100=0.30$, giving $D=0.30-0.25=0.05$. The model predicts 25 treated people on average and 30 were observed.

For the higher-severity group, the observed frequency is $30/50=0.60$, giving $D=0.60-0.50=0.10$. The model predicts 25 treated people on average and 30 were observed. The increasingly positive discrepancies suggest that the fitted model underpredicts treatment frequency in these groups, particularly in the higher-severity group. They are a signal to investigate, not by themselves proof of a particular modeling error.

Overall, 72 of 250 people were treated, so $\hat r=72/250=0.288$. The model predicts $100(0.10)+100(0.25)+50(0.50)=60$ treatments, or $\bar q=60/250=0.24$. The overall discrepancy is $0.048$. The weighted group calculation agrees: $0.4(0.02)+0.4(0.05)+0.2(0.10)=0.048$.

Worked Example: Opposing Errors Across Follow-Up Periods

Suppose 200 people contribute an eligible treatment decision at each of two periods. At period 1, the mean predicted probability is 0.10 and 30 people are treated. At period 2, the mean predicted probability is 0.40 and 70 people are treated.

At period 1, the observed frequency is $30/200=0.15$, and the discrepancy is $0.15-0.10=0.05$. The predicted count is 20, compared with 30 observed.

At period 2, the observed frequency is $70/200=0.35$, and the discrepancy is $0.35-0.40=-0.05$. The predicted count is 80, compared with 70 observed.

Pooling the 400 decisions gives 100 observed treatments, or frequency $100/400=0.25$. The model predicts $20+80=100$ treatments, also $0.25$. The pooled discrepancy is zero even though the model underpredicts treatment at the first period and overpredicts it at the second. The time-specific comparisons expose a pattern that the pooled comparison erases.

Quantifying Uncertainty and Reading the Plot

A nonzero discrepancy is expected from sampling variation. For a fixed group of $m$ independent observations with known true probabilities $e_i$, the observed count $S=\sum_{i=1}^m A_i$ has conditional expectation $\sum_i e_i$. If treatment assignments are conditionally independent given the histories, its conditional variance is $\sum_i e_i(1-e_i)$. Therefore, for the frequency minus mean true probability,

\operatorname{Var}\left(\frac{1}{m}\sum_{i=1}^m\{A_i-e_i\}\,\middle|\,H_1,\ldots,H_m\right) =\frac{1}{m^2}\sum_{i=1}^m e_i(1-e_i).

This follows by adding the variances of conditionally independent Bernoulli variables and dividing by $m^2$. In a simple group with a common probability $e$, the standard deviation of the frequency is $\sqrt{e(1-e)/m}$. This calculation gives a scale for sampling variation, not a complete confidence interval for a diagnostic based on estimated probabilities.

In longitudinal data, observations from the same person are generally dependent. Do not treat every person-period record as independent when quantifying uncertainty across periods. Subject-level resampling, or an uncertainty method that accounts for clustering by person, is more appropriate; when the model is estimated, the uncertainty calculation should also reflect model fitting. A subject-level bootstrap that refits the treatment model in each resample is one practical approach, provided the study design supports it.

A useful display plots observed frequency against mean predicted probability for each group and period, with the identity line as the reference. Points above the line indicate underprediction of treatment frequency; points below indicate overprediction. Show group sizes and uncertainty where feasible. A smooth calibration curve can help reveal gradual departures that arbitrary bins may obscure, but it can also be unstable where few observations contribute.

Worked Example: Interpreting a Group Discrepancy Against Sampling Variation

In a group of 50 independent people, suppose all fitted probabilities are 0.50 and 30 receive treatment. The observed frequency is $30/50=0.60$, the mean predicted probability is 0.50, and $D=0.10$.

Using 0.50 as the probability for a rough sampling-variation benchmark, the standard deviation of the observed frequency is $\sqrt{0.50(1-0.50)/50}=\sqrt{0.005}\approx0.0707$. The discrepancy is about $0.10/0.0707\approx1.41$ such standard deviations from zero. This alone is not compelling evidence of a systematic error. A formal assessment should account for how the group and predictions were constructed, any model fitting, and dependence in the data.

Diagnostics, Alternatives, and Limitations

Choose groups before inspecting their discrepancies when possible, and report how they were formed. Predicted-probability bins can reveal whether errors concentrate in a particular score range; clinical strata can make the same errors easier to assess in context. Avoid creating many tiny groups: their observed rates fluctuate substantially, and a dramatic discrepancy in a handful of people may be mostly noise. Conversely, excessively broad groups can average away meaningful local errors.

In-sample calibration deserves special caution. For example, an unpenalized logistic regression fitted by maximum likelihood with an intercept has an intercept score equation that sets the sum of fitted residuals to zero, when a finite maximum-likelihood estimate exists. Thus, agreement in the overall fitted sample may partly be built into the fitting procedure. It does not guarantee agreement by time or risk group, nor does it guarantee that predictions will calibrate in new data. Out-of-fold or held-out predictions can provide a less optimistic assessment, though their uncertainty and data-splitting scheme still need consideration.

Probability calibration answers a different question from discrimination: discrimination concerns how well a model ranks people by treatment probability, while calibration concerns whether predicted probabilities match observed frequencies. A model can rank well but systematically overpredict, or rank modestly while matching group frequencies. Calibration also differs from post-weighting balance diagnostics, which assess covariate balance after weighting, and from conditional balance within history strata. None substitutes for the others.

Most importantly, calibration is not a certificate that the treatment model is causally adequate. Agreement between observed and predicted treatment frequencies across the chosen groups does not show that all relevant confounders were measured or modeled appropriately. A model can reproduce treatment rates while failing to capture an important feature of treatment assignment within groups. Likewise, a well-calibrated model can still yield unstable weights where estimated probabilities approach zero or one. Interpret calibration alongside the study's substantive assumptions and other diagnostics rather than as a stand-alone pass/fail test.

Common pitfall: “The model predicts the overall treatment rate exactly, so its probabilities are calibrated.” This reasoning ignores cancellation: overprediction in one risk group or period can offset underprediction in another. Compare observed and predicted frequencies within prespecified, clinically meaningful groups and separately over time; then assess uncertainty and remember that calibration does not establish causal adequacy.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a group of 80 people, the mean predicted treatment probability is 0.20 and 22 people are treated. Calculate the observed frequency and discrepancy. Does the sign indicate underprediction or overprediction?
Two equally sized periods have observed frequencies 0.18 and 0.32, while their mean predicted probabilities are 0.23 and 0.27. What are the two period-specific discrepancies and the pooled discrepancy?
State the assumption on group membership needed for the conditional-expectation result, and explain why a fitted model need not satisfy it exactly in a finite sample.
Why can a global in-sample check be uninformative for an unpenalized logistic model with an intercept?

Solutions. (1) The observed frequency is $22/80=0.275$, so $D=0.275-0.20=0.075$. The positive discrepancy means observed treatment is more frequent than predicted: the model underpredicts in this group. (2) The discrepancies are $0.18-0.23=-0.05$ and $0.32-0.27=0.05$. With equal group sizes, the pooled observed and predicted frequencies are both $0.25$, so the pooled discrepancy is zero. (3) Group membership must be determined by the observed pre-treatment history, so the group indicator is measurable with respect to that history. The result concerns the true conditional probability; fitted probabilities can differ because of estimation error, sampling variation, or misspecification. (4) The fitted intercept score equation sets the total fitted residual to zero at a finite maximum-likelihood estimate. This can enforce overall agreement in the fitting sample while leaving subgroup and time-specific discrepancies—and performance in new data—unresolved.