From a Label Change to an Evaluable Question
A label change is intended to affect decisions: who receives a drug, at what dose, with what monitoring, or under what conditions. Postmarketing data can show whether prescribing and exposure changed after the revision, and whether safety or effectiveness outcomes changed. But a difference between the periods before and after a change is not automatically an effect of the change. Treatment guidelines, available alternatives, healthcare use, and data capture can all change over time.
An evaluation should therefore begin by defining what change is being assessed and what effect is of interest. For example, the question might be whether a new dose recommendation changed the proportion of eligible patients starting above that dose, or whether a new warning was followed by a change in a specific safety-event rate. These are distinct outcomes and may require different denominators, time windows, and comparison groups.
The outcome might be a prescribing proportion, a dose distribution, drug exposure, a safety-event rate, or an effectiveness measure. For each, define the eligible population and the unit of analysis. A prescribing proportion needs a denominator such as eligible treatment starts; an event rate needs person-time at risk. Also define when the revised label becomes relevant in practice. The publication or effective date may not be the date when prescribers, pharmacies, and health systems begin acting on it.
Earlier material on integrating real-world evidence into model-based label revisions addresses how evidence informs a label decision. Here the emphasis is different: estimating and interpreting changes in postmarketing outcomes after the decision. The estimate is about the observed implementation in a defined setting, not necessarily the effect of perfect adherence to the revised text.
Pre-Post Comparisons: Useful Description, Weak Counterfactual
A simple pre-post comparison summarizes an outcome before and after a label change. It can establish the direction and approximate size of a change in the observed data, but by itself it cannot separate the label change from secular trends. A secular trend is a change over calendar time that would occur apart from the label revision, such as a gradual change in prescribing practice or event ascertainment.
Worked Example: A Change in Recommended-Dose Prescribing
In a hypothetical database, 62 of 200 eligible treatment starts before a revised dose recommendation followed the new recommendation. In a separate post-change sample, 104 of 200 starts followed it. Assume each sample includes independent treatment starts, the eligibility definition and data capture are unchanged, and the samples are representative of their respective periods.
The pre-change proportion is \(62/200=0.310\), and the post-change proportion is \(104/200=0.520\). The descriptive difference, post minus pre, is \(0.520-0.310=0.210\), or an increase of 21.0 percentage points.
For independent binomial samples, the estimated standard error of the difference is
Substituting the data gives
A normal-approximation 95% confidence interval is \(0.210\pm1.96(0.04814)\), or approximately \((0.116,\,0.304)\). Under the sampling assumptions, this interval describes plausible values for the difference in the two period-specific proportions. It is not an interval for the causal effect of the label change: a concurrent trend or other event could also explain part or all of the increase.
The simple comparison is a useful first description. It is most defensible as an estimate of a label effect when other determinants of the outcome are stable, the before and after populations are comparable, and the timing of implementation is clear. These conditions are often doubtful in routine postmarketing data, so stronger designs are usually needed for causal interpretation.
Interrupted Time Series: Estimating Level and Trend Changes
An interrupted time series (ITS) uses repeated observations before and after the change. Rather than comparing two averages, it estimates the pre-change trajectory and asks whether the outcome departs from that trajectory after implementation. A common segmented regression for a continuous outcome is
Here \(Y_t\) is the outcome at time \(t\), \(\beta_0\) is the extrapolated pre-change level at the interruption, \(\beta_1\) is the pre-change time trend, \(I_t\) is zero before and one after the interruption, and \(T_t\) is zero before and counts time since the interruption after it. Thus \(\beta_2\) estimates an immediate level change and \(\beta_3\) estimates the change in slope. The post-change slope is \(\beta_1+\beta_3\). The error term \(\varepsilon_t\) represents unexplained variation; its dependence over time must be handled appropriately.
Worked Example: A Gradual Change in Prescribing
Suppose a hypothetical monthly series measures recommended-dose prescribing per 1,000 eligible treatment starts. An ITS model fitted to the monthly observations estimates a pre-change level at the interruption of 100, a pre-change slope of 2 additional starts per 1,000 per month, an immediate change of \(-12\), and a slope change of \(-1\) per month. These are illustrative fitted estimates, not data from an actual product.
At three months after the interruption, the model's counterfactual prediction under continuation of the pre-change trend is \(100+2(3)=106\) per 1,000. The fitted post-change prediction is \(100+2(3)-12-1(3)=91\) per 1,000. The estimated departure from the pre-change trend at that time is therefore \(91-106=-15\) per 1,000.
Equivalently, the estimated change at three months is \(\beta_2+3\beta_3=-12+3(-1)=-15\). The immediate fitted change is \(-12\) per 1,000, and the post-change slope is \(2-1=1\) additional start per 1,000 per month. The point estimates alone do not show whether these changes are distinguishable from random variation; uncertainty intervals and time-series diagnostics are also needed.
This interpretation depends on a reasonable continuation of the pre-change trend as the counterfactual, a clearly specified interruption, stable outcome definitions and data capture, and an adequate model for seasonality and serial dependence. If residuals remain correlated over time, ordinary regression standard errors can be too small. If seasonal cycles are present, failing to model them can make a predictable seasonal fluctuation look like an intervention effect. A concurrent intervention at the same time as the label change also weakens attribution.
Comparative Designs and Secular Trends
A concurrent comparison group can help distinguish a label-related change from trends shared across groups. The group should be affected differently by the label change but otherwise have a credible relationship to the target group over time. A group exposed to the same healthcare system changes but not the revised recommendation may be useful; a group with different trends or substantial spillover from the label change may not be.
One common contrast is difference-in-differences (DiD). Let \(Y_{T,pre}\) and \(Y_{T,post}\) be the outcome in the target group before and after the change, and \(Y_{C,pre}\) and \(Y_{C,post}\) the corresponding outcomes in a comparison group. For an additive outcome, the DiD estimate is
The first difference includes both the target group's change and any background trend. Subtracting the comparison group's change removes the shared trend if, absent the label change, the groups would have followed parallel trends on the chosen outcome scale. This is the central identifying assumption; it cannot be verified from a single before-after pair.
The parallel-trends assumption concerns what would have happened without the label change, not whether the groups have identical outcome levels. Multiple pre-change observations can reveal incompatible trends, although similar observed pre-trends do not prove that parallel trends would have continued after the change. Other requirements include consistent outcome measurement, no major group-specific co-intervention, and no substantial spillover that changes the comparator's outcome because of the label revision.
Worked Example: A Comparative Safety-Rate Analysis
A hypothetical label revision adds a safety warning for a target group. In equal observation periods, the target group has 24 events during 10,000 person-years before the change and 30 during 10,000 person-years after it. An unaffected comparison group has 20 events during 10,000 person-years before and 22 during 10,000 person-years after. Assume, for this calculation, independent Poisson event counts, fixed person-time, stable event ascertainment, and no material change in group composition.
The target group's pre-to-post rate ratio is \((30/10{,}000)/(24/10{,}000)=30/24=1.25\). The comparison group's rate ratio is \((22/10{,}000)/(20/10{,}000)=22/20=1.10\). Their ratio, a multiplicative DiD measure, is
This point estimate means the target group's event rate increased by a factor 1.136 relative to the comparison group's change. Under the stated independent-Poisson approximation, the standard error of its logarithm is
Since \(\log(1.136)=0.1278\), a 95% Wald interval on the log scale is \(0.1278\pm1.96(0.4129)=(-0.6814,\,0.9370)\). Exponentiating the endpoints gives an interval of approximately \((0.506,\,2.552)\) for the ratio of rate ratios. It includes 1, so these data do not provide clear evidence of a relative rate change; they also remain compatible with substantial decreases or increases. The result does not establish that the warning had no effect.
This calculation is a compact comparative analysis, not a substitute for checking trends across multiple periods. If person-time differs, the cell-specific rates must use their actual denominators; if events are clustered or overdispersed, the independent-Poisson standard error may be inadequate. A fitted count model can use a log person-time offset and include group, period, and their interaction, while allowing for relevant time patterns and dependence.
Choosing and Checking the Design
Pre-post comparisons, ITS, and comparative designs answer related but different questions. A simple pre-post estimate is transparent and can describe implementation, but it has little protection against secular trends. ITS uses the pre-change trajectory to construct a counterfactual, but remains vulnerable to simultaneous events and misspecified time patterns. A controlled ITS or comparative design can account for shared changes, but depends on a credible comparator and assumptions such as parallel trends. The design should follow the question and the available data, rather than treating the most complex model as automatically best.
| Design | What it estimates well | Key vulnerability |
|---|---|---|
| Pre-post | Observed difference between defined periods | Secular trends and changes in population or measurement |
| Interrupted time series | Departure from an extrapolated pre-change trajectory | Concurrent events, seasonality, or a misspecified trend |
| Comparative DiD or controlled time series | Change beyond that observed in a comparison group | Nonparallel trends, group-specific changes, or spillover |
Diagnostics should address both the data and the model. Plot outcomes over calendar time separately by group, marking the label change and any implementation lag. Check whether the number and characteristics of eligible patients, prescribing records, follow-up, and event coding change at the interruption. For time-series models, inspect residual patterns and seasonality, and assess whether conclusions depend on the time window, functional form, or handling of autocorrelation. For comparative analyses, examine pre-change group trends and consider whether the comparator could be affected indirectly.
When the label change is followed by several changes to clinical practice, the analysis may estimate the combined effect of implementation in that setting rather than the isolated effect of label text. Likewise, prescribing changes do not establish changes in exposure, safety, or effectiveness. Each link requires an appropriately measured outcome and a defensible design. Model uncertainty, missing data, delayed uptake, and limited event counts can also make estimates imprecise. Report what the design supports, and distinguish absence of clear evidence from evidence that an important effect is absent.
Common Pitfall
A second frequent error is to treat a nonsignificant estimate as proof of no effect. An interval that includes no change may also include clinically important increases or decreases, as in the safety-rate example. Interpretation should consider the interval's width, the clinical scale, data quality, and the consequences of uncertainty.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A prescribing proportion rises after a label update. Name two reasons this alone does not identify a label effect.
- In a segmented regression, what do \(\beta_2\) and \(\beta_3\) represent?
- A target and comparison group have changes of \(+8\) and \(+3\) percentage points, respectively. What is the additive DiD estimate, and what assumption is central to its causal interpretation?
- Why can a confidence interval for a safety effect that includes no change not establish that the label change had no effect?
- Name two diagnostics that can reveal problems in an interrupted time-series evaluation.
Solutions. (1) A secular trend could have raised the proportion, or the eligible population or data capture could have changed; a concurrent intervention is another possibility. (2) \(\beta_2\) is the immediate level change at the interruption, while \(\beta_3\) is the change in slope relative to the pre-change trend. (3) The estimate is \(8-3=5\) percentage points. Causal interpretation centrally requires parallel counterfactual trends: without the label change, the target group's trend would have changed like the comparison group's. (4) The interval may include both no effect and clinically important effects, so imprecision prevents a firm conclusion of no effect. (5) Examples include plotting the time series for seasonality or abrupt data shifts, checking residual autocorrelation, examining pre-change group trends, and verifying stable outcome ascertainment and population composition.