Why Uncertainty Needs Special Attention
Adjusting for calendar time can improve the precision and relevance of a treatment-effect estimate, but it does not make its standard error automatic. The estimate may combine information across enrollment periods, use several parameters to represent a trend, or rely on treatment groups that are thinly represented in some periods. In an adaptive trial, assignment probabilities also change in response to the accumulating trial history. An uncertainty estimate is useful only if its assumptions match the analysis that produced the estimate.
The central distinction is between uncertainty calculated from an outcome model, conditional on the observed design variables, and uncertainty that explicitly reflects the sequential assignment process. Neither approach is universally preferable. The appropriate choice depends on the estimand, the assumptions about outcomes and assignment, and the amount of information available in the relevant time periods.
Uncertainty for a Time-Standardized Contrast
Suppose enrollment is divided into \(K\) calendar-time periods. Let \(\hat{\Delta}_k\) be the estimated treatment contrast in period \(k\), and let \(w_k\) be a prespecified target weight, with \(w_k \geq 0\) and \(\sum_{k=1}^K w_k=1\). A standardized contrast is
For example, the weights might represent the proportions of the trial population enrolled in each period. They could instead define another target population, but that target should be stated. The weights are treated as fixed in the standard variance formula below. If the weights are estimated from the same data, their uncertainty and covariance with the period-specific estimates may also matter.
If the period-specific estimates are independent and have estimated variances \(SE(\hat{\Delta}_k)^2\), then
The square on each weight is essential. If the estimates are correlated, the variance also contains covariance terms:
For a simple comparison of independent treatment groups within each period, an estimated period-specific variance is \(s_{1k}^2/n_{1k}+s_{0k}^2/n_{0k}\), where \(n_{1k}\) and \(n_{0k}\) are the treatment and control sample sizes and \(s_{1k}\) and \(s_{0k}\) are their sample standard deviations. This calculation assumes independent outcomes within groups, and that the sample variances are informative about the outcome variances. It can be poor when a group is very small.
Worked Example: Standardizing Across Two Enrollment Periods
Consider a hypothetical trial with a continuous outcome measured in points. The prespecified target weights are 0.60 for the early period and 0.40 for the late period. In the early period, the treatment group has \(n_1=30\), \(\bar{x}_1=8.2\), and \(s_1=2.4\); the control group has \(n_0=30\), \(\bar{x}_0=9.0\), and \(s_0=2.1\). In the late period, the corresponding summaries are \(n_1=8\), \(\bar{x}_1=6.5\), \(s_1=2.8\), and \(n_0=24\), \(\bar{x}_0=7.3\), \(s_0=2.0\). Assume independent outcomes within each treatment-by-period group and use the sample variances to estimate the outcome variances.
The estimated contrasts are \(8.2-9.0=-0.8\) points in the early period and \(6.5-7.3=-0.8\) points in the late period. Thus
The estimated variance for the early-period contrast is \(2.4^2/30+2.1^2/30=5.76/30+4.41/30=0.339\). For the late-period contrast it is \(2.8^2/8+2.0^2/24=7.84/8+4.00/24=1.147\). Therefore,
Using a normal approximation, a 95% interval is \(-0.8\pm1.96(0.553)\), or approximately \((-1.88,\ 0.283)\) points. Under the stated assumptions and approximation, the data are compatible with effects ranging from a reduction of about 1.88 points to an increase of about 0.283 points. The late period contributes more variance despite its smaller target weight because it has only eight treated participants. The normal approximation may itself be inaccurate with so few treated participants.
Model-Based and Sandwich Uncertainty
A calendar-time outcome model might include treatment, a function of enrollment time, and—when treatment effects may vary over time—a treatment-by-time interaction. Let \(\boldsymbol{\beta}\) collect all fitted coefficients, and let \(\hat{\boldsymbol{\beta}}\) be their estimates. A treatment effect standardized over a target set of times can be written as a function \(g(\hat{\boldsymbol{\beta}})\). Its estimated variance can be obtained by the delta method:
The coefficient covariance matrix must include the covariance between treatment and time-related coefficients. Using only the separate coefficient variances can substantially misstate uncertainty. For ordinary least squares, the usual model-based covariance uses a constant conditional residual variance. A sandwich covariance instead uses the fitted design and the observed residual contributions to allow a less restrictive variance pattern. This can help when residual variability differs across observations, but it does not make an incorrect mean model correct.
The distinction can be justified from the estimating equations. In a linear model, the coefficient estimate solves \(\sum_i \boldsymbol{x}_i(Y_i-\boldsymbol{x}_i^\mathsf{T}\boldsymbol{\beta})=0\), where \(\boldsymbol{x}_i\) contains the model terms for participant \(i\). A first-order expansion around the true coefficient gives
If the errors are independent with constant variance, their covariance reduces to a common variance times the design-matrix term used by the model-based calculation. If their variances differ, the covariance of the summed contributions instead depends on the individual residual variances. The sandwich estimate replaces those unknown variances with residual-based quantities. This derivation relies on an adequate conditional mean model and enough information for the empirical covariance to be stable.
Worked Example: Propagating Uncertainty Through a Treatment-by-Time Model
Suppose a hypothetical fitted model uses centered calendar time \(z\), and its treatment contrast at time \(z\) is \(\hat{\tau}(z)=\hat{\beta}_A+\hat{\beta}_{At}z\). The target population has average centered time \(\bar{z}=0.40\). Suppose \(\hat{\beta}_A=0.30\) points, \(\hat{\beta}_{At}=0.12\) points per time unit, \(\widehat{\operatorname{Var}}(\hat{\beta}_A)=0.0400\), \(\widehat{\operatorname{Var}}(\hat{\beta}_{At})=0.0225\), and \(\widehat{\operatorname{Cov}}(\hat{\beta}_A,\hat{\beta}_{At})=-0.0100\). These values are hypothetical and assumed to come from a suitable fitted model.
The standardized contrast is \(g(\boldsymbol{\beta})=\beta_A+0.40\beta_{At}\), so the estimated contrast is
The delta-method variance includes the covariance term:
Thus \(SE=\sqrt{0.0356}=0.189\) points. A normal-approximation 95% interval is \(0.348\pm1.96(0.189)\), or approximately \((-0.022,\ 0.718)\) points. It includes zero. The interval describes uncertainty in the standardized contrast under the model and covariance assumptions; it does not show that the effect is constant over time.
What Adaptive Assignment Changes
Let \(A_i\) indicate whether participant \(i\) is assigned to treatment, and let \(p_i=P(A_i=1\mid\mathcal{H}_{i-1},t_i)\), where \(\mathcal{H}_{i-1}\) is the trial history available before that assignment and \(t_i\) is enrollment time. In an adaptive trial, \(p_i\) can vary with the accumulated history. A regression analysis that treats the observed treatment indicators as fixed can still support valid outcome-model uncertainty estimates when the conditional outcome model and its error assumptions are adequate. But it is not automatically valid to use a variance formula that assumes a simple, constant-probability assignment scheme.
When the analysis uses inverse-probability weighting, the \(p_i\) values enter the estimator itself. Their variation can make contributions highly unequal: a participant assigned to an arm with a small probability receives a large weight. Uncertainty calculations must reflect those weighted contributions and the sequential design assumptions. The assignment probabilities should be the probabilities actually used at each assignment, conditional on information available then—not probabilities reconstructed from the final treatment totals.
For an outcome-regression analysis, a sandwich covariance can be appropriate when the estimating contributions have mean zero given the available history and the relevant design variables, with sufficiently stable information as the trial grows. That sequential condition matters: adaptive assignments are not generally independent of the trial’s past. A sandwich calculation cannot rescue a model whose treatment and time terms are not supported by the observed overlap. Nor does a robust standard error correct bias from a misspecified trend or an inappropriate target population.
Worked Example: Why Assignment Probabilities Matter for Weighted Contributions
In a hypothetical adaptive trial, four participants have treatment probabilities \(p_i\) of 0.50, 0.25, 0.75, and 0.40, calculated from the history available at each enrollment. Their observed assignments and outcomes are: treated with outcome 10, control with outcome 8, treated with outcome 7, and control with outcome 6. For illustration, consider the inverse-probability contrast contribution \(D_i=A_iY_i/p_i-(1-A_i)Y_i/(1-p_i)\). This contribution is not, by itself, a complete uncertainty estimate.
The four contributions are
Their average is \((20-10.667+9.333-10)/4=2.167\) outcome units. The second participant’s control observation is weighted by \(1/(1-p_2)=1/0.75\); the first treated observation is weighted by \(1/p_1=1/0.50\). A small assignment probability would produce an even larger contribution. Consequently, an uncertainty calculation based on treating these weighted observations as though they had equal precision would be inappropriate. A valid variance estimate requires assumptions and a calculation matched to the weighted estimator and sequential assignment process. These four observations alone do not establish that such a variance estimate is reliable.
Complexity, Sparse Periods, and Diagnostics
More flexible calendar-time adjustment can represent nonlinear drift, but it also uses more parameters. Each additional trend parameter and treatment-by-time interaction must be supported by data across the time range. When some periods contain few participants in one arm, coefficient estimates can be highly sensitive to a small number of outcomes, and the estimated covariance matrix may be unstable or nearly singular. A small reported standard error in that situation is not proof that the estimate is precise.
Before interpreting an interval, check whether the calculation reflects the estimand and the fitted model:
- Check treatment counts and assignment probabilities across the time range. Very small probabilities, or periods with few participants in one arm, signal that weighted or model-based uncertainty may be unstable.
- Inspect residual patterns over time and by treatment. Systematic patterns suggest the conditional mean model may not capture the drift or the treatment-by-time relationship.
- Look for unusually large weights, influential observations, and coefficient correlations close to one in magnitude. These can make sandwich estimates and delta-method intervals sensitive to individual observations.
- Report the model terms, target weights, covariance method, reference distribution, and relevant degrees of freedom. With limited data, a normal approximation may understate uncertainty; finite-sample corrections or a suitable small-sample reference distribution may be preferable, but do not create information absent from the data.
| Approach | Useful when | Main caution |
|---|---|---|
| Model-based covariance | The outcome mean and variance assumptions are credible, and the model is adequately supported. | Can be misleading if residual variances differ or the model is misspecified. |
| Sandwich covariance | Mean-model estimating equations are credible and independent or sequential contributions have suitable large-sample behavior. | Can be unstable in small samples or sparse periods; does not fix bias or lack of overlap. |
| Period-specific standardization | The period-specific contrasts and target weights are clear and each period has adequate information. | Can be imprecise when an arm is sparse; covariance terms matter if period estimates are dependent. |
| Uncertainty accounting for adaptive assignment | The estimator uses changing assignment probabilities or the inferential target explicitly concerns the randomization process. | Requires the probabilities used at each assignment and assumptions that match the sequential design. |
Common Mistake and Practical Limits
Wrong reasoning: “The model adjusts for calendar time, so the usual standard error is automatically valid, even though treatment probabilities changed and some periods are sparse.” Adjustment and uncertainty are separate issues. A time term may reduce confounding by drift under the model, but it does not guarantee a correct covariance estimate, stable treatment contrasts, or adequate overlap.
Correction: State the target contrast and time weights, then identify the covariance calculation and the assumptions supporting it. If the estimator uses adaptive probabilities, incorporate those probabilities in a variance procedure appropriate to that estimator. If overlap or period counts are weak, report the resulting uncertainty candidly; a more elaborate covariance formula cannot compensate for missing information.
Model-based and sandwich intervals are generally large-sample approximations, and their reliability can be limited by small enrollment, highly irregular allocation, extreme weights, or a trend model selected to fit noise. These methods also do not account automatically for uncertainty from choosing the model after examining treatment results. The interval should be interpreted as conditional on the analysis choices and assumptions actually used.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why are the period weights squared when calculating the variance of a standardized contrast?
- A treatment-by-time model estimates \(\hat{\tau}(z)=\hat{\beta}_A+\hat{\beta}_{At}z\). What covariance term is needed for the variance of the contrast at a fixed \(z\)?
- Why can a sandwich standard error be unstable when one treatment arm is sparse in a late enrollment period?
- In an adaptive trial, why should the final treatment proportions not be substituted for the probabilities used at each assignment in an inverse-probability analysis?
- Does a robust standard error correct a calendar-time trend model that has the wrong conditional mean? Explain.
Solutions. (1) Multiplying an estimate by \(w_k\) multiplies its variance by \(w_k^2\); covariances between period estimates also contribute when they are dependent. (2) The variance includes \(2z\,\widehat{\operatorname{Cov}}(\hat{\beta}_A,\hat{\beta}_{At})\), as well as the two coefficient variances. (3) Sparse data provide little information about the residual contributions and treatment contrast in that period, so estimated covariance quantities can be highly variable and sensitive to individual outcomes. (4) Adaptive probabilities depend on the history available at each assignment and can change over enrollment; final proportions do not generally recover those conditional probabilities. (5) No. A sandwich calculation can relax some variance assumptions, but it does not correct a misspecified mean model or remove bias from an unrepresented time trend.