Why Adaptation Timing Changes the Design Problem
Adaptive design is often described as if an interim estimate is available as soon as the corresponding observations have been collected. In practice, data may require laboratory processing, adjudication, cleaning, or review before they can inform a decision. In addition, a team may wait for a scheduled batch of results before changing a sampling plan. The design must therefore specify not only what information would trigger a redesign, but also when that information will be available and when a revised plan can take effect.
When a decision is made using the latest available data, the estimate may be stale relative to the observations already collected but not yet reported. Those unreported observations can change what would be optimal, but they cannot be used as though they were known. A sound plan accounts for this gap by modeling the pending information predictively, by waiting for a planned batch when feasible, or by choosing a design that performs acceptably across plausible states of uncertainty.
Make the Information Timeline Explicit
Represent a plan with decision times \(\tau_1,\tau_2,\ldots\), and let \(a_j\) denote the design action chosen at \(\tau_j\), such as allocating remaining samples among candidate sampling times. The action must be a function of the data actually available then: \(a_j=\pi_j(D(\tau_j))\), where \(\pi_j\) is the decision rule. If the action takes \(\ell_j\) units of time to implement, it governs observations only from \(\tau_j+\ell_j\) onward. Data arriving during that interval cannot affect the action already chosen unless the plan explicitly allows another decision before implementation.
This distinction matters because a proposal that says “redesign at the interim” is incomplete unless it identifies the interim’s data cutoff and the action’s effective date. For example, an analysis scheduled when the first 12 participants finish sampling may not be actionable until results from those participants have been checked. If the next participants are enrolled before review is complete, they are not being treated under a design adapted to the interim results.
At a decision time, write the current uncertainty about model parameters \(\theta\) as \(p(\theta\mid D(\tau_j))\). A candidate action \(a\) can be evaluated by integrating its design utility over this uncertainty. If pending observations \(Y_{\mathrm{pend}}\) will arrive later, their values are not part of the current data. They can nevertheless be represented by their predictive distribution under the current information and the design used to collect them.
Here \(a_{\mathrm{current}}\) is the design under which the pending observations are being generated. This expression averages over parameter uncertainty; it does not reveal the pending outcomes. If the delay or probability of a result becoming available depends on its value, that observation process must also be represented. Otherwise, treating the observed data as a random sample from the planned model can be misleading.
Worked Example: Choosing a Design Before a Pending Batch Arrives
Suppose a team must choose one of two designs now, before a pending batch of results is available. Based on current data, the pending batch has a 0.30 probability of indicating a high-response state and a 0.70 probability of a low-response state. Utilities are hypothetical, in normalized design-utility units.
If the team can wait for the batch and then choose, design A has utility 10 in the high state and 2 in the low state. Design B has utility 6 in either state. If the team must commit now, expected utility for A is \(0.30(10)+0.70(2)=3.00+1.40=4.40\). For B it is \(0.30(6)+0.70(6)=1.80+4.20=6.00\). Thus, committing now favors B, even though A would be preferable after a high-state result.
If waiting is feasible, the team can select A after a high-state result and B after a low-state result. The expected utility of that contingent choice is \(0.30\max(10,6)+0.70\max(2,6)=0.30(10)+0.70(6)=7.20\). The expected gain from having the result before choosing is \(7.20-6.00=1.20\) utility units relative to the best immediate commitment. This gain must be weighed against the practical cost of waiting, such as delayed enrollment or missed opportunities to collect informative samples.
The calculation assumes the stated state probabilities and utilities adequately represent the decision, and that the team can act on the batch result before the next design period. If implementation itself takes time, the relevant utilities must reflect the design that will actually be used during that delay.
Evaluate a Decision with Stale Information
If it is impossible to wait, optimize against the information available now rather than substituting a forecast as if it were observed. For a candidate action \(a\), a general decision criterion is its expected utility under current uncertainty:
Here \(U(a,\theta)\) is the utility of using action \(a\) when the model parameter is \(\theta\). The chosen action maximizes \(\Psi\) among actions that satisfy the remaining budget and operational constraints. When pending observations will affect a later action, their predictive distribution can be included in a multi-period objective. The current action remains fixed with respect to those unknown results; a later action may depend on them only if a later decision is genuinely available.
The following comparison makes the distinction precise. Suppose the next action can be selected either now or after pending data \(Y\) arrive. Let \(\Psi(a\mid D,Y)\) be the utility criterion after observing \(Y\), and let \(\Psi(a\mid D)\) be the current criterion. Choosing now gives \(\max_a E[\Psi(a\mid D,Y)\mid D]\) only if the same action must be committed before \(Y\) is known; because that action cannot vary with \(Y\), the appropriate value is \(\max_a E[\Psi(a\mid D,Y)\mid D]\) when the criterion is expressed as the eventual utility of that fixed action. Waiting permits the action to depend on \(Y\), giving \(E[\max_a\Psi(a\mid D,Y)\mid D]\). The latter cannot be smaller, because for every fixed \(a\), \(\max_{a'}\Psi(a'\mid D,Y)\geq\Psi(a\mid D,Y)\); averaging and then maximizing over \(a\) preserves the inequality.
The inequality is a justification for valuing timely information, not a claim that waiting is always best. Delays can consume calendar time, disrupt a sampling schedule, or reduce the number of observations remaining. The operational objective should include those consequences, using the same units or an explicitly defined trade-off.
Worked Example: Comparing Immediate and Batched Decisions
Consider a hypothetical study with 24 participants and a fixed remaining sampling budget. The team can revise the allocation after every result, but doing so requires a separate review; alternatively, it can review every eight results. Suppose a planning calculation gives the following expected design utility, in normalized units, for the two schedules. The values already include the uncertainty in the model parameters and the fact that adaptations occur only at the stated review times.
| Decision schedule | Expected utility | Review effort cost | Net utility |
|---|---|---|---|
| Review after each result | 18.0 | 5.0 | 13.0 |
| Review after each batch of 8 | 16.5 | 1.5 | 15.0 |
| No interim redesign | 14.0 | 0.0 | 14.0 |
For the batched schedule, net utility is \(16.5-1.5=15.0\). Reviewing after every result has the largest gross utility, \(18.0\), but its net utility is \(18.0-5.0=13.0\). Under these hypothetical values and this specified accounting, the batch of eight is preferred: it exceeds the fixed plan by \(15.0-14.0=1.0\) net utility unit and exceeds the immediate-review plan by \(15.0-13.0=2.0\).
The calculation assumes that the review costs are expressed on a scale commensurate with design utility and that the quoted expected utilities reflect the actual review schedule. If operational costs cannot reasonably be converted to utility units, report the design-performance and resource consequences separately rather than subtracting incomparable quantities.
Plan the Batch Size and the Decision Rule Together
Batch size is not merely an administrative choice. Larger batches reduce the frequency of decisions and may reduce review burden, but they leave the design unchanged for longer while information accumulates. Smaller batches can respond sooner, but may produce unstable choices if each update is based on little information, and may create substantial operational work. The preferred schedule depends on the decision’s sensitivity to new observations, the cost of review, the delay to implementation, and the amount of budget remaining.
A useful planning procedure is to specify, before the study begins, when results are expected, when they are considered usable, and which observations each redesign can influence. Then evaluate candidate schedules under the full timing plan, including pending data. This keeps a design decision from being credited with information that arrives too late to change its consequences.
For each planned interim, state which observations will have been collected, quality-checked, and released by the decision time. Identify any result whose availability may be delayed.
State when a redesign can be approved and when it takes effect. Identify observations collected during the implementation lag and the design that will govern them.
Evaluate immediate, batched, and no-redesign options using the same objective, remaining budget, and operational constraints. Include review burden and the consequences of waiting where appropriate.
Define what information enters each update, which decision rule is applied, and how unexpected delays or incomplete batches are handled. Do not silently replace unavailable results with assumed values.
Comparison, Diagnostics, and Limitations
| Approach | Information used | Main advantage | Main limitation |
|---|---|---|---|
| Immediate sequential adaptation | Each newly available observation | Can respond quickly to informative data | Requires timely processing and frequent decisions; updates may be operationally burdensome |
| Batched adaptation | Results available at scheduled batch reviews | Fits coordinated operations and reduces review frequency | Cannot use information until the next decision time; choices may rely on older estimates |
| Fixed robust design | Uncertainty characterized before data collection | Does not depend on interim data availability | Cannot respond to information learned during the study |
These approaches should be compared on a common basis: the same scientific objective, plausible parameter uncertainty, resource limits, and calendar assumptions. Immediate adaptation is not automatically superior if the information arrives too late or decision overhead is substantial. A fixed robust design may be preferable when reliable interim data are unlikely to be available in time, although it gives up the opportunity to respond to observed information.
A practical timing check is to compare the planned collection, release, review, and implementation dates with the data each action is supposed to influence. Recalculate the decision using only the data that would actually be available. If a preferred action changes sharply when plausible pending outcomes are considered, the plan is sensitive to the delay; this is a reason to examine alternative schedules or choose a less brittle action, not to pretend the pending outcomes are known.
A limitation of predictive averaging is that it depends on the model for pending observations and parameter uncertainty. If the model poorly represents the data-generation process, or if availability depends on unmodeled features of the results, the predicted value of waiting or adapting may also be unreliable. Timing plans can also change for non-statistical reasons. A prespecified rule should say how the team will handle a delayed batch, a partial batch, or a review that cannot occur as planned, while keeping the decision tied to the information genuinely available.
Check Your Understanding
Answer each question using the timing and information distinctions developed above. Brief solutions follow.
- At a redesign meeting, six observations have been collected but only four have passed review. Which observations may be used as observed data for the decision?
- A pending batch has two possible states, each with probability 0.50. Action A has utilities 8 and 0 in the two states; action B has utility 3 in both states. What is the best immediate action, and what is the expected utility if the team can wait and choose after seeing the state?
- Why must an implementation lag be included when assessing whether an interim result can improve the next sampling allocation?
- A team reduces review frequency to lower operational burden. Name one potential design benefit and one potential design cost of batching.
Solutions. (1) Use only the four observations available after review; the other two are pending, not observed information for this decision. (2) Immediate expected utility is 4 for A, from \(0.50(8)+0.50(0)\), and 3 for B, so choose A. Waiting gives \(0.50(8)+0.50(3)=5.5\), because the team chooses A in the first state and B in the second. (3) Results arriving during implementation cannot change an action that has already been committed; the lag determines which observations that action can affect. (4) Batching can reduce review burden, while its cost is that the design may remain unchanged as information accumulates between review times.