Why the Replicate Is the Unit of the Output
Operating characteristics describe what a trial procedure does across hypothetical trials generated under specified conditions. A simulation therefore needs to preserve the result of each hypothetical trial, not just a final summary across many runs. The key design choice is to create one output record for each simulated trial replicate.
A replicate is one complete pass through the trial procedure: generate trial data under a specified scenario, apply the planned analysis and decision rule, and retain the resulting outputs. It is not an individual patient, a posterior draw, or an iteration of an algorithm. Keeping these units distinct makes it possible to count decisions and examine the estimates, intervals, and computational outcomes associated with each decision.
The output design should follow the fixed trial procedure and the performance questions it will later support. The data-generating mechanism and scenario matrix specify what is held fixed for a scenario; the replicate record captures what happened in one generated trial. Scenario-level information can be stored in a separate table, but each replicate record must contain an identifier that links unambiguously to that table.
A Practical Record Structure
A useful record separates identifiers, trial results, analysis results, and computation status. Names can differ across software systems, but the meaning and coding of each field should be explicit and consistent.
| Field group | Example fields | Purpose |
|---|---|---|
| Identifiers | scenario_id, replicate_id | Identify the simulated condition and distinguish every trial within it. |
| Scenario link | treatment effect, control response, allocation ratio, sample size | Make the generating conditions accessible directly or through scenario_id. |
| Trial realization | arm-specific event counts, sample means, analysis sample size | Support checks of the generated data and the analysis input. |
| Decision | decision, decision_rule_version | Record the result of applying the specified rule, using a documented coding. |
| Estimator and interval | estimate, lower, upper, interval_level, interval_type | Preserve the target, estimate, scale, and interval interpretation. |
| Computation status | status, failure_reason, diagnostics_summary | Distinguish a completed analysis from a failed or unresolved computation. |
Identifiers should be unique in combination: the pair \((\text{scenario\_id},\text{replicate\_id})\) must identify exactly one record. Replicate numbers may restart at 1 in each scenario, provided that the scenario identifier is retained. A record should also identify the version of the decision rule or analysis code when outputs might otherwise be ambiguous.
The trial decision needs a prespecified representation. For example, define \(D_{si}=1\) if replicate \(i\) in scenario \(s\) meets the stated success rule, and \(D_{si}=0\) otherwise. If the procedure allows another outcome, such as inconclusive, retain that category rather than silently forcing it into success or failure. The rule must specify what the decision means and which posterior quantity or other analysis output determines it.
An estimator field must identify the estimand and its scale. A numeric value alone is not enough: a difference in response probabilities, a log odds ratio, and a ratio of probabilities are not interchangeable. Interval fields likewise need a declared interval type and level, such as a 95% posterior credible interval for a risk difference. Store lower and upper endpoints separately and retain the units or scale in the field definition.
Worked Examples
Worked Example: A Binary Superiority Trial
Suppose a hypothetical calibration scenario generates 100 patients per arm. The decision rule declares success when the posterior probability that the treatment response probability exceeds the control response probability is at least 0.95. In one simulated replicate, there are 36 responses among 100 treated patients and 24 among 100 control patients. The analysis reports a posterior superiority probability of 0.968 and a 95% posterior credible interval for the risk difference from \(-0.010\) to \(0.250\).
The replicate identifier might be scenario 12, replicate 37. The observed response proportions are \(36/100=0.360\) and \(24/100=0.240\). The estimated risk difference is therefore
Its unit is a probability difference (12.0 percentage points), not a percent increase relative to control. Since \(0.968 \geq 0.950\), the decision field is success, or \(D_{12,37}=1\). The record should preserve the posterior probability used by the rule, the estimated risk difference, both interval endpoints, the interval level and type, the response counts, and the scenario and replicate identifiers. The interval includes zero even though the specified posterior-probability rule declares success; that is not an inconsistency, because the decision rule and the interval summary are distinct outputs.
Worked Example: A Continuous Endpoint and Its Interval
Consider a hypothetical scenario with a continuous endpoint measured in days, where the estimator is the treatment-minus-control difference in sample means. In one replicate, the treatment mean is 5.8 days and the control mean is 3.4 days. Suppose the prespecified analysis reports a 90% posterior credible interval from \(-0.2\) to \(5.0\) days, and the decision rule requires the posterior probability that the difference is greater than 1 day to be at least 0.90. The reported posterior probability is 0.88.
The point estimate is calculated directly from the arm means:
Because \(0.88 < 0.90\), the record's decision is no success, \(D=0\). The interval endpoints remain \(-0.2\) days and \(5.0\) days, and its level is 90%, not 95%. A complete record also specifies that positive differences favor treatment, identifies the interval as posterior credible, and links the result to its scenario and replicate. These details prevent a downstream reader from reversing the contrast or mistaking the interval for one on another scale.
Worked Example: Recording an Unresolved Computation
Suppose a hypothetical trial replicate has been generated, but the posterior computation fails a prespecified convergence check. Under the calibration protocol, no decision is assigned unless that check passes. The trial has 50 participants per arm, but the failed computation does not produce a valid posterior estimate or interval.
The record still exists: it contains the scenario and replicate identifiers, the observed data summaries, and a status such as “convergence failure,” along with the relevant diagnostic or failure reason. The decision, estimate, and interval are marked missing using the agreed missing-value representation. They must not be entered as \(D=0\), estimate \(=0\), or interval \([0,0]\): those values would falsely describe a completed analysis with numerical results. Under this protocol, the correct record states that the analysis did not yield a decision. If the operational trial procedure instead specifies that a computational failure triggers a particular action, that action must be represented according to that prespecified rule, while the failure status is still retained.
Why These Fields Are Sufficient and How to Check Them
The reason to retain one row per replicate can be stated directly. For a binary decision, each completed record supplies one value of \(D_{si}\), tied to a known scenario. A later calculation can select the relevant scenario and apply the intended summary to its decision values. If only a pooled total is saved, the scenario membership of each outcome may be lost; if only a final rate is saved, the individual decisions and supporting analysis results cannot be audited.
Similarly, estimates and intervals should remain attached to the replicate that produced them. The record then supports checks such as whether an interval contains its own point estimate when the interval construction guarantees that property, whether endpoints are ordered, and whether the reported decision agrees with the stored decision-rule quantity. Such checks do not establish that the statistical model is appropriate, but they can expose data-handling or implementation errors.
State that one row represents one generated trial and specify a unique scenario–replicate key.
Name the estimand, contrast direction, estimator scale, interval type and level, and the exact decision rule.
Define which statuses count as completed, how failures are recorded, and whether a failed analysis can trigger a decision under the actual procedure.
Check uniqueness, required fields, legal status and decision codes, endpoint order, units, and agreement between stored quantities and the rule.
The procedure used to create the record must be held fixed across replicates within a calibration condition, except for elements deliberately varied by the scenario. Otherwise, a change in outputs could reflect a changed analysis rule rather than random variation in generated trial data. Diagnostics should also be retained at a useful level: a status alone tells whether computation completed, while a short failure reason or diagnostic summary helps identify recurring problems.
Common Design Errors and Alternatives
A frequent mistake is to code every non-success as \(D=0\), including a computation that never produced a valid decision. The reasoning is that “no success was observed, so this must be a failure.” The correction is to distinguish the trial decision from the analysis status. A valid completed analysis may yield \(D=0\); an unresolved computation has no decision under a rule that requires a valid analysis. If the real procedure explicitly treats such a failure as a particular action, encode that rule transparently and keep the computational status as a separate field.
Another mistake is to save only the final operating-characteristic summary. That is compact, but it removes the replicate-level evidence needed to verify the calculation, inspect failures, or calculate a different prespecified metric. At the other extreme, keeping only a large file of individual patient records can make routine checking cumbersome and may make it unclear which analysis result belongs to which simulated trial. A practical alternative is to retain both: a compact replicate-level output table and, when needed, trial-level data linked by scenario and replicate identifiers.
The right amount of detail depends on the calibration questions, but several limitations remain. A well-designed record cannot repair a misspecified data-generating mechanism, an inappropriate estimand, or an incorrectly implemented decision rule. Nor does a successful convergence flag by itself prove that a posterior computation is accurate. The record structure makes assumptions and outputs traceable; statistical validation and computational diagnostics are still needed.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Why is a posterior draw not a replicate-level trial record?
- A record has an estimate of \(-0.15\), but does not identify the contrast direction or scale. What additional information is needed?
- A trial analysis fails before producing a decision. Under a protocol that assigns decisions only after successful analysis, how should the decision and status fields be recorded?
- For a rule requiring posterior probability at least 0.975, a completed replicate reports 0.973. What decision code is appropriate if 1 means success and 0 means no success?
- What is the purpose of retaining both scenario_id and replicate_id?
Solutions. (1) A posterior draw is an algorithmic draw within an analysis; a replicate is one generated trial passed through the full procedure. (2) Specify the estimand, contrast direction, and scale or units, such as treatment-minus-control risk difference. (3) Retain the record and identifiers, record the failure status and reason, and mark the decision as missing under the stated protocol. (4) Since \(0.973 < 0.975\), record no success, \(D=0\). (5) Together the identifiers locate the generating scenario and distinguish the replicate from other trials within that scenario.