From a Design Calculation to a Reproducible Workflow
A group-sequential design may be calculated correctly and still be difficult to review if its inputs are scattered across notes, a spreadsheet, and a report. A later change to one assumption can leave a table showing the old result beside a figure showing the new one. A scripted workflow reduces this risk by making the path from stated inputs to saved outputs explicit and repeatable.
The introductory material on selecting software for group sequential planning addresses choosing a suitable tool. Here the focus is what happens after that choice: how to run the calculation from recorded inputs, retain the results, and regenerate report elements consistently. The workflow does not replace statistical review. It makes the computational steps inspectable and reduces opportunities for untracked manual changes.
The core assumption is that the calculation engine is appropriate for the design being implemented and that its inputs correctly represent the approved design specification. Reproducible execution cannot make an unsuitable statistical method appropriate. The workflow also assumes that input values are complete and internally consistent, that software dependencies are available at recorded versions, and that generated report elements are not subsequently edited by hand. If a calculation contains randomness, its random-number handling must be specified; for a deterministic design calculation, identical inputs and software should yield identical results.
A Minimal Workflow Architecture
Keep the workflow conceptually divided into stages. The input stage holds the values needed by the calculation, such as the number and timing of analyses, the target error rate, the power target, and the effect-size assumptions. The calculation stage passes those values to the selected design software. A validation stage checks both the inputs and the returned results. Finally, an output stage saves the results and builds tables and figures directly from them.
Check required fields, allowed ranges, units, and consistency before starting the calculation. Stop with a clear error if an input is absent or invalid.
Call the chosen software with the validated values. Capture the returned design quantities and any warnings or errors rather than copying results manually.
Test that the output has the expected number of analyses, contains finite values, and satisfies basic ordering or range checks. Save the results in a machine-readable form.
Build tables and figures from the saved results in the same run. Include clear labels, units, and the assumptions needed to interpret them.
Inspect warnings and outputs, then retain the inputs, software version information, results, and generated files together so the run can be checked or repeated.
This order matters. If a table is typed separately after the calculation, it becomes a second source of truth. If instead the table is generated from the saved result object, changing an input and rerunning the workflow updates the calculation and the table through the same path. A short justification for this structure is the dependency chain: report outputs depend on calculation results, which depend on inputs and software. Recomputing downstream elements after an upstream change preserves that dependency; editing a downstream element manually breaks it.
Checks should be tailored to the calculation rather than treated as proof that the design is correct. General checks include confirming that required results are present, values are finite, and the number of returned looks matches the requested design. Contextual checks can verify that information fractions are between 0 and 1 and increase across looks. These checks catch common implementation mistakes; they do not establish that the statistical assumptions or design choices are suitable.
Worked Examples
Worked Example: Converting Information Fractions Into a Report Table
Suppose a hypothetical design has a maximum planned information corresponding to 240 participants. Its three planned analyses occur at information fractions 0.50, 0.75, and 1.00. The calculation software returns those fractions, and a report needs to show the corresponding planned participant counts. The workflow should derive the counts rather than have them entered manually.
Assumption check. For this illustration, assume that information is proportional to the number of participants, the maximum count is 240, and the supplied fractions are valid, strictly increasing fractions no greater than 1. These assumptions make the simple conversion below appropriate for this display. In designs where information is not proportional to participant count, the workflow must use the design's appropriate conversion instead.
For each look, multiply the maximum planned count by its information fraction:
The arithmetic is \(240(0.50)=120\), \(240(0.75)=180\), and \(240(1.00)=240\). The generated table therefore contains planned counts of 120, 180, and 240 participants. A validation check confirms that the counts increase and that the final count equals the maximum. The table is then created from the calculated values, not from a separately maintained copy. If the maximum changes to 260, rerunning the same calculation gives \(260(0.50)=130\), \(260(0.75)=195\), and \(260(1.00)=260\), and the report table updates accordingly.
Worked Example: Generating a Boundary Table and Figure From One Result
Consider a hypothetical three-look design passed to an appropriate group-sequential calculation program. For illustration only, suppose the program returns the following upper efficacy boundaries on the standardized \(Z\) scale: 2.80, 2.30, and 2.02. These numbers are stipulated as returned values for demonstrating the workflow; they are not a design recommendation or a claim that these boundaries apply to any particular error-spending specification.
The workflow first checks that three boundaries were returned for three planned looks, that each value is finite, and that the results are in the expected order for this design. It then constructs a table from the result object:
| Analysis | Upper efficacy boundary, \(Z\) scale |
|---|---|
| 1 | 2.80 |
| 2 | 2.30 |
| 3 | 2.02 |
The figure is generated from the same three pairs: analysis number 1 with boundary 2.80, analysis number 2 with boundary 2.30, and analysis number 3 with boundary 2.02. Thus, the third point in the figure is calculated from the same returned value shown in the third table row. The caption identifies the values as \(Z\)-scale boundaries and states the design assumptions needed to interpret them. If the software returns a different value at the second look after an input change, both table and figure change together on rerun. A visual inspection remains necessary: the figure should have three points, correctly labelled axes, and no unexplained missing or duplicated values.
Worked Example: Detecting a Broken Run Before Reporting
A hypothetical workflow expects four planned analyses, but the calculation step returns only three results because one requested look was omitted from the software call. The returned information fractions are 0.25, 0.50, and 1.00. Without a check, the output stage might create a polished table and figure that conceal the missing analysis.
The validation rule compares the expected and returned numbers of looks. Here, \(4 \ne 3\), so the run fails validation and does not generate final report files. A second check compares the planned and returned fractions. The expected sequence includes four values, while only three are present; the output is therefore incomplete even though the returned fractions are individually between 0 and 1 and increasing.
The correction is to inspect the inputs passed to the calculation and the software call, restore the missing look, and rerun the complete workflow. The workflow should preserve the error message and failed-run record for review rather than silently filling in a value or editing the report table to make it appear complete. Once the calculation returns four valid results and the checks pass, the table and figure can be generated.
Reproducibility Checks, Alternatives, and Limitations
A workflow is useful only if it gives clear evidence of what was run and makes failures visible. At minimum, retain the exact inputs used, the software and relevant dependency versions, warnings, calculation results, and generated files. These are run records, not a substitute for a review of whether the design assumptions are scientifically justified. When a workflow is rerun, compare important outputs with the prior run and investigate unexpected differences rather than assuming that a successful completion means equivalence.
Common diagnostics include:
- Input validation fails. Check spelling, units, missing values, allowed ranges, and whether inputs describe the same design. Do not bypass the check by inserting a plausible-looking default.
- The number of returned results is wrong. Compare requested and returned analyses, then inspect the software call and warnings. Do not generate a partial table as if it were complete.
- Outputs change with no intended design change. Check software versions, dependencies, hidden defaults, and any random component. A version or default change may alter results even when the visible inputs appear unchanged.
- A figure disagrees with its table. Confirm that both are generated from the same saved result object and that neither has been manually edited after generation.
- Files are missing or overwritten. Use a clear output location and retain the run's inputs and results alongside the report files. Verify that expected outputs were actually written.
Manual spreadsheets can be useful for exploration and independent spot checks, but copied values and hand-maintained charts are vulnerable to transcription and synchronization errors. A graphical software interface can be easier to use for one-off calculations, but a reproducible workflow needs a way to retain the settings and recreate all derived outputs. A scripted workflow offers automation and repeatability, at the cost of requiring code review, dependency control, and maintenance. These approaches can complement each other: an independent calculation may help identify an implementation error, but it should not be presented as independent if it uses the same code or assumptions.
Automation also has limits. A script can consistently implement the wrong design, use an inappropriate default, or generate a misleading graph from correctly computed values. Numerical equality across different software platforms may not be exact because of implementation and rounding differences; comparisons should use a pre-specified, defensible tolerance where appropriate. Finally, reproducibility is not the same as auditability or statistical validity. A retained record makes a run traceable, while qualified review must still assess whether the inputs, model, and intended interpretation are appropriate.
Check Your Understanding
Answer each question before consulting the solutions.
- A design has a maximum of 300 participants and information fractions 0.40, 0.70, and 1.00. Under the assumption that information is proportional to participant count, what counts should the generated table show?
- A workflow expects three analyses but receives only two sets of boundaries. What should it do, and why is manually adding a row a poor correction?
- Name two reasons that identical visible inputs might produce different results on a later run.
- A table is generated from a calculation result, but a figure is copied from an old report. Explain the reproducibility problem and give the correction.
Solutions. 1. Multiply each fraction by 300: \(300(0.40)=120\), \(300(0.70)=210\), and \(300(1.00)=300\) participants. The calculation assumes proportional information as stated. 2. The workflow should fail validation, report the mismatch, and stop final output generation until the calculation is corrected and rerun. Manually adding a row does not provide a calculated boundary and can conceal the missing result. 3. Software or dependency versions may have changed, or a hidden default may differ; randomness without controlled random-number handling is another possibility. 4. The table and figure no longer depend on the same current result, so they may represent different runs. Generate both from the same saved result object during the same workflow run.