Why Decision Reporting Matters
A validation result is interpretable only when readers can see how the reported analysis was produced. The relevant account is not just a list of methods: it connects the locked protocol to the data actually analyzed, describes any choices made along the way, and places those choices in time relative to access to validation results. That chronology helps readers judge whether a decision was prespecified, driven by data quality, or made after seeing results.
This tutorial focuses on reporting analysis decisions, not on deciding whether a validation finding has replicated or how to quantify its uncertainty. The introductory material on validation design and cohort characteristics provides context for describing where the data came from and what was measured. Here the emphasis is the audit trail from the protocol to the analyzed samples, features, and variables.
Four kinds of decision should be kept distinguishable. An exclusion removes a sample, observation, or feature from a stated analysis under a rule. A mapping links a recorded variable or molecular feature to the representation required for analysis. An analysis choice specifies how the planned analysis is implemented, such as the coding of a covariate. A deviation is a change from the locked protocol; it may involve an exclusion, mapping, or analysis choice. A decision can belong to more than one category, so report both its nature and whether it departed from the plan.
For every consequential decision, report the rule or choice, its rationale, when it was made, who or what informed it, and what data or analyses it affected. “Before results” is not sufficiently precise: distinguish access to raw validation data, access to technical quality-control summaries, and access to validation outcomes or outcome-linked summaries. State the dates or ordered milestones when available, and say explicitly when outcome status was masked.
Report Exclusions and Changing Denominators
An exclusion report should define the unit being excluded (for example, participants, samples, visits, or features), give the criterion in operational terms, and show how the number eligible for analysis was reached. State whether criteria were applied sequentially or whether categories can overlap. If several omics assays have different coverage, give assay-specific analyzed counts rather than implying that one sample count applies to every modality.
The order matters whenever exclusions can overlap. If each successive rule is applied to the remaining records, the arithmetic should reconcile from the initial count to the final count. If exclusion categories overlap, a simple sum of category counts can overstate the number removed; say so and report the unique total separately. Do not use outcome-dependent exclusions as if they were fixed eligibility criteria.
Worked Example: Reconciling Sample Exclusions
Suppose a hypothetical validation cohort has 120 enrolled participants. Before outcome labels are released, the analysis team applies three sequential, prespecified rules: remove duplicate records, remove samples failing the assay quality threshold, then remove participants without the required baseline outcome measurement. The observed counts are 8, 5, and 3, respectively. The rules are applied to the participants remaining at each step.
After duplicate removal, \(120-8=112\) participants remain. After quality-control removal, \(112-5=107\) remain. After removing records without the baseline outcome, \(107-3=104\) remain. Equivalently, \(120-(8+5+3)=104\), because the categories in this example are sequential and non-overlapping by construction.
A clear report would state the initial count, each rule and count, the order of application, the final analysis count of 104, and that these decisions were made before outcome access. If the molecular assay was available for only 101 of those participants, the report would give 104 as the cohort count and 101 as the assay-specific count, with the reason for the further difference. The latter should not be silently folded into the participant-exclusion total.
Describe Variable and Feature Mappings
Multi-omics validation commonly requires aligning different labels, units, platforms, or feature identifiers across cohorts. Report the source representation, target representation, mapping rule, and handling of unmatched or ambiguous entries. For variables, include units and coding—for example, whether a recorded value in months was converted to years, or whether categories were harmonized. For molecular features, clarify whether one-to-one mappings were required and how duplicate or many-to-one mappings were handled.
A mapping is not automatically harmless preprocessing. A choice to keep one of several candidate matches, combine records, or discard ambiguous features can change what the analysis represents. Give counts before and after mapping and explain the rule without implying that an uncertain correspondence is exact.
Worked Example: Reporting a Feature-ID Mapping
In a hypothetical validation assay, 100 source feature identifiers must be mapped to the feature identifiers used by a locked signature. Before outcome access, a versioned reference table yields 92 unambiguous one-to-one matches, 3 source identifiers with multiple candidate targets, 2 source identifiers that collide on the same target, and 3 identifiers with no target. These categories partition the 100 source identifiers.
The prespecified rule retains only unambiguous one-to-one matches and excludes all ambiguous, colliding, and unmapped identifiers. Thus, the retained count is \(100-(3+2+3)=92\). The report should say that 92 of 100 source identifiers were retained, identify the three reasons for the eight exclusions, name the mapping rule and reference-table version, and state that the mapping decision preceded outcome access.
If the signature requires all specified features, 92 mapped features do not by themselves establish that the signature was fully evaluable; the report must also state whether the required signature features were among those 92. This distinction avoids presenting a partial mapping as complete coverage.
Document Choices, Amendments, and Timing
For each primary analysis choice, identify what the locked protocol specified and what was actually done. Record any amendment or departure in a decision log with a date, rationale, evidence considered, decision-maker, affected variables or samples, and affected analysis. Distinguish a correction that implements the written protocol from a genuine change to it. For example, converting a variable from months to years may implement a specified age adjustment if the protocol defined age in years; changing which covariates are adjusted for is a change in the analysis plan.
Timing should be reported in relation to outcome access, not merely data receipt. A team can access assay files and masked quality-control information without knowing the validation outcomes. Conversely, viewing outcome summaries, group labels, or outcome-linked estimates can constitute outcome access even if the final report has not been generated. If the status is uncertain, describe exactly what was available rather than asserting that a decision was blinded.
Worked Example: A Post-Access Exclusion
A hypothetical locked protocol specifies that all 96 participants passing assay quality control will be included in the primary analysis. After the analyst has viewed outcome-linked results, six participants have unusually large outcome-model residuals. The analyst proposes excluding those six and reports a re-run on the remaining participants.
The count for the proposed re-run is \(96-6=90\). However, the exclusion rule was not in the locked protocol and was proposed after outcome access. It must therefore be reported as a post-access, outcome-informed deviation, not as ordinary quality-control filtering. The report should retain the protocol-specified analysis as the primary analysis if it was carried out, and label the 90-participant re-run as post hoc or exploratory. It should state the residual-based rule, when it was chosen, and the number affected.
The timing changes how the additional analysis should be interpreted; it does not prove that the results are invalid, nor does it justify hiding the analysis. Transparent reporting lets readers distinguish the prespecified analysis from a result-dependent follow-up.
The reason chronology matters can be seen even in a simple illustration. Suppose an analyst chooses between two independent tests after inspecting both, and each test has a 0.05 probability of a false positive under its null hypothesis. The probability that at least one test is significant is
Thus, selecting an analysis after seeing results can alter the operating characteristics of a nominal procedure. This illustration is not a general correction formula; actual dependence and selection rules matter. Its purpose here is to show why reporting when a choice was made is essential.
How to Audit the Report
A compact table or decision log usually makes the account easier to check than a long narrative alone. A narrative remains useful for explaining rationale, but it should not substitute for counts, rules, and chronology. The following fields provide a practical minimum:
| Field | What to report |
|---|---|
| Decision | Exclusion, mapping, coding, model choice, or other change |
| Protocol status | Prespecified as written, clarified by amendment, or deviation |
| Rule and rationale | Operational criterion and reason for applying it |
| Timing | Date or milestone, and whether outcomes were accessible |
| Impact | Samples, features, variables, or analyses affected; before-and-after counts |
Before finalizing the report, check that the initial and final counts reconcile, categories are not accidentally double-counted, every mapping loss has a stated reason, and each primary analysis can be traced to the locked plan or a disclosed amendment. Check that a decision described as blinded was made before access to the relevant outcomes, not merely before a manuscript was written. Where a decision was made after outcome access, label the affected analysis clearly rather than presenting it as prespecified.
Common Pitfalls and Alternatives
- Reporting only the final sample size. This prevents readers from seeing which rules changed the analysis set. Give the starting count and a reconciled sequence of exclusions.
- Calling every change a technical correction. A correction that implements the plan differs from changing the plan. State the original specification and the action actually taken.
- Writing “before analysis” as a timing claim. Analysts may inspect outcome-linked information before running the final model. Report whether outcomes were accessible, and what information was inspected.
- Hiding post hoc analyses. A result-dependent analysis may be informative, but it should be identified as post hoc or exploratory and distinguished from the primary analysis.
- Using prose when a count table is needed. Narrative can explain why; a structured table makes denominators and affected analyses auditable. Use both when decisions are numerous or consequential.
The main alternative to a detailed decision log is a brief methods paragraph. That may suffice when there were no deviations and the mappings and exclusions were simple, but it still needs to say that the protocol was followed and identify the applicable rules. When choices changed, a table or dated log is generally clearer. Neither format can replace disclosure of timing and rationale.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A validation cohort starts with 150 participants. Sequential rules remove 9 duplicate records, 7 samples failing quality control, and 4 participants lacking a required measurement. How many remain, and what should the report state about the counts?
- Of 80 source features, 70 map one-to-one, 4 are ambiguous, 2 collide on the same target, and 4 are unmapped. Under a rule retaining only one-to-one matches, how many are retained?
- An analyst sees only masked assay-quality summaries before deciding to remove failed samples. What should be reported about timing, and why is “before data access” inaccurate?
- After viewing outcome-linked results, an analyst changes a covariate adjustment and reruns the model. How should this analysis be described relative to the locked primary analysis?
Solutions. (1) The sequence is \(150-9=141\), \(141-7=134\), and \(134-4=130\). Report all three rules and counts, their sequential order, and the final count of 130. (2) Retain 70 features; the other \(4+2+4=10\) are excluded, so \(80-10=70\). Report each mapping category and the rule. (3) State that the decision was made after access to assay data or quality-control summaries but before access to validation outcomes, if that is accurate. “Before data access” would incorrectly imply the assay data were unavailable. (4) Report the original specification, the changed covariate adjustment, the reason and timing, and identify the rerun as a post-access deviation or exploratory analysis; do not present it as the protocol-specified primary analysis.