Why Record Analysis Decisions?
A regression analysis often involves choices that cannot all be settled before the data are examined. A variable may be coded differently from what its description suggested; a diagnostic plot may raise a concern; or an initial reporting plan may leave an important qualification unclear. The central question is not whether an analysis ever changes. It is whether a reader can tell what changed, when it changed, why it changed, and what the change affected.
A decision log is a dated record of analysis choices and deviations from the plan. It supplements, rather than replaces, analysis scripts, data provenance records, and the written report. A script can show what calculations were run, for example, without explaining why a variable was redefined or why a diagnostic concern led to one response rather than another. A log preserves that reasoning in context.
A log is not a claim that every decision was made before looking at data. Nor does it make an exploratory choice confirmatory simply because the choice is recorded. Its value is transparency: a later reader can distinguish the original plan from adaptations made during analysis and judge the evidence for each.
What Belongs in an Entry?
Record a decision when it could affect the data being analyzed, the form of the analysis, the interpretation of a diagnostic check, or what readers are told. Routine operations already clear from a script usually do not need separate entries. A useful entry is concise but specific enough that another analyst could understand and, where possible, reproduce the choice.
| Entry element | What to record |
|---|---|
| Date and identifier | When the decision was made and a short label for finding it later. |
| Plan or prior choice | What had been specified before this decision, if anything. |
| Trigger and information | The data, diagnostic output, documentation, or practical constraint that prompted consideration. |
| Decision and reason | What was chosen and why it is appropriate for the analysis question. |
| Alternatives | Reasonable alternatives considered and, when useful, why they were not chosen. |
| Consequences | Which variables, records, model runs, interpretations, or report statements are affected. |
| Follow-up | Where the change was implemented and what needs checking or updating. |
Use dates consistently, such as year-month-day, and identify entries clearly. Keep the original plan visible rather than silently replacing it. If a decision is revised again, add a new dated entry that points back to the earlier one. The sequence should show the history, not just the final state.
Separate Planned Choices from Later Adaptations
A plan may be written before analysis, documented in a protocol, or recorded at the start of a project. The log should state what was planned and preserve the date of that record. Later decisions should be labeled as additions, clarifications, or deviations as appropriate. This distinction matters because a choice prompted by observed data has a different history from a choice made without seeing those data.
For example, changing a variable definition after finding an error in its coding documentation is not the same as trying several definitions and retaining the one that gives the most favorable result. Both situations should be recorded accurately, but their reasons and implications differ. The log should not disguise a data-informed choice as if it had been planned in advance.
A practical entry can follow this sequence:
Name the planned variable, model form, diagnostic response, or reporting choice. If there was no prior plan, say so.
Record what became known and when. Refer to a file, plot, data-dictionary entry, or other identifiable source where useful.
State the decision, the reason, and relevant alternatives. Do not rely on “it looked better” without saying what criterion or analysis purpose that means.
Identify affected analyses and outputs, update the script or documentation, and add a new entry if the decision changes later.
A decision log can help guard against hindsight: once results are known, people may remember a data-informed choice as having been planned all along. A dated entry gives evidence of chronology, especially when it is linked to a versioned project record. A date alone cannot prove when a choice was actually made, so retain the associated plan, files, and analysis versions as appropriate.
Worked Example: Clarifying a Planned Variable Definition
Suppose a hypothetical analysis includes 240 clinic visits and plans to use smoking status as a three-category variable: current, former, or never. On 2026-04-03, the analyst checks the data dictionary and discovers that the recorded value “9” means “not asked,” not “never.” The data contain 18 visits coded 9. Show how the decision should be recorded.
Prior plan. Use the three stated smoking-status categories. The original plan did not explain how code 9 should be handled.
Trigger. The data dictionary, checked on 2026-04-03, defines 9 as “not asked.” That documentation is the evidence prompting the decision; the observed regression result is not.
Decision and reason. Keep code 9 distinct from “never” and treat it as an unavailable smoking-status value for this variable. Combining “not asked” with “never” would assign a meaning contradicted by the data documentation. Record the date, dictionary version or file name, and the coding change in the log.
Consequence. The number with a usable smoking-status value is \(240-18=222\) visits. This arithmetic describes availability for this variable only; it does not by itself determine the analysis sample, which depends on the other analysis requirements. The analyst should update the recoding script and check that code 9 is not inadvertently labeled “never” in summaries or model inputs.
A concise entry could read: “2026-04-03 — Smoking status: the dictionary defines code 9 as ‘not asked’; retain it as unavailable for smoking status rather than recoding it to ‘never.’ There are 18 such records among 240 visits, leaving 222 with a recorded category for this variable. Updated recoding script; verify downstream labels.” The entry preserves both the reason and the scope of the count.
Recording Model and Diagnostic Decisions
A diagnostic finding does not automatically dictate one response. Record what was examined, what was observed, what options were considered, and why the chosen response fits the analysis purpose. For a change in model form, distinguish the reason for examining the alternative from any later evaluation of its results. Avoid presenting a data-prompted change as though it were the original plan.
Similarly, a potentially unusual or influential observation is a reason to investigate, not by itself a reason to delete a record. Check for data errors and relevant context where possible. If an observation is retained, transformed, excluded under a defensible rule, or assessed through an additional analysis, document what was done and why. Keep the explanation factual and avoid implying that a diagnostic flag establishes that a record is invalid.
Worked Example: Logging a Change to Model Form
Imagine a hypothetical study of 84 greenhouse plants. The plan is to relate plant height in centimeters to days since planting using a straight-line regression. On 2026-05-12, a plot of residuals against days since planting shows a curved pattern: residuals tend to be positive at early and late times and negative in the middle. The analyst considers whether a quadratic term is worth examining.
Prior plan. The planned form is \(E(Y_i\mid x_i)=\beta_0+\beta_1x_i\), where \(Y_i\) is height in centimeters and \(x_i\) is days since planting. The plot is the trigger for considering a change; it is not evidence that the alternative is necessarily correct.
Decision and reason. Record that a quadratic form will be examined as a diagnostic response because the residual pattern suggests the straight-line form may not capture the observed relationship. The alternative is \(E(Y_i\mid x_i)=\beta_0+\beta_1x_i+\beta_2x_i^2\). Note that this form is being considered after examining the residual plot, rather than being represented as the original plan.
Alternatives and consequences. The analyst could retain the planned form and report the concern, or examine the quadratic form and assess whether it addresses the pattern without producing an implausible relationship over the observed range. The log should identify both model runs, the plot used to prompt the change, and the script version that implements the added term. If the model form changes again, append another dated entry.
This entry records the reasoning process without claiming that a curved residual pattern proves the quadratic model is correct. The decision and its rationale can be evaluated separately from the estimates the alternative model later produces.
Worked Example: Documenting a Diagnostic Response and Reporting Choice
In a hypothetical analysis of 60 laboratory samples, a residual diagnostic flags one sample for further checking. Its recorded concentration is 48 units, while the other 59 values range from 12 to 35 units. On 2026-06-08, the analyst checks the source record and confirms that 48 is transcribed correctly. The reporting plan also needs to be clarified because a draft summary rounds concentrations to whole units.
Diagnostic response. Record the flagged value, the check performed, and its result: the source record confirms 48 units, so there is no verified transcription error. Retain the observation in the primary analysis rather than removing it solely because it was flagged. If a secondary analysis excluding it is run to assess sensitivity, label that analysis as such and record the reason for it.
Reporting choice. Record that concentrations will be reported with units and one decimal place where appropriate, because rounding to whole units could hide differences smaller than one unit. For example, \(12.4\) units rounds to \(12\) units to the nearest whole unit, losing the recorded tenths place. Apply the chosen rounding rule consistently in the affected summary and check that units remain attached.
Consequence. The log now explains both why the flagged record remained in the primary analysis and why the displayed precision changed. It should identify the diagnostic output, the verified source record, the analysis version, and the summaries affected. It should not imply that retention establishes the observation is typical or that additional analysis was planned in advance if it was prompted by the diagnostic.
Common Mistakes and Practical Safeguards
- Writing only the final choice. A final variable definition or model form hides what changed. Preserve the original plan and add a dated entry describing the revision.
- Giving a vague reason. “Changed after review” does not identify the evidence or purpose. Name the data documentation, diagnostic pattern, or reporting concern that prompted consideration.
- Using results as the sole justification. Choosing a change because it produces a smaller p-value or a more appealing estimate can distort inference. Record data-informed exploration honestly and do not portray it as prespecified.
- Treating a diagnostic flag as a deletion rule. A flag prompts investigation; it does not establish that a record is erroneous. Check the record and justify any action in context.
- Overwriting the log. Replacing an earlier decision erases the sequence. Append a new entry, refer to the previous one, and preserve relevant script or file versions.
- Recording actions without consequences. A change may affect a variable, model run, analysis sample, or report statement. Identify what needs to be updated and checked.
The log should be detailed enough to audit important choices, but it need not become a diary of every keystroke. A useful test is whether a reader could identify the original plan, understand the reason for each consequential change, and locate the analysis materials that implement it. When the answer is yes, the log supports reproducibility and a fair account of how the analysis developed.
Check Your Understanding
For each question, focus on what a reader would need to understand the decision and its history.
- A data dictionary shows that a planned category combines two distinct codes. What should the log say about the prior plan, evidence, revised coding, and affected analysis steps?
- A residual plot suggests curvature, and an analyst adds a quadratic term. Why should the entry identify this as a data-prompted change rather than a planned model form?
- A diagnostic flags an unusual observation, but checking the source confirms its value. What is the difference between recording the flag and justifying deletion?
- Why is an entry that says only “changed the variable coding” insufficient for an auditable decision log?
- An analyst replaces an earlier log entry after reconsidering a choice. What should be done instead, and why?