Why a Clean Rerun Matters
A regression report can look polished and still fail to describe the analysis that produced the current results. A table may have been edited by hand, a figure may come from an earlier model, or the analysis may rely on an object left over from a previous run. Rerunning only the final few commands may not expose these problems if the current session already contains data or model objects created earlier.
A clean rerun means starting a fresh computing session and executing the documented analysis steps in their intended order, using the specified data, settings, and model decisions. The purpose here is verification: check whether the counts, model, tables, figures, and reported values agree with the newly generated outputs. A successful rerun supports computational consistency; it does not, by itself, establish that the model is appropriate or that its assumptions are satisfied.
A useful comparison has two sides. First, establish what the analysis is supposed to do: identify the input data and analysis decisions, including sample restrictions, variable definitions, model terms, coding, and any random-number settings that matter. Second, inspect what the clean execution actually produces. Agreement should be checked at the level of meaning, not merely by noticing that a table or plot exists.
A Structured Rerun Check
Before comparing individual values, confirm that the rerun uses the intended inputs and decisions. A different data file, changed exclusion rule, or altered factor reference group can produce legitimate differences, but then it is not a direct verification of the reported analysis. Keep a record of which data version and analysis specification were used, and do not silently adjust the analysis to make its output resemble the report.
Open a new session, clear reliance on previously created objects, and run the documented steps in order. Note any step that requires an undocumented manual action.
Confirm the data version, number of records, exclusions, missing-data handling, and final analysis count. Reconcile the count using the stated rules.
Compare the fitted model with the reported specification: outcome, predictors, transformations, interactions, factor coding, and reference groups.
Check tables, figures, and written values against the rerun. Verify units, rounding, labels, and the context attached to each result.
Find the cause of each mismatch, correct the underlying analysis or report, and rerun the relevant checks. Do not treat an unexplained mismatch as harmless.
Worked Examples
Worked Example: Reconciling the Analysis Sample
Suppose a hypothetical dataset contains records for 120 participants. The stated analysis rule is to include participants with nonmissing outcome, baseline measurement, and group. Group is complete for everyone. The data check finds 5 missing outcomes, 7 missing baseline measurements, and 2 participants missing both. How many participants should be in the complete-case analysis?
A participant missing both variables is included in both missing-value counts, so subtracting 5 and 7 directly would remove those two records twice. The number excluded for either reason is:
Therefore, the expected analysis count is:
The clean rerun reports \(n=110\), so the count agrees with the stated rule. If the report instead says \(n=108\), the two-person difference requires investigation. For example, there may be an additional exclusion rule that was not stated, or the report may contain an outdated count. The appropriate response is to find and document the cause, then make the report and analysis agree. It would not be appropriate to alter the count in the report without checking which participants were actually analyzed.
Worked Example: Confirming the Model Terms
Consider a hypothetical analysis of an outcome measured in millimeters of mercury. The intended model includes group, baseline outcome, and study site. Group has two levels, usual care and program; site has three levels, A, B, and C. The specification uses usual care and site A as reference groups. The model is:
Here, each \(I(\cdot)\) is an indicator that equals 1 when the condition in parentheses holds and 0 otherwise. The terms \(I(\text{program})\), baseline, \(I(\text{site B})\), and \(I(\text{site C})\) represent the four predictor terms; the intercept is also part of the model. The two site indicators encode the three-level site variable relative to site A.
Suppose the clean rerun shows the program indicator, baseline, and both site indicators, with the intended reference groups, and uses the 110 participants identified in the sample check. Those are the terms and sample expected from the specification. If the rerun omits the site indicators, then the fitted model does not match the stated specification, even if its coefficient table looks plausible. If site C is instead used as the reference group, the model’s parameterization and coefficient labels differ; the report must not describe site A as the reference group.
This check confirms what was fitted, not whether including these predictors is scientifically justified. Nor does a model term alone establish a causal effect. Those questions require interpretation of the study design and model assumptions, beyond a consistency check.
Worked Example: Tracing a Reported Estimate to the Output
Suppose the clean rerun for the hypothetical model reports a program coefficient of \(-4.20\) mmHg, standard error \(1.80\) mmHg, and 104 residual degrees of freedom. A two-sided 95% confidence interval uses the \(t\) critical value \(t^*=1.983\). Verify the interval that should appear in the report.
The interval is the estimate plus or minus the critical value times its standard error:
The margin of error is:
Thus the endpoints are:
Rounded to two decimal places, the interval is \((-7.77,-0.63)\) mmHg. A report showing \((-7.77,-0.63)\) is consistent with these displayed inputs. If the report instead gives \((-7.77,0.63)\), the upper endpoint’s sign is inconsistent with the calculation and should be corrected after checking the underlying output. The negative coefficient describes an adjusted difference in the modeled outcome, in millimeters of mercury, for program versus usual care; it should not be described as a causal effect without a design and assumptions that support that interpretation.
Worked Example: Finding a Stale Figure
A report contains a coefficient plot whose label gives the program estimate as \(-3.60\) mmHg. The clean rerun generates a table with an estimate of \(-4.20\) mmHg, and the verified model specification and sample count have not changed. What should be checked?
The table and figure disagree, so the figure cannot be assumed to represent the current model. Check whether it was generated from an earlier model object or saved output, whether the plotting step ran after the current fit, and whether the report points to the newly generated figure file. If the current model output is verified, regenerate the figure from that output and confirm that its coefficient, axis units, labels, and interval agree with the table. Do not edit the plotted label by hand: that could make the text look consistent while leaving the plotted point at the old value.
After regeneration, the point estimate should be \(-4.20\) mmHg. The plotted interval should also agree with the model’s current interval, and the figure should identify the correct outcome and comparison. This check is about matching the visual output to the analysis; it does not substitute for evaluating whether the model is suitable.
What to Compare, and How Precisely
A practical check is to compare each report item with its source in the rerun. A coefficient table should be checked against the fitted model’s generated estimates, standard errors, confidence intervals, p-values, sample size, and term labels. A figure should be checked against the same model output, including units and reference groups where relevant. Narrative statements should agree with both the numerical results and their direction.
| Report item | Verification question | Possible discrepancy to investigate |
|---|---|---|
| Analysis sample | Does the reported count match the documented inclusion and missing-data rules? | The report has a different \(n\) than the rerun. |
| Model specification | Are the outcome, terms, coding, and reference groups the intended ones? | A predictor is absent or a reference group has changed. |
| Table | Do estimates, uncertainty measures, labels, and units match the generated output? | A rounded value or sign differs. |
| Figure | Does it display the current model and use accurate labels and scales? | The plotted estimate is from an earlier fit. |
| Written interpretation | Does the statement match the estimate’s direction, units, and modeled comparison? | The text reverses the comparison or overstates what the analysis supports. |
The comparison should allow for rounding, but not for unexplained disagreement. If the report displays two decimal places, values that round to the same displayed number can be consistent even when their unrounded forms differ slightly. For example, \(-4.204\) and \(-4.196\) both round to \(-4.20\) using ordinary rounding to two decimal places. Where exact file-to-file comparisons are used, a small numerical tolerance may be appropriate because computations can differ slightly across software environments. Set any tolerance before checking and keep it small enough that it cannot conceal a meaningful change. Differences in sample size, model terms, units, or signs are not rounding differences.
Why the Check Matters
The value of a clean rerun is that it can expose hidden dependencies: an object created manually, a step executed out of order, an old file reused, or a report value typed separately from the analysis output. Each is a route by which the reported results can drift away from the analysis. A rerun also creates an opportunity to verify that all key artifacts belong together: one analysis sample, one model specification, and the corresponding tables, figures, and text.
A rerun is not a stamp of scientific approval. Reproducing an incorrect model or an unjustified interpretation does not make it correct. The check asks whether the report accurately reflects the analysis as specified and executed. Model diagnostics, limitations, uncertainty, and substantive interpretation still need careful consideration.
Common Mistakes
- Rerunning in the old session. Existing objects may conceal missing steps. Start fresh and execute the analysis in order.
- Checking only the final table. Sample counts, model terms, figure files, and narrative claims can disagree even when one table looks right. Compare all of them.
- Fixing a mismatch by editing the displayed number. Trace it to the data, model, output, or report source and correct the underlying cause.
- Treating matching output as proof of a valid analysis. A clean rerun establishes reproducibility under the specified workflow, not that the model assumptions or conclusions are correct.
- Ignoring the analysis sample. The same model formula fitted to a different set of records is not the same analysis. Reconcile the count and the inclusion rules.
Check Your Understanding
Use the checks in this tutorial to answer the following questions.
- Why might rerunning the final commands in an existing session fail to reveal that an analysis depends on an old model object?
- A dataset has 90 records, with 6 missing outcomes, 5 missing predictor values, and 2 records missing both. Under a complete-case rule using those two variables only, what analysis count should a rerun produce?
- A report specifies site A as the reference group, but the clean rerun uses site C. What should be checked before the report is shared?
- A coefficient table and a figure show different estimates. Name two plausible causes and describe an appropriate way to resolve the discrepancy.
- What does agreement between a clean rerun and a report establish, and what does it not establish?