Why Reporting Defects Matter
A model-informed report is useful only when a reader can follow how the analysis supports its conclusions. A result may be numerically correct yet difficult to assess because the report does not explain who was excluded, shows only favorable diagnostics, or states a subgroup conclusion without presenting the comparison that would support it. These are reporting defects: they obstruct interpretation and make it harder to judge whether a claim follows from the evidence.
A reporting defect is not, by itself, proof that the model or analysis is wrong. An unexplained exclusion might have a defensible reason; an omitted diagnostic might be uninformative. But without the relevant information, a reader cannot evaluate those possibilities. Conversely, complete disclosure does not make an analysis valid. The task is to identify what cannot be verified from the report, what evidence is needed, and how the limitation affects the claim.
A useful assessment follows the claim through the report: identify the population and analysis used, check what evidence is shown, look for changes between the analysis and reported result, and ask whether uncertainty and limitations are communicated. The aim is not to demand that every detail appear in the main text. Concise reporting can be appropriate if referenced tables, appendices, or computational records make the necessary information accessible and consistent.
Exclusions and Analysis Populations
An exclusion is informative only when its denominator, timing, reason, and consequences are clear. A report should let the reader determine how the starting data set became the analysis population, whether participants or observations could have more than one reason for exclusion, and whether the excluded information could affect the claim. A single total count may conceal distinct issues, such as missing covariates, protocol deviations, or inadequate measurements.
Worked Example: An Unexplained Reduction in the Analysis Set
In a hypothetical exposure-response analysis, 240 participants entered the data set described in the report. The model analysis included 222, but the report only says that 18 participants were “not evaluable.” Determine what can be calculated and what remains unclear.
Calculate the exclusion proportion. If all 18 exclusions are distinct participants from the 240-person starting set, then:
This calculation describes the reduction in this data set. It does not show that the exclusions are harmless or that they biased the result. The phrase “not evaluable” does not tell the reader whether the exclusions arose from missing exposure measurements, absent outcome data, eligibility criteria, or another reason. Nor does it establish whether the 240 and 222 counts refer to participants, observations, or another analysis unit.
A more informative report would provide a reconciled disposition and reason-specific counts. For example, suppose the hypothetical report clarifies that 10 participants lacked the required exposure measurement and 8 lacked a post-baseline outcome. If these categories are mutually exclusive and exhaustive, then \(10+8=18\), and the counts reconcile with the stated analysis population. If a participant could meet both criteria, the report must say whether the categories overlap; otherwise, summing them could double-count exclusions.
The reader would still need to know how those missing measurements relate to the analysis and its target population. The 7.50% exclusion proportion alone cannot establish that the analyzed participants represent all 240, nor does it quantify the effect of exclusion on the model result. The report should support that evaluation rather than imply that a small percentage is automatically negligible.
This example illustrates a general check: totals should reconcile, but reconciliation is not a substitute for explaining selection. A report can disclose every count and still leave an important limitation if the excluded group differs in ways relevant to the claim.
Diagnostics: Complete Enough to Evaluate
Diagnostic reporting is selective when it presents only a subset that could make model performance appear more favorable, without explaining the selection or making the remaining relevant evidence available. A compact main-text summary is not necessarily selective: it can direct readers to a complete, interpretable diagnostic set. The concern is whether the evidence shown is adequate to assess the stated use and whether omitted results could alter the interpretation.
Worked Example: Four Favorable Diagnostics, Eight Missing
Suppose a hypothetical report states that the model was “well validated” and shows four diagnostic plots. Its analysis plan lists 12 diagnostics covering observed-versus-predicted agreement, residual patterns, and predictive checks. The report neither explains why the other eight are absent nor provides them elsewhere.
The report establishes that four plots were produced and selected for display. It does not establish that all 12 diagnostics were satisfactory. A reviewer should check whether the omitted diagnostics were run, whether they were relevant to the model’s intended use, and whether the report provides their results in an accessible supplement. If an omitted predictive check reveals systematic underprediction in the high-exposure range, that evidence could materially qualify the validation claim.
The appropriate conclusion from the information given is not “the model failed validation,” because the omitted results are unknown. It is that the broad validation statement is not adequately supported by the evidence presented. A balanced report can summarize the most decision-relevant diagnostics, identify limitations or failures, and make the full set available with enough context to interpret it.
Diagnostic displays also need interpretation. A plot without a clear description of the data, model predictions, or relevant reference pattern may be hard to assess even when it is included. The question is not simply how many figures appear, but whether a reader can see what was checked, what the results show, and how those results bear on the claim.
Subgroup Claims Need Direct Support
A report can overstate subgroup evidence by treating a difference in separate estimates, or significance in one group and not another, as proof that groups differ. The relevant evidence is a direct contrast between subgroup effects, with its uncertainty. The claim must also stay within the analyzed subgroups and the scope of the data.
For two subgroup estimates \(\hat{\theta}_1\) and \(\hat{\theta}_2\), the estimated difference is \(\hat{\delta}=\hat{\theta}_1-\hat{\theta}_2\). Its uncertainty depends on both estimates and their covariance:
Thus, looking at the two estimates separately is not a substitute for assessing their contrast. If a model directly estimates a subgroup interaction, that contrast and its uncertainty should be reported. Its interpretation also depends on how the subgroups were defined, how much information supports each estimate, and whether the claim was planned or exploratory.
Worked Example: A Subgroup Difference Without a Supported Claim
In a hypothetical analysis, the estimated treatment effect on a continuous outcome is 0.30 units in subgroup A and 0.18 units in subgroup B. The model directly estimates the difference, A minus B, as 0.12 units, with a 95% confidence interval from -0.08 to 0.32 units. The report says, “Treatment effects are greater in subgroup A.” Assess that statement.
The direct contrast is:
The reported interval ranges from -0.08 to 0.32 units and includes zero. Under the model and interval procedure used, the data are compatible with a difference favoring subgroup B, no difference, or a difference favoring subgroup A. The point estimate is larger in A, but the interval does not establish that the treatment effect is greater in A.
A more calibrated statement is: “The estimated treatment effect was 0.12 units larger in subgroup A, but the 95% confidence interval for the subgroup difference ranged from -0.08 to 0.32 units.” This describes the estimate and its uncertainty without claiming that the groups are equal or that the null hypothesis is true. If the claim is intended for a particular decision, the report should also explain whether the contrast is clinically meaningful and whether the analysis supports that intended use.
This is a reporting issue even if both subgroup estimates and the interaction were calculated correctly: the claim’s wording must match the contrast and its uncertainty. For exploratory subgroup findings, the report should also make the exploratory status clear rather than presenting the result as a confirmed difference.
Model Changes, Uncertainty, and Limitations
A report can conceal a material change when its narrative describes one model but its tables or figures present results from another. Changes may be appropriate, including changes made after a convergence or stability problem. The defect is not necessarily the change; it is failing to identify what changed, when, why, and which results use which specification. Without that account, a reader may attribute an estimate to the wrong analysis or be unable to judge whether the change affected the conclusion.
Worked Example: The Reported Model Is Not the Final Model
In a hypothetical analysis, the methods section describes a model with clearance depending on body weight and a proportional residual-error structure. The results table, however, is generated using a model with body weight and renal function as clearance covariates and a combined residual-error structure. The table’s clearance estimate is 4.8 L/h; an earlier draft, also reproduced in the report, gives 5.3 L/h.
The difference between the two displayed estimates is \(5.3-4.8=0.5\) L/h. Relative to the earlier value, this is \(0.5/5.3\times100\%=9.43\%\). That calculation identifies a numerical difference; it does not tell the reader why it occurred or which estimate supports the report’s claim.
The report should identify the final specification, state that the covariate and residual-error structures changed, explain the reason and analysis stage, and label which estimates and diagnostics correspond to each version. It should also describe whether the change altered the interpretation. The 9.43% difference does not by itself show that the change is important or unimportant; its relevance depends on the claim, uncertainty, and intended use.
Conclusions also fail when they give a point estimate without enough uncertainty or omit limitations that constrain interpretation. Uncertainty may arise from parameter estimation, variability in the data, model assumptions, or linked analyses; a reported point estimate alone does not communicate its precision. Limitations may include sparse observations in a decision-relevant range, weak support for a subgroup, or uncertainty in an extrapolation. State the limitation in a way that explains which claim it qualifies, rather than attaching a generic caution that leaves the reader unsure what is at stake.
A report need not repeat every uncertainty calculation in every paragraph. It should, however, make uncertainty accessible where it matters to the conclusion, state the relevant units, and avoid wording that implies more precision or broader applicability than the evidence supports. A limitation should not be hidden in a distant section if it changes how a central result should be interpreted.
A Practical Diagnostic Sequence
The checks below help separate a documented reporting defect from a claim that the underlying analysis has failed. They are a way to organize the evidence, not a substitute for evaluating the scientific question and intended use.
State precisely what the report asserts and identify the population, outcome, model, and intended scope relevant to that assertion.
Check that the data and analysis populations, exclusions, model version, and reported result can be reconciled from the report and its referenced materials.
Determine whether the displayed diagnostics and subgroup comparisons are relevant and sufficiently complete to evaluate the claim. Ask whether omitted evidence could change the interpretation.
Check whether the conclusion reflects the estimate, uncertainty, limitations, and population actually studied, without implying unsupported causality or generalization.
The main alternative to presenting a comprehensive evidence record in the body is a concise summary with clearly referenced supplementary material. Either approach can work if a reader can locate and interpret the evidence. A brief narrative with no usable details is not made adequate merely by being concise; a very long appendix is not adequate if it fails to identify which analysis supports the claim.
Common Pitfalls and Limits of the Review
- Assuming a disclosed exclusion is unbiased. Reporting a reason and count improves transparency but does not establish that the exclusion has no effect on the target inference.
- Treating missing diagnostics as failed diagnostics. An omission means the evidence is unavailable in the report, not that the omitted result was necessarily unfavorable.
- Inferring a subgroup difference from separate results. A larger estimate in one group, or a statistically distinguishable result in only one group, does not by itself establish that the effects differ.
- Assuming a model change invalidates the analysis. A justified change may improve an analysis. Its implications cannot be assessed if the report does not identify the change and its consequences.
- Confusing disclosure with support. Listing a limitation or presenting a diagnostic is not enough if the report does not explain how it bears on the conclusion.
A report-based review has limits: it can identify what is unclear, unavailable, or inconsistent, but may not determine the truth of a missing result or reproduce an analysis without access to its data and computational record. The appropriate conclusion should reflect that distinction. Describe the evidence gap precisely; do not replace it with an unsupported claim that the analysis is either sound or defective.
Check Your Understanding
For each problem, decide what the report supports and what additional information would be needed. Solutions follow each question.
- A report starts with 160 records and analyzes 152, stating that 8 were excluded for “data quality.” What exclusion percentage is reported, and what can you conclude about possible bias?
Solution: The exclusion percentage is \(8/160\times100\%=5.00\%\). This is the proportion removed from the stated starting set, assuming the counts refer to the same unit and exclusions are distinct. It does not establish that exclusion is unbiased or harmless. The report needs reason-specific information and enough context to assess how the exclusions relate to the analysis population and claim. - A report shows a good observed-versus-predicted plot but states that this proves the model is valid. What is the reporting problem?
Solution: The conclusion overstates what one diagnostic can establish. The plot provides evidence about the aspect of fit it displays, not proof of overall model validity. The report should present and interpret evidence relevant to the intended use, identify important limitations, and avoid a broader claim than the diagnostics support. - Subgroup A has an estimated effect of 0.4 units and subgroup B an estimated effect of 0.2 units. The report gives no direct contrast or uncertainty. Is “the effect is greater in A” established?
Solution: No. The point estimates differ by \(0.4-0.2=0.2\) units, but without uncertainty for the difference the report does not establish that the subgroup effects differ. A direct contrast with its uncertainty, alongside the subgroup definitions and relevant analysis context, is needed to assess the claim. - A results table reports estimates from a model that differs from the model described in the methods section. What should a reader request before interpreting the estimate?
Solution: The report should identify the final model, explain which specification produced the table, describe the changes and their rationale, and clarify how the changes affect the reported claim. Until then, the estimate cannot be confidently traced to the stated analysis.