Why unsuccessful analyses belong in the record
A model report that describes only the selected model can leave an important gap: a reader may not know whether alternatives were never attempted, were attempted and failed, or produced results that were set aside. Those distinctions matter when judging how the analysis was conducted and how strongly its conclusions are supported. Reporting a failed fit does not make it a valid estimate; it makes the history and limits of the analysis visible.
This tutorial focuses on three related records: unsuccessful model fits, changes from the analysis plan, and material candidate analyses that are not presented as the primary analysis. The aim is not to list every keystroke or transient debugging attempt. It is to provide a traceable account of choices that could affect model interpretation, comparison, or conclusions.
These categories can overlap. A planned candidate may fail to fit; a deviation may be made after a planned analysis proves infeasible; and an alternative may converge successfully but not be selected. Name which situation occurred. “The model did not work” is not precise enough to distinguish a numerical problem from a scientific or decision-based reason for not using a model.
What a useful failure record contains
A failure record should identify the attempted analysis and provide enough information to evaluate the failure without suggesting that an unavailable estimate is a result. At minimum, identify the model structure or candidate, the data and analysis population used, the estimation method and relevant settings, the software and version, and the observed stopping or diagnostic evidence. State whether the attempt was repeated with different initial values or settings, and summarize what changed between attempts. If a fit returned some output but did not satisfy the prespecified or otherwise stated criteria for usability, describe that output and the reason it was not treated as a usable result.
Be specific about the evidence. For example, “the optimizer terminated with a rounding-error message; the gradient criterion was not met; and estimates changed materially across the attempted initial values” is more informative than “convergence was poor.” A message alone is not necessarily proof that every estimate is unusable, just as a nominal termination message alone does not establish that all relevant diagnostics are satisfactory. State the criterion applied and the observed evidence. Do not report a parameter estimate as if it were dependable when the analysis did not support that interpretation.
A failure can arise from several sources: an unsuitable model for the available data, weak information about one or more parameters, an overly complex parameterization, poor starting values, or estimation settings. The report should describe what is known and avoid claiming a cause that was not established. If a controlled change to an initial value resolves the issue, report the successful analysis and settings; do not characterize the initial attempt as evidence against the model itself.
Worked Example: Reporting an Unsuccessful Structural Fit
Suppose a hypothetical population PK analysis planned a two-compartment model with first-order elimination for 48 participants. The planned model was attempted from three initial-value sets using the same data, estimation method, and convergence criteria. None met the stated gradient criterion. The objective-function values at termination were 512.4, 511.8, and 512.1, and the clearance estimates were 4.8, 6.2, and 5.1 L/h. A simpler one-compartment model fit successfully with an objective-function value of 520.6. These are hypothetical results.
What the numbers show. The three two-compartment attempts did not satisfy the criterion, and the reported clearance estimates varied by 1.4 L/h between the largest and smallest: \(6.2-4.8=1.4\) L/h. That spread, together with failure to meet the gradient criterion, is evidence that the attempted outputs should not be treated as a stable, usable estimate under the stated procedure. The objective-function values are reported as termination outputs, not as a valid model-comparison result for this failed candidate.
How to report it. A transparent account could say: “The planned two-compartment model was attempted from three initial-value sets in the 48-participant analysis population. None met the prespecified gradient criterion. Termination objective-function values ranged from 511.8 to 512.4, and the corresponding clearance estimates ranged from 4.8 to 6.2 L/h. Because the fit criterion was not met and estimates varied across attempts, these outputs were not interpreted as usable parameter estimates. The one-compartment model met the stated criteria and was used for the primary analysis.”
This wording states what was attempted, gives the relevant evidence, and separates failed-fit output from the usable analysis. It does not conclude that the two-compartment model is biologically wrong; the reported evidence establishes a computational and estimation problem under the described attempts, not the model’s general scientific validity.
Deviations: state the plan, change, timing, and reason
A deviation is easiest to assess when the report makes the comparison with the plan explicit. State the planned decision, the decision actually implemented, when the change was made relative to examining results, and the reason. Also identify who authorized or documented it when that information is relevant to governance. A reason such as “the planned category was too sparse to support the intended estimate” is more useful when accompanied by the observed data feature and the resulting change.
Timing matters because a decision made before seeing analysis results has a different risk of outcome-driven selection from a decision made after inspecting them. This does not mean that every post hoc change is invalid. It means the report should distinguish prespecified choices from choices made after results were available, and explain how the latter were handled. If the change affects the interpretation of the primary analysis, describe that impact rather than presenting the revised procedure as though it had been planned all along.
Maintain the original plan as a reference point. A revised decision does not erase the planned one: readers need both to understand what changed. Where feasible, state whether the planned analysis was still run, whether it was interpretable, and whether conclusions differed. Do not imply that a deviation was harmless merely because the point estimate was similar; uncertainty and decision consequences also matter.
Worked Example: Explaining a Covariate-Analysis Deviation
A hypothetical analysis plan specified evaluating renal function as a continuous covariate on clearance, with a prespecified functional form. Before estimating that covariate model, the team reviewed the analysis dataset and found that renal-function values were missing for 18 of 60 participants. The team documented a deviation to use the planned covariate analysis in the 42 participants with observed values and to describe the missing-data pattern; no imputation was performed. The covariate analysis was not a replacement for the primary model fitted to all 60 participants.
Check the counts. The fraction with missing values is \(18/60=0.30\), or 30%; the observed fraction is \(42/60=0.70\), or 70%. These figures identify the extent of the data limitation. They do not establish that missingness is random or that the 42-person subset is representative.
Report the deviation. The report should state the planned analysis, the actual analysis, and the reason: the covariate could not be evaluated for participants without an observed value under the planned procedure, and the team restricted that evaluation to records with measured renal function. It should state that the change was documented before fitting the covariate model, if that is true, and identify the resulting analysis population as 42 participants. It should separately report the primary model’s population of 60 participants and avoid implying that the covariate result applies without qualification to all 60.
The numbers justify describing the extent of missingness, not attributing a cause to it. A report should not say that the missing values were “random” unless that claim is supported by an appropriate assessment and assumptions. It should also not silently redefine the planned population: doing so would conceal the deviation.
Material candidates that were not selected
A candidate analysis can be relevant even when it did not become the primary model. Include a concise disposition for alternatives that were planned, scientifically plausible, or influential in the selection decision. Useful information includes the candidate’s purpose, whether estimation succeeded, the criterion or rationale used to set it aside, and whether its results changed the substantive interpretation. A table is often more readable than a long narrative.
Not every exploratory variation needs the same prominence. Routine debugging runs that do not represent plausible analysis choices need not be presented as scientific alternatives. However, a candidate that was prespecified, materially affected the selection process, or could reasonably alter the reported conclusion should not disappear merely because it was unsuccessful or inconvenient. The report can distinguish such candidates from technical attempts and explain the scope of what is summarized.
Worked Example: Disposition of Three Candidate Analyses
In a hypothetical analysis, three candidates were considered for the primary PK model: a one-compartment model, a two-compartment model, and a one-compartment model with a prespecified residual-error alternative. The one-compartment model met the estimation criteria and was selected. The two-compartment model had the failed fits described above. The residual-error alternative fit successfully but did not improve the prespecified comparison criterion and yielded similar clearance estimates. For the successful candidates, suppose the clearance estimates were 5.4 L/h and 5.2 L/h, respectively.
Quantify the difference without overstating it. The difference between those estimates is \(5.4-5.2=0.2\) L/h. Relative to 5.4 L/h, this is \(0.2/5.4=0.0370\), or about 3.70%. This calculation describes the point-estimate difference only; it does not show that the estimates are statistically equivalent or that uncertainty is negligible.
A disposition table could read as follows:
| Candidate | Status | Disposition |
|---|---|---|
| One compartment, planned residual-error model | Fit met stated criteria | Selected as primary candidate |
| Two compartments | Three attempts; none met gradient criterion | Not interpreted as a usable fit; failed-fit evidence reported |
| One compartment, alternative residual-error model | Fit met stated criteria | Not selected under the prespecified comparison criterion; clearance estimate was 5.2 L/h versus 5.4 L/h in the selected model |
This account distinguishes an unsuccessful fit from a successfully estimated but unselected model. It also states the reason for the alternative’s disposition and gives enough information to see that the point estimates were close, without asserting that all model implications were identical.
A practical reporting sequence
A consistent sequence helps prevent omissions and keeps explanations proportional to their importance. For each material analysis, record the intended question, the planned procedure, what was attempted, the outcome, and the disposition. For any change, add when and why it was made. The sequence below can be applied to a narrative or a report table.
Name the model or analysis, its purpose, the analysis population, and the relevant plan entry.
State whether the analysis was run, whether estimation met the stated criteria, and which diagnostics or outputs support that description.
Compare the planned procedure with the implemented one and say when the change was made in relation to access to results.
State whether the candidate was selected, set aside, or left uninterpreted, explain why, and summarize any effect on interpretation.
Diagnostics, failure modes, and limits of the record
A report is difficult to evaluate when it uses vague labels, omits unsuccessful planned candidates, or presents a revised procedure without identifying the original plan. Check that each material candidate has a clear disposition and that the stated reason matches the evidence. When reporting a failed fit, make sure the reported outputs are labeled as termination or diagnostic outputs rather than valid estimates if the criteria for interpretation were not met. When reporting a deviation, verify the population, data handling, and decision rule are stated consistently throughout the report.
A complete record does not prove that a model is correct, that missingness is ignorable, or that untried candidates would have failed. It also cannot remove the uncertainty introduced by a post hoc decision. Its value is narrower and important: it lets readers distinguish planned from unplanned analyses, successful from unsuccessful estimation, and evidence from explanation. A failure report should be candid without treating every computational difficulty as a scientific finding.
Common mistake
Wrong reasoning: “The alternative did not converge, so it was a bad model and does not need to be mentioned.”
Correction: A failed estimation attempt is not, by itself, evidence that the model is scientifically implausible. If the alternative was planned or material to the selection process, report what was attempted, the failure evidence, and the limits on interpretation. Conversely, do not present a failed fit as though it supplied reliable parameter estimates or a valid comparison result. Keep the computational outcome and the scientific conclusion separate.
Check Your Understanding
Answer each question by identifying what should be reported, not by assuming that an unsuccessful analysis can be ignored.
- A planned model was attempted twice. One run stopped without meeting the stated convergence criterion; the other met it. What information is needed to describe the first run, and how should the second run be treated?
- A report says, “We changed the analysis because the planned model was unsuitable.” Name two details needed to make this explanation assessable.
- A candidate model’s estimate is 8.0 L/h and the selected model’s estimate is 7.6 L/h. Calculate the absolute difference and the difference relative to 8.0 L/h. What can this calculation establish?
- A planned candidate was not fitted because a required covariate was unavailable for some participants. Should the report call the candidate a failed fit? Explain.
- Why is a successful but unselected candidate different from a failed candidate in a disposition table?
Solutions. (1) Identify the model, data population, method, settings, software where relevant, stopping evidence, and criterion not met; label its outputs accordingly. Treat the second run as a separate attempt that met the stated criterion, and report its result and diagnostics under the same criteria. (2) State the planned and implemented procedures, the observed evidence for the concern, and when the change was made relative to seeing results; also explain the resulting population or interpretation if affected. (3) The absolute difference is \(8.0-7.6=0.4\) L/h. Relative to 8.0 L/h, it is \(0.4/8.0=0.05\), or 5.00%. This describes point-estimate separation, not equivalence, statistical significance, or uncertainty. (4) No. If it was not estimated, report that it was not run and why, distinguishing data availability from estimation failure. (5) The successful candidate produced an estimate under the stated procedure but was not selected for a stated reason; the failed candidate did not meet the criteria for a usable fit. Their evidence and appropriate interpretation therefore differ.