Why Prespecify Decisions?
Model development involves choices: which result to inspect first, what evidence counts as adequate, whether a run is usable, and what to do when evidence is mixed. If those choices are made only after seeing results, it becomes easy to favor an appealing outcome without realizing that the decision rule has changed. Prespecifying decisions makes the path from an analysis objective to an action visible and reviewable.
The earlier material on converting a pharmacometric question into analysis objectives helps establish what the analysis is meant to answer. A decision pathway adds operational detail: it says what evidence will be examined, what threshold or condition will be applied, and what action follows. It does not require every future outcome to be predictable. It requires the plan to say in advance how common outcomes and important exceptions will be handled.
A useful rule is executable by another analyst using the same inputs. “Choose the best model” is not executable because “best” has no stated meaning. “Proceed only if the optimizer reports successful termination, all estimates are finite, and the maximum absolute gradient is no greater than the prespecified tolerance” is more reproducible. The tolerance itself still needs a rationale and must be appropriate to the software and parameterization.
Build the Pathway Before Seeing the Results
Start with the objective, then define decision gates in the order they will be applied. At each gate, distinguish the evidence from the action. A numerical diagnostic, for example, is evidence; rerunning under a specified protocol is an action. A threshold does not make the underlying evidence more meaningful than it is: it makes the planned response to that evidence explicit.
State the question the gate answers, such as whether a run is usable for the next planned assessment.
Identify the output, calculation, data subset, and units to be used. Avoid vague instructions such as “review the diagnostics.”
Define the cutoff, comparison, or qualitative classification before seeing results. Explain whether it reflects a scientific requirement, a technical tolerance, or a conventional statistical criterion.
For pass, fail, borderline, and unavailable evidence, state whether to proceed, repeat a defined step, investigate, or report the issue without making the planned decision.
Save the rule with the analysis plan and record the inputs, result, action, and any deviation in the project’s decision history.
A pathway should have clear precedence when more than one condition applies. For example, a run that fails a numerical gate should not be treated as having passed a later predictive-performance gate merely because its predictions appear favorable. Define an ordered sequence: first establish whether the run is eligible for assessment; only then apply the planned performance rule. This makes the decision a reproducible function of the recorded evidence rather than an informal judgment assembled after the fact.
This sequence is also a practical justification for writing rules in advance. If the rule and its order are fixed, the same evidence yields the same action. If either changes after results are known, the action may depend on the observed result in a way that is difficult to distinguish from an intended plan. Prespecification does not eliminate scientific judgment; it makes the planned judgment and any later change visible.
Worked Decision Pathways
Worked Example: Defining a Numerical Usability Gate
Suppose a team plans to assess whether a hypothetical model run is numerically usable before moving to a prespecified predictive check. Before running the model, the team records these illustrative criteria: the optimizer must report successful termination; all parameter estimates must be finite; the maximum absolute scaled gradient must be no greater than 0.010; and the objective values from qualifying independent starts must differ by no more than 0.10. These tolerances are hypothetical and would need justification for the software and model in an actual project.
Three starts produce objective values of \(-120.03\), \(-120.00\), and \(-120.08\). Their maximum absolute scaled gradients are 0.004, 0.013, and 0.006, respectively. Assume all three report successful termination and all estimates are finite. The second start fails the gradient rule because \(0.013 > 0.010\). The first and third pass that rule.
For the qualifying starts, the objective-value spread is
Since \(0.05 \leq 0.10\), the qualifying starts also meet the planned consistency criterion. The pathway therefore permits the run to proceed to the next planned assessment, while recording that one start failed the gradient gate. It does not permit choosing the failed start simply because its objective value is numerically higher, nor does it establish that the model is scientifically adequate. If no starts qualified, the prespecified action might be to use a documented troubleshooting protocol and rerun; the plan should state whether and how that protocol can be used.
Worked Example: Applying a Predictive-Performance Threshold
A hypothetical analysis plan defines residuals on a held-out set as observed concentration minus predicted concentration, in milligrams per liter. It specifies that this check passes only if root mean squared error is at most 12 mg/L and the absolute mean residual is at most 3 mg/L. These are illustrative, context-specific cutoffs, not general pharmacometric standards. Assume the held-out observations were designated before model assessment and that the same prediction procedure is applied to each observation.
For four held-out observations, suppose the residuals are \(8\), \(-10\), \(12\), and \(-6\) mg/L. The root mean squared error is
The mean residual is
Thus, \(\mathrm{RMSE}=9.274\ \mathrm{mg/L}\leq12\ \mathrm{mg/L}\) and \(|\bar{e}|=1.000\ \mathrm{mg/L}\leq3\ \mathrm{mg/L}\). Both conditions pass, so the planned action is to proceed past this gate. The conclusion is limited to this specified check and held-out set; it does not establish that all predictions are adequate in every clinically relevant setting. A plan should also say what to do if the held-out data or a required prediction is unavailable.
Worked Example: Applying a Prespecified Statistical Threshold
Consider a hypothetical comparison between two candidate models fitted to the same observations. The plan specifies a likelihood-ratio test at \(\alpha=0.05\), provided the models are nested, fitted by maximum likelihood to the same data, and the regularity conditions for the chi-square approximation are reasonable. Assume the difference in model dimension is one, neither tested parameter is on a boundary under the null, and the models pass the planned numerical checks. These assumptions matter: without them, the stated reference distribution may not be justified.
Suppose the log likelihoods are \(-120.0\) for the larger model and \(-122.1\) for the smaller model. The test statistic is
Under the null hypothesis and the stated regularity conditions, this statistic is compared with a chi-square distribution with \(df=1\). The 0.05 upper-tail critical value is approximately 3.841; the corresponding \(p\)-value for 4.2 is approximately 0.0404. Since \(4.2>3.841\), equivalently \(0.0404<0.05\), the prespecified statistical gate is met. The planned action might be to carry the larger model forward to the next planned assessment, not to declare it automatically the final model. If the models are not nested or the regularity assumptions are not credible, the plan must direct the analyst to the stated alternative or to report the comparison as not evaluated under this rule; the chi-square cutoff cannot simply be used anyway.
Choosing and Documenting Thresholds
Thresholds should be chosen for the decision they support, not because a particular number is convenient after results are available. A scientific threshold may represent a maximum tolerable prediction error in a defined context. A numerical tolerance may reflect the precision and scaling of an optimization procedure. A statistical threshold controls a specified testing procedure under its assumptions. These thresholds answer different questions and should not be substituted for one another.
For each threshold, document its source or rationale, the quantity to which it applies, its units or scale, and whether equality counts as passing. State whether the rule applies once or repeatedly, and whether it depends on a particular data set, model fit, or software output. If a threshold is exploratory or provisional, say so. A precise but arbitrary cutoff is not automatically scientifically defensible.
A practical plan also distinguishes a decision rule from a decision log. The plan states what should happen under specified conditions. The log records what actually happened, including the values reviewed, the action taken, and any departure from the plan. Earlier material on decision logs and change history and on run provenance supports traceability; neither record replaces a clearly stated rule.
Diagnostics, Alternatives, and Failure Modes
A rule can be prespecified and still be poorly designed. Before relying on it, check that the evidence can be generated, that its definition is unambiguous, and that its threshold is appropriate to the intended decision. Test the pathway using hypothetical pass, fail, borderline, and missing-output cases. If two analysts could follow the written rule and take different actions, clarify it before results are examined.
| Approach | Strength | Limitation |
|---|---|---|
| Fixed threshold and action | Clear and easy to reproduce when the same conditions apply. | May be too rigid if important scenarios were not anticipated. |
| Staged decision pathway | Orders checks so that eligibility, evidence assessment, and action are not conflated. | Requires explicit precedence and a defined response at each stage. |
| Prespecified conditional rule | Allows different actions for different stated circumstances. | Can become ambiguous if the conditions or fallback are underspecified. |
| Ad hoc judgment after results | Can respond flexibly to an unexpected problem. | Risks outcome-dependent choices and is difficult to reproduce or review. |
A common mistake is to treat a threshold as a complete analysis plan. For example, “use \(p<0.05\)” does not state which comparison will be tested, whether its assumptions apply, what happens if the result is inconclusive, or whether passing the test determines the next action. The correction is to state the evidence, assumptions, cutoff, and action together. Another mistake is to change a cutoff after seeing that the result narrowly misses it. If a change is scientifically necessary, document the reason, timing, and effect on interpretation; do not describe the changed rule as though it had been fixed in advance.
No pathway can cover every unforeseen event. The plan should distinguish foreseeable outcomes, for which an action can be specified, from genuinely unanticipated problems. For the latter, pause the affected decision, record the issue, agree on a justified response, and label any resulting analysis or rule change as a deviation. Prespecification improves transparency; it does not guarantee a correct model, remove uncertainty, or make an unsuitable threshold valid.
Check Your Understanding
Answer each question using the decision-pathway principles above. Solutions follow the questions.
- A plan sets a threshold of \(p<0.05\), but does not identify the comparison or the action if the threshold is not met. What is missing?
- A planned numerical gate requires a maximum absolute gradient of at most 0.010. A run has a value of 0.012. What does the rule imply, and what does it not imply?
- Using the predictive example’s residuals, suppose the prespecified RMSE cutoff were 9.0 mg/L instead of 12 mg/L, while the bias cutoff remained 3 mg/L. Does the gate pass?
- Why should a chi-square likelihood-ratio cutoff not be applied automatically when the compared models fail the stated regularity conditions?
Solutions. (1) The plan must specify the comparison and evidence to be used, any assumptions required for the threshold, and the action for both outcomes. A cutoff alone does not define a decision pathway. (2) The run fails that numerical gate because \(0.012>0.010\). The rule directs the prespecified failure action, such as a defined troubleshooting step; it does not by itself show that the model is scientifically poor or that another model is better. (3) The RMSE remains \(\sqrt{86}=9.274\ \mathrm{mg/L}\), which exceeds 9.0 mg/L. Although the absolute mean residual is \(1.000\ \mathrm{mg/L}\leq3\ \mathrm{mg/L}\), both conditions are required, so the gate does not pass. (4) The chi-square reference distribution relies on assumptions such as nesting and suitable regularity. If those conditions are not credible, the nominal cutoff and \(p\)-value may not have their stated interpretation; follow a prespecified alternative or report that the test is not validly evaluated.