Simulating the Analysis Pipeline

Why the Analysis Pipeline Must Be Simulated

A simulated dataset does not produce a performance result by itself. The result depends on what happens after the data are generated: which model is fitted, what objective is optimized, how the optimizer is initialized, and how parameter constraints are applied. If the eventual study will be analyzed with a particular estimation procedure, a design simulation should reproduce that procedure rather than substitute an easier one.

The data-generating model and the estimation model have different roles. The data-generating model specifies how a replicate is produced under a chosen scenario. The estimation model and fitting procedure specify how the analyst would use the resulting observations. They may be identical when assessing a correctly specified analysis, or intentionally differ when evaluating a prespecified alternative analysis. In either case, the fitting procedure must be fixed before examining simulation results.

For replicate \(r\), denote the simulated dataset by \(Y_r\). The pipeline applies a fixed fitting rule to \(Y_r\) and returns an outcome, which may include parameter estimates, an objective value, and a status. In this way the simulation evaluates not merely an abstract estimator, but the estimator as it would actually be used.

Specify the Fit Before Generating Replicates

A useful specification answers several questions in advance. What likelihood or other objective will be optimized? Which parameters are estimated, and which are fixed? What parameterization is used? What are the permitted parameter ranges? What starting values will be supplied, and will there be one start or a fixed set of starts? What numerical conditions count as a completed fit? Finally, which fields will be retained whether the fit succeeds or not?

These choices matter in nonlinear models because the objective can have difficult geometry, and the optimizer may depend on its starting values. Bounds can also change the estimator: if an unconstrained optimum is outside a permitted range, the constrained estimate is obtained by optimizing only over that range. Starting values are computational inputs, not estimates of the unknown parameters. They should be selected using an explicit rule, not chosen separately after looking at each replicate’s simulated outcomes.

Initialization must not leak the generating truth into the analysis. For example, if a simulated replicate was generated using a clearance of 3 L/h, initializing the fit at 3 L/h solely because that value generated the replicate would give the fitting procedure information unavailable in the intended study. A valid prespecified rule might use fixed typical values, a simple data-based initialization that would also be available in practice, or a fixed set of starts. The rule should be applied consistently across replicates and designs.

Objective Functions, Bounds, and Outcomes

For a likelihood-based fit, an optimizer commonly minimizes the negative log-likelihood or \(-2\) times the log-likelihood. If the model specifies independent observations \(Y_i\sim N(\mu_i(\theta),\sigma^2)\), with known \(\sigma\), then, up to the model’s specified mean structure,

This expression assumes normal errors with the stated variance and conditional independence of observations given the model. If \(\sigma\) is estimated, its contribution must also be included in the likelihood and the estimation problem. A different error model requires its corresponding objective; the objective should not be replaced by least squares unless that is the intended model or a justified equivalent.

A constrained fit can be written as minimizing an objective \(Q(\theta;Y)\) over an admissible parameter set \(\Theta\):

A fit at a bound is not automatically a failed fit. It is a legitimate constrained estimate when the optimizer has completed according to the prespecified criteria and the estimate satisfies the constraints. It is still important to flag the boundary status: repeated boundary estimates can reveal that the data and design do not support interior estimates, or that the parameter range deserves review. Any change to bounds must be justified and made as a prespecified revision, not as a response that selectively improves particular results.

At a minimum, retain a row for every replicate. Useful fields include replicate and design identifiers, fitting status, parameter estimates when available, objective value, initialization identifier, boundary indicators, and a concise reason if no admissible estimate was obtained. Do not silently drop an unsuccessful fit or replace its estimate with the generating value. Later summaries can then use explicit outcome rules rather than an undocumented, filtered set of results.

Worked Examples

Justification and Practical Checks

The reason for fixing the pipeline is straightforward. Let \(A\) denote the prespecified analysis rule and \(Y_r\) a replicate under a chosen design and scenario. The replicate-level result is \(A(Y_r)\). If the rule changes to \(A_r\) after inspecting each dataset, the output becomes \(A_r(Y_r)\), and variation in results can reflect both the simulated data and the changing analysis. Applying the same \(A\) makes the analysis contribution explicit and supports a like-for-like design evaluation.

This does not mean every dataset will produce a successful fit, or that one fitting method is suitable for every scientific question. It means the specified method is applied by a consistent rule and its outcomes are reported transparently. As a practical check, verify that the implemented model matches the written objective, that units and parameter transformations are consistent, and that the starting values fall within the permitted parameter space. Check also that every replicate receives an outcome record, including those without an admissible estimate.

A common mistake is to interpret a simulation as a test of the generating model alone and therefore initialize each fit at the true generating parameters. The reasoning is that a good start merely helps the optimizer. The correction is that the start is part of the operational analysis procedure: unless the truth would be known in the intended study, truth-based initialization gives the analysis information it would not have. Use the prespecified initialization rule instead, and retain the resulting outcome even when a fit is inconvenient.

Another mistake is to treat every boundary estimate as a computational failure, or to remove it before calculating later performance summaries. A boundary estimate may be the correct constrained optimum. Record it distinctly from an unsuccessful fit; the later interpretation depends on the performance question and explicit outcome rules. Conversely, a numeric parameter vector alone does not establish a valid fit: the prespecified completion and admissibility criteria must also be met.

Limitations and Alternatives

A fixed pipeline makes comparisons interpretable but does not guarantee that the chosen estimation model is scientifically adequate or computationally robust. One starting value may be insufficient for a difficult objective; bounds may be too restrictive or too permissive; and the assumed error distribution may not represent the intended analysis. These are reasons to evaluate and justify the procedure before the simulation, not reasons to change it selectively after observing results.

A single-start procedure is simpler and may match the planned analysis exactly. A prespecified multi-start procedure can reduce dependence on one initial point, but costs more computation and requires a fixed selection rule, such as choosing the admissible completed fit with the smallest objective. If a real analysis would use a different approach, such as a distinct estimation method or a different initialization strategy, the simulation should implement that alternative when it is the method whose design performance is being evaluated. The comparison is meaningful only when each procedure is fully specified and outcomes are recorded under explicit rules.