Why Sensitivity Results Can Mislead
A global sensitivity analysis (GSA) quantifies how variation in specified model inputs contributes to variation in a specified output over a specified input distribution. Its results are therefore conditional: they describe the model, input distributions, sampling assumptions, numerical implementation, and output target actually used. A precise-looking ranking can still be misleading if any of these do not represent the scientific question.
The earlier material on input distributions and alternative parameter distributions establishes that sensitivity rankings can change when the distribution changes. Here the emphasis is diagnostic: identify whether a result is scientifically interpretable, numerically reliable, and relevant to the endpoint that motivates the analysis. A sensitivity index is not an intrinsic property of a parameter alone.
For independent inputs and a deterministic model output \(Y=f(X_1,\ldots,X_k)\), a first-order Sobol index for input \(X_i\) is \(S_i=\operatorname{Var}(\operatorname{E}[Y\mid X_i])/\operatorname{Var}(Y)\). It measures the fraction of output variance attributed to variation in \(X_i\) alone under the specified joint input distribution. The total-order index includes the contribution of \(X_i\) and its interactions. These interpretations require a well-defined output with finite, nonzero variance and the sampling structure assumed by the estimator. For dependent inputs, ordinary independent-input Sobol interpretations generally do not apply; methods designed for dependence, such as Shapley effects, are an alternative.
Input Ranges and Physical Feasibility
A narrow range is not automatically an error: it may accurately represent the population or uncertainty relevant to a particular question. The failure occurs when a narrow or misplaced range is treated as evidence that a mechanism is unimportant more generally. Variance-based indices allocate the output variation generated by the chosen ranges. They do not measure mechanistic importance independently of those ranges.
Worked Example: A Narrow Range Changes the Ranking
Consider the hypothetical additive model \(Y=10\theta+2\phi\), with independent inputs. First suppose \(\theta\sim U(-1,1)\) and \(\phi\sim U(-5,5)\). For a uniform variable on \((-a,a)\), the variance is \(a^2/3\). Therefore,
There is no interaction, so the output variance is \(66.67\), and the first-order indices are \(S_\theta=33.33/66.67=0.500\) and \(S_\phi=0.500\). Now suppose the plausible range for \(\theta\) is instead \((-0.1,0.1)\), while the range for \(\phi\) stays the same. Then \(\operatorname{Var}(10\theta)=100(0.01/3)=0.3333\), so \(\operatorname{Var}(Y)=0.3333+33.33=33.67\). The indices become \(S_\theta=0.3333/33.67=0.00990\) and \(S_\phi=33.33/33.67=0.9901\).
The coefficient on \(\theta\) remains 10 in both analyses. The ranking changes because the second analysis assigns much less input variation to \(\theta\), not because the model mechanism changed. A practical check is to show the input distributions alongside the rankings and ask whether the ranges reflect the intended population, uncertainty, or intervention. If several scientifically plausible ranges exist, compare results across them rather than presenting one ranking as universal.
A separate input problem arises when sampled combinations are physically impossible. For example, two quantities may share a conserved total, or a rate and an associated fraction may be constrained by their biological definitions. Independent marginal sampling can generate combinations outside those constraints even when every individual value lies within its marginal range. Such samples may cause solver failures, implausible trajectories, or sensitivity results for states the system cannot occupy.
Check joint feasibility, not only marginal bounds. Plot sampled pairs, apply known conservation relationships, inspect failed simulations, and compare sampled combinations with the scientific meaning of the parameters. When dependencies are real, encode them in the joint sampling distribution or use a dependence-aware analysis. Do not silently discard invalid draws if doing so changes the intended input distribution; document the constraint and define the resulting distribution explicitly.
Simulation Noise and Estimate Stability
Sensitivity estimators are calculated from finite model evaluations. Their estimates vary with the sample, and stochastic simulators add another source of variation. If the output contains simulation noise, a variance-based analysis can attribute less variance to inputs than the deterministic model would, because noise contributes to the observed output variance but is not explained by the inputs.
Worked Example: Noise Dilutes a Sensitivity Index
Suppose a hypothetical model has deterministic output \(f(X)=X\), with \(X\sim U(0,1)\), and the recorded output is \(Y=f(X)+\epsilon\). Assume \(\epsilon\) is independent of \(X\), has mean zero, and has variance \(0.04\). Since \(\operatorname{Var}(X)=1/12=0.08333\), the observed output variance is \(0.08333+0.04=0.12333\). Under these assumptions, the first-order index for \(X\) is
The deterministic model has \(S_X=1\), because all its output variation is explained by \(X\). The lower noisy-output index does not imply that \(X\) has become less influential in \(f\); it reflects the added noise in \(Y\). This calculation assumes independent, zero-mean noise with constant variance. If noise depends on \(X\), the decomposition is more complicated and the simple correction does not apply.
Diagnose noise by repeating simulations at fixed input values and comparing within-input variability with variation across inputs. If the simulator is stochastic, use a design that separates input effects from simulation randomness, such as replicated runs or an appropriate noise-aware estimator. Tighten solver tolerances when the variability is numerical, and verify that the output is stable to reasonable tolerance changes. Replication is not a substitute for diagnosing a structural model error.
Even a deterministic simulator can produce an unstable estimate when the sample is too small, the output has rare extremes, or the estimator is poorly matched to the design. A single estimate does not reveal its own Monte Carlo error. Examine convergence as the sample size increases, compare independent batches, and inspect whether rankings and index magnitudes—not just the output mean—are stable. Batch variability is a diagnostic, not automatically a confidence interval: batches may be dependent, and a few batches provide a weak basis for uncertainty quantification.
Worked Example: Batch Estimates Reveal Instability
Suppose four independent hypothetical batches produce estimates of a total-order index of \(0.41\), \(0.66\), \(0.48\), and \(0.62\). Their mean is
The sample standard deviation across these batch estimates is
The batch estimates range from 0.41 to 0.66, and their sample standard deviation is about 0.117. This spread is substantial relative to the mean, so a claim that the index is precisely 0.54 would not be supported by these batches. The appropriate next step is to increase sampling, investigate estimator and model diagnostics, and check whether the ranking persists across larger independent runs. The batch standard deviation describes observed variation among these four estimates; it is not, by itself, a guaranteed confidence interval for the true index.
Match the Output to the Decision
A technically sound analysis can still answer the wrong question if its output target is poorly chosen. Sensitivity of a peak biomarker concentration need not rank parameters as sensitivity of time above a toxicity threshold, a treatment-response probability, or a crossing time would. The output and time window should be specified before interpreting the ranking, with attention to the biological or decision quantity of interest.
Worked Example: A Stable Ranking for the Wrong Endpoint
In a hypothetical model, a parameter \(\alpha\) mainly controls an early transient peak, whereas \(\beta\) affects late recovery. Suppose a well-converged analysis gives total-order indices of \(0.70\) for \(\alpha\) and \(0.20\) for \(\beta\) for peak concentration. A separate, equally well-converged analysis of concentration at a late decision time gives \(0.15\) for \(\alpha\) and \(0.65\) for \(\beta\). The values are hypothetical, and the unlisted index contributions need not be zero.
Neither ranking is inherently wrong: each describes a different target. If the decision concerns late recovery, the peak-concentration ranking is not an adequate basis for prioritizing parameters. Recompute sensitivity for the late-time output or the prespecified decision metric, and state its units and time definition. If several outcomes matter, report their separate rankings rather than combining them without a defined decision rule.
A Diagnostic Workflow and Its Limits
A useful investigation asks whether the input distribution is scientifically appropriate, whether sampled parameter combinations are feasible, whether the simulator and estimator are numerically reliable, and whether the output represents the decision. These checks address different failure modes; passing one does not establish the others.
Write down the output definition, units, time horizon, input ranges, and dependence assumptions. Ask whether each matches the intended scientific question.
Check joint input samples against physical constraints. Record simulation failures, solver warnings, nonfinite outputs, and implausible trajectories rather than treating them as uninformative noise.
Repeat runs where appropriate, compare independent batches, increase sample size, and test solver tolerance. Check whether index estimates and rankings stabilize.
Report the distributions, dependence structure, estimator, sample size, and target. Treat rankings as conditional evidence, and state unresolved limitations.
There is no universal diagnostic threshold that makes a sensitivity ranking trustworthy. Stability depends on the estimator, output distribution, number of inputs, and intended precision. Independent-input Sobol indices are attractive when their assumptions match the model question and the sampling design; for dependent inputs, dependence-aware measures such as Shapley effects may be more suitable. Morris elementary effects can be useful for screening, but they answer a different question and do not repair an invalid input distribution or a mismatched endpoint. No method compensates for nonphysical simulations or an ill-defined target.
Common Mistakes
- Calling a parameter unimportant because its range was narrow. The index reflects variation over that range. Report the range and examine plausible alternatives before making a broader mechanistic claim.
- Sampling each parameter independently by default. Independent draws can violate known joint constraints. Represent dependencies or explain why independence is scientifically appropriate.
- Treating noisy output variance as parameter-driven variance. Replicate at fixed inputs and separate simulation noise from input effects where the method permits.
- Trusting one converged-looking run. Inspect independent batches and increasing sample sizes; a stable point estimate from one run is not evidence of precision.
- Using a convenient output instead of the decision endpoint. Sensitivity is target-specific. Analyze the outcome that corresponds to the biological or decision question.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A parameter has a small total-order index after being assigned a very narrow range. What conclusion is justified, and what additional check would you make?
- Why can independent sampling be inappropriate when two inputs share a conserved total?
- A noisy simulator produces an index smaller than the deterministic model’s index. Give one explanation and one diagnostic.
- Four independent batches yield index estimates \(0.30\), \(0.34\), \(0.31\), and \(0.35\). Calculate their mean and state one reason not to treat that mean alone as proof of convergence.
- A parameter ranks first for peak concentration but not for late recovery. Is either ranking necessarily incorrect? Explain.
Solutions. (1) The index supports only the claim that the parameter explains little output variance over the narrow specified range. Check whether the range represents the intended uncertainty and compare plausible wider or alternative distributions. (2) Independent draws can produce pairs that violate the fixed-total constraint, so the sampled model states may be impossible; use a joint distribution or parameterization that enforces the constraint. (3) Independent simulation noise can add output variance not explained by the input, reducing the index; repeat simulations at fixed inputs to quantify within-input variability. (4) The mean is \((0.30+0.34+0.31+0.35)/4=1.30/4=0.325\). Four batches alone provide limited evidence, and the sample size may still be too small for reliable convergence assessment. (5) No. The rankings refer to distinct outputs, so each may be valid for its target; use the endpoint relevant to the intended decision.