Why Examine Uncertain Inputs?
A simulation can produce a precise estimate for the model as written while still depending heavily on an input that is not known precisely. For example, a model may estimate how often a system fails using an uncertain daily failure rate. Running the simulation more times can reduce random simulation error, but it cannot resolve whether the chosen rate is appropriate. To investigate that second issue, vary the rate across plausible values and see how the estimated event result responds.
This process is called sensitivity analysis. It helps answer a practical question: which uncertain assumptions have the greatest effect on the model’s conclusions? Sensitivity analysis does not establish which input value is true. Instead, it shows what the model implies under a stated range of possible values.
The input values should be chosen for a reason, such as a range supported by available measurements or expert judgment. A low, central, and high value can make the comparison easy to read, but the values should not be treated as equally likely unless there is a separate basis for doing so. The simulated results describe the consequences of those inputs; they do not assign probabilities to the inputs themselves.
A Basic Simulation Sensitivity Analysis
Suppose a model has uncertain input \(a\), other inputs collected in \(\boldsymbol{\theta}_{-a}\), and an event of interest. Let \(g(a,\boldsymbol{\theta}_{-a})\) denote the event probability implied by the model. A simulation estimates that probability by generating \(M\) runs and recording whether the event occurs in each run:
To examine sensitivity to \(a\), evaluate this estimate at several plausible values of \(a\) while keeping \(\boldsymbol{\theta}_{-a}\) fixed. The difference between the estimates describes how much the model output changes across that input range. Because each estimate is based on a finite number of runs, the comparison should also consider Monte Carlo error.
For independent simulated runs, the estimated standard error of an event proportion \(\hat{g}\) is approximately \(\sqrt{\hat{g}(1-\hat{g})/M}\). This measures simulation-to-simulation variation for a fixed input value. It is distinct from uncertainty about which input value is appropriate.
Worked Examples
Worked Example: Sensitivity to Two System Risks
Consider a hypothetical daily system in which an event occurs if demand is unusually high and a backup unit is unavailable. The model assigns probability \(q\) to unusually high demand and probability \(r\) to backup unavailability. It assumes these outcomes are independent on a given day. For each input setting, simulate \(M=50{,}000\) days and record the proportion of days with both outcomes.
The central inputs are \(q=0.10\) and \(r=0.20\). To examine demand-risk sensitivity, vary \(q\) while keeping \(r=0.20\). To examine backup-risk sensitivity, vary \(r\) while keeping \(q=0.10\). Suppose the simulated event counts are:
| Input varied | Input value | Event days out of 50,000 | Estimated event probability |
|---|---|---|---|
| \(q\), with \(r=0.20\) | 0.08 | 790 | 0.01580 |
| \(q\), with \(r=0.20\) | 0.10 | 1,009 | 0.02018 |
| \(q\), with \(r=0.20\) | 0.12 | 1,197 | 0.02394 |
| \(r\), with \(q=0.10\) | 0.15 | 754 | 0.01508 |
| \(r\), with \(q=0.10\) | 0.20 | 1,009 | 0.02018 |
| \(r\), with \(q=0.10\) | 0.25 | 1,261 | 0.02522 |
For the central setting, the estimated event probability is
or about 2.018% of simulated days. Across the stated range for \(q\), the estimated probability changes from 1.580% to 2.394%, a difference of \(0.02394-0.01580=0.00814\), or 0.814 percentage points. Across the stated range for \(r\), it changes from 1.508% to 2.522%, a difference of \(0.02522-0.01508=0.01014\), or 1.014 percentage points. For these particular ranges, the output is somewhat more sensitive to \(r\) than to \(q\).
At the central estimate, the Monte Carlo standard error is
That is about 0.063 percentage points for a single estimate. The shifts across the tested input ranges are much larger than this single-estimate simulation error. The results therefore suggest that both inputs matter within these ranges, with a larger estimated shift for backup unavailability. This conclusion is limited to the specified ranges and model assumptions; it is not a claim that either input is known to take those values.
Worked Example: A Rate’s Effect Over Repeated Days
Suppose a hypothetical event can occur on each of 30 days, with a constant daily probability \(p\). Assume days are independent and the daily probability remains unchanged throughout the period. The outcome of interest is whether at least one event occurs in the 30-day period. Simulate \(M=20{,}000\) periods for each plausible daily rate.
Suppose the simulation produces 5,206 periods with at least one event when \(p=0.01\), 9,089 when \(p=0.02\), and 11,984 when \(p=0.03\). The estimated event probabilities are:
Thus, increasing the daily rate from 1% to 3% changes the estimated probability of at least one event in 30 days from about 26.03% to 59.92%. A small-looking change in a daily input can produce a substantial change in a cumulative-period output.
The shape of this response follows from the model assumptions. The probability of no event on one day is \(1-p\). By independence, the probability of no event on all 30 days is \((1-p)^{30}\). The event of interest is the complement, so its probability is
This expression explains why the relationship is not a simple one-for-one change: the daily probability acts repeatedly across the period. The simulation estimates are consistent with that increasing pattern. The calculation is a justification for the shape of the result, not a reason to ignore whether the independence and constant-rate assumptions are reasonable.
Worked Example: Sensitivity of a Simulated Count
A hypothetical screening program is modeled for a group of 10,000 people. The event is a missed case: a person has the condition and receives a negative test result. Let \(p\) be the modeled prevalence and \(r\) the probability of a negative result among people who have the condition. For each simulated person, the model first generates condition status using \(p\), then generates the test outcome using \(r\) for people with the condition. This specifies the conditional meaning of \(r\); no assumption about test results among people without the condition is needed to count missed cases.
To estimate the event frequency, suppose the model simulates 100,000 people at each input setting. At the central values \(p=0.05\) and \(r=0.10\), it records 503 missed cases. The estimated proportion is \(503/100{,}000=0.00503\). Expressed as an estimated count per 10,000 people, this is
Now vary one input at a time. Suppose the simulation gives 402 missed cases when \(p=0.04\) and \(r=0.10\), and 596 when \(p=0.06\) and \(r=0.10\). These correspond to 40.2 and 59.6 missed cases per 10,000 people. With \(p=0.05\) held fixed, suppose \(r=0.08\) gives 397 missed cases and \(r=0.12\) gives 604. These correspond to 39.7 and 60.4 per 10,000 people.
Across the chosen prevalence range, the estimated change is \(59.6-40.2=19.4\) missed cases per 10,000. Across the chosen false-negative range, it is \(60.4-39.7=20.7\) per 10,000. The two inputs have similar effects over these particular ranges. The comparison is meaningful because the output uses the same unit and the model holds the other input fixed in each comparison.
For the central event proportion, the approximate Monte Carlo standard error is
On the scale of counts per 10,000, this is about \(10{,}000(0.000224)=2.24\) cases. The changes of roughly 19 to 21 cases across the tested ranges are appreciably larger than this simulation error. The analysis indicates that the modeled missed-case count responds to both uncertain inputs, but it does not show that the inputs are equally uncertain in reality.
Comparing Sensitivity Results Carefully
A useful comparison begins with a clearly defined output and a clearly specified input range. Report the output in context, including units when it is a count or rate. When comparing inputs, compare their effects over stated plausible ranges rather than comparing bare numerical changes in inputs that may use different scales. For instance, a change of 0.02 in a probability and a change of 2 in a time rate cannot be compared directly; their resulting changes in the model output can be.
Simulation noise can also affect the estimated difference between two settings. If two settings use independent simulation runs with \(M\) runs each, their estimated event proportions have separate Monte Carlo variation. For a high setting \(H\) and a low setting \(L\), the estimated difference is \(\hat{\Delta}=\hat{g}_H-\hat{g}_L\). Independence of the two runs means their covariance is zero, so the variance of the difference is the sum of the two variances:
This gives a way to judge whether an observed shift is large compared with the simulation noise in the comparison. A shift that is similar in size to that noise may need more simulation runs or a cautious interpretation. A larger run size can sharpen the simulation estimate, but it cannot fix an implausible input range or an incorrect model assumption.
One-at-a-time analysis is useful for identifying which inputs matter when changed individually, but it can miss interactions. For example, the effect of changing one rate may depend on the value of another rate. If that is plausible for the system, examine combinations of the uncertain inputs as well, while keeping the same outcome definition and model structure. This is still sensitivity analysis: the purpose is to learn how the specified model responds to uncertain inputs, not to silently change its assumptions.
Common Mistakes and Why Sensitivity Matters
- Treating the chosen central value as certain. A central input is a reference point, not proof that the value is correct. Show how results change over a defensible range.
- Changing several inputs at once and attributing the result to one. If other inputs are held fixed, a one-at-a-time comparison isolates the model’s response to the input being varied. If several are changed together, the result reflects all of those changes.
- Confusing Monte Carlo error with input uncertainty. Monte Carlo error describes variation from finite simulation runs at fixed inputs. Input uncertainty concerns which values are plausible. More runs address the first, not the second.
- Calling the largest observed change universally most important. A ranking applies to the particular input ranges, output, and assumptions used. Different plausible ranges or a different outcome can give a different ranking.
- Interpreting sensitivity as evidence that an assumption is true. A large output shift shows that the model depends on that input. It does not validate the input value or establish that the modeled event rate will occur in reality.
Sensitivity analysis matters because it shows whether a model’s conclusion is stable across plausible input values or hinges on one uncertain assumption. A clear report states the event being modeled, the values varied, what was held fixed, the resulting output changes, and whether those changes are large relative to Monte Carlo error.
Check Your Understanding
Use the definitions and examples above to answer these questions.
- A model estimates the probability of a system outage using an uncertain failure probability. What does a one-at-a-time sensitivity analysis vary, and what does it hold fixed?
- Why does increasing the number of simulation runs reduce Monte Carlo error but not resolve uncertainty about the appropriate input value?
- A simulation estimates an event probability as 0.04 for a low input and 0.05 for a high input. What additional information is needed to judge whether the difference is meaningful?
- In the 30-day example, why can a change in daily event probability produce a larger change in the probability of at least one event over the full period?
- Give one reason a one-at-a-time analysis could miss an important feature of a model’s response to uncertain inputs.