Examining Performance Distributions Across Replicates

Why Examine the Distribution Across Replicates?

A simulation summary such as the mean can describe average performance, but it cannot show how that performance varies from replicate to replicate. Two candidate sampling designs can have the same average estimation error while differing substantially in their tails: one may usually perform consistently, while another may have occasional very poor results. A mean can also fall between two clusters of outcomes, even when few or no replicates produced a result near that mean.

The object of interest here is the distribution of a replicate-level performance measure under a specified simulation setup. For example, each replicate might produce one estimate of bias, one confidence-interval width, or one measure of computational time. The distribution describes the observed variation across simulated repetitions; it does not by itself explain which features of the data-generating process caused that variation.

Interpretation depends on what \(Y_r\) measures. An error may be signed, so positive and negative values can cancel in an average; an absolute error is nonnegative and describes error magnitude instead. A confidence-interval width has units of the parameter being estimated, while a computation time has units such as seconds. State the performance measure, its units, and the simulated condition whenever presenting its distribution.

Empirical Quantiles and the Median Convention

To calculate empirical quantiles, first order the replicate results from smallest to largest. Write these as \(Y_{(1)}\leq Y_{(2)}\leq\cdots\leq Y_{(R)}\), where the parentheses indicate order. This tutorial uses the nearest-rank convention: for \(0<p\leq1\), the empirical \(p\)-quantile is the observation at rank \(\lceil Rp\rceil\), where \(\lceil\cdot\rceil\) means round up to the next integer. The empirical 0-quantile is defined as the smallest observation.

For example, the nearest-rank 0.90-quantile is the observation at rank \(\lceil0.90R\rceil\). Approximately 90% of the observed results are at or below that value, with the exact fraction depending on the finite sample and ties. Quantiles from a finite simulation are descriptive estimates; they need not equal the corresponding quantiles of the underlying theoretical distribution.

Be precise about the median when \(R\) is even. The conventional sample median is the average of the two middle ordered values, \( \{Y_{(R/2)}+Y_{(R/2+1)}\}/2 \). Under the nearest-rank convention, however, the empirical 0.5-quantile is \(Y_{(\lceil R/2\rceil)}\), the lower of those two middle values. These are different conventions and can give different values. Report which definition is being used; do not label the nearest-rank 0.5-quantile as the conventional median when \(R\) is even.

A useful compact summary is a set of quantiles, often including the 0.10, 0.25, 0.50, 0.75, and 0.90 quantiles. The difference between the 0.75- and 0.25-quantiles gives an interquartile range under the chosen quantile convention. Quantiles describe selected locations in the distribution, not every detail of its shape. A plot is needed to see features such as gaps or multiple clusters.

Plots That Reveal Distributional Features

A plot should make the replicate-level variation visible rather than merely restating a single average. A histogram groups results into bins and can reveal skewness, gaps, or clusters, though its appearance depends on bin width and placement. A dot plot or strip plot displays individual results directly and is especially useful when the replicate count is modest. An empirical cumulative distribution function (ECDF) plots the fraction of results at or below each possible value; it shows quantiles and tail differences without grouping observations into bins.

A box plot provides a compact comparison of medians and the central spread, but it does not display all individual values or reliably reveal multiple modes. In a conventional box plot, whiskers and points may follow a specified outlier rule; a point flagged by that rule is not automatically erroneous. For replicate-level diagnostics, pairing a box plot with a dot plot, histogram, or ECDF often gives more useful information than using a box plot alone.

• Skewness: A long tail on one side, visible in a histogram or ECDF, suggests that a mean may be pulled away from the bulk of the results. Report quantiles that describe both the center and relevant tail.
• Extreme results: A point far from the rest can be a genuine but rare simulation outcome, a numerical failure, or a data-handling error. Check the replicate and analysis records before deciding how to treat it.
• Possible multimodality: Two or more separated clusters can indicate distinct behavior among replicates. The overall mean or median may lie in a sparse region between clusters. Investigate the simulation and analysis records; do not infer a cause from the plot alone.
• Unexpected discreteness or ties: Repeated values may arise from rounding, boundary estimates, or a discrete performance measure. Preserve the recorded values and consider whether reporting precision obscures meaningful differences.

Replicates should represent the intended simulation condition and analysis pipeline. Interpret the plot in light of how the data were generated and analyzed, and check whether the result being plotted is defined for every replicate. If some analysis runs failed, do not hide that fact by plotting only successful estimates without saying so. A distribution among successful runs is conditional on success and may differ from performance across all attempted runs.

Justification: Why a Mean Can Hide Multiple Clusters

The mean is a balance point, not a guarantee that typical observations lie nearby. Consider twelve hypothetical replicate results, in arbitrary performance units: 4.8, 5.0, 5.1, 5.2, 5.3, 5.4, 14.6, 14.8, 15.0, 15.1, 15.2, and 15.4. The mean is

The six smaller values are all below 5.5, and the six larger values are all above 14.5; no result is close to the mean of 10.075. This follows because the mean averages contributions from both clusters. In a dot plot or histogram with bins that preserve the gap, the separation is visible, whereas the mean by itself gives no indication of it. The nearest-rank quartiles are \(\widehat{Q}_{0.25}=Y_{(3)}=5.1\) and \(\widehat{Q}_{0.75}=Y_{(9)}=15.0\), so the interquartile range is 9.9 units. The conventional sample median is \((Y_{(6)}+Y_{(7)})/2=(5.4+14.6)/2=10.0\), also between the clusters. This example shows why neither a mean nor a median replaces inspection of the distribution's shape.

Choosing and Reporting a Summary

The appropriate summary depends on the question. A mean is useful when average performance is the target, but should not stand alone when the distribution is skewed or has extreme values. A median and interquartile range describe central location and spread more robustly to an isolated extreme, but can conceal tail behavior or separate modes. Selected quantiles make particular parts of the distribution explicit, and plots show patterns that a small set of numbers cannot.

When comparing replicate distributions, use the same performance definition, units, quantile convention, and axis scale. State the number of replicates and whether the results include failed analyses or only defined performance values. For simulation comparisons using common random numbers, paired differences may be central to the comparison; distribution plots of the individual design results do not replace examining those paired differences. Conversely, a summary of paired differences does not reveal the full distribution for either design.

A distribution estimated from a finite number of replicates has simulation variability. A sparse tail may be poorly represented, so an apparently stable 0.99-quantile from a small replicate count should be treated cautiously. This tutorial's summaries describe observed replicate results; they do not provide a formal uncertainty interval for a quantile. The plot is a diagnostic aid, not proof that a pattern will persist under more simulation runs.

Common Pitfalls

• Reporting only the average. Equal means do not imply equal consistency or tail behavior. Add quantiles and a plot suited to the replicate count.
• Mixing median conventions. With an even number of results, the conventional median averages the two central values; the nearest-rank 0.50-quantile is the lower central value. Name the convention and calculate consistently.
• Deleting a surprising replicate automatically. An extreme point may be a genuine outcome. Check its simulation and analysis record, document any correction, and distinguish a verified error from an unusual but valid result.
• Overinterpreting a histogram. A visual cluster can shift or disappear with bin choices, particularly with few replicates. Check individual values or an ECDF and treat the shape as diagnostic evidence, not a definitive classification.
• Ignoring undefined results. Dropping failed replicates can make performance look better. Report failures separately and clearly define whether each displayed distribution is conditional on successful estimation.