Communicating Uncertainty and Statistical Inference

Uncertainty Is Part of the Effect Estimate

An effect estimate summarizes the observed contrast on a chosen scale; an uncertainty measure describes how much that estimate might vary under repeated sampling and the stated statistical model. In an observational comparative study, reporting only the estimate leaves readers unable to assess its precision. Reporting a confidence interval or standard error without explaining how it was calculated can be just as uninformative: matching and weighting change the structure of the analysis, so a variance procedure appropriate for one design may not be appropriate for another.

The uncertainty calculation does not establish that treatment groups are exchangeable, that the propensity-score model is adequate, or that the target population is supported by the data. It quantifies uncertainty under its assumptions. A narrow interval can accompany substantial systematic bias.

A confidence interval is not a statement that the parameter has a specified probability of lying in the particular interval calculated from the data. Rather, the confidence level describes the long-run coverage of the procedure under its assumptions. Similarly, a p-value is not the probability that the null hypothesis is true. State the effect measure and units alongside the uncertainty: for example, a risk difference in percentage points, or a mean difference in days.

Match the Variance Procedure to the Analysis

A variance procedure must reflect how observations contribute to the estimate and which observations may be dependent. In a one-to-one matched design, members of a pair are deliberately linked. Their outcomes may be correlated, so treating all treated and comparison observations as if they came from unrelated groups can give the wrong SE. The within-pair contrast is the relevant unit for estimating uncertainty in a paired analysis.

For \(n\) independent matched pairs, let \(D_i\) be the treated-minus-comparison outcome difference in pair \(i\), and let \(\bar{D}\) and \(s_D\) be the sample mean and sample standard deviation of the \(n\) pair differences. Under a model in which pair differences are independent and identically distributed with finite variance, the estimated SE of the mean difference is \(s_D/\sqrt{n}\). If the pair differences are approximately normal, a \(t\) interval uses \(df=n-1\). With enough pairs, a normal approximation may instead be used when appropriate.

The reason for this formula is that the variance of the average of independent pair differences is the variance of one difference divided by the number of pairs. Since the population variance of the differences is unknown, it is estimated by \(s_D^2\). The \(t\) critical value accounts for that estimation under the stated model. This calculation does not require the two outcomes within a pair to be independent; their dependence is already represented in \(D_i\). It does require an appropriate model for dependence across pairs.

For weighting, the uncertainty calculation should reflect the weights and the estimator. A common approach for inverse-probability weighted estimates is a robust sandwich variance, with an influence-function or estimating-equation calculation that accounts for estimation of the propensity-score model. A simple formula for an unweighted sample mean is not automatically valid after weighting. The report should identify the variance method, whether estimated weights were accounted for, and any clustering or repeated observations handled by the procedure. A bootstrap can be an alternative, but its resampling scheme must respect the sampling structure and, when the analysis procedure is repeated, generally re-estimate the propensity scores and weights in each replicate.

Report Weighted Uncertainty Transparently

Weighting creates a pseudo-population in which observations contribute unequally. The point estimate and its variance are therefore determined by both the outcome data and the weights. Very large or highly variable weights can make an estimate unstable; the nominal number of people may greatly exceed the amount of information effectively contributing to the estimate. Reporting the variance method does not remove this instability, but it lets readers understand how uncertainty was assessed.

For a large-sample, two-sided test of a scalar effect \(\hat{\theta}\), a normal approximation uses \(z=\hat{\theta}/SE(\hat{\theta})\) when the null value is zero. A 95% Wald interval is \(\hat{\theta}\pm1.96SE(\hat{\theta})\). The estimate, SE, interval, test, and p-value should use a consistent variance procedure. If small-sample degrees-of-freedom corrections or a different interval procedure are used, name them rather than implying that the calculation was the standard normal approximation.

Interpret the Interval and Test Together

A confidence interval communicates a range of effect sizes compatible with the data and the stated procedure, while a p-value summarizes evidence against a particular null value under a specified test. When both use the same two-sided procedure, a 95% interval that excludes zero corresponds to a p-value below 0.05 for a test of a zero effect. This correspondence need not hold if the interval and test use different variance estimates, reference distributions, sidedness, or adjustments.

Do not use statistical significance as a substitute for magnitude or relevance. An estimate can be important in context even if its interval includes zero; conversely, a precisely estimated small effect may be of little practical importance. Report enough digits to show the scale without suggesting more precision than the data support, and give the confidence level, sidedness of the test, and variance procedure.

Diagnostics, Failure Modes, and Limitations

Before reporting inferential results, verify that the variance method corresponds to the actual analysis sample and estimator. For matching, check whether the calculation uses pair or matched-set structure, whether matching with replacement or variable-ratio sets creates repeated contributions, and whether clustering is present. For weighting, inspect the weight distribution and report the variance procedure and any treatment of estimated weights. The effective information may be limited by a small number of pairs, highly variable weights, or dependence among observations, even when the total row count appears large.

A common mistake is to calculate an SE as if matched individuals were independent, because there are two groups of observations. That ignores the design: paired outcomes can be correlated, and uncertainty for a paired mean contrast depends on the distribution of within-pair differences. The correction is to use a variance estimator appropriate to the matched sets, or another justified method that preserves their dependence. Similarly, using a conventional unweighted variance formula for a weighted estimate can misstate precision.

Another mistake is to treat a narrow confidence interval as evidence that confounding has been eliminated. The interval generally reflects sampling variation under the specified model and variance procedure. It does not quantify unmeasured confounding, selection bias, outcome misclassification, or model misspecification unless these uncertainties are explicitly incorporated. Report precision and design limitations separately; neither replaces the other.

The main alternatives are not interchangeable: a paired \(t\) procedure uses pair differences and their distribution; a robust sandwich procedure estimates variance under a specified estimating equation while allowing certain forms of heteroskedasticity or dependence; and a bootstrap approximates sampling variation by resampling under a chosen scheme. State which was used and why it fits the estimator and data structure. No procedure is automatically reliable when its assumptions fail, when there are too few independent units for its approximation, or when influential observations dominate the result.