Robustness Of Structural Counterfactuals To Parameter Uncertainty

Why Counterfactuals Need Uncertainty Analysis

A structural model can produce a precise-looking counterfactual: for example, a predicted increase in output after a change in a policy input. But the prediction depends on estimated structural parameters. If those estimates are uncertain, the counterfactual is uncertain too. A point prediction alone does not reveal whether its sign, size, or policy implication is dependable.

The uncertainty question is distinct from whether a counterfactual is identified under the model’s assumptions. Here, suppose a structural specification and intervention are fixed, and focus on how sampling uncertainty in the estimated parameters affects the prediction. Any causal interpretation still depends on the structural model and its identifying assumptions; uncertainty calculations do not establish those assumptions.

The target \(g(\theta;d)\) might be an expected outcome, a difference between two policy scenarios, or a threshold-crossing probability implied by a structural model. The methods below address uncertainty in that target due to estimating \(\theta\). If the target is instead a realized future outcome, uncertainty in future shocks may also matter; it is not included merely by calculating a standard error for \(g(\hat{\theta};d)\).

Delta-Method Uncertainty for a Counterfactual

Suppose the estimator is approximately normal in large samples: \(\hat{\theta}\) has approximate mean \(\theta\) and covariance matrix \(V\), estimated by \(\hat{V}\). If \(g\) is differentiable near the true parameter, a first-order Taylor expansion gives

The gradient records how sensitive the counterfactual is to each parameter. Taking the approximate variance of the linear expression gives

This approximation requires a regular estimator with a valid covariance estimate and a counterfactual function that is sufficiently smooth in the relevant parameter neighborhood. A usual approximate 95% interval is the estimate plus or minus \(1.96\) standard errors. It describes sampling uncertainty under the specified model, estimator, and fixed scenario; it is not an interval that accounts for every possible specification or identifying assumption.

Nonlinear Counterfactuals and Transformations

A counterfactual may be a nonlinear function of the parameters. The delta method remains useful when the function is smooth, but its normal approximation can be poor if the function is strongly curved over the range of plausible parameter values. A monotone transformation can sometimes provide a more natural interval: first construct an interval for the parameter, then transform both endpoints. That preserves the interval’s coverage under the same approximation and avoids some implausible bounds.

Using a Joint Confidence Region

A standard error for one counterfactual summarizes uncertainty in one direction through parameter space. When a conclusion depends on several parameters jointly, it can be useful to evaluate the counterfactual over a joint confidence region. For an approximately normal estimator with estimated covariance matrix \(\hat{V}\), a large-sample 95% region for a \(k\)-dimensional parameter vector is

where \(\chi^2_{k,0.95}\) is the 95th percentile of a chi-square distribution with \(k\) degrees of freedom. For a scalar counterfactual, its range over this region is found by minimizing and maximizing \(g(\theta;d)\) subject to \(\theta\in\mathcal{C}_{0.95}\). The resulting range reflects the whole joint region. It is generally wider than a usual 95% scalar interval because the joint region protects all parameter directions at once.

This optimization approach also works for nonlinear counterfactuals, but the extrema may require numerical optimization. If the region is irregular, unbounded, or includes distinct parameter solutions, the resulting counterfactual range can be wide or disconnected. In such cases, a single standard error may conceal important features of the uncertainty.

Bootstrap Propagation and Specification Comparisons

For a complicated counterfactual, a bootstrap can propagate parameter uncertainty without relying on a first-order Taylor approximation. In each bootstrap replication, resample according to the estimator’s justified sampling scheme, re-estimate the structural parameters, and recompute the counterfactual using the same scenario. Quantiles of the resulting counterfactual estimates can form an interval. The procedure must reproduce the original design and estimation steps; merely drawing parameters independently or holding the estimated parameters fixed does not propagate their sampling uncertainty. Bootstrap inference also does not repair weak identification or an invalid structural specification.

Sampling uncertainty within one specification is only one source of uncertainty. A modeler may also consider plausible alternative functional forms, moment sets, data targets, normalizations, or exclusion assumptions. These alternatives represent specification sensitivity. They should be displayed separately from each specification’s sampling interval: a confidence interval conditional on one model is not automatically a confidence interval that covers across models.

Assessing What Remains Stable

Robustness is best reported in terms of conclusions, not just a list of estimates. For each plausible specification, state the counterfactual estimate, its uncertainty measure, the scenario and units, and whether the relevant conclusion changes. Conclusions might concern the sign of an effect, whether it exceeds a policy threshold, or the ranking of two scenarios. A conclusion is more credible when it persists over parameter uncertainty within specifications and across defensible specifications.

A common mistake is to say, “The counterfactual is robust because its confidence interval excludes zero in the preferred model.” That reasoning checks sampling uncertainty only within one specification. The correction is to examine the target over an appropriate parameter uncertainty set and under plausible alternative specifications. If a sign or policy threshold changes, report that fragility rather than treating one preferred interval as evidence of robustness.