Why Choose a Threshold by Estimated MSE?
Clipping weights limits the influence of observations with large estimated weights, often reducing variance. The same clipping can alter the weighted estimating equation and shift its target away from the causal estimand. Thus, a threshold that produces the smallest standard error need not produce the most accurate estimate. Choosing by estimated mean squared error (MSE) makes the trade-off explicit: it combines estimated squared bias with estimated variance.
This procedure does not make the bias observable. It estimates bias using a specified reference, such as a fitted causal data-generating model or a bootstrap approximation. The resulting threshold is only as credible as that reference and its assumptions. The earlier material on the MSE trade-off for truncation explains why both components matter; here the focus is how to estimate them and make a selection.
Target, Candidates, and Estimated MSE
For an inverse-probability weighted estimator, a common treated-unit clipped weight is \(W_i(c)=A_i\min\{1/\hat e(X_i),c\}\), where \(A_i\) is treatment status and \(\hat e(X_i)\) is the fitted treatment probability. This definition requires \(\hat e(X_i)>0\) for every observation used. Alternatively, define the clipped reciprocal to equal \(c\) when \(\hat e(X_i)=0\), as its limit from positive probabilities. In either case, the estimator must also be defined: for a normalized weighted mean, its denominator must be positive. Clipping does not by itself repair an undefined estimating procedure.
A practical analysis first fixes the estimand, estimator, candidate grid, nuisance-model fitting procedure, and method for estimating bias and variance. The grid should be chosen before looking for its minimum. For example, one could compare \(c\in\{1.5,2,3\}\), with all three estimates computed using the same outcome and treatment-model procedures.
Estimating Bias and Variance Across Thresholds
A bootstrap can approximate the sampling distribution of each \(\hat\psi_c\). In each replicate, refit nuisance models and recompute the entire estimator, including clipping. If the bootstrap procedure supplies a target value \(\psi^*\)—for example, the estimand under a fitted data-generating model—then replicate estimates \(\hat\psi_c^{*(1)},\ldots,\hat\psi_c^{*(B)}\) yield $$ \bar\psi_c^*=\frac{1}{B}\sum_{b=1}^B\hat\psi_c^{*(b)},\qquad \widehat{\operatorname{Bias}}(c)=\bar\psi_c^*-\psi^*, $$ and $$ \widehat{\operatorname{Var}}(c)=\frac{1}{B-1}\sum_{b=1}^B \{\hat\psi_c^{*(b)}-\bar\psi_c^*\}^2. $$ A plug-in estimated MSE is the squared estimated bias plus this estimated variance.
The target \(\psi^*\) may come from a parametric model fitted to the observed data. This estimates bias relative to that model, not necessarily bias in the real data-generating process. A nonparametric bootstrap often instead estimates bias by \(\bar\psi_c^*-\hat\psi_c\). That is a bootstrap approximation to sampling bias, but it does not reveal the true bias without additional assumptions. It can be unreliable when the estimator is nonregular or when the empirical distribution poorly represents important parts of the population.
For a model-based procedure, each replicate should be generated under the fitted model and the target \(\psi^*\) calculated under that same model. For a nonparametric procedure, resample independent sampling units and refit all estimated weights in each replicate. Resampling only the final weighted records while holding fitted weights fixed generally omits part of the estimator's sampling variability.
Choose the target, the estimator, the clipping rule, candidate thresholds, and the bootstrap or model-based reference.
For each candidate and replicate, refit estimated weights, apply that candidate's clipping, and calculate the estimator.
Estimate bias relative to the chosen reference and estimate sampling variance from the replicate distribution.
Choose the candidate with the smallest estimated MSE, then examine the curve, Monte Carlo uncertainty, and sensitivity to the reference.
Worked Example: Selecting Among Three Model-Based Candidates
Suppose \(X\) is binary with \(P(X=1)=0.5\), treatment probabilities \(P(A=1\mid X=0)=0.15\) and \(P(A=1\mid X=1)=0.75\), and potential treated outcomes with conditional means \(E[Y(1)\mid X=0]=0.4\) and \(E[Y(1)\mid X=1]=2.0\). The target treated-potential-outcome mean is $$ \psi=E[Y(1)]=0.5(0.4)+0.5(2.0)=1.2. $$ Assume a fitted parametric model uses this target as \(\psi^*\). After \(B=1000\) replicates, suppose the replicate summaries are as follows.
| Threshold \(c\) | Replicate mean \(\bar\psi_c^*\) | Estimated variance | Estimated squared bias | Estimated MSE |
|---|---|---|---|---|
| 1.5 | 1.31 | 0.040 | \((1.31-1.20)^2=0.0121\) | 0.0521 |
| 2 | 1.25 | 0.030 | \((1.25-1.20)^2=0.0025\) | 0.0325 |
| 3 | 1.18 | 0.025 | \((1.18-1.20)^2=0.0004\) | 0.0254 |
For \(c=2\), for example, the estimated bias is \(1.25-1.20=0.05\), so the estimated MSE is \(0.05^2+0.030=0.0325\). The minimum among the three values is \(0.0254\), so this procedure selects \(c=3\). This conclusion is conditional on the fitted model's reference target and on the quality of the replicate approximations; it is not a proof that \(c=3\) has the smallest true MSE.
What the Bias–Variance Calculation Can Show
A simple population calculation illustrates how clipping can change the target as well as the variance. Let \(X\) be binary with probability \(1/2\) in each stratum, let \(e(0)=0.2\), \(e(1)=0.8\), and suppose \(Y(1)\) is deterministically 0 when \(X=0\) and 2 when \(X=1\). The target mean is \(\psi=1\). For the normalized treated weighted mean with true propensity weights clipped at \(c\), its large-sample limit is $$ \psi_c= \frac{\sum_x P(X=x)e(x)\min\{1/e(x),c\}E[Y(1)\mid X=x]} {\sum_x P(X=x)e(x)\min\{1/e(x),c\}}. $$
At \(c=1\), both clipped weights equal 1. The denominator is \(0.5(0.2)+0.5(0.8)=0.5\); the numerator is \(0.5(0.2)(0)+0.5(0.8)(2)=0.8\). Thus \(\psi_1=0.8/0.5=1.6\), with bias \(0.6\). At \(c=2\), the weights are 2 and 1.25. The denominator is \(0.5(0.2)(2)+0.5(0.8)(1.25)=0.7\), and the numerator is \(0.5(0.8)(1.25)(2)=1\). Hence \(\psi_2=1/0.7=10/7\), with bias \(3/7\). At \(c=5\), neither weight is clipped; numerator and denominator are both 1, so \(\psi_5=1\), with zero bias.
The first-order asymptotic variance of a ratio estimator with numerator contribution \(Z=Aq(X)Y\), denominator contribution \(D=Aq(X)\), and limit \(\psi_c\) is \(\operatorname{Var}(Z-\psi_cD)/\{nE[D]^2\}\). For \(n=100\), the values here are \(0.0128\), approximately \(0.02082\), and \(0.03125\) for \(c=1,2,5\), respectively. For instance, at \(c=5\), \(E[D]=1\), and \(Z-\psi D\) equals \(-5\) with probability \(0.1\), \(1.25\) with probability \(0.4\), and 0 otherwise. Its variance is \(0.1(25)+0.4(1.5625)=3.125\), giving \(3.125/100=0.03125\). The larger thresholds reduce bias in this example but increase first-order variance. Estimated MSE selection is designed to compare these effects on a common scale.
Worked Example: Calculating MSE From Replicates
Suppose a model-based bootstrap has target \(\psi^*=1.2\), and four replicate estimates for one candidate are \(1,2,1,2\). Their mean is \(\bar\psi^*=(1+2+1+2)/4=1.5\), so the estimated bias is \(1.5-1.2=0.3\). The sample variance is $$ s^2=\frac{(1-1.5)^2+(2-1.5)^2+(1-1.5)^2+(2-1.5)^2}{3}=\frac{1}{3}. $$
The plug-in estimate \((0.3)^2+1/3\) is approximately \(0.4233\). Alternatively, the mean squared error around the target computed directly over these four replicates is \(\{0.2^2+0.8^2+0.2^2+0.8^2\}/4=0.34\). These differ because \(s^2\) uses denominator \(B-1\). The exact identity is $$ \frac{1}{B}\sum_{b=1}^B(\hat\psi^{*(b)}-\psi^*)^2 =(\bar\psi^*-\psi^*)^2+\frac{B-1}{B}s^2. $$ For these values, the right side is \(0.09+(3/4)(1/3)=0.34\). With many replicates the factor \((B-1)/B\) is close to 1, but the distinction is worth noting when \(B\) is small.
When Does the Selection Rule Work?
Proof. If the candidate set is a singleton, its only element is necessarily both the estimated and population minimizer. Otherwise, uniqueness and finiteness imply the positive gap $$ \Delta=\min_{c\in\mathcal C,\ c\ne c_0} \{\operatorname{MSE}(c)-\operatorname{MSE}(c_0)\}>0. $$ Because \(\mathcal C\) is finite, pointwise convergence in probability for each candidate implies \(\max_{c\in\mathcal C}|\widehat{\operatorname{MSE}}(c)-\operatorname{MSE}(c)|\xrightarrow{p}0\). On the event that this maximum is less than \(\Delta/3\), every \(c\ne c_0\) has estimated MSE exceeding that of \(c_0\): the population gap is at least \(\Delta\), and the two estimation errors together are less than \(2\Delta/3\). The probability of this event tends to 1, so the estimated minimizer equals \(c_0\) with probability tending to 1. \(\square\)
This result is a selection guarantee only when the estimated MSEs converge to the relevant population MSEs. It does not justify a misspecified model-based bias estimate, a grid chosen adaptively without accounting for that search, or a candidate grid whose size grows without further conditions. If two candidates have nearly equal MSE, the gap is small and considerably more data or more replicates may be needed for stable selection.
Diagnostics, Alternatives, and Limitations
Inspect the estimated-MSE curve rather than reporting only its minimum. A sharp minimum supported by stable replicate summaries is more persuasive than a flat curve whose apparent winner changes across bootstrap runs. Check convergence failures, empty or nearly empty weighted denominators, extreme fitted probabilities, and whether a few resamples dominate the estimated variance. Increase the number of replicates to assess Monte Carlo error; this reduces simulation noise, not sampling uncertainty or model misspecification.
Compare the selected threshold under reasonable alternative bias references or nuisance-model specifications. Large changes indicate that the selection is reference-dependent. Report the candidate grid, replicate scheme, handling of failed fits, bias reference, estimated bias and variance by candidate, and the selected threshold. MSE selection does not remove the need to assess positivity, model adequacy, or the causal assumptions defining \(\psi\).
| Approach | What it prioritizes | Main limitation |
|---|---|---|
| Fixed threshold | Consistency and comparability across analyses | May not suit the sample size or weight distribution |
| Percentile-based clipping | A rule tied to observed weight quantiles | Does not directly optimize target-specific MSE |
| Variance-only selection | Small estimated standard error | Can prefer a threshold with substantial clipping bias |
| Estimated-MSE selection | Estimated squared bias plus variance | Bias estimation requires a credible reference or approximation |
A common mistake is to choose the threshold with the smallest standard error and call it the most accurate. That reasoning ignores the possibility that clipping has shifted the estimator away from \(\psi\). The correction is to compare squared bias and variance together, while stating how bias was estimated. Another mistake is to treat the bootstrap mean minus the original estimate as the true bias. It is only a bootstrap estimate under the resampling approximation; it is not an observed fact about the unknown population.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A candidate has estimated bias \(0.08\) and estimated variance \(0.02\). What is its plug-in estimated MSE?
- Why can a threshold with the smallest estimated variance fail to minimize MSE?
- For a candidate grid containing only one threshold, what does the finite-grid selection result imply?
- What does a model-based bootstrap estimate of clipping bias refer to?
Solutions. (1) The estimated MSE is \(0.08^2+0.02=0.0264\). (2) Its squared bias may be large enough that the sum of squared bias and variance exceeds that of another candidate. (3) The sole candidate is necessarily selected, so no positive-gap argument is needed. (4) It estimates bias relative to the target implied by the fitted data-generating model; it need not equal bias under the unknown true process.