Entropy-Balancing Weights for Covariate Adjustment

Why Entropy Balancing?

Weighting can make the covariate distribution of a comparison group resemble that of a target group. Entropy balancing does this directly: it chooses weights as close as possible to specified base weights while satisfying prespecified covariate-moment constraints. Unlike a procedure that merely seeks similar propensity scores, its constraints state exactly which sample moments must agree.

The constraints are guarantees about the weighted sample, not proof that the groups are comparable in every relevant respect. In particular, balancing observed covariates does not by itself establish a causal effect. The target moments, the feasible set of weights, and the assumptions supporting the causal interpretation all matter.

Setup and Definition

Consider an observational sample with treatment indicator $D_i$, outcome $Y_i$, and pre-treatment covariates $X_i$, for units $i=1,\ldots,n$. Let $c(X_i)$ be a chosen vector of covariate functions: it might contain covariate values, indicators, squares, or interactions. For an average treatment effect on the treated (ATT) analysis, the target is the treated group’s empirical moment vector, $m_T=n_1^{-1}\sum_{i:D_i=1}c(X_i)$, where $n_1$ is the number treated.

Let $i=1,\ldots,n_0$ index the controls, and let $q_i>0$ be their base weights, normalized so that $\sum_{i:D_i=0}q_i=1$. Equal base weights, $q_i=1/n_0$, are a common choice. The entropy-balancing weights $w_i$ are normalized nonnegative control weights, chosen to meet the target moments while minimizing their relative entropy from the base weights.

Definition: Given a target moment vector $m_T$, control base weights $q_i>0$ summing to one, and covariate functions $c(X_i)$, entropy balancing solves $$ \min_{\{w_i\}}\ \sum_{i:D_i=0} w_i\log\!\left(\frac{w_i}{q_i}\right) \quad\text{subject to}\quad w_i\geq 0,\qquad \sum_{i:D_i=0}w_i=1,\qquad \sum_{i:D_i=0}w_i c(X_i)=m_T. $$ A term with $w_i=0$ is defined by continuity as $w_i\log(w_i/q_i)=0$. The objective is the Kullback–Leibler divergence from the chosen-weight distribution to the base-weight distribution, using this stated ordering.

The constraints require the weighted control mean of every included function to equal its treated-group mean. The weights are not required to be equal, but the objective penalizes departures from the base distribution. With equal base weights, this means that among weights meeting the constraints, the procedure selects the distribution with the smallest relative-entropy departure from uniform weighting.

Feasibility, the Exponential-Tilting Form, and a Derivation

Before computing weights, check whether the target is attainable. The vector $m_T$ must lie in the convex hull of the control vectors $c(X_i)$; otherwise, no nonnegative weights summing to one can satisfy the moment constraints. For a finite control sample, a strictly positive feasible solution exists when $m_T$ lies in the relative interior of that convex hull. “Relative interior” means interior within the affine space spanned by the control vectors, so redundant or linearly dependent constraints do not cause confusion.

When a strictly positive feasible solution exists, the minimizer has a finite exponential-tilting representation. This claim does not extend to every feasible target: a target on the boundary can require zero weights, which no finite exponential tilt with strictly positive base weights can produce. The derivation below applies to an interior solution.

Form the Lagrangian.
For the moment constraint multiplier $\lambda$ and normalization multiplier $\lambda_0$, use $L=\sum_i w_i\log(w_i/q_i)+\lambda_0(\sum_iw_i-1)+\lambda^\top(\sum_iw_ic(X_i)-m_T)$.

Differentiate at a positive solution.
The derivative with respect to $w_i$ is $\log(w_i/q_i)+1+\lambda_0+\lambda^\top c(X_i)$. Setting it to zero gives $w_i=q_i\exp(-1-\lambda_0-\lambda^\top c(X_i))$.

Normalize and reparameterize.
Set $\eta=-\lambda$. The common factor is determined by $\sum_iw_i=1$, giving $w_i=q_i\exp(\eta^\top c(X_i))/\sum_jq_j\exp(\eta^\top c(X_j))$. The finite vector $\eta$ is chosen so these weights meet the target moments.

This is an exponential tilt of the base weights. The entropy objective is strictly convex on the probability simplex, so if the constraints are feasible, the minimizing weight vector is unique. The multiplier representation is useful for computation, but its finite form requires strict positivity of the solution. A feasible boundary solution may instead have some $w_i=0$; it can be approached as a limit of tilts whose multipliers diverge.

Worked Examples

Worked Example: Balancing a Binary Covariate

Suppose four hypothetical controls have a binary covariate $X$ with values $0,0,0,1$. Their base weights are $q_i=1/4$. The treated group’s proportion with $X=1$ is $0.5$. Find the entropy-balancing weights, constraining the mean of $X$.

By symmetry, the exponential tilt gives each control with $X=0$ the same weight and the control with $X=1$ a weight multiplied by $r=\exp(\eta)$. After normalization, each zero-valued control has weight $1/(3+r)$, and the one-valued control has weight $r/(3+r)$. The target constraint is $$ \frac{r}{3+r}=0.5 \quad\Longrightarrow\quad 2r=3+r \quad\Longrightarrow\quad r=3. $$

Thus the individual weights are $1/6,1/6,1/6,1/2$. They sum to $1$, and the weighted mean of $X$ is $0(1/6+1/6+1/6)+1(1/2)=0.5$, exactly the treated target. The target is strictly between the control values 0 and 1, so positive weights are feasible and the finite tilt is appropriate.

Worked Example: Checking Balance and Calculating an ATT Contrast

In a hypothetical sample, four treated units have covariates $X=(0,1,1,2)$ and outcomes $Y=(16,18,17,19)$. Four controls have $X=(0,0,1,2)$ and $Y=(10,12,14,20)$. Use equal control base weights and balance the mean of $X$. Then calculate the treated-minus-weighted-control outcome contrast.

The treated target mean is $(0+1+1+2)/4=1$. With equal base weights, an exponential tilt assigns each control weight proportional to $r^{X_i}$, where $r=\exp(\eta)$. The control values have two zeros, one one, and one two, so the normalizing constant after omitting the common base-weight factor is $Z=2+r+r^2$. The target constraint becomes $$ \frac{r+2r^2}{2+r+r^2}=1 \quad\Longrightarrow\quad r+2r^2=2+r+r^2 \quad\Longrightarrow\quad r=\sqrt{2}. $$

Since $Z=2+\sqrt{2}+2=5.4142$, the weights in control order are $1/Z,1/Z,\sqrt{2}/Z,2/Z$, or approximately $0.1847,0.1847,0.2612,0.3694$. They sum to one. The weighted control mean of $X$ is $0(0.1847+0.1847)+1(0.2612)+2(0.3694)=1.000$, matching the treated mean.

The treated outcome mean is $(16+18+17+19)/4=17.50$. The weighted control outcome mean is $$ 0.1847(10)+0.1847(12)+0.2612(14)+0.3694(20) \approx 15.11. $$ The weighted contrast is therefore $17.50-15.11=2.39$ outcome units. This is a descriptive weighted contrast; its causal interpretation requires the identification assumptions discussed below. The calculation verifies balance only for the specified mean of $X$, not for every possible feature of the covariate distribution.

Worked Example: A Feasible Boundary Target

Consider two hypothetical controls with covariate values $c(X_1)=0$ and $c(X_2)=1$, equal base weights $q_1=q_2=0.5$, and target moment $m_T=0$. Find the feasible weights and determine whether they arise from a finite exponential tilt.

The constraints are $w_1+w_2=1$, $w_1,w_2\geq0$, and $0w_1+1w_2=0$. They force $w_2=0$ and $w_1=1$. The objective is finite: $$ 1\log(1/0.5)+0\log(0/0.5)=\log(2)\approx0.6931. $$

For any finite tilt parameter $\eta$, the weight on the second control would be $\exp(\eta)/(1+\exp(\eta))$, which is strictly positive. It approaches zero only as $\eta$ tends to negative infinity. Thus this feasible entropy-minimizing solution has finite objective but no finite exponential-tilt multiplier. The target is on the boundary of the controls’ moment convex hull, rather than in its relative interior.

Using the Weights for Causal Adjustment

For an ATT analysis, the empirical weighted contrast is $$ \widehat{\tau}_{ATT} = \frac{1}{n_1}\sum_{i:D_i=1}Y_i - \sum_{i:D_i=0}w_iY_i. $$ The treated mean is left unweighted, and the control weights sum to one and target the treated covariate moments. If outcomes are measured in, for example, dollars, the contrast is in dollars. The weighting construction alone does not supply an uncertainty estimate or guarantee that this contrast identifies a causal effect.

For causal interpretation, let $Y_i(1)$ and $Y_i(0)$ denote unit $i$’s potential outcomes under treatment and control. Consistency requires the observed outcome to equal the potential outcome under the treatment actually received. Conditional exchangeability for the ATT requires that, among treated units, the untreated potential outcome is conditionally comparable with that of controls given the measured pre-treatment covariates. Positivity requires controls to be available at the covariate values represented in the treated target. Under these assumptions, the weighted control outcomes can represent the treated group’s counterfactual untreated outcomes only if the weights adequately reproduce the treated covariate distribution, or if additional outcome-model assumptions make the balanced moments sufficient (for example, if the conditional untreated outcome mean is in the span of the balanced functions). Entropy balancing enforces selected empirical moment equalities; it cannot establish exchangeability, repair unmeasured confounding, or guarantee adequate overlap.

Moment choice is consequential. Matching means does not in general match variances, nonlinear functions, or interactions. If those features matter for the adjustment, include the corresponding functions in $c(X)$, then check each constraint after weighting. Adding constraints can improve measured balance but can also make the target infeasible or push it to the boundary, producing extreme or zero weights. In finite samples, inspect both the achieved balance and the distribution of weights rather than treating the existence of a numerical solution as sufficient evidence of a sound design.

Common Mistakes and Practical Checks

Assuming feasibility guarantees a finite tilt. Feasibility only means some nonnegative weights satisfy the constraints. A boundary target can require zero weights; finite exponential tilts are justified when a strictly positive feasible solution exists.
Claiming that all covariates are balanced. Exact equality holds for the functions included in $c(X)$. State what those functions are and check their weighted moments explicitly.
Forgetting normalization. The control weights must sum to one for the weighted mean and ATT contrast given here. If weights are reported on another scale, normalize them before interpreting them as a weighted mean.
Treating balance as proof of causality. Balance on measured functions is a design property, not a test of unmeasured confounding or conditional exchangeability.
Ignoring overlap and weight concentration. A target outside the control moment convex hull is infeasible. A target near its boundary may be technically feasible but require highly concentrated weights; inspect the actual weights and the covariate support.

Key takeaway: Entropy balancing finds the relative-entropy-closest normalized control weights that satisfy chosen covariate-moment constraints. Check feasibility and achieved balance, distinguish interior tilts from boundary solutions, and treat causal interpretation as dependent on assumptions beyond the weighting calculation.

Check Your Understanding

Use the definitions and conditions in this tutorial to answer the following questions.

What does the entropy objective penalize, and what three constraints define the normalized ATT control weights?
Why does a target moment outside the convex hull of the control covariate vectors make exact entropy balancing impossible?
For what kind of feasible target does the finite exponential-tilting representation apply? Why does the boundary example not have a finite multiplier?
If the constraints balance the means of two covariates, which further balance claims are justified without additional constraints or checks?
What assumptions, beyond empirical moment balance, are needed to interpret the weighted control outcome mean as the treated group’s counterfactual untreated mean?