Limited Predictor Overlap and Fixed Comparisons

Why Similar Profiles Matter

An adjusted comparison asks how two groups differ when their control-predictor values are held fixed. That question is clearest when the data include observations from both groups with similar control values. If one group is observed mainly at low values and the other mainly at high values, the comparison may still be calculated, but it can rely heavily on a model’s predictions for profiles that are poorly represented—or absent—in one group.

This is a question of support: do the observed profiles provide a sound basis for comparing the groups at the chosen values of the controls? A regression program may return a coefficient even when overlap is limited. The existence of a number does not, by itself, show that the data strongly support the comparison it represents.

Fixed Comparisons and Overlap

Let \(G\) indicate group membership, with \(G=1\) for one group and \(G=0\) for the other, and let \(X\) be a control predictor. Write \(m_g(x)\) for the model’s predicted mean outcome for group \(g\) at control value \(x\). A fixed comparison at \(x_0\) is the difference between the two predicted means at that same value:

\Delta(x_0)=m_1(x_0)-m_0(x_0)

The comparison fixes \(X\) at \(x_0\), so it does not contrast one group at its typical control value with the other group at a different control value. In an additive regression model, this kind of comparison is represented by comparing fitted outcomes after setting the control predictor to the same value for both groups.

Definition: The overlap between groups is the extent to which their observed control-predictor values—or, with several controls, their observed combinations of control values—are similar or shared. A comparison has direct support at a profile when observations from both groups occur at, or near, that profile. Limited overlap means that many relevant profiles are represented in one group but not the other.

With several controls, a profile is a combination such as a baseline score of 70 and an age of 40. Finding overlap on each predictor considered separately does not guarantee overlap in their combinations. For example, two groups might both include people aged 40 and people with baseline scores of 70, but few or none with both characteristics together. The comparison is made at a joint profile, so joint support matters.

A Model Assumption Does Not Create Missing Profiles

A simple way to describe an adjusted comparison is with a model for the mean outcome given group and control value. For example, an additive linear model is

E(Y\mid G,X)=\beta_0+\beta_GG+\beta_XX

Here \(Y\) is the outcome, \(G\) is the group indicator, and \(X\) is the control predictor. Under this model, comparing the predicted means at the same \(X=x_0\) gives \(\beta_G\). This equation is an assumption about the conditional mean: it says that the specified linear form adequately describes the mean outcome over the profiles being compared. Regression estimation also relies on the observations and design providing enough information to distinguish the model terms; for usual model-based uncertainty calculations, additional error assumptions are needed.

The key point here is narrower: even if the model form is accepted, it does not make an unobserved group profile directly observed. With limited overlap, the model can produce a fixed comparison by extending a fitted relationship into regions where one group has little or no data. The result then depends more on the assumed form of the relationship. Overlap is not a guarantee that a model is correct, and limited overlap does not make calculation impossible; it changes how directly the data support the comparison.

Worked Example: A Comparison at a Shared Baseline Score

Suppose a hypothetical program evaluation compares two programs, \(A\) and \(B\), using baseline score \(X\) as a control. At \(X=50\), observations from both programs are available, and the estimated mean outcome is 70 points for program \(A\) and 74 points for program \(B\). What is the fixed comparison at this score, and how well is it supported by the profiles described?

The comparison holds baseline score fixed at 50:

\Delta(50)=m_B(50)-m_A(50)=74-70=4\text{ points}

The fitted mean outcome is 4 points higher for program \(B\) than for program \(A\) at baseline score 50. Because both programs have observations at this same score, the profiles directly support a comparison there. This does not establish that the programs caused different outcomes; it describes the model-based group contrast at the specified score.

Interpolation and Extrapolation in Profile Comparisons

A simple profile calculation shows why overlap matters. Suppose a group’s fitted mean outcome is represented by a straight line between two observed predictor values, \(x_L\) and \(x_H\), where \(x_L<x_H\). To estimate the mean at \(x_0\), write it as a weighted combination of the fitted endpoint values. The weights are chosen to sum to one and to reproduce \(x_0\) when applied to the endpoint predictor values:

w_L=\frac{x_H-x_0}{x_H-x_L},\qquad w_H=\frac{x_0-x_L}{x_H-x_L}

Indeed, \(w_L+w_H=1\), and

w_Lx_L+w_Hx_H =\frac{(x_H-x_0)x_L+(x_0-x_L)x_H}{x_H-x_L} =x_0

So the corresponding fitted value at \(x_0\) is \(w_Lm(x_L)+w_Hm(x_H)\). If \(x_0\) lies between \(x_L\) and \(x_H\), both weights are between 0 and 1: this is interpolation. If \(x_0\) lies outside that interval, one weight is negative: this is extrapolation beyond the endpoint predictor values. A negative weight does not always mean the fitted value lies outside the endpoint fitted values. That is true when the endpoint fitted values differ, but if they are equal the weighted result remains equal to them.

Worked Example: A Comparison Requiring Extrapolation

In a hypothetical comparison, program \(A\) has observed profiles at baseline scores 20 and 30, with fitted mean outcomes of 50 and 60 points. Program \(B\) has profiles at scores 50 and 60, with fitted mean outcomes of 80 and 90 points. Use straight-line extensions of these profiles to compare the programs at score 40.

For program \(A\), the endpoints are \(x_L=20\) and \(x_H=30\), and the target is \(x_0=40\). Its weights are

w_L=\frac{30-40}{30-20}=-1,\qquad w_H=\frac{40-20}{30-20}=2

Therefore the predicted mean for \(A\) at 40 is

m_A(40)=(-1)(50)+(2)(60)=-50+120=70\text{ points}

For program \(B\), the endpoints are 50 and 60, so

w_L=\frac{60-40}{60-50}=2,\qquad w_H=\frac{40-50}{60-50}=-1

Its predicted mean at 40 is

m_B(40)=(2)(80)+(-1)(90)=160-90=70\text{ points}

The fixed comparison at score 40 is \(70-70=0\) points. But neither group has observed profiles near 40: the target lies above \(A\)’s observed range and below \(B\)’s. Both predictions are extrapolations, so the zero-point contrast is not a direct comparison of groups observed at that profile. It depends on extending the straight-line profiles. In this example the endpoint fitted values differ within each group, so each extrapolated fitted value lies beyond that group’s endpoint fitted-value range.

To see why the last statement needs that qualification, if the endpoint fitted values were both 5, the same weights \(-1\) and 2 would give \((-1)(5)+(2)(5)=5\). The calculation would still extrapolate beyond the endpoint predictor values, but the fitted value would equal both endpoint fitted values.

Worked Example: A Comparison Supported by Nearby Profiles

Now suppose a hypothetical evaluation has fitted mean outcomes for program \(A\) of 68 points at score 40 and 78 points at score 60. Program \(B\) has fitted means of 75 points at score 45 and 81 points at score 55. Estimate each program’s mean at score 50 using a straight line between its two listed profiles, then compare them.

For program \(A\), score 50 is halfway between 40 and 60, so the weights are both \(0.5\):

m_A(50)=0.5(68)+0.5(78)=34+39=73\text{ points}

For program \(B\), score 50 is halfway between 45 and 55, giving

m_B(50)=0.5(75)+0.5(81)=37.5+40.5=78\text{ points}

The fixed comparison is \(78-73=5\) points in favor of program \(B\) at score 50. Both predictions use interpolation between nearby observed profiles, rather than extending beyond the observed score range. This provides more direct profile support than in the previous example, though the straight-line interpolation remains a modeling choice rather than a guarantee that the mean relationship is truly linear.

How to Assess Support for a Fixed Comparison

Before interpreting an adjusted comparison, identify the exact profile at which it is made and check whether each group is represented near that profile. A useful review is:

State the fixed profile.
Name the control value—or values—at which the group predictions are compared. A comparison is not fully described by saying only “adjusted.”

Check both groups’ observed profiles.
Ask whether each group has observations at or near the target. With multiple controls, inspect combinations of values, not only each control separately.

Distinguish interpolation from extrapolation.
Determine whether the target lies within the observed range for each group. A prediction outside a group’s range relies on extending the modeled relationship for that group.

Qualify the conclusion.
Report the contrast in context and note when support is limited. A precise numerical output does not remove dependence on the model form.

These checks do not replace evaluating the regression model or the study design. They address a specific question: how much does the comparison draw on profiles actually represented in both groups? A model can use all observations to estimate its coefficients, yet a particular fixed comparison can still be weakly supported if the groups occupy different parts of the predictor space.

Common Mistake: Treating Adjustment as a Substitute for Overlap

A common mistake is to say, “The regression adjusts for baseline score, so the comparison at any score is equally grounded in the data.” The correction is that adjustment defines a model-based comparison; it does not create observations at profiles that were not observed. When both groups are represented near the fixed profile, the comparison has direct support there. When one or both groups are far from it, the fitted comparison can depend substantially on extrapolation and on the assumed mean relationship.

Limited overlap does not automatically invalidate a regression calculation, and good overlap does not prove that the model is correct. It is a reason to interpret the particular adjusted comparison with care, especially when its target profile lies beyond the observed values for one group. The comparison remains an association described by the model, not by itself evidence of a causal effect.

Key takeaway: An adjusted comparison fixes the control profile for both groups. Its data support is strongest when both groups have observations at or near that profile; limited overlap can make the same numerical comparison rely on extrapolation and stronger modeling assumptions.

Check Your Understanding

Use the ideas of fixed profiles, overlap, and interpolation to answer these questions.

In your own words, what does a fixed comparison at \(X=x_0\) hold constant?
Group A is observed at control values 10 and 20, while group B is observed at 18 and 28. Is a comparison at 19 interpolation or extrapolation for each group?
For endpoints \(x_L=2\) and \(x_H=6\), calculate the interpolation weights for \(x_0=5\). Do the weights sum to one?
Why can a regression produce an adjusted comparison even when the groups have little profile overlap?
If an extrapolation uses a negative weight, what does that establish about the predictor value, and what does it not necessarily establish about the fitted outcome value?