An Observational Evaluation of a Transit Policy

Why Evaluate a Policy With Adjusted Regression?

Suppose some transit zones have introduced a new fare policy and others have not. Commute times may differ between the zones, but the zones may also differ in age composition, vehicle availability, transit access before the policy, or employment density. A raw comparison can therefore mix the policy association with differences that were already present.

Regression can estimate the association between policy exposure and commute time while holding measured baseline characteristics fixed. That adjusted estimate is still observational: adjustment addresses only the covariates included and modeled, and does not by itself show that the policy caused a change.

Set the Model and the Adjustment Plan

Definition: Let \(Y_i\) be the commute time, in minutes, for commuter \(i\), and let \(A_i=1\) indicate that the commuter's transit zone has the policy in place and \(A_i=0\) indicate that it does not. An adjusted policy association is the estimated difference in mean commute time between policy and comparison commuters with the specified adjustment variables held fixed in the regression model.

For this hypothetical evaluation, the adjustment set is chosen before examining the policy coefficient: age, vehicle availability, transit access measured before the policy, and employment density. The model is

Y_i=\beta_0+\beta_1A_i+\beta_2(\text{age}_i)+\beta_3(\text{vehicle}_i)+\beta_4(\text{baseline access}_i)+\beta_5(\text{employment density}_i)+\epsilon_i.

Here vehicle availability is coded 1 if a commuter has access to a vehicle and 0 otherwise. The coefficient \(\beta_1\) is the model's adjusted policy association, measured in minutes. For example, \(\beta_1=-4\) means that the model estimates a four-minute shorter mean commute for policy-exposed commuters than for comparison commuters with the same modeled covariate values. This is an association, not automatically a causal effect.

The regression estimate and its usual standard error rely on assumptions. The conditional mean of commute time must be adequately represented by the stated linear form; in particular, any important nonlinear relationships or interactions must be modeled rather than ignored. For the usual \(t\)-based confidence interval in the example below, errors are assumed independent, have a common variance, and are approximately normal conditional on the predictors. The data must also contain policy and comparison commuters with sufficiently similar covariates to support the adjusted comparison. Because policy status is not randomized, causal interpretation would additionally require strong assumptions about confounding and policy assignment, discussed below.

Worked Example: Estimate and Interpret the Adjusted Association

Imagine a hypothetical observational evaluation with 120 commuters, one sampled from each of 120 distinct transit zones. Sixty commuters are in zones with the policy and 60 in comparison zones. Commute time is recorded in minutes, and the four adjustment variables are measured before policy implementation. A linear regression using the model above gives a policy coefficient of \(-4.20\) minutes, with \(SE=1.55\) minutes. The residual degrees of freedom are \(df=114\).

The point estimate says that, after adjustment for age, vehicle availability, baseline transit access, and employment density, the policy group is estimated to have a mean commute time 4.20 minutes shorter than the comparison group. The estimate describes the fitted conditional association over covariate combinations represented in the data.

A 95% confidence interval is the estimate plus or minus the \(t\) critical value times its standard error. For \(df=114\), use \(t^*=1.981\):

-4.20\pm 1.981(1.55) =-4.20\pm 3.071 =(-7.27,-1.13)\text{ minutes}.

The multiplication is \(1.981(1.55)=3.07055\), which rounds to 3.071. The interval is consistent with an adjusted association ranging from 7.27 minutes shorter to 1.13 minutes shorter. Under the regression assumptions, the usual 95% confidence procedure would cover the model coefficient in 95% of repeated samples. It does not mean there is a 95% probability that this fixed coefficient lies in this particular interval. The corresponding test statistic for \(H_0:\beta_1=0\) is \(-4.20/1.55=-2.710\); a two-sided \(t\) test with 114 degrees of freedom gives approximately \(p=0.008\). This is evidence against a zero adjusted association under the model, not evidence that the policy caused the difference.

Why Overlap Matters

Adjustment is informative when policy and comparison commuters can be compared at similar values of the adjustment variables. If, for example, all policy commuters live in dense areas with excellent baseline access while all comparison commuters live in sparsely populated areas with poor access, the model must rely heavily on extrapolation to estimate a policy difference at otherwise similar covariate values. A fitted coefficient can still be produced, but the data may provide little direct support for that comparison.

Overlap can be examined using plots, ranges, tables of covariate summaries, and counts within covariate groups. Standardized differences give one compact summary for numerical and binary variables. For a numerical variable, a common descriptive measure is the difference between group means divided by a pooled standard deviation. For a binary variable, the pooled standard deviation can be calculated from the pooled proportion. These summaries help identify imbalances; they do not prove that all relevant confounding has been removed, and no single cutoff guarantees adequate overlap.

Worked Example: Summarize Overlap in Baseline Characteristics

In the same hypothetical data, policy and comparison groups each contain 60 commuters. Their age ranges are 20–64 and 19–66 years, respectively. Their baseline transit-access scores both range from 2 to 10. The policy group has a mean age of 38 years and the comparison group 40 years; the pooled standard deviation is 10 years. The standardized age difference is

\frac{38-40}{10}=-0.20.

Thus, the policy group is two years younger on average, a difference of 0.20 pooled standard deviations. For baseline access, suppose the group means are 6.2 and 5.8, with pooled standard deviation 2.0. Its standardized difference is \((6.2-5.8)/2.0=0.20\). Both variables have overlapping observed ranges, though the means are not identical.

Suppose 24 of the 60 policy commuters and 33 of the 60 comparison commuters have vehicle access. The corresponding proportions are \(24/60=0.40\) and \(33/60=0.55\). The pooled proportion is \((24+33)/120=0.475\), so the pooled standard deviation for this binary variable is

\sqrt{0.475(1-0.475)}=0.4994.

The standardized difference is \((0.40-0.55)/0.4994=-0.300\). This shows a noticeable difference in vehicle availability that the model should adjust for. The ranges and summaries suggest some overlap, but should be supplemented by checking whether important combinations of characteristics are represented in both groups. For instance, age and vehicle availability considered separately may overlap even if older commuters without vehicles occur almost exclusively in one group.

Check the Adjustment Set Before Comparing Estimates

The prespecified adjustment set should reflect the evaluation question and knowledge of how policy exposure and commute time are related. Baseline characteristics that could help explain both policy placement and commute time are plausible adjustment variables. The timing matters: a variable measured after implementation may itself have changed because of the policy. Adjusting for such a variable can remove part of the association one intends to describe.

A short derivation helps clarify what the policy coefficient measures. Let \(C\) denote the intercept and the four adjustment variables, and let \(\tilde A_i\) and \(\tilde Y_i\) be the residuals from regressing \(A_i\) and \(Y_i\), separately, on \(C\). In ordinary least squares, after the linear contribution of \(C\) has been removed, the remaining coefficient on policy minimizes the residual sum of squares

\sum_i(\tilde Y_i-b\tilde A_i)^2.

Differentiating with respect to \(b\) and setting the derivative equal to zero gives

-2\sum_i\tilde A_i(\tilde Y_i-b\tilde A_i)=0 \quad\Longrightarrow\quad \hat\beta_1=\frac{\sum_i\tilde A_i\tilde Y_i}{\sum_i\tilde A_i^2},

provided \(\sum_i\tilde A_i^2>0\). In words, the fitted policy coefficient relates the part of policy status not linearly explained by the adjustment variables to the corresponding adjusted part of commute time. If policy status is almost completely predicted by those variables, little residual variation in policy remains; the estimate then depends on limited overlap and may be imprecise or sensitive to model choices.

Worked Example: Compare Prespecified and Alternative Adjustments

Suppose the prespecified four-variable model gives the estimate \(-4.20\) minutes and standard error 1.55 minutes. As a sensitivity check, a model omitting baseline transit access gives \(-5.10\) minutes, with standard error 1.42 minutes. The change in the estimate is

-5.10-(-4.20)=-0.90\text{ minutes}.

The estimate is 0.90 minutes more negative when baseline access is omitted. This difference indicates that the policy estimate depends to some extent on whether baseline access is included. It does not establish which estimate is correct: the prespecified model is preferred for the stated adjustment plan, while the alternative is useful for showing sensitivity to that choice.

Now suppose a separate model adds the number of transit services used after policy implementation and produces an estimate of \(-1.80\) minutes. The change from the prespecified estimate is \(-1.80-(-4.20)=2.40\) minutes. Post-policy service use may have been influenced by the policy; if so, adjusting for it changes the comparison and can block part of a pathway through which the policy might relate to commute time. This model is therefore not simply a better-adjusted version of the original. It answers a different conditional question and should not replace the prespecified model without a clear reason.

These alternative estimates are illustrative fitted results, not calculations recoverable from the group summaries above. In a real analysis, each would need to be computed from the data, and the covariates, estimates, standard errors, and sample used should be reported transparently.

When Is a Causal Interpretation Defensible?

The model estimates an adjusted association. To interpret it as the effect of the policy would require additional assumptions that an observational comparison cannot guarantee. In particular, after adjustment, policy and comparison commuters would need to be comparable with respect to all causes of policy exposure and commute time that matter for the question. Important unmeasured differences—such as service changes introduced at the same time, local travel patterns, or reasons zones were selected for implementation—could remain.

A causal interpretation also requires adequate overlap: for relevant combinations of baseline covariates, both policy and comparison exposure must be possible and represented. The policy itself must be sufficiently well defined that exposure has a consistent meaning, and one zone's policy status should not substantially alter another zone's commute outcomes in a way ignored by the analysis. The timing of measurements must distinguish baseline characteristics from consequences of implementation. These assumptions require substantive knowledge of the policy and data collection, not just a satisfactory regression output.

The result should therefore be worded in proportion to the design. “The adjusted model estimated a 4.20-minute shorter mean commute in policy zones” describes the result. “The policy reduced commutes by 4.20 minutes” asserts causation and would require stronger support than the adjusted association alone provides.

Common Pitfalls

Treating adjustment as randomization. Including measured covariates can reduce certain measured differences, but it does not eliminate bias from unmeasured confounding or misspecified relationships.
Assuming overlap because a regression returned a coefficient. A numerical estimate can be generated even when policy and comparison observations occupy very different covariate regions. Inspect ranges, distributions, and combinations of covariates.
Choosing covariates by whichever model gives the preferred result. State the primary adjustment set in advance, then label other specifications as sensitivity analyses and explain their rationale.
Adjusting automatically for every available variable. A post-policy variable may be affected by the policy; controlling for it can change the question rather than improve the original comparison.
Calling the coefficient a causal effect without qualification. Report an adjusted association unless the design and assumptions justify causal language.

Key takeaway: Report the adjusted policy coefficient in commute-time units, state the prespecified adjustment set, examine whether policy and comparison groups overlap, and show how reasonable alternative adjustments affect the estimate. Regression adjustment improves the comparison only to the extent that its data, model, and assumptions support it.

Check Your Understanding

Use the model and hypothetical evaluation described above to answer these questions.

In context, what does a policy coefficient of \(-4.20\) minutes mean, and what does it not establish?
Why is overlapping observed age range not, by itself, enough to establish adequate overlap across all adjustment variables?
For the vehicle-access proportions 0.40 and 0.55, what does the standardized difference of \(-0.300\) summarize?
Why might adding a post-policy measure of transit use change the question addressed by the regression?
Name two additional assumptions needed before an adjusted observational association could be given a causal interpretation.