Comparing Schools After Student Adjustment

Why Compare Schools After Student Adjustment?

A school’s average exam score reflects more than instruction. The students enrolled at one school may, on average, have different prior achievement or other background characteristics from students at another. A regression comparison can adjust for measured student characteristics, making the school contrast a comparison at a common set of those characteristics.

This adjustment has a specific meaning: it describes how fitted exam scores differ between schools for students with the same values of the included predictors. It does not automatically estimate what would happen if the same students were assigned to different schools. Also, students in the same school may have more similar outcomes than students in different schools. Treating every student’s outcome as independent can therefore make uncertainty appear smaller than it is.

A Regression Model for Adjusted School Differences

Suppose student \(i\) attends school \(j\). Let \(Y_{ij}\) be the student’s exam score, measured in points, and let \(X_{1ij},\ldots,X_{pij}\) be measured student background predictors, such as prior achievement. With school A as the reference school, one additive linear model is:

Y_{ij}=\beta_0+\sum_{k=2}^{J}\gamma_k I(\text{school }j=k)+\sum_{\ell=1}^{p}\beta_\ell X_{\ell ij}+\varepsilon_{ij}.

Here \(J\) is the number of schools in the comparison, \(I(\text{school }j=k)\) equals 1 for a student at school \(k\) and 0 otherwise, and \(\varepsilon_{ij}\) is the part of the score not represented by the fitted mean. School A is the reference because it has all school indicators equal to zero. The coefficient \(\gamma_k\) is the adjusted difference in fitted exam score between school \(k\) and school A, in score points, when the included predictors are held fixed. For example, \(\gamma_B=-4\) means the fitted score for a student at school B is 4 points lower than for a student with the same included predictor values at school A.

Model assumptions for this comparison: The conditional mean of the score is represented adequately by the specified school indicators and student predictors. The model’s common predictor slopes mean that each predictor has the same fitted relationship with score across schools; if that is not plausible, school-by-predictor interactions may be needed. For cluster-robust uncertainty, schools are treated as independent clusters, while errors may be dependent within a school. The predictors must also have adequate overlap across schools for a common-profile comparison to be informative. These are modeling assumptions, not facts guaranteed by fitting the equation.

A school contrast is often easiest to understand by comparing fitted scores at a stated common profile of background characteristics. The contrast is then the difference between those fitted scores. If the model includes interactions, that difference can depend on the chosen profile; without interactions, it is constant across profiles.

Worked Example: Comparing Two Schools at a Common Profile

Suppose a hypothetical fitted model uses prior mathematics score and a background index, as well as school indicators. It estimates an intercept of 20 points, a prior-score slope of 0.55 points per prior-score point, and a background-index slope of 1.6 points per index unit. Its school coefficients are \(\gamma_B=-4.8\) and \(\gamma_C=2.1\), relative to school A. Consider a student profile with prior score 70 and background index 5, values that lie within the observed range at each school.

For school A, both school indicators are zero. The fitted score is:

\widehat{Y}_A=20+0.55(70)+1.6(5)=20+38.5+8=66.5\text{ points}.

For school B, add its school coefficient:

\widehat{Y}_B=20-4.8+0.55(70)+1.6(5)=61.7\text{ points}.

Thus, at this common student profile, the fitted B-minus-A difference is \(61.7-66.5=-4.8\) points. This is an adjusted descriptive difference under the specified model. The calculation does not show that attending school B causes a student’s score to fall by 4.8 points.

Contrasts and Their Uncertainty

Regression output may report each school coefficient relative to the reference, but a question can instead ask for a difference between two non-reference schools. For schools B and C, the B-minus-C contrast is \(\gamma_B-\gamma_C\). Its uncertainty depends on the uncertainty in both coefficient estimates and on their covariance; the two coefficients must not be treated as independent unless that is justified.

Worked Example: A Cluster-Adjusted Contrast Between Two Schools

In a hypothetical analysis of 24 schools, the fitted coefficients are \(\widehat{\gamma}_B=-4.8\) points and \(\widehat{\gamma}_C=2.1\) points. A covariance estimate based on school-clustered residuals gives standard errors 2.0 and 1.8 points, respectively, and covariance \(1.20\) points squared. Suppose a small-sample procedure reports \(df=23\) for this contrast.

The estimated B-minus-C difference is:

\widehat{\gamma}_B-\widehat{\gamma}_C=-4.8-2.1=-6.9\text{ points}.

Its estimated variance includes the covariance term:

\widehat{\operatorname{Var}}(\widehat{\gamma}_B-\widehat{\gamma}_C) =2.0^2+1.8^2-2(1.20)=4.00+3.24-2.40=4.84.

Therefore, \(SE=\sqrt{4.84}=2.20\) points, and the test statistic for a zero contrast is \(t=-6.9/2.20=-3.14\). With \(df=23\), the two-sided p-value is approximately 0.0046. A 95% interval, using \(t^*=2.069\), is:

-6.9\pm 2.069(2.20)=-6.9\pm4.55=(-11.45,-2.35)\text{ points}.

Under the stated model and uncertainty procedure, the estimated adjusted score at school B is 6.9 points lower than at school C; the interval ranges from 11.45 to 2.35 points lower. The p-value describes how unusual a statistic at least this extreme would be under a zero-contrast null model and the procedure’s assumptions. It is not the probability that the null hypothesis is true.

Why School Clustering Changes Uncertainty

The school indicator and student predictors describe the fitted mean. They do not guarantee that outcomes from students in the same school are independent. Shared teachers, policies, facilities, or local conditions can make students’ unexplained scores move together. Ordinary standard errors that treat all student records as independent can then misstate the uncertainty of estimated contrasts.

One common approach is to calculate a cluster-robust covariance estimate, grouping residual information by school. In matrix notation, a basic form is:

\widehat{V}_{\text{cluster}} =(X'X)^{-1}\left(\sum_{j=1}^{J}X_j'\hat{\varepsilon}_j\hat{\varepsilon}_j'X_j\right)(X'X)^{-1}.

Here \(X_j\) is the model matrix for students in school \(j\), and \(\hat{\varepsilon}_j\) is the vector of fitted residuals for those students. This estimate allows residual dependence within a school while relying on independence across schools. In practice, finite-sample corrections and suitable degrees of freedom matter, particularly when there are few schools. A large number of student records cannot by itself compensate for having very few independent schools.

A simple design-effect calculation illustrates the role of cluster size and within-school correlation. It applies to the variance of a mean under equal cluster sizes and a common within-cluster correlation; it is an illustration, not a substitute for the regression covariance calculation above.

Worked Example: Cluster Size, Correlation, and Effective Sample Size

Suppose 10 hypothetical schools each contribute 2 students, giving \(N=20\) students. Assume each student outcome has variance \(\sigma^2\), and the correlation between two students in the same school is \(\rho=0.2\). The variance of the overall mean is the variance of the sum divided by \(N^2\). There are \(N\) individual variances and, within each of 10 schools, \(2(2-1)\) ordered pairs with covariance \(\rho\sigma^2\). Thus:

\operatorname{Var}(\bar{Y}) =\frac{N\sigma^2+10\{2(2-1)\}\rho\sigma^2}{N^2} =\frac{\sigma^2}{N}\{1+(2-1)\rho\}.

The multiplier \(1+(m-1)\rho\), with cluster size \(m=2\), is the design effect. Here it is \(1+(2-1)(0.2)=1.2\), so the effective sample size for this mean is approximately:

N_{\text{eff}}=\frac{20}{1.2}=16.67.

The effective amount of independent information is closer to 20 students than to 10 schools in this example. If instead \(\rho=0.9\), the design effect is \(1.9\) and \(N_{\text{eff}}=20/1.9\approx10.53\), which is closer to the number of schools. The effect therefore depends on both cluster size and within-school correlation; positive correlation does not by itself make the effective sample size close to the number of schools.

Descriptive Adjustment Is Not a Causal Effect

A regression-adjusted school contrast is a model-based comparison of students with the same recorded predictor values. It can be useful for describing how scores differ after accounting for measured background, but it does not remove differences in unmeasured background or guarantee that the included predictors capture all relevant differences between schools. Students are not necessarily assigned to schools at random, and families may choose schools based on characteristics that also relate to exam performance.

A causal claim would need additional justification, such as a credible design and assumptions that address confounding, along with a clear definition of the intervention being considered. Merely including several student predictors in a regression does not establish those conditions. In a report, label the result as an adjusted association or difference unless a causal interpretation is supported by the study design and assumptions.

Common Mistake and Practical Reporting

Wrong reasoning: “There are many students, so the standard error must be small; each student contributes an independent observation.” Correction: students within a school can share residual variation. Report uncertainty that accounts for the school clusters, and consider both the number of schools and within-school dependence.
Wrong reasoning: “The adjusted coefficient is the causal effect of attending that school.” Correction: it is a conditional model comparison unless causal assumptions and design support that interpretation.
Wrong reasoning: “A negative school coefficient means every student there scores lower.” Correction: it describes a difference in fitted conditional means, not a claim about every individual student.
Wrong reasoning: “The coefficient for B compared with A also gives the B-versus-C difference.” Correction: calculate the contrast \(\gamma_B-\gamma_C\), including its covariance when finding uncertainty.

Key takeaway: State the outcome, student predictors, school contrast, and uncertainty method. Interpret the coefficient as an adjusted difference in score points for comparable measured student backgrounds, and avoid causal language unless the design and assumptions justify it.

Check Your Understanding

Use the model and distinctions in this tutorial to answer the following.

In a model with school A as the reference, what does a school B coefficient of \(-3.2\) points mean when student predictors are held fixed?
Why does the variance of a contrast \(\widehat{\gamma}_B-\widehat{\gamma}_C\) include a covariance term?
For equal clusters of size \(m=4\) and within-cluster correlation \(\rho=0.25\), calculate the design effect and state what it illustrates.
Why can treating students in the same school as independent lead to misleading uncertainty?
What additional issue prevents an adjusted school difference, by itself, from being interpreted as a causal effect?