Judge Assignment As A Natural Experiment

Why Judge Assignment Needs an Audit

A judge-assignment design can look compelling: cases appear to be distributed across judges, and judges differ in how often they impose a particular treatment, such as pretrial detention or a custodial sentence. But a judge’s identity is useful as a source of quasi-random variation only if the process sending cases to judges is credible. If attorneys, court staff, or defendants can direct particular cases toward particular judges, differences in outcomes may reflect case selection rather than judge assignment.

The relevant question is not simply whether two judges have similar average caseloads. It is whether the cases assigned to each judge are comparable given the court’s actual assignment rules. Assignment may depend on courthouse, courtroom, case type, filing date, shift, or judge availability. Those features determine the comparisons that need to be audited.

Definition: Let $Z_i$ denote the judge assigned to case $i$, $D_i$ a treatment received by the defendant, $Y_i$ an outcome, and $X_i$ pretreatment case characteristics. Let $G_i$ denote the assignment block, such as courthouse, case category, and docket session. Judge assignment is plausibly quasi-random within blocks if, among eligible cases in a block, assignment is determined by a procedure unrelated to the cases’ potential outcomes and treatment responses, apart from chance. This is a design claim to investigate, not a conclusion established by observing different outcomes across judges.

The audit starts by defining the eligible cases and the point at which judge assignment occurs. If a case enters the analysis only after a judge has accepted it, for example, the assignment records should be checked for cases rejected, transferred, or omitted. The analysis should also establish whether judge identity means the initially assigned judge or the judge who ultimately heard the case. Replacing initial assignments with later assignments can introduce selection if transfers occur in response to case characteristics.

A useful working assumption is conditional exchangeability of assignment: within each relevant block $G_i$, the assigned judge is independent of the potential outcomes and treatment responses for the eligible cases. A second requirement is assignment overlap: within the blocks used for comparison, eligible cases have a genuine chance of being assigned to the judges being compared. If one judge handles every case of a certain type or works only in a particular period, there may be no within-block comparison for that group. Finally, the assignment process must not allow consequential sorting by case difficulty, expected outcome, or other unmeasured features.

Start With the Assignment Procedure

Administrative records and written rules are often more informative than a regression table. Determine who assigns cases, whether assignment follows a rotation or another mechanical rule, when the rule is applied, and which exceptions are permitted. Then compare the documented procedure with what actually happened: count cases assigned, transferred, reassigned, or removed from the queue, and identify why each departure occurred.

A rotation is not automatically random. For instance, a nominal rotation can be bypassed when a judge is unavailable, when a case is unusually complex, or when counsel requests a particular judge. Some departures may be innocuous, but they matter if the reason for departure is associated with potential outcomes. A judge’s unavailability can also create systematic assignment patterns by time of day or case type. The audit should therefore preserve the original assignment record and examine deviations rather than treating the final judge as if selected by an uncomplicated lottery.

Worked Example: Auditing a Rotation Rule

Hypothetical data. A court’s written procedure assigns eligible cases to the next judge in a rotating list. An audit of 500 cases finds that 486 went to the next judge and 14 did not. Of those 14 exceptions, records identify 11 attorney requests and 3 instances in which the next judge was unavailable.

The observed exception rate is

\frac{14}{500}=0.028=2.8\%.

The high rate of compliance is encouraging but does not establish quasi-random assignment. Attorney-requested deviations are a direct reason to investigate sorting: requests may be related to counsel’s knowledge of the judge, case characteristics, or anticipated outcomes. The three unavailability cases also need to be examined. If unavailability is unrelated to the case after conditioning on docket session, those cases may be manageable through appropriate blocks or exclusions; if not, they may create selection.

The evidence supports a qualified conclusion: most recorded cases followed the rotation, but the 14 exceptions require attention before assignment is treated as quasi-random. In particular, the analyst should report whether results change when request cases are excluded, and should verify that the excluded cases are not a selectively different subset. A small exception percentage alone cannot establish that the remaining assignments satisfy the design assumptions.

Check Case-Mix Balance

A second audit compares pretreatment characteristics across judges. Relevant variables can include age, prior convictions, charge severity, representation, filing time, and other features measured before assignment. The selection of characteristics should follow substantive knowledge of what predicts treatment or outcome, not only which variables happen to be available or statistically significant.

For a continuous characteristic $X$, a common summary is the standardized mean difference (SMD), the difference between judge-specific means divided by a pooled standard deviation. For a binary characteristic, the same idea can be expressed using the difference in proportions divided by a pooled standard deviation for a binary variable. These summaries describe the size of a difference in standard-deviation units; they are not tests that prove or disprove random assignment.

Balance summary: For two judge groups with sample sizes $n_1$ and $n_0$, sample means $\bar{x}_1$ and $\bar{x}_0$, and sample standard deviations $s_1$ and $s_0$, the continuous-variable standardized mean difference is $$ \mathrm{SMD}= \frac{\bar{x}_1-\bar{x}_0} {\sqrt{\left[(n_1-1)s_1^2+(n_0-1)s_0^2\right]/(n_1+n_0-2)}}. $$ For a binary variable with proportions $\hat{p}_1$ and $\hat{p}_0$, a frequently used denominator is $$ \sqrt{\frac{\hat{p}_1(1-\hat{p}_1)+\hat{p}_0(1-\hat{p}_0)}{2}}. $$ The sign gives the direction of the difference; the absolute value describes its magnitude.

No single threshold makes a design valid. A sizeable SMD is a warning that the observed case mix differs; small SMDs are reassuring only for the measured characteristics and comparisons examined. Check distributions, not just means: similar averages can hide different tails or nonlinear patterns. For multiple judges, examine all relevant judge comparisons or use an overall assessment that reflects the assignment mechanism. A collection of separate significance tests is a poor substitute for this inspection: results depend on sample size and multiple testing, and a nonsignificant test is not evidence that a difference is negligible.

Worked Example: Computing Case-Mix Differences

Hypothetical data. Suppose 120 cases are assigned to a relatively lenient judge and 120 to a relatively strict judge. Mean age is 34 years with standard deviation 10 years in the first group, versus 36 years with standard deviation 9 years in the second. The proportion with a prior conviction is 0.30 versus 0.42. For age, the pooled standard deviation is

s_{\mathrm{pool}}= \sqrt{\frac{119(10^2)+119(9^2)}{238}} =\sqrt{\frac{119(181)}{238}} =\sqrt{90.5} \approx 9.513\text{ years}.

The age SMD, comparing the lenient-judge group with the strict-judge group, is

\mathrm{SMD}_{\mathrm{age}} =\frac{34-36}{9.513} \approx -0.210.

For prior conviction, the binary-variable denominator is

\sqrt{\frac{0.30(0.70)+0.42(0.58)}{2}} =\sqrt{0.2268} \approx 0.4762, \qquad \mathrm{SMD}_{\mathrm{prior}} =\frac{0.30-0.42}{0.4762} \approx -0.252.

Both observed differences favor fewer older cases and fewer cases with prior convictions in the lenient-judge group. The magnitudes deserve investigation, especially alongside other case characteristics and the assignment records. They do not, by themselves, identify why the groups differ or prove that judge assignment was nonrandom. The analyst should examine whether age and prior convictions differ within the actual assignment blocks, and whether the same pattern appears for other pretreatment predictors.

Caseloads, Timing, and Sorting

Caseload is part of the assignment story, not merely a descriptive statistic. If cases are directed to the judge with the shortest queue, assignment probability may vary with time, staffing, case volume, and judge availability. Those variables can also be related to case composition or outcomes. Equal total caseloads at the end of a year do not rule out such patterns: two judges can each receive 100 cases while receiving them under different conditions.

Sorting can be deliberate or indirect. Attorneys might file or request a hearing at a time when a preferred judge is more likely to be sitting. Court staff might route complex cases to a specialist. A clerk might skip an unavailable judge in the rotation, causing cases from a particular shift to concentrate with another judge. Useful diagnostics include assignment shares by session, day, case category, attorney, and judge availability; queue length at assignment; frequencies and reasons for transfers; and whether characteristics measured before assignment predict judge identity after accounting for the documented blocks.

Worked Example: Why Equal Caseloads Can Mislead

Hypothetical data. Two judges each receive 100 cases. The court has two sessions, morning and afternoon, with 100 cases in each. Judge A receives 80 morning cases and 20 afternoon cases; Judge B receives the other 20 morning cases and 80 afternoon cases. Of the 100 morning cases, 80 are high-severity; of the 100 afternoon cases, 20 are high-severity. Suppose the cases assigned to each judge within each session reflect the session’s case mix: Judge A receives 64 high-severity morning cases and 4 high-severity afternoon cases, while Judge B receives 16 and 16, respectively.

The high-severity proportion is $68/100=0.68$ for Judge A and $32/100=0.32$ for Judge B. The pooled binary-variable denominator is

\sqrt{\frac{0.68(0.32)+0.32(0.68)}{2}} =\sqrt{0.2176} \approx 0.4665.

Thus the pooled SMD for high severity is

\mathrm{SMD} =\frac{0.68-0.32}{0.4665} \approx 0.772.

The equal totals of 100 cases per judge conceal a large difference in overall case mix, arising here because the judges receive very different shares of morning and afternoon cases. The relevant next check is within-session assignment: the example’s case proportions match the session-specific mix, so a comparison that conditions on session could remove this particular imbalance if both judges genuinely have assignment overlap in each session. If one judge is never available in a session, there is no within-session comparison there. The analyst should not treat equal annual caseloads as evidence of random assignment, nor treat a pooled imbalance as proof of sorting without inspecting the assignment blocks.

What the Evidence Can Establish

A persuasive assessment combines the procedure audit, balance checks, and evidence about caseload and sorting. The assignment rule should be described precisely; exceptions should be counted and explained; and comparisons should respect the blocks within which cases were actually assigned. Balance diagnostics should cover important pretreatment case characteristics, with both magnitudes and distributions reported. Where records permit, analyses can be repeated after excluding clearly nonrandom exceptions or within narrower assignment blocks, while making clear how those choices change the target population.

These checks are diagnostic rather than definitive. Balance on recorded variables cannot rule out imbalance in unrecorded characteristics, and an apparently mechanical rule may still be selectively bypassed. Conversely, a chance imbalance in a finite sample does not automatically invalidate a genuinely random process. The assignment mechanism and audit trail therefore matter alongside the numerical summaries.

Judge assignment as a natural experiment is also not a complete causal argument for every use of judge identity. If judge identity is used as an instrumental variable for a treatment, credible assignment is one part of the justification. Relevance and the exclusion restriction are separate requirements: in particular, assignment to a judge must affect the outcome through the treatment of interest in the way the proposed analysis requires. An audit of case assignment does not establish those other conditions.

Approach	What it contributes	Key limitation
Assignment-procedure audit	Shows how cases were routed and why exceptions occurred.	Written rules may not match actual practice or reveal informal sorting.
Pretreatment balance diagnostics	Reveal observed case-mix differences across judges or blocks.	Cannot establish balance on unmeasured characteristics.
Adjustment for observed covariates	Can account for measured differences under an appropriate model.	Does not repair unmeasured sorting or a fundamentally nonrandom process.
Lottery-based assignment	Provides a direct randomization mechanism when implemented and documented.	Still requires attention to compliance, exclusions, and the target population.
Judge-based instrumental-variable analysis	Can use judge-assignment variation to study treatment effects when its assumptions hold.	Quasi-random assignment alone does not establish all instrumental-variable assumptions.

Common Pitfalls

“The caseloads are equal, so assignment was random.” Equal totals say nothing about which types of cases arrived under which conditions. Examine assignment by block, case characteristics, and availability.
“The balance tests are nonsignificant, so the judges received identical cases.” Failure to detect a difference is not proof of equivalence. Report differences and uncertainty, and assess their substantive size.
“A mechanical rotation guarantees random assignment.” Exceptions, transfers, and strategic timing can defeat the intended procedure. Verify how the rule operated for the cases analyzed.
“Good balance proves the instrument is valid.” Balance on observed pretreatment variables is supportive evidence about assignment, not proof of unmeasured balance or of the exclusion restriction.

Key takeaway: Judge assignment is credible as a natural experiment only to the extent that the documented process makes assignment plausibly unrelated to case potential outcomes within appropriate blocks. Procedure, balance, caseload, and sorting diagnostics should be interpreted together, and their limits stated explicitly.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A rotation assigns most cases as scheduled, but attorneys can request a judge. What records and comparisons would you examine before treating assignment as quasi-random?
Two judge groups have mean ages of 40 and 42 years, with a pooled standard deviation of 8 years. Compute the age SMD for the first group minus the second and interpret its sign.
Why might equal total caseloads fail to establish comparable case assignment?
A judge comparison has small SMDs for recorded covariates. What important conclusion does this not establish?
If judge identity is proposed as an instrument, what does a credible assignment audit establish, and what does it leave unresolved?

Solutions. (1) Inspect initial and final assignment records, request frequency and reasons, transfers, eligibility rules, and assignment shares within docket sessions and case categories. Check whether requests or deviations are related to pretreatment characteristics; consider a transparent analysis excluding request cases as a sensitivity check. (2) $\mathrm{SMD}=(40-42)/8=-0.25$. The first group’s mean age is one-quarter of a pooled standard deviation lower; the number is a descriptive balance summary, not a test of random assignment. (3) Judges can have equal totals but receive cases at different times, in different case categories, or under different availability and queue conditions. (4) Small observed differences do not rule out imbalance in unmeasured characteristics, nor do they verify the assignment procedure. (5) The audit supports the claim that assignment is plausibly unrelated to case potential outcomes under the stated blocking and eligibility rules. It does not by itself establish relevance, the exclusion restriction, or every other requirement for an instrumental-variable interpretation.