Why Dependence Matters in DiD
Difference-in-differences (DiD) compares outcome changes across treated and comparison groups. A data set may contain many outcome records, but those records do not necessarily represent equally many independent pieces of information. The same person can contribute outcomes in several periods; people in the same clinic can face the same policy and local conditions; and a policy can be assigned to an entire state even when outcomes are recorded for thousands of residents.
These connections matter because statistical inference depends on how much independent variation the data contain. Treating related observations as independent can make the data appear more informative than they are. The DiD estimate itself may still be computed, but its uncertainty cannot be assessed correctly by counting every row as a separate, unrelated observation.
In a quasi-experimental DiD study, “assignment” need not mean randomized assignment. It refers to the level at which the policy or intervention is applied and treatment status differs. If a state adopts a policy for all its residents, the state is the treatment-assignment unit even if researchers measure outcomes for individual residents.
Three Levels to Keep Separate
A useful first step is to distinguish the row in the data from the unit being followed over time and from the unit whose treatment status changes. These can be different entities. In a clinic-policy study, for instance, each row might be one patient-period, the same patient might appear in several periods, and the policy might be assigned at the clinic level.
| Level | Question | Example |
|---|---|---|
| Observation | What does one row record? | One patient’s outcome in one quarter |
| Repeated-measurement unit | Which records belong to the same entity over time? | The patient observed in several quarters |
| Treatment-assignment unit | At what level does treatment status vary? | The clinic that adopts the policy |
| Potential dependence cluster | Which observations may share unmeasured causes or shocks? | Patients treated in the same clinic and quarter |
These levels can coincide, but often do not. If treatment is assigned to clinics, patients within a clinic may share resources, staff, protocols, and local conditions. If outcomes are repeated for each patient, observations can also be related within patient. Dependence may therefore arise both among measurements of the same patient and among different patients at the same clinic.
Common Sources of Dependence
Repeated outcomes on the same unit. A person’s outcome this quarter is often related to that person’s outcome last quarter because stable characteristics, history, and circumstances persist. In panel DiD, multiple records for the same person are not automatically independent just because they have different time labels. The same issue arises when the unit is a school, business, or region measured repeatedly.
Shared treatment and setting. People in a treated clinic or school receive the same policy or implementation environment. Even if their measured outcomes differ, they may experience common staffing, management, eligibility rules, or service capacity. Their outcomes can consequently move together. A common treatment status alone does not prove outcomes are dependent, but shared context makes dependence plausible and important to consider.
Common shocks. Outcomes in the same place and period can be affected by a local event, such as a hospital closure, a regional economic change, or severe weather. A shock can affect treated and comparison units differently depending on where they are located. Common time shocks can also affect many units at once. In DiD, the possibility of shared shocks is especially relevant when exposure and treatment vary at a geographic or institutional level.
Nested or overlapping connections. Patients may be grouped within clinics, clinics within health systems, and systems within states. Two observations might share a clinic, while others share only a state-level environment. In some studies, connections cross rather than nest—for example, a provider may serve patients from several locations. The natural dependence structure therefore needs to be understood from the study design, not inferred from the shape of a spreadsheet alone.
Worked Examples
Worked Example: A Policy Assigned to Clinics
Hypothetical setting. A study compares a new discharge protocol in 8 clinics with usual care in 8 clinics. Each clinic contributes 50 patients in a pre-policy period and 50 in a post-policy period. The treatment policy is introduced clinic-wide.
There are \(16 \times 50 \times 2 = 1{,}600\) patient-period records. But treatment status changes at the clinic level: 8 clinics adopt the protocol and 8 do not. For the question “how many clinics were assigned to each policy condition?”, the arithmetic is \(8+8=16\), not 1,600.
The DiD contrast can be calculated from the group-period means. Suppose the hypothetical mean outcome, in days, is 6.0 before and 5.0 after in adopting clinics, and 6.2 before and 5.8 after in comparison clinics. Then
The estimated difference in changes is a reduction of 0.6 days in the adopting clinics relative to the comparison clinics. This calculation describes the point estimate; it does not establish that 1,600 independent treatment assignments support its precision. Patients in the same clinic may share implementation and local shocks, and the policy itself varies across only 16 clinics. An inference procedure must respect the dependence and assignment structure rather than equating patient-period records with independently assigned clinics.
Worked Example: Repeated Measurements on the Same Person
Hypothetical setting. A DiD data set follows each participant before and after a service change. For one participant, the pre-change outcome is 12 units and the post-change outcome is 9 units, so that person’s observed change is \(9-12=-3\) units. The records are two measurements of one person, not two different people.
The reason this matters can be seen from the variance of a change. Suppose, for illustration, that the variance of each period’s outcome is 25 square units and the covariance between the same person’s pre- and post-change outcomes is 20 square units. Then
If the measurements were incorrectly treated as independent, their covariance would be set to zero, giving \(25+25=50\) square units instead. The arithmetic shows that dependence changes the variance of a within-person change; the direction and size of the change depend on the covariance. These numbers illustrate the calculation, not a general claim that repeated outcomes always have positive covariance.
In an actual DiD study, the target is usually a contrast across groups as well as time, not this one participant’s change. The example isolates one source of dependence: repeated observations from the same person. If policy is assigned at a higher level, such as the clinic, that is a separate source to consider.
Worked Example: Many Residents, Few Policy Units
Hypothetical setting. A state-level policy is adopted in 3 states, while 3 other states remain comparison states. Researchers observe 2,000 residents in each state before and after the policy. There are \(6 \times 2{,}000 \times 2=24{,}000\) resident-period records, but the policy varies across 6 states.
Suppose the state-period means (in hypothetical index points) are:
| Group | Before mean | After mean | Change |
|---|---|---|---|
| Policy states | 50 | 54 | 4 |
| Comparison states | 48 | 50 | 2 |
The estimated contrast is
This is a 2-point difference in changes. It does not mean that 24,000 independent policy assignments were observed. All residents within a state share the state’s policy status, and residents may also share state-level economic or administrative shocks. The treatment-assignment level therefore directs attention to the six states as the units across which policy status varies. A large resident sample can measure state-period outcomes with detail, but it cannot by itself create additional state-level treatment variation.
A Covariance Justification
The basic reason dependence affects uncertainty follows from the variance of a sum. Let \(E_1,\ldots,E_m\) be observation-level errors and let \(w_i\) be fixed weights in a contrast. Then
This identity follows by expanding the variance of the weighted sum: each term contributes its own variance, while each pair contributes twice its weighted covariance. Under independence, all pairwise covariances are zero, leaving only the first sum. When observations share a person, clinic, state, or shock, some covariance terms may be nonzero. A calculation that omits those terms is not justified merely because the observations appear on separate rows.
For DiD, the estimate is built from multiple group-time means or an equivalent regression contrast. Its uncertainty therefore depends on how errors co-move within and across the units contributing to that contrast. The identity does not prescribe a particular estimator of uncertainty; it explains why the dependence structure must be considered before choosing one.
How Assignment Level Guides Inference
The treatment-assignment level identifies where treatment variation comes from. If treatment varies among individuals independently, individual-level treatment variation may be available. If treatment is assigned to schools, clinics, or states, then individuals within those units share treatment status. Treating them as independent treatment assignments would overstate the number of distinct units providing evidence about the treated-versus-comparison contrast.
This is a guide, not a mechanical rule that every dependence problem ends exactly at the assignment boundary. Measurements can be correlated within individuals over time, and shocks can be shared by larger regions or periods. Conversely, assignment to a cluster does not imply that every pair of outcomes in that cluster has identical dependence. The study design and plausible outcome-generating processes both matter.
Identify what one row represents, which entities recur over time, and how observations are grouped by site or location.
Name the level at which the policy or intervention status differs. In a state policy study, for example, this may be the state rather than the resident.
Consider common people, providers, institutions, places, and time-specific shocks that could make outcomes move together.
Do not count every record as an independent contribution by default. Use the assignment and dependence structure to determine what uncertainty calculation is appropriate.
The last step is a design principle, not a recipe for a specific standard-error method. Cluster-robust procedures and adjustments for very small numbers of clusters require additional choices and conditions. Here, the essential point is to identify the dependence before selecting an inferential procedure.
Common Mistake and Practical Consequences
Wrong reasoning: “There are 10,000 patient records, so there are 10,000 independent observations for the policy comparison.” This confuses record count with treatment-assignment count. If the policy was adopted by 10 clinics, the policy status varies across those 10 clinics, not across all 10,000 patients.
Correction: State separately how many outcome records were collected and how many units received or did not receive the intervention. Then identify plausible dependence within repeated-measurement units, treatment clusters, and broader shared environments. The number of records remains useful for describing outcomes, but it does not alone determine inferential precision.
This distinction matters because uncertainty that is understated can produce intervals that are too narrow and tests that reject a null hypothesis too often under that null. Those consequences are not repaired by a large number of individual records if treatment variation comes from only a small number of groups. Likewise, recognizing clustering does not guarantee that a causal interpretation is valid: the identifying assumptions for DiD, including parallel trends and appropriate treatment timing, remain separate requirements.
Check Your Understanding
Use the study structure, rather than the number of rows alone, to answer these questions.
- A policy is introduced in 12 schools, and 40 students per school are measured before and after. What is the treatment-assignment unit, and why are the student-period records not 960 independent policy assignments?
- Give two distinct sources of dependence that could occur in a panel DiD study of patients observed quarterly.
- If the pre- and post-period outcomes for the same person have positive covariance, how does that covariance enter the variance of the person’s change?
- In a state-policy study, explain why collecting outcomes from more residents in the same states does not create more states with different policy status.
- Why is treatment-assignment level an important guide to inference but not a complete description of every possible dependence in the data?