Why Permute Labels in a Two-Way Table?
A two-way table summarizes counts for two categorical variables measured on the same units. For example, a clinic might record both a patient’s appointment type and whether the patient arrived on time. The central question is whether the distribution of one variable differs across categories of the other. A permutation test addresses this by comparing the observed table with tables that could arise if the labels were rearranged in a way justified by the null hypothesis.
The key is to preserve what was observed about each variable separately. When one variable’s labels are shuffled across units, the row totals and column totals remain fixed, while the way the two variables line up can change. A discrepancy statistic measures how far the observed table is from what independence would predict.
The Table and a Discrepancy Statistic
Suppose the first variable has \(r\) categories and the second has \(c\) categories. Let \(O_{ij}\) be the observed count in row \(i\), column \(j\). The row total is \(R_i\), the column total is \(C_j\), and the total number of units is \(n\). Under independence, the expected count in cell \((i,j)\), given the margins, is \(E_{ij}=R_iC_j/n\).
One common choice of test statistic is the Pearson chi-square discrepancy statistic. It is large when the observed counts differ substantially from the counts expected under independence. Here it is used as a statistic for ranking permuted tables; a permutation test does not need to use a chi-square reference distribution.
The statistic is nonnegative, and it is zero if every observed count equals its expected count. For testing general association, larger values indicate greater discrepancy from independence, so the permutation p-value is the proportion of permuted tables with \(T\) at least as large as the observed value. Another discrepancy statistic could be used, but its direction and tail must be specified in advance.
Worked Example: Calculating the Table Discrepancy
Suppose a hypothetical survey records preferred appointment type and arrival status for 50 people. The table is:
| On time | Late | Total | |
|---|---|---|---|
| Morning | 18 | 12 | 30 |
| Afternoon | 7 | 13 | 20 |
| Total | 25 | 25 | 50 |
For the morning/on-time cell, the expected count under independence is \(E_{11}=(30)(25)/50=15\). The other expected counts are 15, 10, and 10, respectively. Thus the statistic is
The morning group’s observed on-time proportion is \(18/30=0.600\), compared with \(7/20=0.350\) in the afternoon group. The statistic summarizes this table-wide departure from independence. By itself, \(T=3.00\) is not a p-value and does not establish whether the discrepancy is unusual under the permutation null.
How the Permutation Test Works
Keep each unit’s row-category label fixed, and randomly reassign the observed column labels among the units. This preserves the number of units in every row and column category. For each rearrangement, form the table and calculate the same statistic \(T\). Repeating the operation creates a permutation distribution representing the discrepancies expected under the null, conditional on the observed margins.
For an exact test, consider every allowed rearrangement and calculate the fraction whose statistic is at least the observed statistic. For a Monte Carlo test, randomly sample \(B\) rearrangements; if \(b\) have statistics at least as large as observed, the commonly used estimate is \((b+1)/(B+1)\). The added one prevents a simulated estimate of zero and treats the observed arrangement as part of the reference set.
For an association question, the null is that the two categorical variables are independent. State which labels are permuted and why those labels are exchangeable under this null.
Use the observed table to find its margins, expected counts, and discrepancy statistic \(T_{\text{obs}}\).
Reassign one variable’s labels across the units using the specified scheme. Each table retains the observed margins.
Find the proportion of permutation statistics at least as large as \(T_{\text{obs}}\), then describe what that p-value says about the null model in context.
Why the Margins and Exchangeability Matter
The margin-preserving scheme has a precise justification when, under the null, the permuted labels are exchangeable across the units being shuffled. In particular, if units are independent and identically distributed and the two variables are independent, then conditional on the observed counts of each column label, every allocation of those labels to the fixed row-labeled units has the same probability. Consequently, uniformly permuting the labels reproduces the conditional null distribution.
Independence of units alone is not enough. Under the null, different units could have different probabilities of receiving the column categories; then some reallocations would be more likely than others. Nor is a generic shuffle suitable when observations are clustered or when a study’s design restricts which labels could be assigned to which units. In those settings, the permutation scheme must reflect the actual exchangeability or assignment structure. Do not use a margin-preserving shuffle merely because the data can be put in a table.
Worked Example: An Exact Test with Fixed Margins
Consider a hypothetical \(2\times2\) table with six units in each row and six units in each column. The observed table has six units in each diagonal cell and none in either off-diagonal cell:
| Column A | Column B | Total | |
|---|---|---|---|
| Row 1 | 6 | 0 | 6 |
| Row 2 | 0 | 6 | 6 |
| Total | 6 | 6 | 12 |
All four expected counts are \((6)(6)/12=3\), so
Let \(X\) be the number of Column A labels assigned to the six Row 1 units. Given the margins, \(X\) can be 0 through 6. For a particular value \(x\), choose \(x\) of the six Column A labels for Row 1, then choose the remaining \(6-x\) Row 1 labels from the six Column B labels. The total number of allocations of six Column A labels among 12 units is \(\binom{12}{6}=924\). Therefore,
For a table with these margins, its four counts are \(x,6-x,6-x,x\). Substitution into the statistic gives \(T=4(x-3)^2/3\). Thus \(T\geq12\) only when \(x=0\) or \(x=6\). There is one allocation for each of those two cases, so the exact upper-tail p-value is
Under the stated exchangeability condition, this small p-value indicates that a table at least this discrepant would be unusual if the variables were independent. It does not give the probability that independence is true.
A Monte Carlo Permutation Example
For larger tables, enumerating every permitted allocation may be impractical. Monte Carlo sampling approximates the permutation distribution. Each replicate must follow the same label-shuffling rule, and its table statistic must be calculated in the same way as the observed statistic.
Worked Example: Estimating a Permutation P-Value
In a hypothetical dataset of 80 units, two categorical variables produce this table:
| Category A | Category B | Category C | Total | |
|---|---|---|---|---|
| Group 1 | 18 | 12 | 10 | 40 |
| Group 2 | 7 | 13 | 20 | 40 |
| Total | 25 | 25 | 30 | 80 |
The expected counts in each row are 12.5, 12.5, and 15. For Group 1, the three contributions to \(T\) are \(30.25/12.5=2.42\), \(0.25/12.5=0.0200\), and \(25/15=1.6667\). Group 2 has the same squared deviations, so
Suppose 9,999 random permutations of the category labels are generated, and 173 have \(T\geq8.213\). The Monte Carlo p-value estimate is
This estimate suggests that, under the null model and the stated permutation scheme, about 1.74% of reference tables have a discrepancy at least as large as the observed one. Because it is based on a finite simulation, it has simulation error; a different set of random permutations could give a somewhat different estimate.
Assumptions, Interpretation, and Common Pitfalls
Before reporting a result, check that every unit contributes one pair of categorical observations, the categories are defined consistently, and the chosen labels are exchangeable under the null and design. The ordinary scheme also presumes that the permitted category combinations are not restricted in a way the shuffle ignores. If these requirements fail, the resulting permutation distribution may not represent the null distribution.
A permutation p-value is a probability calculated under a specified null model and permutation mechanism: it is the chance of obtaining a statistic at least as extreme as observed under that reference process. It is not the probability that the null hypothesis is true. A small p-value is evidence against independence under the assumed scheme, but does not by itself measure the strength or practical importance of association. The table’s cell counts and conditional proportions help describe its pattern.
- Mistake: assuming independent units guarantee a valid shuffle. Independent observations may still have nonidentical category probabilities. The correction is to justify exchangeability under the null, not merely independence among units.
- Mistake: changing the margins while permuting. The intended scheme reassigns the observed labels, so their category totals—and the other variable’s group sizes—remain fixed.
- Mistake: treating \(T\) as the p-value. The statistic measures discrepancy; the p-value comes from comparing it with the permutation distribution.
- Mistake: claiming causation from association. A table may show evidence that variables are associated, but an observational association alone does not establish that one variable caused the other.
Check Your Understanding
Answer each question using the definitions and procedures in this tutorial.
- What stays fixed when one variable’s labels are permuted in the usual two-way-table scheme?
- For a cell with row total 24, column total 15, and overall total 60, calculate its expected count under independence.
- Why does independence of units alone not guarantee that permuting labels is valid?
- If 48 of 4,999 simulated permutation statistics are at least as large as observed, calculate the plus-one Monte Carlo p-value.
- In a permutation test using \(T\) as the Pearson chi-square discrepancy, which tail is used for a general test of association, and why?