Dependence And The Effective Evaluation Sample

Why the Number of Evaluation Rows Can Mislead

An evaluation set with 500 rows does not necessarily contain as much information as 500 independent observations. Several rows may come from the same policyholder, clinic, employer, or time period. Shared characteristics can make their prediction errors similar. In that case, treating every row as independent overstates how much evidence the evaluation provides.

This matters both when interpreting an evaluation score and when designing a split. If the intended prediction task concerns new policyholders, for example, records for one policyholder should not be allowed to appear on both sides of a training-test split. A row-wise split may instead measure performance on familiar policyholders. The score may be calculated correctly for those test rows, yet answer the wrong generalization question.

Definition: The effective evaluation sample size is the number of independent observations that would give the same variance for an evaluation average as the dependent sample actually used. It is a variance-equivalent summary, not a replacement count of the observed rows. Its value depends on the dependence pattern, the score being averaged, and the target population.

The calculations below concern an average evaluation score, such as an average loss or error. They do not change the observed score itself. They explain why that score may be less precisely estimated than an independence calculation suggests. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Clustered Observations and the Design Effect

Suppose evaluation rows are arranged in $m$ independent clusters, with $b$ rows in each cluster, so the total row count is $n=mb$. Assume every row-level score $Z_{gj}$ has mean $\mu$ and variance $\sigma^2$, and two different scores in the same cluster have common correlation $\rho$. Scores in different clusters are assumed independent. This is an equal-cluster-size, exchangeable-correlation model; it is a simplifying assumption, not a universal description of clustered data.

The average score is $\bar{Z}=(1/n)\sum_g\sum_j Z_{gj}$. Its variance depends on both individual variances and within-cluster covariances. Under the stated assumptions, the variance inflation relative to $n$ independent scores is called the design effect.

\operatorname{Var}(\bar{Z}) =\frac{\sigma^2}{n}\left[1+(b-1)\rho\right], \qquad D=1+(b-1)\rho, \qquad n_{\mathrm{eff}}=\frac{n}{D}.

The effective size follows by equating the dependent-sample variance, $\sigma^2D/n$, with the variance $\sigma^2/n_{\mathrm{eff}}$ that would result from $n_{\mathrm{eff}}$ independent scores. When $\rho>0$, $D>1$, so $n_{\mathrm{eff}}<n$. The formula assumes equal cluster sizes and a common within-cluster correlation. If cluster sizes differ, correlations vary, or clusters are themselves dependent, this simple expression need not apply.

Derivation: Within one cluster, the variance of the sum of $b$ scores is the sum of their variances plus twice the sum of their pairwise covariances. There are $b$ variance terms and $b(b-1)/2$ pairs, each with covariance $\rho\sigma^2$. Thus the cluster-sum variance is $b\sigma^2+b(b-1)\rho\sigma^2$. The $m$ clusters are independent, so the variance of the total sum is $m$ times this quantity. Dividing by $n^2=(mb)^2$ gives $$ \operatorname{Var}(\bar{Z}) =\frac{m\{b\sigma^2+b(b-1)\rho\sigma^2\}}{(mb)^2} =\frac{\sigma^2}{mb}\left[1+(b-1)\rho\right]. $$ Since $n=mb$, this is the stated variance formula.

Worked Example: Repeated Evaluations Within Employers

A hypothetical model is evaluated on 120 employees from 12 employers, with 10 employees per employer. The standard deviation of the row-level loss scores is 0.40, and the within-employer correlation is estimated as 0.30. Under the equal-size cluster model, calculate the effective sample size and compare the standard error with the independence calculation.

Method. Use the cluster design effect with $b=10$, $\rho=0.30$, $n=120$, and $\sigma=0.40$. The model assumes independent employers and a common correlation among different employees at the same employer.

Step 1: Calculate the design effect. $D=1+(10-1)(0.30)=1+2.70=3.70$.

Step 2: Calculate effective sample size. $n_{\mathrm{eff}}=120/3.70=32.4324$, or about 32.43 independent-score equivalents. The dataset still has 120 rows and 12 employers; 32.43 is not a new row count.

Step 3: Compare standard errors. If the rows were independent, the variance of the average would be $0.40^2/120=0.16/120=0.0013333$, giving standard error $\sqrt{0.0013333}=0.0365$. With clustering, the variance is $0.0013333(3.70)=0.0049333$, giving standard error $\sqrt{0.0049333}=0.0702$. The standard error is larger because employees at the same employer provide overlapping information.

Check and trap. Dividing by 120 without the design effect would report 0.0365, understating the model-based standard error. The corrected variance is also $0.16/32.4324=0.0049333$, which checks the effective-size calculation. This method avoids treating clustered rows as independent evidence.

Serial Dependence in Time-Ordered Evaluation

For time-ordered scores, dependence is often strongest between nearby observations and weaker at longer lags. One common model is a stationary first-order autoregressive correlation structure, abbreviated AR(1): every score has variance $\sigma^2$, and the correlation between scores $k$ time steps apart is $\rho^k$, with $|\rho|<1$. Scores are assumed to have the same mean and variance across the period. The formula below applies to the average of one consecutive sequence under these assumptions; it is not a general formula for every time series.

There are $n-k$ pairs separated by lag $k$. Adding the variance contributions and the covariances of all such pairs gives the following variance ratio relative to independent observations:

D_n =1+2\sum_{k=1}^{n-1}\left(1-\frac{k}{n}\right)\rho^k, \qquad n_{\mathrm{eff}}=\frac{n}{D_n}.

For large $n$ and positive $\rho$, the lag weights are close to 1 for the lags that matter most, and the geometric series gives the approximation $D_n\approx1+2\sum_{k=1}^{\infty}\rho^k=(1+\rho)/(1-\rho)$. This approximation can be useful for a long series, but it should not be presented as the exact finite-sample design effect. For short series, substantial negative dependence, trends, or changing variance, the AR(1) assumptions and approximation require particular care.

Worked Example: Consecutive Monthly Loss Scores

A hypothetical evaluation uses 80 consecutive monthly loss scores. Their standard deviation is 0.30, and an AR(1) model with lag-one correlation $\rho=0.40$ is specified. Estimate the design effect and effective sample size using the finite-sample formula, then compare standard errors.

Method. Sum the weighted lag correlations. Because $0.40^{79}$ is negligible at the displayed precision, the geometric sums can be evaluated to that precision using their infinite-series values.

Step 1: Evaluate the lag sums. The unweighted sum is $\sum_{k=1}^{\infty}0.4^k=0.4/(1-0.4)=0.666667$. The weighted part uses $\sum_{k=1}^{\infty}k(0.4)^k=0.4/(1-0.4)^2=1.111111$. Therefore, $\sum_{k=1}^{79}(1-k/80)(0.4)^k\approx0.666667-(1.111111/80)=0.652778$.

Step 2: Calculate the finite-sample design effect. $D_{80}\approx1+2(0.652778)=2.305556$. Thus $n_{\mathrm{eff}}\approx80/2.305556=34.70$.

Step 3: Compare standard errors. Under independence, the average has standard error $0.30/\sqrt{80}=0.0335$. With the specified serial dependence, it is approximately $0.0335\sqrt{2.305556}=0.0509$. Equivalently, $0.30/\sqrt{34.70}=0.0509$.

Check and trap. The large-sample approximation gives $D\approx(1+0.40)/(1-0.40)=2.3333$, slightly above the finite-sample value 2.3056. Using it as if exact would give $n_{\mathrm{eff}}\approx34.29$, not 34.70. Both may be reasonable approximations in context, but the finite-sample calculation is the requested one here.

Diagnosing a Split That Treats Related Rows as Independent

A dependence calculation is only part of the evaluation question. The split must also reflect what is meant by future or unseen data. If the target is performance on new groups, the evaluation groups should be kept out of training. If the target is a later period, the split should respect time order. Earlier material on grouped cross-validation and time-series cross-validation addresses these split structures; the key point here is why row count alone cannot establish that the resulting evaluation is informative.

A useful diagnosis starts by identifying what can make two rows related. Look for repeated identifiers, multiple records per account, patients nested in clinics, claims from the same policyholder, overlapping observation windows, or adjacent periods. Then ask whether related rows can cross the training-evaluation boundary. The issue is not merely correlation within the evaluation set: if a related record is in training, the model may have information about the test row's group that would not be available for a genuinely new group.

Worked Example: Clinic Records in a Row-Wise Split

A hypothetical dataset contains 8 patient records from each of 30 clinics, for 240 rows. A random row-wise split puts 75% of rows in training and 25% in testing. The goal is to estimate performance at clinics not represented in training. Assume, for a rough information calculation, that test-row scores within a clinic have correlation 0.50. Diagnose the split and quantify the test-set information under a grouped alternative.

Method. First calculate the chance that a clinic appears on both sides under independent row assignment. Then compare that design with holding out entire clinics. The probability calculation assumes each row independently enters training with probability 0.75 and testing with probability 0.25.

Step 1: Assess clinic overlap in the row-wise split. For one clinic, the probability all 8 records are in training is $0.75^8=0.100113$. The probability all 8 are in testing is $0.25^8=0.0000153$. Therefore, the probability that the clinic appears on both sides is $1-0.75^8-0.25^8=0.899872$. Across 30 clinics, the expected number appearing on both sides is $30(0.899872)=26.996$, approximately 27 clinics.

Step 2: Construct a split aligned to the target. Hold out 6 entire clinics. The test set then has $6(8)=48$ rows, and no held-out clinic contributes records to training. This design targets performance at new clinics more directly than the row-wise split.

Step 3: Calculate the row-average effective size for the grouped test set. With $b=8$ and $\rho=0.50$, $D=1+(8-1)(0.50)=4.50$. Hence $n_{\mathrm{eff}}=48/4.50=10.67$ independent-score equivalents for a row-weighted average under the stated correlation model.

Check and trap. There are 48 test rows but only 6 independent held-out clinics. The effective size of 10.67 is a variance-equivalent calculation for the average of correlated patient scores; it does not mean there are 10.67 independent clinics. A row-wise split would put about 27 clinics on both sides and could answer a familiar-clinic question rather than the intended new-clinic question. The correction is to split by clinic and describe the limited number of held-out clinics, not to report 48 as if it were 48 independent new-clinic evaluations.

Interpreting Effective Size in Practice

Effective sample size is useful for communicating how dependence changes precision, but it should not be treated as a universal adjustment that repairs a poor evaluation design. It does not remove information leakage, change the population being tested, or make a small number of independent groups equivalent to a large group sample. A model can also have different dependence in different score components or segments, so a single estimated correlation may not summarize every feature of the evaluation.

Before using an independence-based interpretation of an evaluation average, check these points:

State the target unit. Is the objective performance on a new person, a new organization, or a later time period?
Identify the dependence structure. Determine which rows share an entity, setting, or time history.
Inspect the split. Confirm that related records do not cross the split when the target requires unseen groups or future periods.
Separate row count from independent-unit count. Report both where useful, especially when few groups are held out.
State assumptions behind a numerical adjustment. The equal-cluster formula and AR(1) calculation require specific structures; do not apply them mechanically to unequal or more complex data.

Key takeaway: The nominal number of evaluation rows describes the dataset, not necessarily its information. Positive dependence generally increases the variance of an average score; matching that variance to an independent sample gives an effective size smaller than the row count. A suitable split must also respect the unit and time horizon to which predictions are meant to generalize.

Check Your Understanding

For each problem, identify the relevant dependence assumption and calculate or diagnose the evaluation information. Answers are provided for checking.

A hypothetical evaluation has 15 independent branches, each contributing 4 score rows. Within-branch correlation is 0.20. Find the design effect and effective sample size. (Answer: $D=1.60$; $n_{\mathrm{eff}}=60/1.60=37.5$.)
A long sequence of monthly scores is modeled with AR(1) correlation $\rho=0.25$. Using the large-sample approximation, find the design effect and effective sample size for 100 months. (Answer: $D\approx(1.25)/(0.75)=1.6667$; $n_{\mathrm{eff}}\approx60.0$.)
A dataset has 10 records per member and a random row split. The intended target is performance for new members. What is the main design concern, and what split unit is appropriate? (Answer: Records from a member can appear in both training and test data, so use a member-grouped split with whole members held out.)
A test set has 5 independent sites with 6 records each. Assume equal cluster size and within-site correlation 0.40. Calculate the row-average effective sample size. (Answer: $n=30$, $D=1+5(0.40)=3$, and $n_{\mathrm{eff}}=30/3=10$; there are still only 5 independent sites.)
For 50 consecutive scores under an AR(1) model with $\rho=0$, state the design effect and effective sample size. Explain why. (Answer: $D_{50}=1$ and $n_{\mathrm{eff}}=50$, because zero correlation makes the scores independent under this model.)

What You'll Learn