Independence At The Participant, Study, And Measurement Levels

What Does Independence Mean in Validation?

A validation data set can contain records that look new while still sharing people, measurements, or study processes with calibration data. Independence therefore cannot be established by counting rows or checking that two files have different names. It requires tracing how the observations were generated and whether information or experimental units are shared.

The introductory material on calibration and validation distinguishes using data to estimate a model from evaluating predictions without refitting. The question here is narrower: what kinds of overlap or shared conduct undermine a claim that validation evidence is independent of calibration evidence? The answer depends on the level being considered.

Definition: Two random quantities \(X\) and \(Y\) are statistically independent if, for every pair of events \(A\) and \(B\) defined from their possible values, \(P(X\in A, Y\in B)=P(X\in A)P(Y\in B)\). In a validation assessment, independence must be considered in relation to the units and processes that generated the calibration and validation observations. Distinct records alone do not establish this condition.

Three levels are especially important. Participant-level independence concerns whether the same people contribute to both calibration and validation data. Study-level independence concerns whether observations come from genuinely separate study conduct, rather than different records or arms of a shared experiment. Measurement-level independence concerns whether observations are distinct measurements or repeated measurements on the same underlying unit. These levels answer related but different questions; passing one does not imply passing the others.

Trace the Units and Their Provenance

Start by defining the unit at each level. A participant may contribute several visits; a study may include multiple cohorts, sites, or arms; and a measurement may be one concentration at one time point. The relevant unit depends on the prediction question and the data-generating process. For example, separate concentration records from one participant are different measurements, but they are not independent participants.

Participant overlap can be direct, such as the same person appearing in both data sets under the same identifier, or indirect, such as a participant reappearing under a changed identifier. Study overlap can occur when calibration and validation observations come from different arms or time points of one protocol, or share study-specific procedures and conduct. Measurement dependence can arise from repeated samples within a participant, samples processed in a shared batch, or measurements derived from the same underlying specimen.

Shared conduct is evidence to investigate, not an automatic proof that every observation is statistically dependent. Two studies at the same site may use the same laboratory but enroll different participants under separate protocols. Conversely, two studies with different names may reuse participants or specimens. Independence claims should therefore describe the actual overlap found and the level at which it matters, rather than treating labels such as “external” or “new study” as sufficient evidence.

Worked Example: Detecting Participant Overlap

In a hypothetical PBPK assessment, the calibration file contains 40 participants and the proposed validation file contains 30. After identifiers are harmonized, four participants occur in both files. Determine whether the validation set is participant-independent and quantify the overlap.

Let \(C\) and \(V\) denote the sets of calibration and validation participants. The intersection contains four people: \(|C\cap V|=4\). The validation data therefore are not disjoint at the participant level.

The fraction of validation participants who also appear in calibration is \[ \frac{|C\cap V|}{|V|}=\frac{4}{30}=0.1333, \] or 13.3%. The fraction of calibration participants reused is \(4/40=0.1000\), or 10.0%. These are different denominators and answer different descriptive questions.

The counts establish participant overlap; they do not by themselves quantify the resulting bias in any prediction assessment. The four shared participants may contribute different visits or measurements, but that does not restore participant-level independence. A defensible report would state the overlap, explain how identifiers were reconciled, and avoid describing the full validation set as participant-independent.

Repeated Measurements and Dependence

Repeated observations from the same participant often share stable characteristics and study conditions. Thus a participant’s concentration at one time may be informative about that participant’s concentration at another time beyond what would be expected for two independently sampled people. The dependence can be summarized in a simple exchangeable model: all observations have variance \(\sigma^2\), and any two measurements from the same participant have correlation \(\rho\). This is an illustrative model, not a universal description of longitudinal data.

For \(N\) participants with \(m\) measurements each, the average of all \(Nm\) measurements has variance

\operatorname{Var}(\bar{Y})= \frac{\sigma^2}{Nm}\left[1+(m-1)\rho\right].

This expression assumes equal cluster size, equal measurement variance, and a common within-participant correlation, with different participants independent of one another. The bracketed multiplier shows why treating all records as independent can understate uncertainty when \(\rho>0\). It does not mean the observations are unusable; it means the participant, rather than each record, is the independent sampling unit for this simplified structure.

Worked Example: Four Measurements per Participant

Suppose a hypothetical validation cohort has \(N=4\) participants, each measured at \(m=4\) times. Assume the concentration measurement variance is \(\sigma^2=4\ (\mathrm{mg/L})^2\), and the within-participant correlation is \(\rho=0.60\). Compare the variance of the grand mean under the repeated-measurement model with the variance obtained by incorrectly treating all 16 measurements as independent.

Under the stated equal-cluster model,

\operatorname{Var}(\bar{Y})= \frac{4}{4(4)}[1+(4-1)(0.60)] =\frac{4}{16}(2.80) =0.7000\ (\mathrm{mg/L})^2.

The corresponding standard error is \(\sqrt{0.7000}=0.8367\ \mathrm{mg/L}\). If all 16 records were incorrectly treated as independent, the variance would be \(4/16=0.2500\ (\mathrm{mg/L})^2\), giving a standard error of \(0.5000\ \mathrm{mg/L}\). Under these assumptions, the naive calculation is too small because it ignores the positive correlation among measurements from the same participant.

This calculation is a diagnostic illustration, not a complete analysis of a PBPK validation profile. Real concentration-time data may have correlations that vary with time, dose, and participant, and may include study-level clustering as well. The key conclusion is that 16 measurement records here do not provide the same independence as 16 independently sampled participants.

Study Conduct and Shared Data Sources

Study-level separation asks whether the validation evidence comes from a distinct episode of data collection and measurement, not merely whether it is stored in a separate table. Relevant provenance includes protocol, enrollment period, site, investigator team, dosing and sampling procedures, laboratory, assay batch, specimen identity, and data source. Some shared features may be routine and harmless; others may couple observations or indicate reuse. Their meaning depends on how they enter the data-generating process.

For example, separate participants enrolled in two independently conducted studies can support participant-level separation. If both studies use the same central laboratory, however, shared assay procedures or batch effects may remain relevant to measurement independence. Likewise, participants in separate arms of one protocol are not automatically evidence from independent studies: allocation, conduct, and common study conditions may link the arms. Study labels should be supported by a provenance account.

Worked Example: Distinct Participants, Shared Study Conduct

Consider a hypothetical calibration cohort of 18 participants and a validation cohort of 20 different participants. Both cohorts were enrolled under one protocol at one site, during the same period, and their samples were measured by the same laboratory batch. No participant identifiers overlap. Assess what can and cannot be concluded.

The participant sets are disjoint, so the data support a claim of no observed participant overlap. That is a useful and specific finding. But both cohorts share a protocol, site, period, and assay batch. They are therefore not independent study conduct in the ordinary design sense, and shared batch or protocol conditions may induce dependence or common systematic effects.

The conclusion is not that the validation observations must be statistically dependent in every possible model. Rather, participant disjointness alone does not justify a broad claim of independent validation evidence at the study and measurement levels. A report should identify the shared conditions and state that the evidence is participant-distinct but study-conduct and assay-source shared. Whether the shared features are consequential requires scientific assessment; the records alone cannot resolve that question.

A Practical Independence Audit

A clear assessment separates verifiable overlap from interpretation. The following sequence makes the basis of the claim explicit.

Define the units.
List the participant, study, specimen, and measurement units represented in calibration and validation data. Specify which unit is relevant to the prediction question.

Trace identifiers and source records.
Compare harmonized participant and specimen identifiers, including documented aliases or replacements. Record unresolved identity matches rather than silently treating them as unique.

Map shared conduct.
Compare protocols, cohorts, sites, periods, staff, procedures, laboratories, batches, and data repositories. Distinguish shared background infrastructure from shared experimental units or processes.

Describe dependence at each level.
State what is demonstrably disjoint, what is shared, and what remains uncertain. Do not infer measurement independence merely from participant separation or study independence from different file labels.

Qualify the claim.
Report the evidence and its limitation in precise terms, such as “no participant overlap was identified; both cohorts used the same protocol and assay batch.”

How the Levels Compare

Level	Question to ask	Evidence that informs the answer	What it does not establish by itself
Participant	Do the same people contribute to both data sets?	Harmonized identifiers, enrollment records, documented repeat participation	Independent study conduct or independent measurements
Study	Were observations generated in distinct study conduct?	Protocols, sites, periods, cohorts, procedures, and study records	Absence of shared participants, specimens, or laboratory effects
Measurement	Are records repeated or linked through the same participant, specimen, or batch?	Visit and sample identifiers, time points, assay and batch records	Independence merely because measurement rows are separate

The main alternatives are not interchangeable. A participant-separated assessment focuses on person-level reuse. A study-separated assessment focuses on the design and conduct that generated the evidence. A measurement-separated assessment focuses on repeated or technically linked observations. A strong independence argument states which of these is supported, rather than collapsing them into a single yes-or-no label.

Diagnostics, Limitations, and Common Mistakes

Warning signs include duplicate or inconsistent identifiers, unexplained participant counts, validation measurements at visits also represented in calibration, shared specimen codes, or records whose study and assay provenance is missing. Sudden differences between row counts and unique participant counts are a reason to investigate, not proof of a problem. Similarly, a shared site or laboratory is a reason to document and assess possible common effects, not automatic proof that data are unusable.

An independence audit is limited by the quality of its metadata. De-identified data may make cross-file matching impossible; records may omit sample lineage; and common protocol conditions can be difficult to separate from general practice. In such cases, the correct conclusion is that independence is uncertain at the affected level, not that it has been demonstrated. Statistical models can represent some clustering, but modeling does not change the historical fact that participants, samples, or study processes were shared.

Common mistake: “The validation file has different rows and no repeated concentration values, so every observation is independent.” This reasoning confuses unique records with independent units. Multiple time points from one participant are distinct rows but share the participant; distinct participants in one study can still share study-level conditions. The correction is to trace units and provenance separately, then make a level-specific claim supported by the available evidence.

Key takeaway: Independence is a property of how observations are generated, not a label attached to a file. Assess participant overlap, shared study conduct, and repeated or linked measurements separately, and report precisely what is known at each level.

Check Your Understanding

Answer each question, then compare with the solutions below.

Calibration and validation contain different rows but include the same participant at different visits. Which level of independence is not supported?
There are 10 participants with three measurements each. Under equal within-participant correlation \(\rho=0.50\), by what factor does the variance of the grand mean exceed the variance under an independent-record calculation, assuming equal measurement variance?
Two disjoint participant cohorts use separate protocols but the same assay batch. What can be said about participant-level and measurement-source independence?
Why does a different study name not, by itself, prove study-level independence?

Solutions. (1) Participant-level independence is not supported because the same person contributes to both data sets, even though the visits differ. (2) With \(m=3\), the variance multiplier is \(1+(m-1)\rho=1+2(0.50)=2.00\); the variance is twice the independent-record value under the stated model. (3) The cohorts are participant-disjoint, but the shared batch is a common measurement source that should be documented and assessed; separate protocols do not establish measurement-source independence. (4) A name does not establish distinct participants, specimens, protocols, conduct, or data sources. Those features must be checked from provenance records.