Observation Records and Covariate Records

Why Separate Observation Records from Covariate Records?

A pharmacometric data set must preserve what was measured, for whom, and when. A concentration recorded from a sample is an observation of the response; a weight recorded at a visit is a covariate measurement. They may refer to the same subject and occasion, but they are not necessarily the same kind of record. Treating them as if they were interchangeable can obscure measurement times, create duplicate response rows, or attach a covariate value to the wrong observation.

The earlier material on the pharmacometric record grain and record keys establishes that the meaning of a row must be explicit and that a key must distinguish rows at that grain. Here the central task is to maintain separate meanings for response observations and covariate measurements, then combine them only when the relationship between their keys and times is clear.

Definition: An observation record represents one measured response for a specified subject at a specified observation time, with any additional identifiers needed to distinguish the measurement, such as analyte or endpoint. A covariate record represents a measured or defined predictor for a specified subject and, when it varies over time, a specified measurement time. A covariate value is not itself a response observation.

The record grain follows the question the row answers. In an observation table, a row might mean “this subject’s plasma concentration of the specified analyte at this sampling time.” In a covariate table, a row might mean “this subject’s measured body weight at this visit.” A subject-level attribute such as sex may have one value per subject; a time-varying attribute such as weight can have multiple measurements per subject.

A compact observation table and separate covariate tables are often easier to audit than one wide table with many repeated fields. A combined analysis table can still be useful, but it should be derived from well-defined source records using a documented matching rule. The choice of layout does not remove the need to define the grain and keys of each table.

Record Layouts and Their Meaning

A long observation layout stores each response measurement in its own row. The subject identifier, observation time, analyte or endpoint identifier when needed, response value, and units identify the observation. A separate long covariate layout stores each covariate measurement in its own row, with a covariate name, value, units, subject identifier, and measurement time if applicable. This layout accommodates different measurement schedules without pretending that every covariate was measured whenever a response was observed.

A wide covariate layout instead has one row per subject or per subject-time, with different covariates in separate columns. This can be convenient when covariates share a common schedule. It can be misleading when those columns were measured at different times: the row may appear to imply simultaneity that did not occur. In either layout, record the actual measurement time and units rather than relying on a column label or visit name to carry that information.

For subject-level covariates, a one-row-per-subject table is natural if each subject has one defined value for the analysis period. For time-varying covariates, a subject-time table is appropriate only when the row truly represents measurements at that time, and the key must distinguish any additional covariate records at the same time. If different covariates have different timestamps, a covariate-name/value layout can preserve each timestamp explicitly.

Table or layout	Typical row meaning	Key elements to consider	Useful when
Observation table	One response measurement	Subject, observation time, and identifiers distinguishing measurements	Response measurements are the primary records
Subject-level covariate table	One defined covariate profile per subject	Subject and covariate identity if stored long	Values do not vary over the relevant period
Time-varying covariate table	One covariate measurement at a stated time	Subject, covariate, measurement time, and any required distinguishing identifier	Covariates have their own measurement schedule
Combined analysis table	One observation with explicitly matched covariate values	Observation key plus a documented matching rule	A modeling workflow requires covariates beside responses

Matching Covariate Records to Observations

A join combines records using specified key fields. Before joining, decide whether a covariate should match by subject alone, by subject and exact time, or by a prespecified time-based rule. A subject-only match is usually appropriate for a truly subject-level covariate. It is generally not sufficient for a time-varying covariate: if a subject has several covariate measurements, joining only on subject can associate every observation with every covariate record for that subject.

An exact-time match means the subject and defined time fields agree. A time-based match, such as selecting the most recent eligible covariate measured before an observation, is a different rule and must be specified in advance. It must also have a direction, a time window if one is required, and a policy for ties. Interpolation, forward-carrying values, and nearest-time matching make additional assumptions about how the covariate behaves between measurements. They should not be introduced implicitly by a data operation.

Check cardinality before and after a join. For each observation, count the number of covariate records that match its key. If a one-to-one match is expected, each observation should have exactly one match. If a subject-level value is joined to several observations, one covariate record may appropriately match many observation records; that is a many-to-one relationship from observations to covariates. Multiple matching covariate records for a single observation require investigation unless the intended design explicitly calls for them.

Join-count result: Let \(N\) be the number of observation records, and let \(m_i\) be the number of covariate records matching observation \(i\). An inner join returns \(N_{\text{joined}}=\sum_{i=1}^{N}m_i\) rows. A left join, retaining every observation, returns \(N_{\text{joined}}=\sum_{i=1}^{N}\max(1,m_i)\) rows. These formulas assume each observation is a distinct input row and matches are counted under the specified join keys. A left join retains an unmatched observation once, with no matching covariate values; it does not create a match.

The formulas follow by counting the output contribution from each observation. An inner join emits one row for every matching pair, so observation \(i\) contributes \(m_i\) rows, including zero if it has no match. A left join emits one row per matching pair when \(m_i>0\), but emits one retained row when \(m_i=0\); its contribution is therefore \(\max(1,m_i)\). Summing those contributions over observations gives the respective totals. The formulas make clear why a join’s type must be stated when diagnosing row counts.

State each table’s grain.
Write down what one observation row means and what one covariate row means, including the actual measurement time for time-varying values.

Specify the matching rule.
Name the key fields, the join type, and any temporal rule. Do not use a subject-only key for a time-varying covariate unless that is genuinely the intended relationship.

Check matches and row counts.
Count zero, one, and multiple matches per observation. Compare the output row count with the formula for the join type used.

Inspect the resulting meaning.
Confirm that each response still has its correct subject, time, analyte or endpoint identity, and covariate value with the intended measurement time and units.

Worked Examples

Worked Example: Keeping Concentration and Weight Records Distinct

All values in this example are hypothetical. A study has two concentration observations for subject S01 and two weight measurements. The concentration records are: at 2 hours, 4.8 mg/L; at 8 hours, 2.1 mg/L. The weight records are: at 0 hours, 60 kg; at 6 hours, 62 kg.

The observation table has two rows, each representing one concentration measurement for S01 at its stated sampling time. The covariate table has two rows, each representing one weight measurement for S01 at its own measurement time. A direct exact-time join on subject and time yields no matches because neither weight time, 0 or 6 hours, equals an observation time, 2 or 8 hours. That result is not a software failure; it accurately reports that the records do not share exact timestamps.

If the analysis protocol specifies “use the most recent weight measured at or before each concentration time,” then the 2-hour concentration matches the 0-hour weight of 60 kg, and the 8-hour concentration matches the 6-hour weight of 62 kg. The time differences are \(2-0=2\) hours and \(8-6=2\) hours, respectively. The combined records retain concentration units of mg/L and weight units of kg, and the covariate’s measurement time remains traceable.

This match is justified only under that stated rule. Without a prespecified rule, assigning 60 kg to the 2-hour observation and 62 kg to the 8-hour observation would be an undocumented assumption. The example also shows why covariate time should not be overwritten with response time: the weight was measured at 0 or 6 hours, not at 2 or 8 hours.

Worked Example: Diagnosing Inner-Join and Left-Join Row Counts

All values are hypothetical. There are two observation records, O1 and O2. Under the chosen matching keys, O1 has one matching covariate record (\(m_1=1\)) and O2 has none (\(m_2=0\)).

For an inner join, the row count is

N_{\text{joined}}=\sum_i m_i=1+0=1.

O1 contributes one matched row, and O2 contributes none because an inner join drops unmatched observations. For a left join, the row count is

N_{\text{joined}}=\sum_i \max(1,m_i)=\max(1,1)+\max(1,0)=1+1=2.

O1 contributes its one matched row, while O2 remains once with no matched covariate values. Thus the same keys produce one row under the inner join and two under the left join. Reporting “two records were retained” without naming the join would be ambiguous.

Worked Example: Finding Row Multiplication from Duplicate Matches

All values are hypothetical. Three concentration observations have match counts \(m_1=1\), \(m_2=2\), and \(m_3=0\) against a time-varying covariate table. The intended rule was an exact subject-and-time match, but the second observation has two covariate rows with the same matching key.

An inner join gives

N_{\text{inner}}=1+2+0=3.

A left join gives

N_{\text{left}}=\max(1,1)+\max(1,2)+\max(1,0)=1+2+1=4.

The left-joined output has four rows although the input has three observations. The second observation was duplicated by its two matches, and the third was retained once despite having no match. These effects do not cancel here: the output has one more row than the input. The duplicate key must be investigated—for example, the covariate table may contain repeated records or the key may omit a field that distinguishes them. Choosing one value arbitrarily would conceal the unresolved record-grain problem.

Diagnostics, Alternatives, and Limitations

A row-count check is necessary but not sufficient. Two different problems can offset each other under an inner join: an observation with multiple matches increases the count, while an unmatched observation contributes zero. Therefore a plausible total does not prove that every record is correctly linked. Inspect match counts by observation and review examples of unmatched and multiply matched keys.

Unexpected row multiplication: one or more observations match multiple covariate rows. Check duplicate keys, omitted key fields, and whether the covariate table’s grain was defined correctly.
Unexpected losses: an inner join removes observations with no match. Determine whether the absence reflects an incorrect key, a different measurement schedule, or a genuinely unavailable covariate record.
Misleadingly complete values: a subject-only join may attach several time-varying values to each observation. Verify that time is part of the key or that a defensible temporal matching rule was applied.
Unit or timestamp mismatch: confirm that compared times use the same origin and units, and that covariate values retain their own units and measurement times. A numerically equal time in different conventions is not necessarily the same time.

An inner join is appropriate when only observations with a covariate match should remain, and the resulting exclusions are understood. A left join is appropriate when the observation table is the backbone and all observation records must be retained for review or subsequent processing. A full outer join can be useful for reconciling two source tables because it retains unmatched records from both sides, but it is not automatically a suitable modeling table: unmatched covariate-only rows do not represent response observations. In every case, the join type and matching rule determine the output’s meaning.

A separate covariate table preserves its own measurement schedule and can be audited independently. A wide, precombined table may be convenient for downstream modeling, but it can hide repeated values, mismatched timestamps, or accidental replication. Neither layout establishes that a covariate is scientifically appropriate, that a time-based carry-forward assumption is valid, or that the resulting records satisfy all later data requirements. Those questions require explicit study and analysis decisions.

Common Mistake

Common mistake: “The combined table has the same number of rows as the observation table, so the join must be correct.” This conclusion is not justified. A left join can increase rows when an observation has multiple matches, while an inner join can reduce rows by dropping unmatched observations; in an inner join, these changes can even offset. The correction is to name the join type, calculate or inspect matches for every observation, verify the expected key cardinality, and check that the matched measurement time and value make sense.

Key takeaway: Keep response observations and covariate measurements identifiable at their own record grains. Combine them only with explicit keys and a justified temporal rule, then validate individual matches as well as the total row count.

Check Your Understanding

Answer each question, then compare with the solutions below.

An observation table has four rows. Their covariate match counts are 1, 0, 2, and 1. How many rows result from an inner join and from a left join?
A subject has concentration observations at 3 and 9 hours and weight measurements at 0 and 7 hours. What information is needed before assigning weights to the concentration observations?
Why might joining a time-varying covariate to observations using subject identifier alone create incorrect records?
A left join returns five rows from a table with four observations. What is one possible match-count pattern, and what should be checked next?

Solutions. For question 1, the inner join returns \(1+0+2+1=4\) rows. The left join returns \(\max(1,1)+\max(1,0)+\max(1,2)+\max(1,1)=1+1+2+1=5\) rows. For question 2, an explicit time-alignment rule is needed, such as the most recent measurement at or before the observation, along with any required time window and tie policy. Under the stated most-recent-prior rule, the 3-hour observation would use the 0-hour weight and the 9-hour observation the 7-hour weight, but that rule must be justified rather than assumed. For question 3, one subject may have multiple covariate measurements, so each observation can match all of them and be duplicated or linked to values from inappropriate times. Include the relevant time key or apply a documented temporal matching rule. For question 4, match counts of 1, 2, 1, and 1 give a left-join total of \(1+2+1+1=5\). Check the observation with two matches for duplicate covariate keys, an incomplete join key, or an intended one-to-many relationship before treating the output as valid.