Tutorials › High-Dimensional Genomic Data Analysis › Paired Designs With Negative Binomial Models

Count-Based Differential Expression for RNA-seq · Tutorial 451 of 795

Paired Designs With Negative Binomial Models

Specify and interpret paired negative binomial designs, including the assumptions, within-subject cancellation, and diagnostics that determine whether pairing is informative.

Intermediate 10 min read

What You'll Learn

  • Write a negative binomial log-mean model with subject fixed effects and a condition coefficient.
  • Interpret the condition coefficient as a within-subject log fold change.
  • Explain why pairing can remove stable between-subject differences from the condition comparison.
  • Check the design for confounding, missing pairs, and assumptions about a common condition effect.
  • Distinguish a paired model from an unpaired model and a random-intercept approach.
  • Interpret hypothetical fitted effects, standard errors, and Wald intervals in context.

Why Pair Samples by Subject?

In RNA-seq, samples from the same person, animal, or other biological subject can be measured under two conditions. If subjects differ substantially in their baseline expression, an analysis that ignores which samples belong together can mix those baseline differences into the condition comparison. A paired design makes the subject identity part of the model, so the condition effect is estimated after accounting for stable differences among subjects.

The pairing is useful only when the experimental structure supports it. In the basic design considered here, each subject contributes one sample under each of two conditions. The model compares conditions within subjects while allowing each subject to have a distinct baseline for each gene. It does not automatically address time trends, condition-specific responses that vary across subjects, or other complexities of repeated-measures data.

Definition: A paired negative binomial design models the count for each gene in each sample using a negative binomial generalized linear model (GLM), with subject indicators and a condition term. The subject indicators account for subject-specific baseline expression; the condition coefficient represents the shared condition effect across subjects.

The Model and Its Assumptions

Let \(Y_{gkc}\) be the observed count for gene \(g\), subject \(k\), and condition \(c\), where \(c=0\) denotes the reference condition and \(c=1\) the other condition. Let \(s_{kc}\) be the sample’s size factor, used as an offset to account for differences in sequencing depth or composition. A common gene-wise model is

$$ Y_{gkc}\sim\operatorname{NB}(\mu_{gkc},\alpha_g),\qquad \operatorname{Var}(Y_{gkc})=\mu_{gkc}+\alpha_g\mu_{gkc}^{2}, $$

with log mean

$$ \log(\mu_{gkc})=\log(s_{kc})+\gamma_g+ \sum_{j=2}^{K}\delta_{gj}I(k=j)+\beta_g c. $$

Here \(K\) is the number of subjects, subject 1 is the reference subject, and \(I(k=j)\) equals 1 when the sample belongs to subject \(j\), and 0 otherwise. The coefficient \(\gamma_g\) is the reference-subject baseline log mean after accounting for the offset; \(\delta_{gj}\) is the baseline difference for subject \(j\); and \(\beta_g\) is the common condition effect for gene \(g\). The dispersion \(\alpha_g\) allows the count variance to exceed the mean. The subject-specific terms are fixed effects in this model.

The condition coefficient is on the natural-log scale. Holding the subject and size factor fixed, \(\exp(\beta_g)\) is the expected-count ratio for condition 1 versus condition 0. Thus \(\beta_g=0\) corresponds to no condition effect, and a positive value corresponds to a larger expected count in condition 1, after accounting for the model terms.

The model requires a full-rank design: condition must not be perfectly determined by subject. In the basic paired design, each subject should have samples in both conditions. Counts are treated as conditionally independent given the model, size factors should be suitable offsets, and the common condition coefficient assumes the same log-scale condition effect across subjects. The negative binomial mean-variance relationship and dispersion specification must also be adequate. These assumptions do not imply that all biological variation is removed by including subject.

Worked Example: Writing the Model for Three Subjects

Suppose three hypothetical subjects each have one control and one treated sample. For a particular gene, set control to \(c=0\) and treated to \(c=1\). The model is

$$ \log(\mu_{gkc})=\log(s_{kc})+\gamma_g+\delta_{g2}I(k=2)+ \delta_{g3}I(k=3)+\beta_g c. $$

For subject 2’s control sample, \(I(k=2)=1\), \(I(k=3)=0\), and \(c=0\). Therefore \(\log(\mu_{g,2,0})=\log(s_{2,0})+\gamma_g+\delta_{g2}\). For subject 2’s treated sample, only the condition indicator changes, so \(\log(\mu_{g,2,1})=\log(s_{2,1})+\gamma_g+\delta_{g2}+\beta_g\).

Subtracting the control equation from the treated equation gives \(\log(\mu_{g,2,1}/\mu_{g,2,0})=\log(s_{2,1}/s_{2,0})+\beta_g\). After accounting for the size factors, the subject-specific baseline cancels. The same cancellation applies to subjects 1 and 3. This is the algebraic reason the design compares conditions within subjects rather than comparing unrelated baselines.

Why Pairing Can Improve Estimation

The cancellation can reduce the influence of stable between-subject differences. A simple log-scale argument makes this clear. Suppose a measurement can be represented approximately as a subject baseline plus a condition effect and an error: \(Z_{kc}=a_k+\beta c+e_{kc}\). Within subject \(k\), the difference is \(D_k=Z_{k1}-Z_{k0}=\beta+(e_{k1}-e_{k0})\): the baseline \(a_k\) drops out.

If the two errors have equal variance \(\sigma^2\) and correlation \(r\), then

$$ \operatorname{Var}(D_k) =\operatorname{Var}(e_{k1})+\operatorname{Var}(e_{k0}) -2\operatorname{Cov}(e_{k1},e_{k0}) =2\sigma^2(1-r). $$

An unpaired comparison also carries variation in subject baselines. In this simplified argument, if \(a_k\) has variance \(\tau^2\) and subjects are independent across conditions, the variance of a difference between two group means with \(K\) subjects per condition is \(2(\tau^2+\sigma^2)/K\). The paired difference of means has variance \(2\sigma^2(1-r)/K\). Pairing is especially helpful when baseline variation \(\tau^2\) is substantial and measurements within a subject are positively correlated.

This argument is an explanation, not an exact variance formula for a negative binomial GLM. Counts have a mean-dependent variance, and the fitted model estimates effects and dispersion from the observed data. Pairing does not guarantee a smaller standard error: noisy within-subject measurements, weak correlation, missing samples, or a poorly specified model can reduce or eliminate the benefit.

Worked Example: Reading Within-Subject Count Ratios

Consider hypothetical normalized counts for one gene, with size factor 1 in all samples:

SubjectControl countTreated countTreated/control ratio
1801201.50
240601.50
31602401.50

For each subject the observed log ratio is \(\log(1.50)=0.4055\). The mean of these three log ratios is also \((0.4055+0.4055+0.4055)/3=0.4055\), corresponding to a fold change of \(\exp(0.4055)=1.50\). The counts differ greatly in scale across subjects, yet their within-subject ratios agree.

This calculation illustrates the paired comparison; it is not a substitute for fitting the negative binomial GLM. In an actual analysis, sampling variability, dispersion, and the offsets affect the fitted coefficient and its uncertainty. If size factors differed between the two samples, the raw count ratio would not equal the offset-adjusted ratio.

Interpreting a Fitted Condition Effect

After fitting the model for a gene, the estimated condition coefficient and its standard error can be used in a Wald calculation. The coefficient estimates the log fold change, and exponentiating it puts the effect on a fold-change scale. A confidence interval on the log scale can also be exponentiated, preserving the ordering because the exponential function is increasing.

Worked Example: A Hypothetical Wald Result

Suppose a fitted paired model for one gene gives \(\hat{\beta}_g=0.4055\) with \(SE=0.1800\). Under the null value \(\beta_g=0\), the Wald statistic is

$$ z=\frac{\hat{\beta}_g-0}{SE} =\frac{0.4055}{0.1800}=2.253. $$

Using a two-sided standard normal reference distribution gives \(p=2P(Z\geq 2.253)\approx0.02427\), where \(Z\sim N(0,1)\). This p-value measures how unusual a statistic at least this far from zero would be under the null model and the Wald approximation; it is not the probability that the null hypothesis is true.

A 95% Wald interval on the log scale is \(0.4055\pm1.96(0.1800)=0.4055\pm0.3528\), or \([0.0527,0.7583]\). Exponentiating gives the fold-change interval \([\exp(0.0527),\exp(0.7583)]=[1.054,2.135]\). In this hypothetical analysis, the estimated treated-to-control expected-count ratio is \(1.50\), with a 95% Wald interval from about \(1.05\) to \(2.14\), conditional on the model and its assumptions.

Diagnostics, Alternatives, and Failure Modes

The subject term is only useful when it represents the sample structure correctly. Before interpreting results, confirm that sample labels identify the right subject-condition pairs, that condition is not confounded with subject or another design variable, and that the size factors and count-model diagnostics are reasonable. Residual patterns or unusually influential samples may indicate that the model does not capture the data well; dispersion and extreme-count diagnostics remain relevant in paired analyses.

A missing member of a pair need not make a GLM mathematically impossible, but it weakens the within-subject comparison and can make the design less informative. If an entire condition occurs only for different subjects, condition is confounded with subject and the paired condition effect cannot be separated from subject baselines. Likewise, a subject term cannot correct a batch effect that is perfectly aligned with condition.

ApproachWhat it representsMain consideration
Paired fixed-effect GLMA separate baseline for each subject and a shared condition effectControls stable subject differences when the design supports within-subject comparisons
Unpaired GLMCondition groups without subject termsMay lose precision or be confounded by baseline differences when samples are paired
Random-intercept modelSubject baselines drawn from a distributionUses a different model for subject variation and requires assumptions about that distribution

The fixed-effect design is a direct way to control for the observed subjects, but it estimates a baseline term for each subject and relies on the shared-effect assumption. An unpaired model may be appropriate when samples are genuinely independent, but ignoring known pairing can discard useful structure. A random-intercept approach is an alternative when its assumptions and intended generalization are appropriate; it is not interchangeable with simply adding fixed subject indicators.

  • Common mistake: treating paired samples as independent replicates. The wrong reasoning is that all control samples form one group and all treated samples another, so subject identity is irrelevant. The correction is to encode subject in the design when the same subjects contribute both conditions; otherwise stable baseline variation is left in the condition comparison.
  • Common mistake: claiming pairing always increases power. Pairing can help when within-subject comparisons remove substantial baseline variation, but it does not guarantee a smaller standard error. Examine the design and model diagnostics rather than assuming a benefit.
  • Common mistake: using subject indicators when condition is confounded with subject. If each subject appears in only one condition, subject and condition may be inseparable. Check the design matrix and sample allocation before interpreting a condition coefficient.
Key takeaway: In a paired negative binomial GLM, subject indicators absorb gene-specific baseline differences and the condition coefficient estimates a shared within-subject log fold change. Pairing can improve precision, but only when the sample structure, offsets, and model assumptions support that comparison.

Check Your Understanding

Answer each question, then compare with the solutions below.

  1. In the paired model, what does \(\exp(\beta_g)\) represent?
  2. Why does subtracting the two log-mean equations for the same subject remove the subject effect?
  3. A gene has \(\hat{\beta}_g=-0.2231\). What is the estimated fold change for condition 1 versus condition 0, approximately?
  4. What design problem arises if every subject is observed in only one condition and the subject indicators are included?

Solutions. (1) It is the condition-1 versus condition-0 expected-count ratio after accounting for the subject and size-factor terms. (2) The same subject-specific baseline appears in both equations, so its difference is zero; the offset difference remains unless the size factors are equal. (3) \(\exp(-0.2231)\approx0.8000\), so the expected count in condition 1 is estimated to be 0.80 times that in condition 0, or about 20% lower, conditional on the model. (4) Condition may be confounded with subject, so the design cannot distinguish the condition effect from subject baselines; the condition coefficient is not separately identifiable in a fully confounded design.

Clinical Biostats