Intermittently Missing Longitudinal Assays

Why Intermittent Missingness Needs Care

A participant may provide an omics assay at baseline, miss the next scheduled collection, and provide an assay again later. This is intermittent missingness: an assay is absent at one or more occasions between observed occasions. It differs from monotone dropout, where measurements stop after a participant leaves follow-up. The distinction matters because a later observed assay can inform a participant’s trajectory even when an intermediate value is absent.

An analysis that discards every participant with any missing assay wastes observed information and may change the population being analyzed. But simply using the available values is not automatically valid either. The key question is how assay availability relates to the observed and unobserved assay values. The earlier material on Informative Dropout In Longitudinal Omics concerns a related threat; here the emphasis is on gaps within otherwise observed repeated measurements.

Definition: Let \(Y_{ij}\) be the assay value for participant \(i\) at scheduled occasion \(j\), for \(j=1,\ldots,J\). Let \(R_{ij}=1\) if that assay is observed and \(R_{ij}=0\) otherwise. Write \(Y_i^{\mathrm{obs}}\) and \(Y_i^{\mathrm{mis}}\) for the observed and missing components of that participant’s assay vector, and \(X_i\) for observed covariates. Missingness is MCAR if the observation pattern is independent of the assay values and covariates; MAR if, conditional on observed assays and covariates, the pattern does not depend on the missing assay values; and MNAR if it still depends on unobserved assay values after conditioning on the observed information.

Formally, MAR means the probability of a pattern \(R_i\), given the full assay vector and covariates, can be written as a function of observed data only:

P(R_i\mid Y_i,X_i)=P(R_i\mid Y_i^{\mathrm{obs}},X_i).

For example, if the probability of missing the second assay depends on the first assay, which was observed, that dependence can be MAR. If it depends on the unobserved second assay value itself even after accounting for observed assays and covariates, it is MNAR. A missingness pattern need not be monotone to satisfy MAR.

Likelihood With Gaps in the Assay Record

Likelihood-based methods use the distribution of the values that were actually observed. Suppose a longitudinal model specifies a density \(f_\theta(Y_i\mid X_i)\), with parameters \(\theta\) governing the assay trajectory. The observed-data contribution to the likelihood is obtained by integrating over the missing assay values:

L_i(\theta)=f_\theta(Y_i^{\mathrm{obs}}\mid X_i) =\int f_\theta(Y_i^{\mathrm{obs}},Y_i^{\mathrm{mis}}\mid X_i)\,dY_i^{\mathrm{mis}}.

This is not the same as filling in a guessed assay value. The model accounts for the range of possible missing values through integration. In a correctly specified multivariate normal model, for instance, the likelihood contribution from observed visits is the marginal normal density for those visits, using the corresponding subvector of means and submatrix of covariances.

Ignoring the missingness mechanism in this likelihood requires conditions. The assay model must be correctly specified, including the relevant mean and covariance structure; the observation process must be MAR (MCAR is a special case); and the parameters of the assay model and missingness mechanism must be distinct. The visit schedule and covariate information used in the model must also be handled appropriately. Under these conditions, the missingness-process factor does not change the likelihood information about \(\theta\). MAR is an assumption about the data-generating process, not something established merely by fitting a model.

Worked Example: A Likelihood Contribution With One Missing Visit

Hypothetical data. Consider one participant measured at three occasions. Suppose the assay vector follows a multivariate normal model with mean vector \((1,2,3)\) assay units and covariance matrix

\Sigma= \begin{pmatrix} 1&0.5&0.3\\ 0.5&1&0.4\\ 0.3&0.4&1 \end{pmatrix}.

The observed values are \(Y_{i1}=2\) and \(Y_{i3}=4\); \(Y_{i2}\) is missing. Assume for this calculation that the model parameters are known and that the observation pattern is ignorable under MAR. The observed residual vector is \((2-1,4-3)=(1,1)\), and its covariance matrix is the submatrix for occasions 1 and 3:

\Sigma_{\mathrm{obs}}= \begin{pmatrix} 1&0.3\\ 0.3&1 \end{pmatrix}.

Its determinant is \(1-(0.3)^2=0.91\). The quadratic form for the residuals is

(1,1)\Sigma_{\mathrm{obs}}^{-1}(1,1)^\mathsf{T} =\frac{2}{1+0.3} =1.5385.

Therefore the observed-data density contribution is

\frac{1}{2\pi\sqrt{0.91}} \exp\left(-\frac{1.5385}{2}\right) \approx 0.0773.

This calculation uses the joint distribution of the two observed occasions. It does not set the missing second assay to its model mean of 2, nor does it need to reconstruct that assay before forming the likelihood. In practice, model parameters are estimated from all participants’ observed-data contributions.

Multiple Imputation: A Different Route

Multiple imputation (MI) creates \(M\) completed data sets by drawing plausible values for the missing assays from an imputation model. The analysis model is fitted separately in each completed data set. The variation between imputations contributes to uncertainty about the result; a single completed data set treated as fully observed does not capture that uncertainty.

For a scalar estimate \(Q_m\) and its estimated variance \(U_m\) in imputation \(m\), Rubin’s rules combine the results as follows:

\bar{Q}=\frac{1}{M}\sum_{m=1}^{M}Q_m,\qquad \bar{U}=\frac{1}{M}\sum_{m=1}^{M}U_m,\qquad B=\frac{1}{M-1}\sum_{m=1}^{M}(Q_m-\bar{Q})^2,\qquad T=\bar{U}+\left(1+\frac{1}{M}\right)B.

Here \(\bar{Q}\) is the pooled estimate, \(\bar{U}\) is the average within-imputation variance, \(B\) is the between-imputation variance, and \(T\) is the total variance. MI relies on a suitable imputation model that preserves important features of the repeated data, including time, covariates, and relevant within-person dependence. For valid inference under MAR, the imputation model must also be sufficiently compatible with the analysis and include variables needed to explain missingness and the outcome.

Worked Example: Pooling an Imputed Longitudinal Association

Hypothetical data. Suppose five imputations produce estimates of an association between an assay and a clinical outcome, measured as outcome units per one assay unit: \(Q_m=0.20, 0.30, 0.40, 0.50, 0.60\). Each analysis has estimated variance \(U_m=0.04\) outcome-units-squared. Then \(\bar{Q}=(0.20+0.30+0.40+0.50+0.60)/5=0.40\), and \(\bar{U}=0.04\).

The deviations from 0.40 are \(-0.20,-0.10,0,0.10,0.20\), so

B=\frac{0.20^2+0.10^2+0^2+0.10^2+0.20^2}{4} =\frac{0.10}{4}=0.025.

Thus \(T=0.04+(1+1/5)(0.025)=0.070\), and the pooled standard error is \(\sqrt{0.070}=0.2646\) outcome units per assay unit. Using the usual large-sample normal approximation, a 95% interval is \(0.40\pm1.96(0.2646)=(-0.119,0.919)\), approximately. This interval describes uncertainty under the imputation and analysis assumptions; it is not a probability statement about the fixed association parameter. With few imputations or limited complete-data degrees of freedom, a finite-degrees-of-freedom \(t\) reference is preferable to the normal approximation.

Diagnosing Assumptions and Comparing Approaches

Observed data can reveal patterns that deserve investigation, but they generally cannot distinguish MAR from MNAR conclusively. Useful checks include tabulating missingness by occasion and participant, comparing observed baseline covariates and earlier assays across later observation patterns, and examining whether assay availability varies by batch, clinical status, or other recorded factors. Such comparisons can identify predictors that should be included in the longitudinal or imputation model. They cannot show that missingness is independent of the assay value that was never observed.

A complete-case analysis uses only participants with every scheduled assay. It is simple, but loses information and can be biased if completeness is related to the assay trajectory or covariates. Likelihood analysis uses each participant’s observed measurements under the specified model and MAR assumptions. MI can accommodate a broad analysis model and propagates imputation uncertainty when performed properly, but depends on a credible imputation model. Neither likelihood nor MI automatically fixes MNAR. Weighting methods instead reweight observed records using a model for observation probabilities; they also require assumptions, including a correctly specified observation model and adequate positivity. They are not a universal remedy for unobserved-value dependence.

In high-dimensional omics settings, missingness may differ by feature, assay platform, or batch. A single mechanism for every feature may be implausible. Values absent because an assay is below a detection limit may be censored rather than ordinary missing observations; replacing them with a constant or treating them as MAR can distort the distribution. The measurement process may need explicit modeling. Also, if the chance of collecting an assay depends on the participant’s evolving clinical state, the observation process may carry information beyond a simple intermittent gap.

Worked Example: A Pattern-Mixture Sensitivity Analysis

Hypothetical data. At one visit, 10 participants were eligible for an assay. Six observed values sum to 66 assay units, giving an observed mean of \(66/6=11\) assay units. For a transparent illustration, suppose a MAR imputation model predicts a mean of 10 assay units for each of the four missing values. The resulting completed-data mean is

\frac{66+4(10)}{10}=10.6\text{ assay units}.

Now consider a sensitivity scenario in which, conditional on the observed information, missing values average 2 assay units below those MAR predictions. The assumed mean for each missing value is then \(10-2=8\), and the completed-data mean becomes

\frac{66+4(8)}{10}=\frac{98}{10}=9.8\text{ assay units}.

The 0.8-assay-unit difference between 10.6 and 9.8 is induced by the sensitivity assumption, not learned from the missing values. Varying this offset over a scientifically plausible range shows how conclusions depend on departures from MAR. It does not identify which scenario is true.

Deriving the Ignorability Result

The likelihood result follows by factoring the joint model for assays and missingness. Let \(\psi\) denote parameters of the observation process. The observed joint density for a participant is

f_{\theta,\psi}(Y_i^{\mathrm{obs}},R_i\mid X_i) =\int f_\theta(Y_i^{\mathrm{obs}},Y_i^{\mathrm{mis}}\mid X_i) f_\psi(R_i\mid Y_i^{\mathrm{obs}},Y_i^{\mathrm{mis}},X_i) \,dY_i^{\mathrm{mis}}.

Under MAR, the second factor inside the integral does not depend on \(Y_i^{\mathrm{mis}}\), so it can be taken outside:

f_{\theta,\psi}(Y_i^{\mathrm{obs}},R_i\mid X_i) =f_\psi(R_i\mid Y_i^{\mathrm{obs}},X_i) \int f_\theta(Y_i^{\mathrm{obs}},Y_i^{\mathrm{mis}}\mid X_i) \,dY_i^{\mathrm{mis}} =f_\psi(R_i\mid Y_i^{\mathrm{obs}},X_i)f_\theta(Y_i^{\mathrm{obs}}\mid X_i).

If \(\theta\) and \(\psi\) are distinct parameters, the first factor contains no information about \(\theta\) once observed data are given. Thus the likelihood for \(\theta\) can be based on \(f_\theta(Y_i^{\mathrm{obs}}\mid X_i)\). If missingness depends on the unobserved assay values, the factor generally remains inside the integral and this simplification does not follow.

Key takeaway: Intermittent gaps do not by themselves invalidate longitudinal analysis. Likelihood and MI can use incomplete assay records under appropriate model and MAR conditions. Because MAR cannot generally be verified from observed data alone, report the assumptions and assess whether plausible MNAR departures could change the substantive conclusion.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A participant has assays at visits 1 and 3 but not visit 2. What makes this intermittent missingness, and is it necessarily MNAR?
Under MAR, why does the missingness factor separate from the observed-data likelihood for the assay-model parameters?
In the pooling example, what happens to the pooled standard error if between-imputation variance increases while the within-imputation variance stays fixed?
Can comparisons of observed assay values between participants with different missingness patterns prove MAR? Explain.
In the sensitivity example, what completed-data mean results if the four missing values are assumed to average 9 assay units instead of 8?

Solutions. (1) The missing visit is between two observed visits, so the pattern is intermittent; it may be MCAR, MAR, or MNAR depending on what drives assay availability. (2) Under MAR the missingness probability depends only on observed assays and covariates, so it can be taken outside the integral over missing assay values; with distinct model parameters it does not alter the likelihood for the assay parameters. (3) Total variance \(T=\bar{U}+(1+1/M)B\) increases, so the pooled standard error increases. (4) No. Observed comparisons can reveal associations with recorded information, but cannot establish how missingness depends on unobserved values. (5) The mean is \([66+4(9)]/10=102/10=10.2\) assay units.