Small Reference Arms and Unstable Predictive Distributions

Why Small Reference Arms Need Special Attention

Reference-based imputation uses information from a chosen reference arm to model outcomes that are missing in another arm. When the reference arm is small, its estimated mean and covariance can be imprecise. That imprecision matters: an imputed outcome is not generated from a known reference distribution, but from one whose parameters have themselves been estimated. Treating those estimates as fixed can make predictive distributions too narrow and conceal how sensitive imputations are to a few reference participants.

The issue is especially important for repeated outcomes. A covariance estimate governs how the model relates visits and how uncertainty in one outcome is carried into predictions of another. With few participants, a plausible change in an estimated correlation can substantially change a conditional prediction. A reference-arm sample size should therefore be assessed against the complexity of the distribution being estimated, not just against the total trial sample size.

Definition: Parameter uncertainty is uncertainty about the unknown mean and covariance of the reference distribution, given the observed reference-arm data. Outcome variability is the remaining variation among individual outcomes even if those parameters were known. A predictive distribution for an imputed outcome should account for both sources, under the model used to generate it.

Here, \(n\) denotes the number of independent reference-arm participants contributing the relevant outcome history, not necessarily the number randomized to that arm. In a longitudinal model, the relevant distribution may be conditional on observed histories, covariates, or visit patterns; its effective information can be less than the nominal arm size. The calculations below first use a simple normal model so the contribution of estimation uncertainty is explicit.

A Small-Sample Predictive Distribution

Suppose \(n\) observed reference-arm outcomes \(Y_1,\ldots,Y_n\) are independent draws from \(N(\mu,\sigma^2)\). Assume the model is appropriate for these outcomes and there are no unmodeled strata or covariates. Let \(\bar{x}\) be their sample mean and \(s\) their sample standard deviation, calculated with denominator \(n-1\). Under the usual normal model with prior density proportional to \(1/\sigma^2\) for \((\mu,\sigma^2)\), the predictive distribution for one new outcome is:

Y_{\mathrm{new}}\mid \text{reference data} \sim t_{n-1}\left(\bar{x},\ s\sqrt{1+\frac{1}{n}}\right).

The second argument here is the Student t scale, not its standard deviation. The factor \(s\) represents estimated individual outcome variation; the additional factor \(\sqrt{1+1/n}\) accounts for uncertainty in the estimated mean. The predictive distribution also has heavier tails than a normal distribution with the same scale when \(n\) is small, reflecting uncertainty in the estimated variance. The result relies on the normal sampling model and the stated prior; it is not a universal correction for every fitted reference model.

Key distinction: A plug-in imputation draws from \(N(\bar{x},s^2)\), treating the estimated mean and variance as known. The small-sample predictive distribution instead allows for uncertainty in those estimates. For a single new outcome, its scale includes the mean-uncertainty factor, and its tails are t rather than normal.

Worked Example: Predicting One Missing Outcome From Five Reference Participants

Suppose five hypothetical reference-arm participants have outcomes with \(\bar{x}=10\) points and \(s=4\) points. Assume the independent normal model and prior described above are reasonable for this illustration. Find a 95% predictive interval for one missing outcome.

Here \(n=5\), so the predictive degrees of freedom are \(n-1=4\). The predictive scale is:

s\sqrt{1+\frac{1}{n}} =4\sqrt{1+\frac{1}{5}} =4\sqrt{1.2} \approx 4.38178046\ \text{points}.

The 97.5th percentile of a \(t_4\) distribution is approximately 2.776445. Retaining the unrounded scale through the multiplication, the margin is:

2.776445\left(4\sqrt{1.2}\right) \approx 12.1658\ \text{points}.

Thus the interval is:

10\ \mathbin{\pm}\ 12.1658 =[-2.1658,\ 22.1658] \approx [-2.17,\ 22.17]\ \text{points}.

This interval describes a predictive range for one outcome under the stated model, not a confidence interval for the reference-arm mean. Its breadth reflects both variation among individual outcomes and uncertainty from estimating the distribution with only five participants. A negative lower endpoint may be impossible for some clinical outcomes; that would be a warning that a normal model on this scale may be unsuitable, rather than a reason to silently truncate the interval.

Mean Uncertainty and Imputed Values

Uncertainty in the reference mean is also visible when estimating a mean, although a confidence interval for a mean is not itself a predictive interval for an individual. For normal observations with unknown variance, a two-sided \(100(1-\alpha)\%\) confidence interval for \(\mu\) is \(\bar{x}\pm t_{n-1,1-\alpha/2}s/\sqrt{n}\). The standard error \(s/\sqrt{n}\) shrinks as the reference sample grows; the predictive scale for an individual does not shrink in the same way, because individual outcome variability remains.

Worked Example: Separating Mean Precision From Individual Prediction

In a second hypothetical reference arm, \(n=4\), \(\bar{x}=6\) points, and \(s=2\) points. Assume independent normal outcomes. Calculate a 95% confidence interval for the reference mean, then compare its uncertainty with the scale for an individual predictive outcome.

For the mean, \(df=3\), \(t_{3,0.975}\approx3.182446\), and \(SE=s/\sqrt{n}=2/2=1\) point. Therefore:

6\pm3.182446(1) =[2.817554,\ 9.182446] \approx[2.818,\ 9.182]\ \text{points}.

This is a confidence interval for the population mean under repeated sampling and the assumed model. For one new outcome, the predictive scale is larger:

s\sqrt{1+\frac{1}{n}} =2\sqrt{1.25} \approx2.236\ \text{points}, \qquad df=3.

The mean interval concerns uncertainty about a common location; the predictive distribution also includes the outcome's own variability. In an imputation, drawing a parameter and then an outcome is one way to preserve both sources of uncertainty. Repeating the same fitted mean and variance for every imputation would not represent parameter uncertainty.

Covariance Uncertainty in Repeated Outcomes

For a vector of \(p\) outcomes, a multivariate normal model uses a mean vector and a covariance matrix. The covariance matrix describes both marginal variances and the relationships among visits. Estimating it requires more information than estimating one variance. In particular, a sample covariance matrix from \(n\) complete vectors has rank at most \(n-1\). If \(n\leq p\), it is singular, so an unrestricted covariance estimate cannot support the usual multivariate calculations that require its inverse.

A useful reference calculation makes the impact of finite-sample estimation explicit. Assume \(n\) independent, complete \(p\)-dimensional reference outcomes follow a multivariate normal distribution with unknown mean vector and positive-definite covariance matrix. Let \(\bar{\mathbf{x}}\) be the sample mean vector and \(S\) the sample covariance matrix with denominator \(n-1\). Under the multivariate analogue of the prior used above, when \(n>p\), the predictive distribution for a new vector is a multivariate t with \(n-p\) degrees of freedom and scale matrix:

\mathbf{Y}_{\mathrm{new}}\mid\text{data} \sim t_{n-p}\left( \bar{\mathbf{x}}, \frac{(n+1)(n-1)}{n(n-p)}S \right).

In this parameterization, the matrix argument is the scale matrix. If the degrees of freedom exceed 2, the predictive covariance matrix is the scale matrix multiplied by \((n-p)/(n-p-2)\). A derivation follows from estimating the mean conditionally on the covariance and then integrating over covariance uncertainty: the posterior for the covariance is inverse-Wishart with degrees of freedom \(n-1\) and sum-of-squares matrix \((n-1)S\). Integrating the uncertain mean contributes the factor \(1+1/n\); integrating the covariance gives a multivariate t with \(n-p\) degrees of freedom. Thus small \(n\) relative to \(p\) produces heavier tails and greater predictive uncertainty. This result is a reference calculation, not a substitute for the fitted conditional model used in a particular imputation.

Worked Example: Predictive Inflation for Three Outcomes

Suppose eight hypothetical reference participants each contribute a complete three-outcome vector. Assume the independent multivariate normal model is appropriate, and suppose the sample covariance matrix is \(S=\operatorname{diag}(1,4,9)\) in squared outcome units. Here \(n=8\), \(p=3\), and the predictive degrees of freedom are \(8-3=5\). The scale multiplier is:

\frac{(n+1)(n-1)}{n(n-p)} =\frac{9\cdot7}{8\cdot5} =1.575.

The predictive scale matrix is therefore \(1.575S=\operatorname{diag}(1.575,6.300,14.175)\). Since \(df=5>2\), the predictive covariance matrix is \(5/(5-2)=5/3\) times that scale matrix:

\operatorname{Cov}(\mathbf{Y}_{\mathrm{new}}\mid\text{data}) =\frac{5}{3}(1.575S) =2.625S =\operatorname{diag}(2.625,\ 10.500,\ 23.625).

The predictive variances are larger than the corresponding entries of \(S\), because prediction includes uncertainty in estimated parameters as well as individual variability. The t distribution also has heavier tails than a normal distribution. These calculations rely on complete independent vectors and an unrestricted multivariate normal model; they should not be applied mechanically to an unbalanced longitudinal data set with partially observed histories.

Worked Example: How a Covariance Estimate Changes a Conditional Imputation

Consider two hypothetical covariance estimates for a pair of visits in a small reference arm. Both have estimated standard deviations of 2 and 3 points, so their variances are 4 and 9 points squared. One estimate has correlation 0.60; the other has correlation 0.90. Under a bivariate normal model, the conditional mean of the second outcome given the first is \(x_2\)'s mean plus the covariance-to-variance ratio times the first outcome's deviation from its mean. The conditional variance is \(\sigma_{22}-\sigma_{21}^2/\sigma_{11}\).

For correlation 0.60, the estimated covariance is \(0.60(2)(3)=3.6\) points squared. If the first outcome is 7 points and its reference mean is 5 points, the conditional mean shift is:

\frac{3.6}{4}(7-5)=0.9(2)=1.8\ \text{points}.

If the second outcome's reference mean is 10 points, the conditional mean is 11.8 points. Its conditional variance and standard deviation are:

9-\frac{3.6^2}{4}=5.76\ \text{points}^2, \qquad \sqrt{5.76}=2.4\ \text{points}.

For correlation 0.90, the covariance is \(0.90(2)(3)=5.4\) points squared. The corresponding conditional mean is \(10+(5.4/4)(2)=12.7\) points, and the conditional variance is \(9-5.4^2/4=1.71\) points squared, giving a conditional standard deviation of approximately 1.308 points. Both covariance matrices are positive definite because their correlations have magnitude below 1. The differing predictions demonstrate why a small-sample correlation estimate can materially affect an imputed value and its dispersion. They are alternative estimates, not evidence that either correlation is correct.

Diagnostics, Alternatives, and Limits

No single sample-size cutoff establishes that a reference distribution is stable. Diagnostics should connect the amount of information to the number and structure of parameters being estimated. In particular, inspect:

Effective information. Count participants contributing the relevant history, and examine missingness and visit-pattern imbalance. A nominally large arm can still provide little information for a particular conditional covariance.
Covariance feasibility and conditioning. Check that the estimated matrix is positive definite when required, and inspect its eigenvalues or condition number. A very small eigenvalue or a large condition number can signal near-singularity and sensitivity to minor data changes.
Parameter sensitivity. Compare imputed distributions under reasonable alternative covariance structures or fitting choices. Large shifts in conditional means, variances, or tail behavior indicate that conclusions rely strongly on a weakly estimated structure.
Predictive behavior. Review simulated or imputed outcomes for implausible ranges, extreme tails, and unrealistic visit-to-visit patterns. Such behavior may indicate model mismatch as well as imprecision.

A direct comparison is with a plug-in predictive distribution, which treats the estimated parameters as fixed. It is simpler but generally understates uncertainty when parameters are imprecise. Parameter-draw multiple imputation carries parameter uncertainty into completed data sets, while the particular draw mechanism must match the model and account for covariance estimation. A more constrained covariance structure can reduce estimation instability but imposes stronger assumptions. Shrinkage or regularization can stabilize estimates, but changes the estimator and requires a defensible specification; it does not create information that the reference arm lacks.

These procedures do not guarantee a trustworthy imputation. The normal model may poorly represent skewed, bounded, or heavy-tailed outcomes. A covariance estimate may be unstable even when it is technically positive definite. Further, parameter uncertainty within a chosen reference model is not the same as uncertainty about whether that reference arm or its assumptions are appropriate. Interpret imputed outcomes and trial inferences in light of both the data available and the modeling choices.

Common pitfall: “The sample mean and covariance are estimated, so I can substitute them into a normal distribution and proceed as if they were known.” This reasoning omits uncertainty in the estimated parameters. For a small reference arm, use an imputation procedure that carries parameter uncertainty through the predictive distribution, and check whether the covariance structure is estimable and stable. A plug-in calculation can be a useful diagnostic comparison, but should not be mistaken for a full predictive analysis.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a univariate normal reference model, what are the degrees of freedom and scale for predicting one new outcome from \(n\) observations with sample standard deviation \(s\)?
With \(n=6\) complete reference vectors and \(p=6\) outcomes, can the unrestricted sample covariance matrix be inverted? Explain.
For the three-outcome example above, why is the predictive covariance not equal to the predictive scale matrix?
A plug-in imputation uses the same fitted mean and covariance for every imputed data set. What uncertainty can this miss?

Solutions. (1) Under the stated normal model and prior, the predictive degrees of freedom are \(n-1\), and the scale is \(s\sqrt{1+1/n}\). (2) No. The sample covariance rank is at most \(n-1=5\), which is less than \(p=6\), so it is singular. (3) The multivariate t scale is not its covariance; with \(df=5\), the covariance equals the scale matrix multiplied by \(5/(5-2)=5/3\). (4) It can miss uncertainty in the estimated mean and covariance, making the imputed predictive distribution appear more precise than warranted by the reference-arm data.