Means and Dependence Describe Different Features
For clustered responses, it is useful to separate two questions. The first is about the average response at each measurement occasion or for each member of a cluster. The second is about how responses from the same cluster vary together. These are different features of a joint distribution: knowing one does not, in general, determine the other.
Write the responses in cluster \(i\) as \(Y_{i1},\ldots,Y_{im_i}\). Their marginal means are \(\mu_{ij}=E[Y_{ij}]\). Their joint distribution describes probabilities involving two or more responses at once. A covariance such as \(\operatorname{Cov}(Y_{ij},Y_{ik})\) summarizes part of that joint behavior, but it is not itself a marginal mean.
This distinction is central in marginal mean modeling. A model for \(\mu_{ij}\) specifies a feature of each response’s distribution. A covariance model describes how responses vary together. In general, choosing a different joint distribution can change the covariance while leaving all the marginal distributions—and hence all the marginal means—unchanged.
Why Preserving Marginals Preserves Means
A joint distribution with specified marginal distributions is sometimes called a coupling of those marginals. The result says that changing the coupling cannot change a marginal mean if the marginal distributions themselves are kept fixed.
Proof. Let the two joint distributions be \(P\) and \(Q\). For any response \(Y_j\), let its common marginal distribution be \(F_j\). The expectation of \(Y_j\) under either joint distribution is the integral of the identity function with respect to that marginal:
These integrals exist by the assumed finiteness of \(E[|Y_j|]\). The joint distributions can assign different probabilities to combinations of response values, but summing or integrating over the other coordinates leaves the same marginal distribution \(F_j\). Thus the mean of each response is unchanged. \(\square\)
This is a direct consequence of marginalization, not an approximation or a large-sample statement. It applies to binary, count, and continuous responses whenever the relevant means exist. It does not say that the marginal distributions can be chosen arbitrarily in every joint model. A model may impose constraints linking them; that issue is separate from whether dependence alone sets a mean.
Worked Example: Three Joint Tables With the Same Means
Let \(Y_1\) and \(Y_2\) be binary responses, with \(P(Y_1=1)=P(Y_2=1)=0.4\). Compare three possible joint distributions. In each table, rows give \(Y_1\) and columns give \(Y_2\).
Independent responses: independence gives \(P(Y_1=a,Y_2=b)=P(Y_1=a)P(Y_2=b)\). The joint probabilities are \(P(0,0)=0.6(0.6)=0.36\), \(P(0,1)=0.6(0.4)=0.24\), \(P(1,0)=0.4(0.6)=0.24\), and \(P(1,1)=0.4(0.4)=0.16\). They sum to 1, and each response has mean 0.4. Since \(E[Y_1Y_2]=0.16\),
Responses equal with probability 1: set \(P(0,0)=0.6\) and \(P(1,1)=0.4\), with the other two probabilities zero. Each response still has mean 0.4. Now \(E[Y_1Y_2]=0.4\), so the covariance is \(0.4-0.16=0.24\).
Responses never both equal 1: set \(P(0,0)=0.2\), \(P(0,1)=0.4\), \(P(1,0)=0.4\), and \(P(1,1)=0\). These probabilities sum to 1, and both marginal probabilities of 1 remain 0.4. Here \(E[Y_1Y_2]=0\), so the covariance is \(0-0.16=-0.16\).
The same two marginal means are compatible with positive, zero, or negative covariance. In particular, the dependence pattern is not enough to recover the means: the equal-response pattern has means 0.4, but that same pattern could be used with any common binary mean \(p\) by assigning probabilities \(1-p\) to \((0,0)\) and \(p\) to \((1,1)\).
What Covariance Does—and Does Not—Tell You
For responses with finite second moments, covariance is defined by
This identity relates a second-order feature of the joint distribution to the two means and a cross-moment. Knowing the covariance alone does not generally identify either mean: the identity also contains the cross-moment and both means. Nor does specifying a covariance matrix generally specify the entire joint distribution. For example, distinct joint distributions can share the same means and covariances while differing in other features.
Covariance is not entirely unconstrained by the marginals. For binary responses, for instance, probabilities must remain between 0 and 1, restricting which covariances are possible for given marginal probabilities. The point is not that means and covariance can always be selected independently. Rather, covariance is not a general rule that determines the marginal means. Constraints come from the full set of distributional assumptions, not from the word “dependence” by itself.
Worked Example: Independence Does Not Specify a Mean
Consider two possible models for a binary pair. In Model A, \(Y_1\) and \(Y_2\) are independent Bernoulli responses with success probability 0.2. In Model B, they are independent Bernoulli responses with success probability 0.8.
For a Bernoulli response with success probability \(p\), its mean is \(E[Y_j]=p\). Thus the marginal means are 0.2 in Model A and 0.8 in Model B. Both models have the same dependence description: independence. In each model,
Independence therefore does not set a particular mean. It says that the joint probabilities factor into the marginal probabilities; those marginal probabilities still need to be specified. Similarly, a zero covariance is not by itself a statement of independence in general. It only states that the centered responses have expectation of their product equal to zero.
Conditional Models and the Law of Total Covariance
A shared latent variable or cluster-level factor can induce dependence. The law of total covariance shows how this can happen. It also separates this source of covariance from the marginal means.
Proof. Write \(a_j(U)=E[Y_j\mid U]\) and \(a_k(U)=E[Y_k\mid U]\). Decompose each response into its conditional residual and conditional mean:
Multiply these expressions and take expectations. Each cross-term has expectation zero. For example,
because \(E[Y_j-a_j(U)\mid U]=0\). The other cross-term is zero by the same argument, conditioning on \(U\). The first product term has expectation \(E[\operatorname{Cov}(Y_j,Y_k\mid U)]\), and the last has expectation \(\operatorname{Cov}(a_j(U),a_k(U))\). Adding the two terms gives the stated identity. Finite second moments ensure the products and covariances are integrable. \(\square\)
The corresponding law of total expectation is \(E[Y_j]=E[E[Y_j\mid U]]\), provided \(E[|Y_j|]\) is finite. A shared \(U\) can make the conditional means move together, producing the second covariance term in the theorem. But the unconditional mean still requires the distribution of \(U\) and the conditional mean function. The covariance decomposition alone does not supply them.
Worked Example: A Shared Latent Probability Creates Covariance
Let \(U\) take values 0 and 1 with probability \(1/2\) each. Conditional on \(U\), let \(Y_1\) and \(Y_2\) be independent Bernoulli responses with success probability \(p(U)\), where \(p(0)=0.2\) and \(p(1)=0.8\).
For either response, the law of total expectation gives
Conditional independence makes \(E[Y_1Y_2\mid U]=p(U)^2\). Therefore
It follows that \(\operatorname{Cov}(Y_1,Y_2)=0.34-(0.5)(0.5)=0.09\). The total-covariance identity gives the same result: the conditional covariance is zero, and the covariance of the two conditional means is \(\operatorname{Var}(p(U))=E[p(U)^2]-E[p(U)]^2=0.34-0.25=0.09\). The mean 0.5 comes from averaging the conditional probability over the distribution of \(U\); the positive covariance comes from the fact that this shared probability varies across clusters.
If instead \(p(0)=0.1\) and \(p(1)=0.9\), with the same equally likely values of \(U\), the marginal mean remains \(\tfrac12(0.1)+\tfrac12(0.9)=0.5\). The covariance becomes \(\tfrac12(0.01+0.81)-0.25=0.16\). Thus even within this conditional construction, the same marginal mean can accompany different dependence.
When Joint Assumptions Matter
The conclusion is not that joint assumptions are irrelevant to means. A joint model can include restrictions that link responses, and those restrictions can rule out some marginal means. What matters is to identify the assumption that creates the restriction rather than attributing it to dependence in the abstract.
Worked Example: A Conditional Joint Model Restricts the Marginal Mean
Suppose \(Y_1\) and \(Y_2\) are binary, \(P(Y_1=1)=0.3\), \(P(Y_2=1\mid Y_1=1)=0.9\), and \(P(Y_2=1\mid Y_1=0)=0.2\). By the law of total probability,
Because \(Y_2\) is binary, \(E[Y_2]=P(Y_2=1)=0.41\). Here the specified conditional probabilities, together with the marginal distribution of \(Y_1\), determine the marginal mean of \(Y_2\). This is a consequence of the full conditional and marginal assumptions and the law of total probability. It is not a general consequence of saying only that the responses are dependent.
In a model that requires \(Y_2=Y_1\) with probability 1, for example, the means must be equal. That is a structural restriction on the joint distribution. It can be useful and substantive, but it must be stated as an assumption. If it is relaxed, the equality of means need not follow.
Common Mistake and Practical Interpretation
Wrong reasoning: “The responses within a cluster are strongly correlated, so their marginal mean must be higher (or lower).” Correlation describes how responses co-vary relative to their individual means and variances. It does not specify where those means are located.
Correction: State the marginal mean model and the dependence assumptions separately. Use expectation identities to derive a mean from a specified joint or conditional model when the assumptions warrant it. Do not infer a mean merely from a covariance, correlation, or label such as “exchangeable” or “clustered.”
This separation helps clarify what a marginal model claims. Its mean specification concerns the average response at the covariate values under study. A working description of within-cluster covariance concerns the precision and joint variability of those responses. A particular probability model may link these features, but such a link comes from that model’s assumptions—not from dependence alone.
Check Your Understanding
Use the distinction between marginal distributions and joint distributions to answer these questions.
- Two joint models have the same marginal distribution for \(Y_1\), but different covariances between \(Y_1\) and \(Y_2\). What can be concluded about \(E[Y_1]\), assuming it exists?
- For binary responses, explain why knowing that \(Y_1\) and \(Y_2\) are independent does not specify either response’s mean.
- In the law of total covariance, what are the two terms, and what kind of dependence can the second term represent?
- Can zero covariance alone establish independence? Explain why or why not.
- A model assumes \(Y_2=Y_1\) with probability 1. What restriction does this impose on their marginal means, and which assumption creates that restriction?