Sampling Regimes for GLM Asymptotics

Why the Sampling Regime Must Be Stated

Large-sample reasoning begins by specifying what a larger sample means. In a generalized linear model, the number of response units may increase while the coefficient vector stays fixed; alternatively, the covariates may themselves be sampled from a population. These are different sampling regimes, and they determine what is random and what likelihood is being used.

A GLM equation alone does not settle those questions. The statement \(g(E[Y_i\mid x_i])=x_i^\mathsf{T}\beta\) describes a conditional mean, but it does not say whether the \(x_i\) are fixed by design or random, whether the responses are independent, or which quantities change as the sample grows. This tutorial makes those choices explicit without drawing conclusions about estimator behavior.

Definition: A sampling regime is a specification, for each sample size \(n\), of the random variables observed, their joint sampling assumptions, the covariates treated as fixed or random, and the quantities allowed to depend on \(n\). An asymptotic sequence consists of experiments indexed by \(n\), with \(n\) increasing.

A Common GLM Setup

For response unit \(i\) in experiment \(n\), write the response as \(Y_{ni}\) and its covariate vector as \(x_{ni}\in\mathbb R^p\), for \(i=1,\ldots,n\). A familiar conditional GLM specification is

g(\mu_{ni})=\eta_{ni}=x_{ni}^{\mathsf T}\beta+o_{ni}, \qquad \mu_{ni}=E[Y_{ni}\mid x_{ni}], \qquad \operatorname{Var}(Y_{ni}\mid x_{ni}) =\frac{\phi}{w_{ni}}V(\mu_{ni}).

Here \(g\) is the link, \(o_{ni}\) is a known offset, \(w_{ni}\) is a known weight under the chosen model convention, and \(\phi\) is a dispersion parameter. Not every GLM uses every one of these terms. This notation simply makes the potential ingredients visible.

A common starting regime keeps the dimension \(p\), the target coefficient vector \(\beta\), and any dispersion parameter \(\phi\) fixed as \(n\) increases. The observed number of response units grows. Covariate rows, offsets, or weights may form a changing sequence, but whether they are fixed or random must be specified. This is a convention for describing a regime, not a universal requirement: models with growing \(p\), changing parameters, or changing measurement schemes require their own explicit asymptotic setup.

Independent Responses With Fixed Design

In a fixed-design regime, the covariate rows \(x_{n1},\ldots,x_{nn}\) are treated as nonrandom constants. The design can be chosen by an investigator or regarded as fixed for the analysis. The response model commonly assumes that \(Y_{n1},\ldots,Y_{nn}\) are independent under the specified design, with each response distributed according to its GLM distribution conditional on its own row.

Definition: An independent-response, fixed-design GLM sequence has nonrandom design quantities \(x_{ni}\) (and any specified nonrandom offsets or weights) for each \(n\), and independent responses \(Y_{n1},\ldots,Y_{nn}\) whose distributions are specified by the GLM at a fixed parameter value such as \(\beta\). The rows may differ across units and across sample sizes.

“Fixed design” does not mean that every row must be identical for every \(n\), nor that the covariates cannot vary among units. It means the design values in a given experiment are treated as given rather than as draws from a covariate population. Thus, the responses are independent but usually not identically distributed: different covariate rows can produce different means or response distributions.

If the response mass or density for unit \(i\), under its row, is \(f_{ni}(y;\beta)\), conditional independence gives the likelihood

L_n(\beta;y) =\prod_{i=1}^{n} f_{ni}(y_{ni};\beta), \qquad \ell_n(\beta) =\sum_{i=1}^{n}\log f_{ni}(y_{ni};\beta).

This product is the response likelihood for the specified design. The index \(n\) emphasizes that the experiment and its collection of rows can change with sample size.

Worked Example: Fixed-Design Logistic Regression

Suppose a study records a binary outcome for each subject and uses the logistic model \(P(Y_{ni}=1\mid x_{ni})=\operatorname{expit}(x_{ni}^{\mathsf T}\beta)\), where \(\operatorname{expit}(u)=1/(1+\exp(-u))\). Consider four subjects with an intercept and one binary covariate, so the rows are \((1,0),(1,0),(1,1),(1,1)\). Let \(\beta=(0,\log 3)^\mathsf T\).

For the first two rows, the linear predictor is \(0\), and the event probability is \(\operatorname{expit}(0)=1/2\). For the last two rows, the linear predictor is \(\log 3\), and the probability is \(3/(1+3)=3/4\). If the observed outcomes are \((0,1,1,0)\), their likelihood at this parameter value is

\left(\frac12\right)\left(\frac12\right) \left(\frac34\right)\left(\frac14\right) =\frac{3}{64}.

The design rows are fixed, the four responses are independent under the model, and the coefficient vector is a fixed model parameter. In a sequence of such experiments, \(n\) increases as additional response units are included. The proportions of the two row types may be held constant or may change; either choice is a design sequence that should be stated.

Independent Responses With Random Design

In a random-design regime, covariates are random variables. A common formulation treats the observed pairs \((X_i,Y_i)\), \(i=1,\ldots,n\), as independent and identically distributed, with a conditional GLM for \(Y_i\) given \(X_i\). The covariate distribution is part of the sampling description, even if the modeling target concerns only the conditional response distribution.

Definition: In an independent, identically distributed random-design GLM regime, the pairs \((X_i,Y_i)\) are independent and share a common joint distribution. The conditional distribution of \(Y_i\) given \(X_i=x\) follows the stated GLM with a common parameter such as \(\beta\). The distribution of \(X_i\) must also be specified if the joint sampling law is being described.

Independence of the pairs implies independence of the responses across units, but the responses need not have the same marginal distribution in every model setup if the pair distributions differ. In the identically distributed formulation, the pairs share a joint law; conditionally, different realized covariate values can still give different response distributions. More generally, researchers may use independent but non-identically distributed random covariates. That is a valid regime too, but it is not the identically distributed one just defined.

For random design, the conditional response likelihood given the observed covariates is

L_n(\beta;y\mid x) =\prod_{i=1}^{n} f(y_i\mid x_i;\beta).

If the covariate distribution has density or mass function \(q(x)\) and does not involve \(\beta\), the joint likelihood for the observed pairs factors as

L_n(\beta;y,x) =\prod_{i=1}^{n}q(x_i)f(y_i\mid x_i;\beta).

The factor involving \(q\) then does not change which \(\beta\) maximizes the likelihood. If the covariate distribution has parameters of its own, or depends on \(\beta\), that factor cannot simply be ignored when forming a joint likelihood for all parameters.

Result: If the observed pairs are independent, their joint density or mass function factors into the product of their pairwise joint distributions. If each pair has conditional response law \(f(y_i\mid x_i;\beta)\) and covariate law \(q(x_i)\), then the joint likelihood is \(\prod_i q(x_i)f(y_i\mid x_i;\beta)\). When \(q\) is parameter-free with respect to \(\beta\), maximizing this joint likelihood over \(\beta\) is equivalent to maximizing the conditional response likelihood.

Proof. Independence of the pairs means their joint distribution is the product of their pairwise distributions. For each pair, the definition of a conditional distribution gives the factorization \(p(x_i,y_i)=q(x_i)f(y_i\mid x_i;\beta)\), with the analogous density factorization when variables are continuous. Multiplying over \(i\) gives the displayed joint likelihood. If \(q\) contains no \(\beta\), then \(\prod_i q(x_i)\) is constant as a function of \(\beta\). Multiplication by this nonnegative, parameter-independent factor does not change the maximizing values of \(\beta\) when the factor is positive on the observed data. Equivalently, the joint and conditional log-likelihoods differ only by a term that does not involve \(\beta\). \(\square\)

Worked Example: A Random-Design Poisson Model

Let \(X_i\sim N(0,1)\) independently, and suppose \(Y_i\mid X_i=x\sim\operatorname{Poisson}(\exp(\beta_0+\beta_1x))\). Take \(\beta_0=0\) and \(\beta_1=1/2\). Conditional on a realized covariate value \(x\), the response mean is \(\exp(x/2)\). For example, at \(x=0\) it is \(1\), while at \(x=2\) it is \(\exp(1)\).

The conditional likelihood for observed pairs \((x_i,y_i)\) is

\prod_{i=1}^{n} \frac{\exp\{-\exp(\beta_0+\beta_1x_i)\} \exp\{y_i(\beta_0+\beta_1x_i)\}}{y_i!}.

If the normal covariate distribution is part of the joint model, its density contributes the additional factor \(\prod_{i=1}^{n}(2\pi)^{-1/2}\exp(-x_i^2/2)\). That factor is parameter-free here, so it does not change likelihood maximization over \((\beta_0,\beta_1)\).

The marginal mean of a response is not obtained by substituting the average covariate into the conditional mean. In this example,

E[Y_i] =E\!\left[E[Y_i\mid X_i]\right] =E[\exp(X_i/2)] =\exp\!\left(\frac{(1/2)^2}{2}\right) =\exp(1/8).

The final equality uses the moment-generating function of \(N(0,1)\). Since \(\exp(E[X_i]/2)=1\), this calculation also shows that the marginal mean differs from the conditional mean evaluated at \(E[X_i]=0\). The model is stated conditionally; averaging over a random design is a separate operation.

What Increases: Units, Trials, and Exposure

The symbol \(n\) should count the response units in the stated sampling scheme. It does not automatically count individual Bernoulli trials inside a grouped binomial response, measurement occasions within a subject, or exposure time. How those quantities behave as \(n\) grows is part of the regime.

Worked Example: More Units Versus More Binomial Trials

Suppose independent units have responses \(Y_i\mid x_i\sim\operatorname{Binomial}(m_i,p_i)\), with \(\operatorname{logit}(p_i)=x_i^\mathsf T\beta\). If there are \(n\) units and each has \(m_i=10\) trials, then increasing \(n\) adds response units while the number of trials per unit stays at 10. The total number of Bernoulli trials is \(10n\).

A different sequence could keep \(n=1\) and increase the number of trials \(m_1\) from 10 to 100. The number of response units has not increased: there is still one binomial response. The conditional variance in either case is \(m_i p_i(1-p_i)\), but the sampling sequences are distinct and should not be described by the same unqualified statement that “the sample size grows.”

Similarly, for a Poisson model with exposure \(t_i\), a log-link specification may use \(\log(t_i)\) as a known offset. Increasing the number of independent units differs from increasing \(t_i\) for a fixed unit. Which quantity grows changes the experiment being described; the model notation does not make those regimes interchangeable.

Two Sampling Regimes at a Glance

Feature	Fixed design	Random design
Covariates	Nonrandom values specified for each experiment	Random variables sampled as part of the experiment
Responses	Often independent conditional on the fixed rows	Often modeled through independent pairs and a conditional response law
Likelihood commonly used	Response likelihood for the specified design	Conditional response likelihood, or joint likelihood for pairs
What may vary with \(n\)	Number of rows and the design sequence	Number of sampled pairs; possibly the pair distribution in non-identical regimes

Common Mistake: Treating “Sample Size” as Self-Explanatory

A common mistake is to say, “As \(n\) grows, the data are independent observations from the same GLM,” without saying whether the covariates are fixed or random, or what counts as an observation. In a fixed-design GLM, independent responses can have different distributions because their covariate rows differ. In a grouped binomial analysis, \(n\) may count groups while the total number of trials is \(\sum_i m_i\). In a random-design model, specifying only the conditional response distribution does not specify the distribution from which covariates are sampled.

The correction is to state the sampling unit, the independence assumption, the status of the covariates, and the quantities held fixed or allowed to change. For example: “For each \(n\), there are \(n\) independent Bernoulli responses at nonrandom design rows \(x_{ni}\); the coefficient vector is fixed, and the design rows follow a specified sequence.” Or: “There are \(n\) independent, identically distributed pairs \((X_i,Y_i)\), and the conditional law of \(Y_i\) given \(X_i\) follows the GLM with fixed coefficients.”

Key takeaway: An asymptotic GLM regime is an explicit sequence of sampling experiments, not just a model equation with a large \(n\). State what a unit is, what is independent, whether covariates are fixed or random, and which model and design quantities remain fixed or change.

Check Your Understanding

For each question, identify the sampling assumption that must be stated or the distinction that matters.

In a fixed-design logistic regression, must the response distributions be identical across units? Explain.
What additional sampling component is present in a random-design GLM but not in a fixed-design response likelihood?
If the number of groups stays fixed while the number of binomial trials per group increases, has the number of response units increased?
When does the joint likelihood for random covariates reduce, for coefficient estimation, to the conditional response likelihood?
State one sentence describing an independent-response fixed-design regime with fixed coefficients and growing sample size.