Why Use the Matérn Family?
A spatial latent effect can represent residual similarity between nearby locations after measured covariates have been included. The exponential and Gaussian covariance models provide useful examples, but they impose particular degrees of local smoothness. The Matérn family includes the exponential model as a special case and allows the analyst to vary smoothness while retaining a flexible distance-based covariance structure.
This distinction matters in a GLMM because the latent field enters the linear predictor, not usually the response directly. For example, a spatial Gaussian field may be added to a logit linear predictor for a binary outcome. Changing its covariance smoothness changes how quickly the latent predictor can vary over nearby locations; it does not, by itself, change the response distribution or establish a causal spatial effect.
The expression at \(h=0\) is defined by its limit, which gives \(R(0)=1\). For every \(\nu>0\), \(\kappa>0\), and \(\sigma_b^2>0\), the Matérn function defines a valid covariance on Euclidean space: covariance matrices formed from any finite set of locations are positive semidefinite. The stationarity and isotropy assumptions mean that covariance depends on the pair of locations only through their separation, and specifically through distance rather than direction.
Scale and Smoothness Have Different Roles
The parameter \(\rho\) sets the distance scale over which correlation declines. It is not defined as the distance at which correlation reaches a particular value: the correlation at a given multiple of \(\rho\) also depends on \(\nu\). The parameter \(\nu\) controls the local smoothness of realizations of the field. A larger \(\nu\) produces a smoother field, all else held fixed; it does not simply multiply correlations by a constant.
Several special cases make the family easier to interpret. With the scale convention above, \(\nu=1/2\) gives the exponential correlation, while \(\nu=3/2\) and \(\nu=5/2\) give simple closed forms:
The Gaussian spatial covariance is another common model, but it is not obtained by choosing a finite \(\nu\) in the same way as the three expressions above. It imposes very high local smoothness. The Matérn family lets the analyst represent a wider range of field textures, from rough exponential fields through increasingly smooth alternatives.
Worked Example: Comparing Correlation at One Scale Length
Suppose a hypothetical spatial field has scale \(\rho=10\) km. Compare the correlation between locations 10 km apart under three Matérn smoothness values.
At \(h=\rho=10\) km, the exponential case gives $$ R_{1/2}(10)=\exp(-1)\approx 0.3679. $$ For \(\nu=3/2\), substitute \(h/\rho=1\): $$ R_{3/2}(10)=\left(1+\sqrt{3}\right)\exp(-\sqrt{3}) \approx 0.4834. $$ For \(\nu=5/2\): $$ R_{5/2}(10)=\left(1+\sqrt{5}+\frac{5}{3}\right)\exp(-\sqrt{5}) \approx 4.9027\times 0.1068779\approx 0.5240. $$ Thus, in this comparison, the correlation at a distance of 10 km is approximately 0.3679, 0.4834, and 0.5240 for \(\nu=1/2,3/2,5/2\), respectively. These values are consistent with the chosen parameterization; they do not imply that \(\rho\) is a universal “correlation range” with a fixed correlation threshold across smoothness choices.
Why \(\nu\) Describes Smoothness
A useful way to understand smoothness is to examine how variable the difference \(b(s)-b(s')\) is at nearby locations. Since both locations have variance \(\sigma_b^2\), $$ \operatorname{Var}\{b(s)-b(s')\} =2\sigma_b^2\{1-R(h)\}. $$ If \(R(h)\) approaches 1 rapidly as \(h\) approaches 0, the field’s nearby values tend to differ less. The Matérn smoothness parameter determines the small-distance behavior of this correlation and, more generally, whether spatial derivatives exist in the mean-square sense.
A short justification uses the spectral representation. For a Matérn field in \(d\) dimensions, its spectral density is proportional to \((\kappa^2+\|\omega\|^2)^{-(\nu+d/2)}\), where \(\omega\) is spatial frequency. At large frequency its radial tail is proportional to \(r^{-2\nu-d}\). A mean-square derivative of order \(m\) requires the spectral integral weighted by frequency to the power \(2m\) to be finite. At high frequencies its radial part behaves as $$ \int_1^\infty r^{2m}\,r^{-2\nu-d}\,r^{d-1}\,dr = \int_1^\infty r^{2m-2\nu-1}\,dr. $$ This integral converges exactly when \(2m-2\nu-1<-1\), or \(\nu>m\). Thus a Matérn field has mean-square derivatives through order \(m\) when \(\nu>m\); at the boundary \(\nu=m\), the integral diverges. This is a statement about mean-square differentiability, not a guarantee that a finite observed map will look visibly smooth.
Worked Example: Scale Changes Correlation at Different Distances
Consider a hypothetical field with \(\nu=3/2\) and scale \(\rho=20\) km. Calculate correlation at 10 km and 20 km.
For \(\nu=3/2\), let \(x=\sqrt{3}\,h/\rho\), so \(R(h)=(1+x)\exp(-x)\). At \(h=10\) km, $$ x=\frac{\sqrt{3}(10)}{20}\approx 0.8660, \qquad R(10)=(1+0.8660)\exp(-0.8660)\approx 0.7849. $$ At \(h=20\) km, $$ x=\sqrt{3}\approx 1.7321, \qquad R(20)=(1+1.7321)\exp(-1.7321)\approx 0.4834. $$ The shorter separation has the higher correlation, as expected. This example also shows why the scale and smoothness must be interpreted together: \(\rho\) sets the distance scale, while the Matérn form, determined in part by \(\nu\), sets how correlation changes across distances.
Worked Example: Smoothness and Latent Increments in a GLMM
Suppose a hypothetical binary-outcome GLMM uses a spatial field \(b(s)\) in a logit linear predictor, and the field has marginal variance \(\sigma_b^2=0.64\) on the logit scale and scale \(\rho=10\) km. Compare the variance of the latent-field difference between locations 10 km apart for \(\nu=1/2,3/2,5/2\). Assume the Matérn covariance is the only source of covariance between these latent effects.
For each value, use \(\operatorname{Var}\{b(s)-b(s')\}=2\sigma_b^2\{1-R(h)\}\), with \(h=\rho\). For \(\nu=1/2\), \(R(\rho)=0.3679\), so $$ 2(0.64)(1-0.3679)=1.28(0.6321)\approx 0.8091. $$ For \(\nu=3/2\), \(R(\rho)=0.4834\), giving $$ 2(0.64)(1-0.4834)=1.28(0.5166)\approx 0.6612. $$ For \(\nu=5/2\), \(R(\rho)=0.5240\), giving $$ 2(0.64)(1-0.5240)=1.28(0.4760)\approx 0.6093. $$ The increment variance is smaller for the smoother choices in this comparison. These are variances of differences in latent logit effects, not variances of binary outcomes or changes in outcome probability. Conditional response probabilities also depend on the fixed effects and the inverse-logit transformation.
Modeling Choices, Diagnostics, and Limitations
The Matérn family is useful when a continuous spatial field is plausible and its degree of smoothness should not be forced to the exponential or Gaussian extreme. It is a parametric model: stationarity and isotropy may be poor approximations if the spatial process changes across the region or behaves differently by direction. It also assumes a Euclidean distance is meaningful for the locations. A Matérn model can be inappropriate on a network, across barriers, or over a domain where straight-line distance does not capture dependence.
In practice, \(\nu\) may be difficult to estimate separately from the scale, especially when locations are sparse or tightly clustered over a limited spatial extent. Similar covariance patterns can be produced by different combinations of smoothness and scale. Estimation can therefore be unstable, with a flat likelihood or large uncertainty in one or both parameters. Fixing \(\nu\) at a scientifically defensible value and comparing a small set of alternatives is sometimes more reliable than attempting to estimate every covariance parameter freely.
Useful checks include plotting fitted correlations against distance, examining residual spatial structure after fitting, and checking whether estimated parameters are near optimization boundaries or highly sensitive to initial values. A fitted covariance should also produce valid covariance matrices at the observed locations; a correctly specified Matérn construction does so, but numerical approximations or an incorrectly assembled matrix can still cause computational failures. Good marginal fit alone does not establish that the spatial dependence model is adequate.
| Approach | Useful feature | Main caution |
|---|---|---|
| Exponential covariance | Simple form; the Matérn case \(\nu=1/2\) | Imposes a rougher local field than smoother Matérn choices |
| Matérn covariance | Allows explicit control of scale and smoothness | Smoothness and scale can be hard to distinguish with limited spatial information |
| Gaussian covariance | Represents very smooth fields | May impose more local smoothness than the data support |
| Discrete adjacency-based spatial effects | Can match areal data whose natural structure is a neighborhood graph | Does not use continuous Euclidean distance in the same way |
The comparison is about modeling assumptions, not a rule that one covariance is always preferable. In particular, choosing a more flexible smoothness parameter does not solve nonstationarity, anisotropy, incorrect distance measurement, or omitted covariates. Those issues require separate modeling decisions.
Common Pitfalls
- Treating \(\rho\) as a fixed-threshold range. The correlation at \(h=\rho\) depends on \(\nu\); interpret \(\rho\) as the scale defined here, not as a universal distance where correlation reaches a preset value.
- Equating high correlation with smoothness. Correlation at a particular distance depends on both scale and smoothness. Smoothness concerns local behavior and differentiability, not one correlation value alone.
- Calling every fitted surface “smooth.” A plotted surface depends on the smoothness parameter, the locations, the fitted values, and the plotting method. Mean-square differentiability is a model property; visual appearance is not a diagnostic proof.
- Interpreting latent-field covariance as response correlation. In a GLMM, the field acts on the linear predictor. A latent covariance value does not directly equal the correlation between binary or count outcomes.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For a Matérn field with \(\nu=1/2\), \(\rho=5\) km, and separation \(h=5\) km, calculate the correlation.
- For \(\nu=3/2\), state the condition for a first mean-square derivative to exist.
- A fitted spatial field has \(\sigma_b^2=0.50\) and correlation \(R(h)=0.40\) at a specified separation. Find the variance of the difference between the two latent effects.
- Why should an analyst avoid interpreting \(\rho\) as the distance at which correlation always equals 0.1?
Solutions. (1) Since \(\nu=1/2\), \(R(h)=\exp(-h/\rho)\). Therefore \(R(5)=\exp(-1)\approx 0.3679\). (2) A first mean-square derivative exists when \(\nu>1\); here \(3/2>1\), so the condition holds. (3) The increment variance is \(2(0.50)(1-0.40)=0.60\), in squared units of the latent effect. (4) The correlation at a given multiple of \(\rho\) depends on \(\nu\), and \(\rho\) is defined as a scale parameter rather than as a fixed correlation-threshold distance.