Why HMC Needs a Mass Matrix
An NLME posterior can have parameters with very different scales, strong correlations, or both. For example, the log of a population clearance parameter and a random-effect correlation parameter may vary by quite different amounts and may be associated in the posterior. Hamiltonian Monte Carlo (HMC) uses a momentum variable to propose movement through this posterior. The mass matrix controls how momentum is translated into movement, and therefore how well the movement matches the posterior’s scale and orientation.
The mass matrix does not change the posterior distribution being sampled. It changes the dynamics used to explore that distribution. A poor choice can make movement inefficient: some directions may be traversed too cautiously while others are traversed too aggressively. A suitable choice can make typical movement more balanced.
The positive-definite condition ensures that the kinetic energy is well-defined and positive for nonzero momentum. For a diagonal \(M\), each parameter’s scale can be adjusted separately, but cross-parameter correlations cannot be represented. A dense \(M\) can also account for linear relationships between parameters. Some software and references use the term “metric” for \(M^{-1}\), or describe a matrix by its role in the velocity update. Check the convention before comparing numerical matrices: the equations above define the convention for this tutorial.
Why the Gaussian Target Gives a Useful Guide
A multivariate Gaussian gives a precise way to see what the mass matrix does. Suppose the target is \(q\sim N(\mu,\Sigma)\), where \(\Sigma\) is positive definite. Its negative log density, up to a constant, is $$ U(q)=\frac{1}{2}(q-\mu)^\mathsf{T}\Sigma^{-1}(q-\mu). $$ The gradient is \(\nabla U(q)=\Sigma^{-1}(q-\mu)\). Substituting this and \(\dot q=M^{-1}p\) into the equations of motion gives the following justification for the ideal Gaussian choice: $$ \ddot q=M^{-1}\dot p=-M^{-1}\Sigma^{-1}(q-\mu). $$ If \(M=\Sigma^{-1}\), then \(M^{-1}=\Sigma\), so $$ \ddot q=-(q-\mu). $$ Every direction then has the same oscillation frequency in these idealized continuous dynamics. This is the sense in which the target’s precision matrix is the ideal mass matrix under the stated convention. It balances differences in scale and, when dense, accounts for linear correlations.
This result is a guide, not a claim that a real NLME posterior is exactly Gaussian. Real posteriors may be skewed, heavy-tailed, constrained, or curved. A mass matrix can match a global linear approximation, but it cannot remove all such features.
Worked Example: Matching Two Different Marginal Scales
Suppose a hypothetical two-parameter Gaussian target has covariance \(\Sigma=\begin{pmatrix}0.04&0\\0&9\end{pmatrix}\). The first parameter has standard deviation \(0.2\), while the second has standard deviation \(3\). Find the ideal diagonal mass matrix and compare it with the identity mass matrix.
Because the covariance is diagonal, its inverse is obtained by taking the reciprocal of each diagonal entry: $$ M=\Sigma^{-1}= \begin{pmatrix}1/0.04&0\\0&1/9\end{pmatrix} = \begin{pmatrix}25&0\\0&0.1111\end{pmatrix}. $$
For the identity choice \(M=I\), the negative log density has curvature \(1/0.04=25\) in the first direction and \(1/9=0.1111\) in the second. The corresponding frequencies are the square roots of these values: \(5\) and \(0.3333\). Their ratio is \(5/0.3333\approx15\), indicating a substantial scale mismatch in the dynamics.
With \(M=\Sigma^{-1}\), each frequency is \(1\), as the Gaussian derivation predicts. The mass matrix therefore adjusts to the very different marginal scales. These calculations concern the idealized Gaussian dynamics; they do not guarantee a particular acceptance rate or sampling efficiency for a nonlinear posterior.
Diagonal and Dense Choices
An identity mass matrix makes no parameter-specific adjustment. A diagonal mass matrix can compensate for different marginal variances, but it treats the coordinates as if their linear posterior relationships did not matter. A dense matrix can adapt to both marginal scales and linear correlations. In the Gaussian case, the ideal dense choice is the full precision matrix \(\Sigma^{-1}\), not the covariance \(\Sigma\), under the convention defined above.
| Choice | What it can adapt to | Main limitation |
|---|---|---|
| Identity | Nothing beyond the original coordinates | Scale differences and correlations remain unaddressed |
| Diagonal | Different marginal scales | Does not represent cross-parameter correlation |
| Dense | Marginal scales and linear correlations | Requires estimating many matrix entries; may be unstable with limited information |
Worked Example: A Correlated Gaussian Target
Consider a hypothetical target with covariance \(\Sigma=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). Both parameters have marginal standard deviation \(1\), so a diagonal mass matrix based only on marginal variances is \(I\). Compare that choice with the ideal dense mass matrix.
The determinant is \(1(1)-0.8^2=0.36\). Therefore, $$ \Sigma^{-1}=\frac{1}{0.36} \begin{pmatrix}1&-0.8\\-0.8&1\end{pmatrix} = \begin{pmatrix}2.7778&-2.2222\\-2.2222&2.7778\end{pmatrix}. $$ This is the ideal dense mass matrix in the convention used here.
To see what the identity mass leaves uncorrected, the covariance has eigenvalues \(1.8\) along the direction proportional to \((1,1)\) and \(0.2\) along the direction proportional to \((1,-1)\). The negative-log-density curvatures in those directions are their reciprocals: \(0.5556\) and \(5\). With \(M=I\), the frequencies are \(\sqrt{0.5556}\approx0.7454\) and \(\sqrt{5}\approx2.236\), a ratio of \(3\).
With \(M=\Sigma^{-1}\), the product \(M^{-1}\Sigma^{-1}=\Sigma\Sigma^{-1}=I\), so both frequencies are \(1\). A diagonal matrix cannot rotate the coordinates to align with the long and short directions of this correlated target. The dense matrix represents that orientation as well as the scales.
Scaling NLME Parameters and Reading Diagnostics
The same principle applies to NLME parameters, including population parameters and individual random effects when they are included in the sampled vector. Consider dimensionless coordinates \(q_1=\log(CL/5)\) and \(q_2=\log(V/20)\), where \(CL\) and \(V\) are clearance and volume and the constants set reference values in their respective units. Such transformations can make parameter scales more comparable, but they do not guarantee weak posterior correlation.
Worked Example: A Mass Matrix for Two Log-Scale NLME Parameters
Suppose a hypothetical local Gaussian approximation for \((q_1,q_2)\) has covariance \(\Sigma=\begin{pmatrix}0.09&0.12\\0.12&0.64\end{pmatrix}\). The marginal standard deviations are \(0.3\) and \(0.8\), and the correlation is \(0.12/(0.3\times0.8)=0.5\). Find diagonal and dense mass-matrix choices.
A diagonal approximation based on marginal variances is $$ M_{\mathrm{diag}}= \begin{pmatrix}1/0.09&0\\0&1/0.64\end{pmatrix} = \begin{pmatrix}11.111&0\\0&1.5625\end{pmatrix}. $$ It adjusts the two scales but leaves the correlation unrepresented.
For the dense choice, the determinant is \(0.09(0.64)-0.12^2=0.0432\). Inverting the covariance gives $$ M_{\mathrm{dense}}=\Sigma^{-1} =\frac{1}{0.0432} \begin{pmatrix}0.64&-0.12\\-0.12&0.09\end{pmatrix} = \begin{pmatrix}14.815&-2.7778\\-2.7778&2.0833\end{pmatrix}. $$ The off-diagonal entries encode the linear correlation. This matrix is positive definite because \(\Sigma\) has positive diagonal entries and positive determinant. It is an ideal choice only to the extent that this hypothetical Gaussian covariance adequately represents the posterior geometry.
In practice, poor scaling may be suggested when movement appears disproportionately constrained in some parameter directions, or when the observed joint parameter cloud is elongated or tilted relative to the coordinate axes. These clues are not proof that the mass matrix is the sole problem: posterior curvature can vary across the parameter space, gradients can be inaccurate, and parameterizations can create difficult geometry.
A useful comparison is between changing coordinates and changing the mass matrix. Rescaling or reparameterizing changes the coordinates in which the posterior is represented; a mass matrix changes the HMC dynamics in the current coordinates. Either can improve linear scaling. In hierarchical models, centered and noncentered parameterizations can also differ in their posterior geometry. If a posterior has a funnel shape or strong location-dependent curvature, a single global matrix may not resolve the problem. That calls for investigating parameterization and model geometry, rather than assuming a denser matrix will solve everything.
Practical Choices, Assumptions, and Pitfalls
The Gaussian derivation assumes a smooth, differentiable target with a positive-definite covariance and an unconstrained Euclidean parameter vector. HMC also requires a valid positive-definite mass matrix so that its momentum distribution and kinetic energy are well-defined. Constrained parameters are often represented through unconstrained transformations; the mass matrix then applies to the coordinates actually used by the sampler.
A diagonal mass matrix is often a practical alternative when a dense estimate would be unreliable or costly. A dense matrix has \(d(d+1)/2\) distinct entries for \(d\) parameters, so estimating it becomes more demanding as dimension grows. If a covariance estimate is noisy or nearly singular, its inverse can be unstable. Regularization or a simpler diagonal choice can be more reliable than using an ill-conditioned dense estimate. Also, an empirical covariance estimate of the target is not itself the mass matrix under this convention: the Gaussian ideal is its inverse.
The mass matrix and the step size have different roles. The matrix adjusts how momentum maps to parameter movement; the step size controls the resolution of the numerical integration. A well-scaled matrix does not remove the need for an appropriate step size, and changing the step size does not make a correlation disappear. These are complementary tuning choices.
The key limitation is that one fixed mass matrix can correct only global linear scale and correlation. It does not change the posterior, fix weak identifiability, make a non-Gaussian target Gaussian, or eliminate curvature that varies across the parameter space. Treat it as a useful preconditioner, then assess whether the resulting dynamics and the model’s parameterization are behaving sensibly.
Check Your Understanding
Answer each question using the convention \(p\sim N(0,M)\) and \(H(q,p)=U(q)+p^\mathsf{T}M^{-1}p/2\). Short solutions follow.
- For a Gaussian target with covariance \(\operatorname{diag}(4,0.25)\), what is the ideal diagonal mass matrix?
- A target has covariance \(\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}\). Can a diagonal mass matrix represent its correlation?
- What is the main practical advantage of a diagonal mass matrix over a dense one?
- Does using \(M=\Sigma^{-1}\) make an arbitrary NLME posterior exactly Gaussian?
Solutions. (1) The ideal mass is the inverse covariance: \(\operatorname{diag}(1/4,1/0.25)=\operatorname{diag}(0.25,4)\). (2) No. A diagonal matrix can adjust marginal scales, but its off-diagonal entries are zero and it cannot represent the target’s correlation. (3) It is simpler to estimate and requires fewer entries, which can be valuable when a dense estimate is unreliable or costly. (4) No. The Gaussian result motivates a mass matrix for scale and linear correlation; it cannot make a non-Gaussian posterior Gaussian or remove location-dependent curvature.