Profiling a Gaussian Scale-Shape Model

Why Separate Scale From Shape?

A covariance matrix determines both the overall magnitude of variation and its pattern across coordinates. In some models it is useful to write these features separately: a positive scalar controls the overall scale, while a matrix indexed by a parameter describes the shape. Profiling means maximizing the likelihood over the scale for each fixed shape, then using the resulting likelihood to assess the shape parameter.

We use independent observations \(X_1,\ldots,X_n\) from a \(p\)-dimensional Gaussian distribution with known mean \(\mu\). The covariance is \(\tau\Omega(\eta)\), where \(\tau\) is positive and \(\eta\) indexes the shape. Thus this tutorial concerns covariance estimation with the mean held fixed. The key calculation is that the best scale for any fixed shape has a closed form.

The scatter matrix \(Q\) is positive semidefinite. If at least one observed residual is nonzero, then \(A(\eta)>0\) for every admissible \(\eta\): the trace is a sum of positive quadratic forms in the residuals, and a positive-definite inverse shape matrix gives a strictly positive value for each nonzero residual. This condition is important for a finite maximizing scale.

Profiling the Scale

For a fixed \(\eta\), the Gaussian log-likelihood is obtained by summing the \(n\) observation log-densities. Since \(|\tau\Omega(\eta)|=\tau^p|\Omega(\eta)|\), it can be written as

Only the last two terms depend on \(\tau\). The next result identifies their unique maximum when \(A(\eta)>0\).

Proof. Fix \(\eta\) and abbreviate \(A=A(\eta)>0\). Differentiating the log-likelihood with respect to \(\tau\) gives

Because \(2\tau^2>0\), the derivative is positive for \(0<\tau<A/(np)\) and negative for \(\tau>A/(np)\). The log-likelihood therefore increases up to \(\tau=A/(np)\) and decreases afterward, so this is its unique maximum. Substitution gives \(A/(2\hat\tau)=np/2\), yielding the stated profile log-likelihood. \(\square\)

Equivalently, the profile likelihood itself is

For fixed data, maximizing this expression over \(\eta\) is equivalent to minimizing \(\log|\Omega(\eta)|+p\log A(\eta)\), or, equivalently, minimizing \(|\Omega(\eta)|A(\eta)^p\). These equivalent forms make clear that the shape is judged not by its determinant or its fit to the scatter alone, but by their combination.

Estimating the Shape After Profiling

Once the scale has been maximized out, an estimate of \(\eta\) maximizes \(\ell_p(\eta)\) over the admissible shape set. The likelihood calculation does not guarantee that such a maximum exists: the parameter set may be open, or the profile likelihood may approach its supremum only at a boundary. When a maximizing shape \(\hat\eta\) does exist, the corresponding joint maximum-likelihood scale estimate is \(\hat\tau(\hat\eta)\).

A simple model illustrates how profiling can reduce a joint optimization to a one-dimensional calculation.

Scale and Shape Are Not Automatically Identifiable

A scalar multiple of a shape matrix can be absorbed into the scale parameter. This is not merely a notational issue: unless the shape family is normalized or otherwise restricted, different parameter pairs can describe exactly the same covariance matrix. Profiling makes the corresponding likelihood invariance explicit.

Proof. Under this replacement, the determinant and quadratic form become \(|c\Omega|=c^p|\Omega|\) and \(\operatorname{tr}\{(c\Omega)^{-1}Q\}=A/c\). Consequently, the determinant factor in \(L_p\) is multiplied by \(c^{-np/2}\), while its quadratic-form factor is multiplied by \(c^{np/2}\). The factors cancel, so \(L_p\) is unchanged. The scale estimate becomes \((A/c)/(np)=\hat\tau/c\), and \((\hat\tau/c)(c\Omega)=\hat\tau\Omega\). \(\square\)

A common way to distinguish scale from shape is to impose a normalization such as \(|\Omega(\eta)|=1\) or \(\operatorname{tr}\{\Omega(\eta)\}=p\), provided that normalization is compatible with the model. The precise choice depends on the application. Without such a restriction, the likelihood can still identify a covariance matrix while failing to identify a unique division of that matrix into a scale and a shape.

Boundary Case and Common Pitfall

If every residual is zero, then \(Q=0\) and \(A(\eta)=0\) for every admissible shape. In that case the log-likelihood contains the scale-dependent term \(-np\log(\tau)/2\) but has no quadratic penalty \(-A/(2\tau)\). As \(\tau\) tends to zero, the log-likelihood tends to \(+\infty\); there is no finite maximizing scale. The formula \(\hat\tau=A/(np)\) would give zero, which is outside the parameter space \(\tau>0\), not a valid maximum-likelihood estimate.

Common mistake: It is tempting to maximize the likelihood over \(\tau\) by setting the quadratic-form term \(A(\eta)/(2\tau)\) as small as possible, and conclude that the best scale is arbitrarily large. That reasoning ignores the determinant contribution \(-np\log(\tau)/2\), which penalizes large scales. The derivative of the complete log-likelihood balances these two terms and gives \(\hat\tau=A(\eta)/(np)\) when \(A(\eta)>0\).

Profiling is useful because it removes a nuisance scale parameter exactly, leaving a likelihood involving only the shape. It also provides a diagnostic: if a proposed shape parameterization contains an unconstrained overall multiplier, the resulting shape is not separately identified from scale. Always check both the residual condition for a finite scale estimate and the identifiability of the scale-shape decomposition.