Profiling Dispersion Out Of A GLM Likelihood

Why Profile Dispersion?

A GLM likelihood may contain both regression coefficients $\beta$ and a dispersion parameter $\phi$. If the inferential target is $\beta$, one way to remove the nuisance parameter is to maximize the likelihood over $\phi$ separately for each fixed value of $\beta$. This produces a likelihood that depends only on the regression coefficients, while retaining the fact that dispersion is estimated rather than known.

This operation is called profiling. It is not the same as fixing dispersion at an arbitrary value, and it does not require a closed-form dispersion estimate. One solves the dispersion score equation when an interior solution exists, substitutes that solution into the likelihood, and then compares the resulting likelihood values across $\beta$.

Definition: Let $\ell(\beta,\phi)$ be a log-likelihood, with regression coefficient vector $\beta$ and scalar dispersion parameter $\phi\in\Phi$. The profile log-likelihood for $\beta$ is $$ \ell_p(\beta)=\sup_{\phi\in\Phi}\ell(\beta,\phi). $$ If the supremum is attained at a value $\hat\phi(\beta)$, then $\ell_p(\beta)=\ell(\beta,\hat\phi(\beta))$. When that maximizer is an interior point and the likelihood is differentiable, it must satisfy $\ell_\phi(\beta,\hat\phi(\beta))=0$; a score root must still be checked to ensure it is a maximum.

The distinction between a supremum and an interior solution matters. A dispersion score equation may have no root, a root may fail to be a maximum, or the likelihood supremum may occur at the boundary of the parameter space. In those cases, profiling is still defined by the supremum, but it cannot be carried out by blindly substituting an interior score root.

Profiling as a Calculation

For independent observations, write the log-likelihood as the sum of their log densities, with the means determined by the GLM regression structure. At a fixed $\beta$, the fitted means are fixed, so the dispersion score is differentiated with respect to $\phi$ while holding $\beta$ constant. The resulting equation can depend on the family and on the observations; there is no universal dispersion formula for every GLM.

Hold the regression coefficients fixed.
Evaluate the likelihood as a function of $\phi$, with the corresponding fitted means unchanged.

Optimize over dispersion.
Solve $\ell_\phi(\beta,\phi)=0$ for interior candidates and check their maximizing property and any boundary values.

Substitute the maximizing value.
Use the maximizer $\hat\phi(\beta)$, if attained, to obtain $\ell_p(\beta)=\ell(\beta,\hat\phi(\beta))$.

Compare regression coefficients.
Maximize the profile log-likelihood over $\beta$, or use its shape to form likelihood-based inferences.

Worked Examples

Worked Example: Gaussian Identity-Link Regression

Suppose $Y_i$ are independent with $Y_i\sim N(x_i^\mathsf{T}\beta,\phi)$, where $\phi>0$, and let $RSS(\beta)=\sum_{i=1}^n(y_i-x_i^\mathsf{T}\beta)^2$. Assume $RSS(\beta)>0$. Find the profile likelihood for $\beta$.

The log-likelihood is $$ \ell(\beta,\phi)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log\phi-\frac{RSS(\beta)}{2\phi}. $$

At fixed $\beta$, differentiating with respect to $\phi$ gives $$ \ell_\phi(\beta,\phi)=-\frac{n}{2\phi}+\frac{RSS(\beta)}{2\phi^2} =\frac{RSS(\beta)-n\phi}{2\phi^2}. $$

Since $\phi^2>0$, the score is positive when $0<\phi<RSS(\beta)/n$ and negative when $\phi>RSS(\beta)/n$. Thus the unique interior maximum is $\hat\phi(\beta)=RSS(\beta)/n$. Substitution yields $$ \ell_p(\beta)=-\frac{n}{2}\left[\log(2\pi)+1+\log\left\{\frac{RSS(\beta)}{n}\right\}\right]. $$

The profile log-likelihood decreases as $RSS(\beta)$ increases, so maximizing it over $\beta$ gives the least-squares coefficient estimate. The variance estimate at that coefficient estimate is $RSS(\hat\beta)/n$, not the degrees-of-freedom-adjusted estimator often used for unbiased variance estimation.

Worked Example: A Numerical Gaussian Profile

For $x=(0,1,2,3)$, suppose the observed responses are $y=(1,2,2,5)$. Fit a straight line with an intercept and profile out the Gaussian variance.

The sample means are $\bar x=1.5$ and $\bar y=2.5$. The least-squares slope is $$ \hat\beta_1=\frac{\sum_i(x_i-\bar x)(y_i-\bar y)}{\sum_i(x_i-\bar x)^2} =\frac{2.25+0.25-0.25+3.75}{2.25+0.25+0.25+2.25} =\frac{6}{5}=1.2. $$

The intercept is $\hat\beta_0=\bar y-\hat\beta_1\bar x=2.5-1.2(1.5)=0.7$. The fitted values are $0.7,1.9,3.1,4.3$, giving residuals $0.3,0.1,-1.1,0.7$ and $$ RSS(\hat\beta)=0.3^2+0.1^2+(-1.1)^2+0.7^2=0.09+0.01+1.21+0.49=1.8. $$

There are $n=4$ observations, so the profiled variance estimate is $\hat\phi=1.8/4=0.45$. The maximized profile log-likelihood is $$ \ell_p(\hat\beta)=-2\left[\log(2\pi)+1+\log(0.45)\right]\approx -4.079. $$

Worked Example: Profiling Gamma Dispersion at Fixed Means

Consider independent gamma observations with means $\mu_i$ and variance $\phi\mu_i^2$. The shape parameter is $k=1/\phi$, and the scale is $\phi\mu_i=\mu_i/k$. For two observations $y_1=1,y_2=2$, hold both fitted means fixed at $\mu_1=\mu_2=1.5$. Derive the dispersion equation.

In terms of $k$, the density for observation $i$ is $$ f(y_i;\mu_i,k)=\frac{k^k}{\Gamma(k)}\frac{y_i^{k-1}}{\mu_i^k}\exp\left(-\frac{k y_i}{\mu_i}\right),\qquad y_i>0,\ k>0. $$

Writing $\psi(k)=\frac{d}{dk}\log\Gamma(k)$, the log-likelihood for these two observations is $$ \ell(k)=2[k\log k-\log\Gamma(k)-k\log(1.5)]+(k-1)\log 2-2k. $$

Differentiating gives $$ \ell_k(k)=2[\log k+1-\psi(k)-\log(1.5)]+\log 2-2 =2[\log k-\psi(k)]+\log(8/9). $$ The interior dispersion equation is therefore $\log k-\psi(k)=\frac12\log(9/8)$. The left side is approximately $0.5772$ at $k=1$ and tends to $0$ as $k$ tends to infinity, while the right side is approximately $0.05889$. A numerical solution is $k\approx8.65$, or $\hat\phi\approx1/8.65\approx0.116$. Since $\psi_1(k)=\sum_{j=0}^\infty(k+j)^{-2}>\int_0^\infty(k+x)^{-2}\,dx=1/k$, the function $\log k-\psi(k)$ is strictly decreasing; it tends to infinity as $k\downarrow0$ and to $0$ as $k\to\infty$. Thus this is the unique score root, and the score changes from positive to negative there, confirming it is the maximum. To obtain the profile likelihood value, substitute that maximizing dispersion into the displayed log-likelihood.

Worked Example: A GLM With No Dispersion to Profile

For independent Poisson observations with means $\mu_i=\exp(x_i^\mathsf{T}\beta)$, the log-likelihood is $$ \ell(\beta)=\sum_{i=1}^n\left[y_i x_i^\mathsf{T}\beta-\exp(x_i^\mathsf{T}\beta)-\log(y_i!)\right]. $$

There is no free dispersion parameter in this Poisson likelihood: its dispersion is fixed at $1$. Consequently there is no $\phi$-score equation to solve and no nuisance dispersion value to substitute. The profile likelihood over dispersion is just $\ell_p(\beta)=\ell(\beta)$. Treating an extra dispersion estimate as though it belonged to this likelihood would change the model, rather than profile the stated Poisson likelihood.

How Profiling Changes Coefficient Curvature

Profiling can affect coefficient inference even when it does not change the coefficient estimate. The precise relationship follows from differentiating the maximizing dispersion as $\beta$ varies. The result requires smoothness and a nondegenerate interior optimum; an interior maximum alone does not guarantee the required strict curvature.

Theorem: Let $\beta\in\mathbb R^p$, and suppose $\ell(\beta,\phi)$ is twice continuously differentiable on an open parameter region. In a neighborhood of a given $\beta$, assume that the selected conditional maximizer agrees locally with the unique stationary branch through the point, is an interior conditional maximizer, satisfies $\ell_\phi(\beta,\hat\phi(\beta))=0$, and has $\ell_{\phi\phi}(\beta,\hat\phi(\beta))<0$. Then $\hat\phi$ is differentiable locally and $$ \nabla_\beta\ell_p=\ell_\beta,\qquad \nabla_\beta^2\ell_p=\ell_{\beta\beta}-\ell_{\beta\phi}\ell_{\phi\phi}^{-1}\ell_{\phi\beta}, $$ where likelihood derivatives on the right are evaluated at $(\beta,\hat\phi(\beta))$. Under the stated strict-curvature assumption, $\ell_{\beta\phi}\ell_{\phi\phi}^{-1}\ell_{\phi\beta}$ is negative semidefinite.

Proof. Define $g(\beta,\phi)=\ell_\phi(\beta,\phi)$. At the point under consideration, $g=0$ and $g_\phi=\ell_{\phi\phi}<0$, so $g_\phi\ne0$. The implicit function theorem gives a differentiable local solution $\hat\phi(\beta)$ to $g(\beta,\hat\phi(\beta))=0$. Differentiating this identity with respect to the vector $\beta$ gives $$ \frac{\partial\hat\phi}{\partial\beta}=-\ell_{\phi\phi}^{-1}\ell_{\phi\beta}. $$

By the chain rule, $\nabla_\beta\ell_p=\ell_\beta+\ell_\phi\,\frac{\partial\hat\phi}{\partial\beta}$. The second term is zero at the conditional maximizer, proving the profile score identity. Differentiating $\ell_\beta(\beta,\hat\phi(\beta))$ once more gives the stated Hessian formula after substituting the derivative of $\hat\phi$.

Finally, for any vector $v\in\mathbb R^p$, let $a=\ell_{\phi\beta}v$, a scalar. Then $$ v^\mathsf{T}\left(\ell_{\beta\phi}\ell_{\phi\phi}^{-1}\ell_{\phi\beta}\right)v =\frac{a^2}{\ell_{\phi\phi}}\leq 0, $$ because $\ell_{\phi\phi}<0$. This proves the negative semidefinite claim under the theorem's assumption. A strict negative second derivative is an additional nondegeneracy condition, not a consequence of being an interior maximum. $\square$

The first identity explains why, at an interior joint maximum, the profile score for $\beta$ agrees with the joint likelihood's $\beta$-score evaluated at the maximizing dispersion. The second identity shows that the profile curvature is not generally the curvature obtained by pretending dispersion is fixed. In observed-information terms, the profile information is $$ -\nabla_\beta^2\ell_p=-\ell_{\beta\beta}+\ell_{\beta\phi}\ell_{\phi\phi}^{-1}\ell_{\phi\beta}. $$

Under the theorem's strict-curvature assumption, the added term is negative semidefinite. Thus profiling generally reduces the observed curvature for coefficients relative to the fixed-dispersion curvature. This is the local likelihood expression of accounting for uncertainty in the nuisance dispersion. It is one reason that coefficient uncertainty should not automatically be calculated as if a fitted dispersion were known.

Worked Example: Curvature Adjustment in a Smooth Likelihood

Consider the smooth log-likelihood function $$ \ell(\beta,\phi)=-\frac12(\beta-\phi)^2-2(\phi-2)^2,\qquad \phi>0,\ \beta>-7. $$ This simple function illustrates the curvature calculation. At fixed $\beta$, $$ \ell_\phi=(\beta-\phi)-4(\phi-2)=\beta+8-5\phi, $$ so $\hat\phi(\beta)=(\beta+8)/5>0$ and $\ell_{\phi\phi}=-5<0$. The joint maximum occurs at $(\hat\beta,\hat\phi)=(2,2)$.

The relevant second derivatives are $\ell_{\beta\beta}=-1$, $\ell_{\beta\phi}=1$, and $\ell_{\phi\phi}=-5$. Hence the profile Hessian is $$ \ell_p''(\beta)=-1-(1)(-1/5)(1)=-4/5. $$

Direct substitution confirms the result: $\beta-\hat\phi(\beta)=4(\beta-2)/5$ and $\hat\phi(\beta)-2=(\beta-2)/5$, so $$ \ell_p(\beta)=-\frac12\left[\frac{4(\beta-2)}5\right]^2-2\left[\frac{\beta-2}{5}\right]^2=-\frac25(\beta-2)^2, $$ whose second derivative is $-4/5$. If $\phi$ were incorrectly held fixed at $2$, the curvature in $\beta$ would be $-1$, a different value.

Using the Profile for Inference

Once $\ell_p(\beta)$ has been formed, the profile maximum gives the regression coefficient estimate when the joint maximum exists and the maximization is carried out over the same parameter space. Likelihood comparisons for $\beta$ can then be made using changes in $\ell_p$, rather than changes in the joint likelihood with an arbitrary fixed dispersion. The profile also makes explicit that each candidate coefficient value is assessed at its own best-fitting dispersion.

In the Gaussian identity-link example, profiling leaves the least-squares coefficient estimate unchanged because the conditional variance maximizer is a function of the residual sum of squares, and the profile likelihood is a decreasing function of that sum. This convenient property should not be assumed for every GLM. In other models, the likelihood may couple $\beta$ and $\phi$ in a way that makes their joint maximization consequential.

Common Pitfall

Using a score root without checking it. A root of $\ell_\phi=0$ could be a minimum or another stationary point. Compare likelihood values and account for boundary behavior before calling it the profile maximizer.
Assuming an interior maximum has strictly negative curvature. An interior maximum can have $\ell_{\phi\phi}=0$, as in a flat-topped function. The Hessian formula using $\ell_{\phi\phi}^{-1}$ requires the additional nondegeneracy condition $\ell_{\phi\phi}<0$.
Substituting a single estimated dispersion for all coefficient values. Profiling means maximizing over dispersion separately at each fixed $\beta$, yielding $\hat\phi(\beta)$, not holding dispersion at its joint estimate throughout the coefficient likelihood.
Profiling a parameter that is not free in the likelihood. Standard Poisson and Bernoulli likelihoods have fixed dispersion. Adding a dispersion parameter is a different model, not a profiling step.

Key takeaway: Profiling eliminates dispersion by optimizing it at every fixed regression coefficient value. Its score identity is an envelope result, while its curvature includes a nuisance-parameter adjustment under explicit smoothness and strict-curvature conditions.

Check Your Understanding

Use these questions to check both the calculation and the interpretation.

For a Gaussian identity-link model, why does the fixed-$\beta$ dispersion score give $RSS(\beta)/n$, and what condition ensures this is an interior maximum?
In the gamma example, why is the fitted mean held fixed while differentiating the dispersion score?
State the regularity condition that permits the implicit-function derivation of the profile Hessian. Why does an interior maximum alone not guarantee it?
Explain why profiling dispersion can alter coefficient curvature even when the profile and joint maximizers for $\beta$ agree.
Why is there no dispersion profiling step in the ordinary Poisson likelihood?