Rounding-Induced Changes in Likelihood Evaluation

Why Intermediate Precision Matters

Likelihood-based estimation depends on repeatedly evaluating a model and its likelihood at changing parameter values. If an intermediate prediction, probability, or individual contribution is rounded before the calculation is complete, the optimizer is not evaluating the intended likelihood exactly. The resulting changes can be small in objective value yet substantial for a gradient, especially when the true gradient is itself small or when rounding creates flat regions.

This is different from rounding a final estimate for presentation. Formatting an estimate to three significant figures after fitting does not change the likelihood that was optimized. Rounding a predicted concentration to two decimals before calculating residuals does change that likelihood. The location of rounding in the workflow is therefore central to diagnosis.

Definition: Let \(\ell(\theta)\) denote the exact log likelihood at parameter vector \(\theta\), and let \(\tilde{\ell}(\theta)\) denote the value computed by a particular implementation. Rounding-induced likelihood change is a difference between \(\tilde{\ell}(\theta)\) and \(\ell(\theta)\) caused by finite precision in an intermediate quantity, such as a prediction, probability, or contribution, rather than solely by the final display of the result. A gradient change occurs when the corresponding derivatives, or numerical approximations to them, also differ.

The exact likelihood is a mathematical function defined by the model and the data. A computed likelihood is an algorithmic result: it depends on the operations, the precision at which intermediate results are stored, and where any rounding is applied. If no explicit decimal rounding occurs, computer arithmetic still has finite precision, but this tutorial focuses on understanding and diagnosing intermediate precision loss, including explicit rounding or low-precision storage.

How Rounding Changes the Objective and Its Gradient

Suppose a smooth function \(f(z)\) is evaluated after its input \(z\) has been rounded to a grid with spacing \(h\). Write \(R_h(z)\) for rounding to the nearest grid point and \(e=R_h(z)-z\) for the rounding error. Then \(|e|\le h/2\). If \(f\) is differentiable at \(z\), the local change has the first-order form

f(z+e)-f(z)=f'(z)e+o(|e|),\qquad e\to 0.

The remainder is \(o(|e|)\) under differentiability at \(z\); differentiability alone does not justify replacing it by \(O(e^2)\). If \(f'\) is continuous near \(z\), the mean value theorem also gives \(|f(z+e)-f(z)|\le M|e|\), where \(M\) is an upper bound for \(|f'|\) on the segment between \(z\) and \(z+e\). This gives a useful scale for the objective error, but it does not guarantee that the gradient error is small.

In a likelihood calculation, the rounded quantity is often a function of the parameters, such as a prediction \(g(\theta)\). Rounding \(g(\theta)\) before applying a likelihood contribution produces a computed objective of the form \(f(R_h(g(\theta)))\). As \(\theta\) changes within a region where \(R_h(g(\theta))\) stays fixed, this computed objective can be constant even when the exact objective changes smoothly. At the boundaries between rounding bins it can jump. Consequently, a derivative of the rounded objective may be zero within bins and undefined at jumps. A numerical gradient estimated from nearby rounded evaluations can therefore fail to represent the gradient of the exact likelihood.

Key distinction: An optimizer tolerance is a stopping rule applied to computed quantities, such as the change in objective, the change in parameters, or a gradient norm. Intermediate rounding changes those computed quantities before the stopping rule is checked. Tightening a stopping tolerance cannot recover information already discarded by rounding.

Worked Examples

Worked Example: Rounded Prediction in a Normal Likelihood

Consider one hypothetical observation \(y=1.234\), modeled as normally distributed with mean \(\theta\) and known standard deviation \(\sigma=0.1\). The log likelihood is

\ell(\theta)=-\log(\sigma\sqrt{2\pi})-\frac{(y-\theta)^2}{2\sigma^2}.

Suppose an implementation rounds the prediction \(\theta\) to two decimal places before calculating the residual. At \(\theta=1.234\), the exact residual is \(1.234-1.234=0\). The rounded prediction is \(1.23\), so the residual used by the implementation is \(1.234-1.23=0.004\). The constant term in the likelihood is unchanged. Thus the computed log likelihood minus the exact log likelihood is

\tilde{\ell}(1.234)-\ell(1.234) =-\frac{(0.004)^2}{2(0.1)^2} =-\frac{0.000016}{0.02} =-0.0008.

The value change is small, but the gradient can be more seriously affected. For this model, the exact derivative is \(\ell'(\theta)=(y-\theta)/\sigma^2\). At \(\theta=1.230\), it is \((1.234-1.230)/0.01=0.4\) per unit of \(\theta\). A central finite difference with step \(d=0.001\), using the exact likelihood, gives

\frac{\ell(1.231)-\ell(1.229)}{2d} = \frac{\big[-(0.003)^2+(0.005)^2\big]/0.02}{0.002} = \frac{0.0008}{0.002} =0.4.

But the predictions \(1.229\), \(1.230\), and \(1.231\) all round to \(1.23\) at two decimal places. The rounded residual, and hence the rounded log likelihood, is identical at the two finite-difference points. The computed central difference is therefore zero, not \(0.4\). The model assumptions for this illustration are a single normal observation with the stated known \(\sigma\), and the calculation uses no parameter-dependent variance.

Worked Example: Rounded Probability Gives a Misleading Gradient

Suppose one hypothetical Bernoulli observation has outcome \(y=1\), with success probability \(p(\theta)=1/(1+\exp(-\theta))\). Its exact log likelihood is \(\ell(\theta)=\log p(\theta)\). At \(\theta=0\), \(p=0.5\), so \(\ell(0)=\log(0.5)\approx-0.693147\). Differentiation gives

\ell'(\theta)=1-p(\theta),\qquad \ell'(0)=0.5.

Now suppose the probability is rounded to three decimal places before taking its logarithm. With finite-difference step \(d=0.001\), \(p(0.001)\approx0.500250\) and \(p(-0.001)\approx0.499750\). Both round to \(0.500\). The central difference from the rounded calculations is

\frac{\log(0.500)-\log(0.500)}{0.002}=0.

For comparison, the exact central difference is \(0.5\). Indeed, \(\log p(\theta)=\theta-\log(1+\exp(\theta))\), and direct substitution at \(d\) and \(-d\) gives \(\log p(d)-\log p(-d)=d\). Dividing by \(2d\) gives \(0.5\). Thus the rounded calculation removes the local gradient signal even though the exact likelihood is smooth and has a nonzero slope. This example assumes one Bernoulli observation and a correctly specified logistic probability; the failure is introduced only by rounding the intermediate probability.

Worked Example: Rounding Contributions Can Hide a Total Difference

Consider two hypothetical candidate parameter settings for \(n=100\) independent Bernoulli observations, all with outcome \(y=1\). At the first setting, each observation has probability \(p=0.5000\); at the second, each has probability \(p=0.5001\). Independence and the common probability within each setting imply that the total log likelihood is \(n\log(p)\). The exact difference, second setting minus first, is

100\left[\log(0.5001)-\log(0.5000)\right] =100\log(1.0002) \approx100(0.00019998) =0.019998.

Suppose instead that each individual log-likelihood contribution is rounded to three decimal places before summation. The first contribution is \(\log(0.5000)\approx-0.693147\), which rounds to \(-0.693\). The second is \(\log(0.5001)\approx-0.692947\), which also rounds to \(-0.693\). The two computed totals are both \(100(-0.693)=-69.3\), so the difference appears to be zero. The exact likelihoods distinguish the settings by about \(0.0200\) log-likelihood units, but the rounded contributions erase that difference. This is not a claim that the second setting is statistically or practically better; it demonstrates how a computational comparison can lose resolution.

Distinguishing Precision Effects from Optimizer Tolerance

Optimizer tolerances govern when an algorithm stops, not how accurately its objective has been evaluated. For example, an objective-change tolerance may declare convergence when successive computed log likelihoods differ by less than a chosen threshold. If intermediate rounding makes these values identical, the optimizer may satisfy that stopping rule even while the exact likelihood still has a meaningful slope. Conversely, demanding a much smaller tolerance may cause repeated iterations or unstable behavior without repairing the rounded evaluation.

A useful diagnosis holds the parameter values fixed and compares the objective and gradient under controlled changes in evaluation precision. Keep the data, model, parameter vector, optimizer settings, and gradient method unchanged. Then recompute using full-precision intermediates, or progressively finer precision if the implementation permits it. If the likelihood or gradient changes materially while the parameters and model are fixed, the issue is evaluation precision rather than an optimizer stopping threshold. Gradient comparisons should use the same differentiation method and step sizes; otherwise the comparison confounds precision with the gradient procedure.

Locate intermediate rounding.
Trace whether predictions, probabilities, transformations, or individual likelihood contributions are rounded or stored at reduced precision before the objective is complete.

Compare at fixed parameters.
Evaluate the objective at the same parameter vector using the current precision and a higher-precision calculation. A difference here cannot be attributed to a different optimizer path.

Check gradient stability.
Compare analytic or numerical gradients across precision settings. For finite differences, repeat with reasonable step sizes and watch for gradients that change sharply or become exactly zero only under coarse precision.

Change one factor at a time.
After establishing whether evaluation precision matters, adjust optimizer tolerances separately. This distinguishes a stopping-rule issue from loss of resolution in the objective or gradient.

Preserve precision until reporting.
Avoid rounding model predictions and likelihood contributions during estimation. Round parameter estimates and summaries only when formatting final output.

Diagnostics, Alternatives, and Limitations

Warning signs include objective values that repeat exactly across nearby parameter values, finite-difference gradients that are zero over a range where the model should vary, and estimates that change when only intermediate storage precision changes. Another warning is a mismatch between an analytic gradient computed from the smooth model and a finite-difference gradient computed from rounded objective evaluations. Such a mismatch is evidence to investigate, not by itself proof that rounding is the only problem.

A practical comparison is between (1) optimizing the likelihood as currently implemented, (2) evaluating the same likelihood with higher-precision intermediates, and (3) reporting rounded results only after fitting. The third approach is generally appropriate for presentation; it does not impair fitting. The first and second approaches can differ if the first rounds before the full likelihood is assembled. When available, an analytic gradient of the intended smooth likelihood can avoid finite-difference noise, but it cannot make a rounded objective value accurate; the objective and gradient implementations should describe the same model and precision.

There are limits to what a precision check can establish. A small difference in objective values does not automatically imply a material change in parameter estimates, and a large gradient difference should be interpreted in the parameter’s scale and in the context of the fitting problem. Higher precision can help identify rounding effects, but it cannot resolve weak information in the data or make an unsuitable model appropriate. Nor should a rounded evaluation be blamed merely because an optimizer stops: the optimizer’s stopping criteria, gradient calculation, model implementation, and precision all need to be examined as distinct parts of the workflow.

Common Mistake

Common mistake: “The optimizer met its gradient tolerance, so the likelihood must have been evaluated accurately.” A stopping rule only assesses the quantities supplied to it. If a rounded objective produces a zero or small computed gradient, meeting the tolerance does not show that the gradient of the intended likelihood is small. Re-evaluate the objective and gradient at fixed parameters with adequate intermediate precision, then assess optimizer tolerances separately.

Check Your Understanding

Answer each question, then compare with the solutions below.

In the normal-likelihood example, what caused the computed finite-difference gradient to be zero even though the exact gradient at \(\theta=1.230\) was \(0.4\)?
Does rounding a final estimate for display change the likelihood used during fitting? Explain the distinction.
For 100 Bernoulli observations with probabilities \(0.5000\) and \(0.5001\), what is the exact total log-likelihood difference, approximately, and why can rounding each contribution to three decimals conceal it?
Why is tightening an optimizer’s gradient tolerance not a reliable remedy for a gradient flattened by intermediate rounding?
Name two checks that help distinguish intermediate precision loss from an optimizer stopping-rule issue.

Solutions. (1) The predictions at the two finite-difference points, \(1.229\) and \(1.231\), both round to \(1.23\), making their computed likelihoods equal and the difference quotient zero. (2) No, if rounding is applied only after fitting for presentation; rounding a prediction or contribution before the objective is complete changes the evaluated likelihood. (3) The difference is \(100\log(1.0002)\approx0.019998\). Each contribution rounds to \(-0.693\), so the rounded totals tie at \(-69.3\). (4) A tighter tolerance uses the same potentially rounded gradient and cannot restore information discarded before the gradient is calculated; it may only alter stopping behavior. (5) Compare objective and gradient values at fixed parameters under current and higher evaluation precision, and change optimizer tolerances separately while holding the evaluation method fixed.