Detecting Bias From Model Misspecification

Why compare the planned model with alternative data-generating models?

A trial analysis is designed using assumptions about how outcomes arise. For example, a dose-response analysis may assume a linear mean response, or a survival analysis may assume that the treatment hazard ratio is constant over time. If the data arise differently, the planned analysis can still return an estimate and a p-value. The important question is whether those results remain useful for the decision the trial is meant to support.

In simulation, this question can be examined directly: generate trial data from a specified alternative model, apply the planned analysis exactly as intended, and record both estimation performance and the resulting decisions. This is a focused check of bias and decision consequences under named departures from the planned model; it is not a claim that every possible departure has been explored.

Definition: Let \(\theta\) be the scientific quantity targeted by the trial decision, and let \(\hat{\theta}_A\) be the estimate produced by planned analysis model \(A\). Under data-generating model \(G\), the estimator's bias for that target is \(E_G(\hat{\theta}_A)-\theta_G\), where \(\theta_G\) is the target value implied by \(G\). A model is misspecified for this purpose when its assumptions do not adequately describe \(G\), or when its reported parameter does not represent the target \(\theta_G\).

The distinction between target and fitted parameter matters. A misspecified model may estimate a well-defined quantity, such as a best-fitting linear slope, but that quantity may not equal the clinically relevant contrast. Bias is always relative to a stated target—not simply a property of an estimate in isolation.

How to define the comparison

Specify the planned analysis and the alternative data-generating model before interpreting simulation results. The planned analysis includes the estimator, its standard error or interval procedure, and the decision rule. The alternative model specifies how simulated participants' outcomes are generated, including the treatment effect, variability, covariates, and any relevant follow-up or censoring.

The target also needs a precise definition and units. It could be a mean response difference at a specified dose and time, a 12-month survival difference, or another decision-relevant effect. Do not compare an estimate of one quantity with the value of a different quantity merely because both are called “treatment effect.”

For \(R\) simulated trials, let \(\hat{\theta}_{A,r}\) be the estimate in replicate \(r\), and let \(D_r\) equal 1 when the prespecified decision is made and 0 otherwise. The estimated bias and decision probability are:

\widehat{\operatorname{Bias}}_G = \frac{1}{R}\sum_{r=1}^{R}\hat{\theta}_{A,r}-\theta_G, \qquad \hat{\pi}_G = \frac{1}{R}\sum_{r=1}^{R}D_r.

For a binary decision, \(\hat{\pi}_G\) is the simulated probability of making that decision under \(G\). If the decision is undesirable for that scenario—for example, declaring success when the target is null—it estimates the corresponding decision-error rate. As in the earlier material on evaluating estimator bias and interval coverage, simulation summaries have Monte Carlo error: for \(R\) independent replicates, the approximate standard error of an estimated decision probability is \(\sqrt{\hat{\pi}_G(1-\hat{\pi}_G)/R}\).

Deriving bias from a nonlinear response

Consider a randomized, three-dose trial with doses 0, 50, and 100 mg. Suppose the true mean response, in a specified response unit, is quadratic in dose, but the planned analysis fits an intercept and a linear dose effect. With equal numbers of participants at each dose, the expected fitted slope is the slope of the least-squares line through the three dose-specific means. That line need not reproduce the true mean at every dose.

Justification: For fixed doses \(d_i\) and mean responses \(\mu_i\), the least-squares slope is \(\sum_i(d_i-\bar{d})\mu_i/\sum_i(d_i-\bar{d})^2\). This follows by minimizing \(\sum_i(\mu_i-a-bd_i)^2\) over intercept \(a\) and slope \(b\), then solving the slope equation after centering dose at \(\bar{d}\).

Worked Example: A Linear Analysis of a Curved Dose-Response

Suppose the true mean response is \(\mu(d)=2+0.08d-0.0006d^2\), with response measured in units, and individual residuals have mean zero and standard deviation 4 units. The scientific target is the mean response difference between 50 mg and 0 mg.

The true means at 0, 50, and 100 mg are \(2\), \(4.5\), and \(4\) units, respectively. Thus the target is \(\theta_G=\mu(50)-\mu(0)=4.5-2=2.5\) units. With equal allocation, \(\bar{d}=50\) mg, so the expected slope under the planned linear fit is:

E_G(\hat{b}) = \frac{(-50)(2)+(0)(4.5)+(50)(4)} {(-50)^2+0^2+50^2} = \frac{100}{5000} = 0.02\ \text{units/mg}.

The linear analysis therefore estimates a 50 mg versus 0 mg contrast with expected value \(50(0.02)=1\) unit. Its bias for the target is \(1-2.5=-1.5\) units. This is not a numerical accident: the linear fit averages over the pattern of means, while the target is specifically the middle-dose contrast.

For a concrete decision rule, suppose the trial advances only if the estimated contrast exceeds 2 units. If there are 100 participants at each dose, the dose-centered sum of squares is \(100(5000)=500{,}000\ \text{mg}^2\). Under the stated independent residual errors, the standard deviation of the estimated 50 mg contrast is \(50(4)/\sqrt{500{,}000}=0.283\) units. Thus the probability of advancing is approximately \(1-\Phi((2-1)/0.283)=1-\Phi(3.53)\approx0.0002\). The true target is above the threshold, yet this planned analysis almost never meets it in this example: a substantial false-negative decision risk.

This derivation also clarifies a diagnostic point: a narrow interval around the fitted linear contrast would not remove its bias for the middle-dose target. Looking only at uncertainty around the fitted parameter can conceal a systematic mismatch between the model and the question being asked.

Nonproportional hazards and time-dependent effects

A proportional-hazards model assumes that the treatment-to-control hazard ratio is constant over follow-up. If the true hazard ratio changes with time, a single fitted Cox-model hazard ratio is generally a summary whose value can depend on event timing, censoring, and follow-up. It should not automatically be interpreted as the treatment effect at every time point.

Worked Example: Crossing Hazards with Equal 12-Month Survival

Consider a hypothetical trial with piecewise-constant hazards. During months 0–6, the control hazard is \(0.10\) per month and the treatment hazard is \(0.06\) per month. During months 6–12, these hazards are \(0.05\) and \(0.09\) per month, respectively. The hazard ratio is \(0.60\) early and \(1.80\) later, so proportional hazards does not hold.

For a constant hazard \(h\) over an interval of length \(t\), survival over that interval is \(e^{-ht}\). Control cumulative hazard through month 12 is \(0.10(6)+0.05(6)=0.90\), giving \(S_C(12)=e^{-0.90}=0.4066\). Treatment cumulative hazard is \(0.06(6)+0.09(6)=0.90\), giving \(S_T(12)=e^{-0.90}=0.4066\). The 12-month survival difference is therefore \(S_T(12)-S_C(12)=0\).

The curves nevertheless differ earlier: \(S_C(6)=e^{-0.60}=0.5488\), whereas \(S_T(6)=e^{-0.36}=0.6977\). A single hazard ratio cannot describe both the early benefit and the later hazard increase. For this scenario, a simulation should record the fitted Cox-model result and any resulting decision, but also calculate the prespecified 12-month survival target from the generating model. Calling the fitted summary a constant treatment effect would misrepresent the data-generating pattern.

Whether a decision is erroneous depends on the decision's target. If the intended claim is “hazard is lower throughout follow-up,” the late hazard ratio of 1.80 contradicts that claim. If the target is specifically 12-month survival difference, its value here is zero. These are different questions; simulation should label and evaluate the decision against the one actually intended.

Quantifying errors in a simulation

A simulation can make decision consequences explicit by applying the planned analysis and decision rule to every replicate, then comparing each decision with the target defined under the generating model. The following calculation uses stipulated hypothetical simulation output to illustrate the arithmetic; it is not a universal error rate for crossing hazards.

Worked Example: Estimating a False-Decision Rate

Suppose 5,000 hypothetical replicates are generated under the crossing-hazards scenario above. The planned rule declares success when a fitted Cox-model analysis has an estimated hazard ratio below 1 and a two-sided \(p\)-value below 0.05. Suppose 430 replicates meet that rule. Since the 12-month survival difference under the generating model is zero, the estimated rate of declaring success when that target is null is \(430/5000=0.086\), or 8.60%.

The approximate Monte Carlo standard error is:

SE(\hat{\pi}_G) = \sqrt{\frac{0.086(1-0.086)}{5000}} = 0.00396.

That is about 0.40 percentage points. It describes simulation precision, not uncertainty about whether any particular trial will succeed. The rate is a false-decision rate relative to the 12-month survival target. It is not automatically the Type I error rate for every hypothesis the Cox analysis might be interpreted as testing.

For each alternative generating model, report at least the target, estimated bias, interval coverage if relevant, decision rule, decision-error rate, and Monte Carlo error. If the analysis sometimes fails to produce an estimate, report that frequency and state how those replicates enter the decision calculation; silently dropping them can make performance look better than it is.

Diagnostics, alternatives, and limitations

Useful checks begin with the difference between what the planned model assumes and what the alternative model generates. For a dose-response, compare generated mean responses with the fitted response curve at each dose and inspect dose-specific residual patterns. For survival, inspect whether treatment effects vary over follow-up and whether a single summary obscures changes in direction. In both cases, compare the planned decision with decisions based on the prespecified clinical target.

Alternatives address different problems:

Fit a more appropriate model. A nonlinear dose-response model or a survival analysis that represents time-varying effects may align better with the generating pattern. Its assumptions still need evaluation.
Report a target-specific contrast. A mean contrast at a chosen dose or survival difference at a chosen time can answer a clearer clinical question than an unqualified model coefficient.
Use robust standard errors where appropriate. These can improve uncertainty estimates for some forms of variance misspecification, but they do not generally correct a biased point estimate, restore a wrong functional form, or turn a fitted parameter into a different estimand.

No finite set of generating models establishes that the planned analysis is unbiased in all possible settings. Results also depend on the chosen effect sizes, sample size, variability, follow-up, and decision rule. Interpret a small simulated bias relative to the target's units and clinical importance, and consider its Monte Carlo error before treating it as a precise finding.

Common pitfall: “The model converged and returned a small p-value, so its assumptions must be adequate.” Convergence only indicates that the fitting procedure returned a result; it does not establish that the model represents the data-generating process or estimates the intended target. Compare the fitted quantity and decision with the target under each specified alternative model.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A planned model estimates a linear dose slope. What additional definition is needed before calling that estimate biased under a nonlinear generating model?
In the nonlinear-response example, what are the true 50 mg contrast and the expected contrast from the linear fit?
Why is a single hazard ratio potentially misleading when the treatment hazard ratio changes from 0.60 to 1.80?
In 5,000 replicates, 250 decisions meet a specified rule. Calculate the estimated decision probability and its approximate Monte Carlo standard error.
Would a robust standard error generally remove bias caused by fitting a linear mean to a curved dose-response? Explain.

Solutions. (1) State the scientific target \(\theta_G\), including the contrast and its units; bias is the expected estimate minus that target. (2) The true contrast is \(4.5-2=2.5\) response units, while the expected fitted contrast is \(50(0.02)=1\) response unit. (3) The hazard ratio is time-dependent, so no single constant value describes both periods; a fitted Cox summary depends on event timing and follow-up. (4) The estimated probability is \(250/5000=0.05\). Its standard error is \(\sqrt{0.05(0.95)/5000}=0.00308\), or about 0.31 percentage points. (5) No, not generally. A robust standard error may address certain variance-estimation problems, but it does not make the fitted linear mean correct or remove the point estimate's bias for a target contrast.