Standardized Effects And Plausible Trial Differences

Why Standardized Effects Need a Clinical Scale

A prior on a standardized treatment effect can be easier to elicit or compare across settings than a prior stated in endpoint units. But standardization does not make the clinical scale irrelevant. A standardized effect of \(-0.3\) corresponds to different mean differences when the endpoint standard deviation is 5 units versus 20 units. Prior predictive checking therefore asks a concrete question: what raw-scale treatment differences does this prior imply for this endpoint?

This tutorial considers a continuous endpoint measured at a specified follow-up time and a two-arm trial. The earlier material on predictive means under Normal endpoint models covers the basic construction of predictive arm means. Here the focus is translating standardized effects into raw units and judging the resulting differences against the endpoint’s measurement scale and clinical context.

The sign convention here is treatment minus control. Thus, if lower endpoint values are better, negative values of \(\Delta\) and \(\delta\) favor treatment; if higher values are better, positive values favor treatment. State the direction explicitly in an application. Standardization changes the scale, not the clinical meaning of the endpoint.

Model and Prior Predictive Translation

For the basic model, suppose that each participant contributes one endpoint measurement, participants are independent within and across randomized arms, and the endpoint has a Normal distribution with a common standard deviation:

The \(N(\mu,\sigma^2)\) notation uses the second argument as the variance. The model assumes the same endpoint definition, measurement units, and follow-up time in both arms; a common within-arm spread; and no unmodeled clustering or dependence among participants. The prior must describe plausible values of the parameters in this model. In particular, a prior on \(\delta\) does not by itself determine \(\Delta\): the scale \(\sigma\), fixed or drawn from a prior, is also needed.

In a prior predictive simulation, draw the parameters from their joint prior and calculate \(\Delta=\delta\sigma\) for each draw. If trial data are also simulated, draw participant outcomes under the model and calculate the observed arm-mean difference. These are related but distinct quantities: \(\Delta\) is the parameter contrast, while the observed difference varies around it because of sampling.

The fixed-scale result follows directly from the rules for a linear transformation of a Normal random variable. With uncertain scale, the product \(\delta\sigma\) must be evaluated under the joint prior. If \(\delta\) and \(\sigma\) are independent, their independent draws can be paired in the simulation. If they are dependent, the simulation must preserve that dependence rather than sample from separate marginal priors.

Why the Observed Trial Difference Is More Variable

For a trial with \(n_C\) and \(n_T\) participants in the two arms, let \(D=\bar{Y}_T-\bar{Y}_C\) be the observed difference in sample means. Under the Normal model and conditional on the parameters, independent observations give \[ D\mid\delta,\sigma\sim N\!\left(\delta\sigma,\ \sigma^2\left(\frac{1}{n_C}+\frac{1}{n_T}\right)\right). \] The conditional mean is the parameter difference; the conditional variance is sampling variance. This distinction matters when a prior predictive check concerns trial-level summaries rather than only the clinical effect parameter.

With fixed \(\sigma\), a Normal prior on \(\delta\) also gives a Normal prior predictive distribution for \(D\). The derivation uses the law of total expectation and the law of total variance:

The first variance term is trial sampling variability; the second is prior uncertainty about the treatment effect. This result assumes the specified Normal model, fixed common \(\sigma\), and a Normal prior for \(\delta\). With an uncertain \(\sigma\), the predictive distribution generally requires integrating over its prior or simulating; the fixed-scale Normal formula should not be applied as if the scale were known.

When the Endpoint Scale Is Uncertain

The scale may itself be uncertain, particularly before a trial has collected endpoint data. In that case, simulation makes the translation explicit: draw \(\sigma\) and \(\delta\) jointly from their prior, calculate \(\Delta=\delta\sigma\), and inspect the resulting raw-unit differences. For trial-level checks, then simulate observations under the endpoint model and calculate \(D\). The output should retain the endpoint’s units so it can be judged against relevant clinical knowledge.

Diagnostics, Alternatives, and Limitations

A useful check is to compare the simulated raw differences with quantities that have meaning on the endpoint’s scale: the instrument’s possible range, known measurement resolution, relevant historical variability, and clinically credible differences for the specified population and follow-up. The basis for any comparison range should be stated. For example, a difference larger than the instrument can record signals a support or scale problem; a difference that conflicts with clinical knowledge may instead signal an implausible prior or an inappropriate model. Check both directions when effects can favor either arm.

Inspect the scale draws as well as the effect draws. A modest standardized effect multiplied by an unexpectedly large \(\sigma\) can yield a large raw difference. Conversely, an apparently substantial standardized effect may imply a small change in original units when \(\sigma\) is small. If an endpoint has natural bounds, a Normal model may also generate impossible individual outcomes even when the mean difference looks reasonable; this tutorial’s contrast calculation does not remove that model concern.

A prior directly on \(\Delta\) is a reasonable alternative when elicitation is naturally expressed in endpoint units or when clinical decisions are tied to a raw-unit threshold. A prior on \(\delta\) can aid communication across scales, but it requires a relevant \(\sigma\) model and can obscure the implications when that scale is uncertain. Neither parameterization is automatically preferable: compare the resulting prior predictive implications in the endpoint’s units. For unequal arm variances, a single common \(\sigma\) no longer defines the model’s standardized effect in the same way; the standardization convention and variance model must be specified rather than silently applying the common-SD formula.

The central diagnostic is not whether a standardized effect looks familiar in isolation. It is whether the specified joint prior and endpoint model generate parameter differences, and if needed trial-level differences, that are credible for this particular measurement scale and study setting. This check cannot establish that the prior is correct; it can expose consequences that deserve reconsideration before the data are analyzed.