Choosing the Reference Prediction in a pcVPC

Why the Reference Prediction Matters

A prediction-corrected visual predictive check (pcVPC) rescales outcomes to make records with different typical predictions more comparable. This is useful when typical predictions vary across time, dose, or covariate values. But the correction needs a reference prediction: the scale to which each record is rescaled. Choosing that reference is not just a plotting detail. A single reference used everywhere and a different reference for each bin can produce different across-bin patterns, even when both preserve the same within-record proportional discrepancy.

The reference should be chosen to match the comparison the pcVPC is meant to support. A pooled reference gives one scale across the analysis set. Bin-specific references put each bin on its own typical scale. Group-specific references put each defined subgroup on its own scale. These choices determine which differences remain visible and which are normalized away.

Definition: Let $Y_q$ be the observed outcome for record $q$, and let $m_q$ be its positive typical prediction, calculated using the fixed effects and the record’s design and covariates, with individual random effects set to their typical value. For a chosen positive reference prediction $r_q$, the prediction-corrected outcome is $$ Y_{q,\mathrm{pc}} = Y_q\frac{r_q}{m_q}. $$ The same scaling is applied to simulated outcomes for the corresponding records. A reference is pooled if it is common to the analysis set, bin-specific if it is chosen separately for each time or design bin, and group-specific if it is chosen separately for each defined subgroup. The reference may, for example, be the median typical prediction in the relevant set.

This tutorial concerns the choice of $r_q$, not how to form or summarize bins. For a valid comparison, define the reference set and the typical predictions used to calculate it, then apply that choice consistently to observed and simulated outcomes. In particular, do not choose the reference after inspecting which choice makes the observed plot look most favorable.

Assumptions Behind the Correction

The usual multiplicative correction has a clear interpretation when the outcome is positive and departures from the typical prediction are meaningfully expressed as relative differences. It assumes that scaling an outcome by $r_q/m_q$ is scientifically interpretable. The correction does not make unlike outcome scales comparable automatically, and it does not repair a misspecified model.

Positive, usable typical predictions. The ratio $r_q/m_q$ must be defined and reasonably stable. Predictions near zero can make small outcome differences produce large corrected differences.
A meaningful multiplicative scale. The correction preserves proportional deviations from typical prediction. It is not generally neutral to additive deviations.
Comparable records. The records being pooled or compared should belong to the intended target population and have a meaningful common interpretation after correction.
Consistent application. Use the same reference definition for observed and simulated outcomes. If a reference is estimated from a specified design set, keep that definition fixed across the predictive check.

The typical prediction $m_q$ is not the observed outcome and is not a subject-specific empirical Bayes prediction. It represents the model’s typical expected level for that record under the stated covariates and design. The reference $r_q$ changes the scale of the corrected outcome; it does not change the original model prediction or the simulated data-generating process.

What the Choice Preserves

The algebra shows why a common reference and a bin-specific reference answer different visual questions. Suppose an observed outcome has a relative deviation $\delta_q$ from its typical prediction, so that $Y_q=m_q(1+\delta_q)$. Then:

Y_{q,\mathrm{pc}} = m_q(1+\delta_q)\frac{r_q}{m_q} = r_q(1+\delta_q).

Thus the corrected outcome retains the relative deviation $ \delta_q $, expressed around the chosen reference. If every record uses the same $r$, the corrected level associated with that relative deviation is on one common scale. If each bin uses a different $r_b$, the same relative deviation is expressed around a different level in each bin. As a result, between-bin differences in corrected levels can change simply because the references differ.

This is a scale statement, not a claim that any particular reference makes the model fit better. The simulations must still be compared with the observations after the same correction. A reference choice can emphasize within-bin agreement while removing typical differences between bins, or preserve a common scale while leaving variation in typical predictions across records.

Worked Example: One Pooled Reference Versus Bin-Specific References

Consider four hypothetical records in two bins. Their typical predictions are 8 and 12 units in bin 1, and 18 and 22 units in bin 2. Suppose each observed outcome is exactly 20% above its typical prediction. The observed outcomes are therefore 9.6, 14.4, 21.6, and 26.4 units. Compare a pooled median reference with a median reference within each bin.

Pooled reference. The median of the four typical predictions, 8, 12, 18, and 22, is $r=15$ units. For the first record:

Y_{1,\mathrm{pc}}=9.6\frac{15}{8}=18.0\text{ units}.

For the other records, the correction likewise gives $14.4(15/12)=18.0$, $21.6(15/18)=18.0$, and $26.4(15/22)=18.0$ units. Each record is 20% above the common reference of 15 units.

Bin-specific references. The bin medians are 10 units for bin 1 and 20 units for bin 2. In bin 1, the corrected outcomes are $9.6(10/8)=12.0$ and $14.4(10/12)=12.0$ units. In bin 2, they are $21.6(20/18)=24.0$ and $26.4(20/22)=24.0$ units. Again, each outcome is 20% above its own reference.

The two corrections preserve the same proportional deviation, but their across-bin levels differ: the pooled correction puts both bins at 18 units, whereas the bin-specific correction puts them at 12 and 24 units. The references, rather than a change in the observed relative deviations, create that difference.

Choosing Among Pooled, Bin-Specific, and Group-Specific References

A pooled reference is suitable when the intended interpretation uses one common corrected scale across the analysis set. It avoids assigning a separate scale to each bin, but does not remove all effects of differences in typical prediction. If typical predictions vary substantially across bins or groups, corrected levels and their interpretation should be considered in that context.

A bin-specific reference is useful when the intended comparison is primarily within bins and typical prediction changes systematically across those bins. It expresses each bin relative to its own typical level. Consequently, it should not be treated as preserving the original between-bin typical level differences. The reference can also be unstable when a bin contains few records or a narrow, unrepresentative range of typical predictions.

A group-specific reference is useful when comparing model performance within covariate or design groups while placing each group on its own typical scale. This can help prevent differences in typical levels between groups from dominating within-group comparisons. But if the question concerns whether the model captures differences in typical levels between those groups, group-specific normalization may remove part of the pattern of interest. A reference can also be defined for group-by-bin cells; that gives still more local scaling and requires enough relevant records in each cell.

Reference choice	Scale used	Most directly supports	Main caution
Pooled	One reference for the analysis set	Comparisons on a common corrected scale	Differences in typical predictions can remain relevant to across-bin interpretation
Bin-specific	A separate reference for each bin	Comparisons relative to each bin’s typical level	Across-bin corrected levels depend on the changing references
Group-specific	A separate reference for each defined group	Within-group comparisons on group-specific scales	Between-group typical-level differences may be normalized away

Worked Example: Preserving or Normalizing Group Differences

Suppose two hypothetical covariate groups have typical predictions of 10 and 30 units. Their observed outcomes are 11 and 33 units, respectively, so both are 10% above their own typical predictions. Compare a pooled median reference with group-specific median references.

With a pooled reference, the median of 10 and 30 is 20 units. The corrected outcomes are $11(20/10)=22$ and $33(20/30)=22$ units. The two groups appear at the same corrected level because each has the same proportional excess relative to its typical prediction.

With group-specific references, the references are 10 and 30 units. The corrected outcomes are $11(10/10)=11$ units and $33(30/30)=33$ units. Their original group-level difference remains visible, while each is still 10% above its own reference.

Neither scale is universally preferable. If the question is whether relative departures from typical prediction differ by group, a pooled scale makes the equal proportional departures easy to compare. If the question requires retaining the groups’ different typical levels, group-specific normalization is not a substitute for examining those levels on an appropriate common scale.

A Practical Selection and Review Process

State the comparison.
Decide whether the goal is a common scale across the analysis set, comparisons relative to each bin’s typical level, or comparisons within groups.

Define the reference sets.
Specify which records contribute to each reference and how it is calculated, such as the median typical prediction. Check that the sets reflect the intended target population.

Check prediction support.
Inspect the typical predictions and references for values near zero, extreme values, and sparse groups or bins. A mathematically defined ratio may still yield unstable corrections.

Apply the choice consistently.
Use the defined reference rule for both observed and simulated outcomes. Do not let observed outcomes determine different reference choices for different simulation replicates.

Interpret the scale explicitly.
Report whether corrected values use a pooled, bin-specific, or group-specific reference. Explain which differences that choice preserves and which it normalizes.

As a useful sensitivity check, compare a small number of scientifically defensible reference choices. If the substantive interpretation changes, report that dependence rather than selecting the plot that looks most reassuring. Such a comparison assesses sensitivity to scaling; it does not replace evaluation of the model’s predictive performance.

Derivation and a Key Limitation

The correction is motivated by proportional variation in typical prediction. Suppose an outcome can be represented as a typical level multiplied by a relative factor, $Y_q=m_q U_q$, where $U_q$ is the record’s outcome relative to its typical level. Applying the correction gives:

Y_{q,\mathrm{pc}} = m_q U_q\frac{r_q}{m_q} = r_q U_q.

The record-specific typical level $m_q$ cancels, while its relative factor $U_q$ remains on the reference scale. This explains why prediction correction is natural for multiplicative differences and why reference selection determines the level at which those relative differences are displayed.

For an additive representation, $Y_q=m_q+\varepsilon_q$, the same operation instead gives:

Y_{q,\mathrm{pc}} = (m_q+\varepsilon_q)\frac{r_q}{m_q} = r_q + r_q\frac{\varepsilon_q}{m_q}.

The additive error is rescaled by $r_q/m_q$, so its corrected magnitude depends on the typical prediction. If additive errors are the scientifically relevant scale, this multiplicative correction may change the pattern one wants to assess. The correction should therefore be justified by the outcome scale and model structure, not chosen solely because it makes records easier to place on one graph.

Common Mistake and Failure Modes

Common mistake: “Bin-specific references are more accurate because each bin gets its own typical prediction, so a corrected difference between bins must reflect model performance.” This is not generally correct. A bin-specific reference changes the scale from bin to bin; it can remove or create apparent differences in corrected levels that are not differences in proportional deviations. The correction does not make across-bin levels directly comparable when the reference itself changes.

A second failure mode is using an unstable reference. If a typical prediction is close to zero, multiplying by $r_q/m_q$ can greatly amplify the outcome and its variability. If predictions can be zero or negative, this ratio-based correction may be undefined or lack a sensible interpretation. In such cases, do not silently apply the formula: reconsider whether prediction correction is appropriate for the endpoint and scale.

Finally, a reference selected from a small subgroup can be noisy, and a subgroup-specific reference can conceal a poorly represented group-level trend. Check the number and range of typical predictions contributing to each reference. State the reference definition alongside the pcVPC interpretation so readers can distinguish observed predictive patterns from consequences of rescaling.

Key takeaway: Choose the reference prediction to match the intended comparison. A pooled reference puts records on one corrected scale; bin-specific or group-specific references make comparisons relative to local typical levels and can normalize away differences between those bins or groups.

Check Your Understanding

Answer each question using the stated reference definition. Solutions are given below.

An outcome of 24 units has a typical prediction of 20 units and is corrected to a reference of 15 units. What is the corrected outcome, and what is its relative deviation from the reference?
Two bins have typical predictions of 5 and 25 units. If both observations are 20% above their typical predictions, what happens to their corrected values under a common reference of 15 units? What happens under bin-specific references equal to the bin predictions?
Why can a group-specific reference be unsuitable when the target is to assess differences in typical levels between groups?
Name two conditions or diagnostics to check before using a multiplicative prediction correction.

Solutions. (1) $24(15/20)=18$ units; this is $18/15-1=0.20$, or 20% above the reference. (2) The observations are 6 and 30 units. Under the common reference, both correct to $6(15/5)=18$ and $30(15/25)=18$ units. Under bin-specific references equal to the predictions, they remain 6 and 30 units, each 20% above its own reference. (3) Each group is rescaled to its own typical level, so between-group typical-level differences can be normalized away. (4) Check that typical predictions and references are positive and not near zero, and that multiplicative deviations are scientifically meaningful; also verify that reference sets are adequately supported and that the same rule is applied to observed and simulated outcomes.