Selecting Diagnostic Evidence For A Regulatory Report

Why Diagnostic Evidence Must Be Selected

A regulatory report needs to let a reviewer judge whether the population PK model describes the data adequately, where its predictions are most uncertain, and whether its apparent fit depends heavily on a small number of observations or subjects. A large appendix of plots is not automatically persuasive: repeated versions of the same display can obscure a meaningful pattern, while a plot without a clear definition may be impossible to interpret. The goal is a concise set of complementary evidence, selected to answer specific questions.

This is a reporting task, not a claim that one diagnostic plot can prove a model correct. Diagnostics can reveal incompatibilities between the fitted model and the data; they cannot establish that every model assumption is true or guarantee accurate predictions in a new population. The report should connect each display to a question, define what was plotted, and describe patterns without overstating what they show.

A Compact, Complementary Diagnostic Set

A useful core set often includes an observed-versus-population-prediction display, an observed-versus-individual-prediction display, residual displays against time and prediction, and summaries of the estimated random effects. The exact selection depends on the model, study design, and regulatory questions already framed for the analysis. These displays are complementary rather than interchangeable.

The population-prediction display asks whether the model describes the overall concentration pattern across the data without using each subject's fitted random effects. The individual-prediction display asks whether the model can describe subject-specific profiles after those effects are incorporated. Individual predictions will commonly look closer to observations because they use subject-specific information; that fact alone is not evidence that the population model predicts new individuals equally well.

Residual displays can help reveal structure hidden by an overall observed-versus-predicted trend. For example, a residual-versus-time plot can show systematic underprediction late in the dosing interval, while a residual-versus-prediction plot can expose a trend related to concentration magnitude. If standardized or weighted residuals are shown, define their construction and reference lines. A residual scale that is approximately centered at zero with spread near one may be expected under a suitable residual model, but this is not a universal exact rule: the residual definition, estimation method, sampling design, and dependence among observations matter.

Random-effects summaries commonly include distributions of empirical Bayes estimates (EBEs) and displays of those estimates against relevant covariates or design features. EBEs are subject-level estimates of latent random effects conditional on the fitted model and that subject's observations. They are not direct observations of the underlying random effects. In particular, shrinkage toward the population mean can make an EBE distribution appear narrower than the underlying distribution, especially when individual data are sparse or uninformative. A histogram of EBEs should therefore not be treated as a definitive test of the assumed random-effects distribution.

What a Caption and Summary Need to Say

A caption should be short enough to read but complete enough to remove ambiguity. Identify the analysis population or data subset, the quantities on each axis and their units, any transformations, the prediction or residual definitions, and any stratification. Explain reference lines and symbols, including whether points represent records or subjects. If panels use different scales or subsets, say so. Mention the number of subjects and observations when those counts help a reviewer understand how much information is represented.

The accompanying text should report what the diagnostic shows, not merely repeat the figure title. A balanced description might say that population predictions broadly followed observed concentrations, while residuals showed a modest positive pattern at late times in one dose group. It should identify the scope of the pattern and avoid implying that visual agreement proves validity. Where a summary statistic is supplied, define its denominator and aggregation: a record-weighted summary gives more weight to subjects with more records, whereas a subject-weighted summary gives each subject equal weight.

Worked Interpretation of Residual Evidence

A residual display is most useful when the report distinguishes a pattern from an isolated large value. For illustration, suppose a model produces standardized residuals for 120 records. Under the stated residual definition, values near zero are expected if predictions are appropriate on average; a systematic run of positive values can indicate underprediction in the corresponding region. The following example uses a simple count to make the evidence concrete, not to create a universal pass/fail threshold.

Random Effects and Potentially Influential Data

For random-effects reporting, choose displays that relate directly to the model's random-effects structure. A distribution plot of EBEs can show whether estimates are concentrated near zero or include unusual values; an EBE-versus-covariate display can help reveal remaining structure worth considering. Report the number of subjects contributing and explain that individual estimates have unequal precision. If shrinkage is substantial, apparent clustering or a narrow distribution should be interpreted cautiously rather than presented as evidence that the population distribution is truly narrow.

Potentially influential data require equally careful language. An observation can have a large residual without strongly affecting parameter estimates; conversely, a subject with an unusual dose-time design or sparse information may affect an estimate even without the largest residual. A report may identify records or subjects flagged by the prespecified influence diagnostics and state the exact flagging method. If a focused leave-one-subject-out diagnostic is used, define the quantity compared and report the resulting change transparently. Such a flag indicates that the fit may depend on that subject; it does not establish data error, justify exclusion, or show that the subject is unrepresentative.

Justifying a Concise Evidence Set

The reason to use complementary diagnostics can be expressed in terms of what each summarizes. A pooled residual mean weights each record equally. If subjects contribute unequal numbers of records, this is not the same as an average that weights each subject equally. Specifically, let subject \(j\) contribute \(n_j\) residuals with mean \(\bar{r}_j\), and let \(N=\sum_j n_j\). The record-weighted mean is

The subject-weighted mean is

where \(J\) is the number of subjects. These summaries coincide when all \(n_j\) are equal, but can differ when record counts vary. For example, a subject with many observations contributes more to the record-weighted summary. This is why an overall plot or pooled summary should not substitute for displays that reveal subject-level fit or subject-level patterns.

A report need not include both summaries in every case. It should choose based on the question and make the weighting clear whenever a numerical aggregate is shown. If a concern is isolated to a subset—such as a route, formulation, dose range, or sampling window—a targeted stratification may be more informative than another all-data plot. Avoid creating many small strata that invite unstable conclusions from very few subjects or records.

Diagnostics, Alternatives, and Common Pitfalls

Check whether the chosen displays cover the relevant concentration and time ranges, include the intended analysis set, and use consistent axes and definitions. Sparse regions, missing observations, extreme values, and concentrations near assay limits can make a smooth-looking plot less informative than it appears. A residual pattern should be checked against the corresponding records and study design before assigning it a model-based explanation. If an individual-prediction plot looks good while the population-prediction plot does not, report that distinction rather than presenting only the more favorable display.

The main alternative to a compact evidence set is to report every exploratory output. That may be appropriate in supplementary material when a complete archive is needed, but it is usually less effective in the main narrative: repetitive plots dilute attention, and unexplained displays shift interpretive work to the reviewer. The opposite alternative—showing only one summary statistic—is also inadequate when it hides time, concentration, or subject-level structure. A concise main set, accompanied by clearly identified supporting material where needed, balances traceability and readability.

• Mistake: treating a close individual-prediction plot as proof of population-level adequacy. Individual predictions incorporate subject-specific fitted effects. The correction is to show and interpret population predictions separately.
• Mistake: calling an unusual EBE a measured biological outlier. An EBE is an uncertain model-based estimate that may be affected by shrinkage. Describe it as an unusual estimate and examine the supporting data and diagnostics.
• Mistake: excluding a subject because a residual is large. A large residual is a reason to investigate, not an automatic exclusion rule. Check data quality, model assumptions, and the prespecified analysis procedures, and report decisions transparently.
• Mistake: reporting a residual mean without its weighting or coverage. State whether records or subjects are weighted equally, give the relevant counts, and avoid interpreting correlated records as independent evidence.