Why a Checklist Is Needed
A binary PQL fit can converge and produce plausible-looking coefficients even when its approximation is consequential. The practical question is therefore not simply whether software returned estimates, but whether the data and fitted model give reason to trust those estimates for the intended purpose. Event frequency, the distribution of events across clusters, the cluster design, fitted probabilities, estimated random-effect variability, and sensitivity to the estimation method all contribute evidence.
No single event-rate cutoff, cluster-size rule, or random-effect standard deviation establishes that PQL is adequate. A checklist is a structured way to identify accumulated warning signs and to decide whether further comparison is needed. The decision is also purpose-specific: a modest change in a descriptive fixed-effect estimate may matter less than a change that reverses a substantive conclusion or materially alters an interval.
The checklist below assumes the model specification is scientifically defensible and the data are correctly coded. It assesses the approximation used to fit that model; it cannot repair confounding, an inappropriate link, missing-data problems, or a random-effects structure that does not represent the design.
The Decision Checklist
Report the total numbers of events and non-events, then show how many clusters have zero, few, or many events. A reassuring overall event fraction can conceal many event-free clusters or a small number of clusters carrying nearly all events.
Report the number of clusters and the distribution of cluster sizes, not only the total sample size or average cluster size. Note whether information is spread across clusters or concentrated in a few large ones. Neither many observations in a few clusters nor many very small clusters automatically guarantees stable PQL estimates.
Look for conditional fitted probabilities close to zero or one, separation-like patterns, convergence warnings, and extreme working quantities. These are clues to investigate, not standalone proof of failure.
Report the estimated random-intercept variance \(\hat{\tau}^2\) or standard deviation \(\hat{\tau}\), and consider whether the estimate is near a boundary, large relative to the logit scale, or unstable across fits. A large estimate can combine with sparse outcomes to produce highly disparate cluster probabilities.
Compare the PQL result with a suitable alternative, such as likelihood-based integration, using the same data and model specification. Examine changes in coefficients, standard errors, variance estimates, intervals, and substantive conclusions. A stable fixed effect does not imply a stable variance estimate.
State which evidence is reassuring, which is concerning, and whether the remaining uncertainty is acceptable for the scientific purpose. If key conclusions change across reasonable methods, do not present PQL as providing a method-insensitive result.
Why Extreme Fitted Probabilities Deserve Attention
For a Bernoulli outcome with a logit link, a common working-response linearization at current fitted probability \(p\) and linear predictor \(\eta=\operatorname{logit}(p)\) uses $$ z=\eta+\frac{y-p}{p(1-p)},\qquad w=p(1-p). $$ The response \(z\) is a local linear approximation, and \(w\) is its working weight. This calculation helps explain why fitted probabilities near the boundaries can make the approximation delicate.
Worked Example: A Rare Event and a Large Working Response
Suppose a hypothetical observation has outcome \(y=1\) and current fitted probability \(p=0.01\). Its linear predictor is \(\eta=\log(0.01/0.99)\approx -4.595\). The working response is $$ z=-4.595+\frac{1-0.01}{0.01(0.99)} =-4.595+\frac{0.99}{0.0099} =-4.595+100 =95.405. $$ The working weight is \(w=0.01(0.99)=0.0099\).
Thus, one event under a very small fitted probability generates a large working-response value but a small weight. The linearization is local, so this combination is a warning to inspect the fit, especially when such observations or clusters are common. It does not by itself prove that the final coefficient is biased by a particular amount: the full pattern of data, iterations, and model structure matters.
This justification does not create a universal probability threshold. A few boundary fitted values may be unsurprising in a large data set, while moderate fitted probabilities cannot compensate for an event pattern concentrated in very few clusters. The relevant question is whether multiple warning signs converge and whether the target estimates are sensitive.
Worked Decisions in Three Hypothetical Settings
Worked Example: Several Reassuring Signals
Suppose a hypothetical study has 30 clusters, each containing 20 observations, for \(30(20)=600\) binary outcomes. There are 300 events and 300 non-events, so the observed event fraction is $$ \hat{p}=\frac{300}{600}=0.500. $$ Each cluster has between 8 and 12 events, so events are not concentrated in a handful of clusters. A fitted PQL model has conditional probabilities from 0.12 to 0.82, no reported convergence warning, and estimated random-intercept standard deviation \(\hat{\tau}=0.35\) on the logit scale.
For the focal fixed effect, PQL gives \(\hat{\beta}=0.40\) with \(SE=0.16\). A likelihood-based integration fit of the same model gives \(\hat{\beta}=0.42\) with \(SE=0.17\). The coefficient difference is \(0.42-0.40=0.02\), small relative to either standard error; both estimates indicate a positive conditional log-odds association.
The balanced cluster sizes, spread of events, non-extreme fitted probabilities, and close sensitivity results are collectively reassuring for this fixed effect. They do not certify every model feature: for example, the random-effect distribution and covariate specification still require substantive assessment. A defensible conclusion is that PQL appears reasonable for the stated fixed-effect purpose, with the alternative fit providing useful corroboration.
Worked Example: Sparse Events and Method Sensitivity
Consider a hypothetical data set with 40 clusters of 10 observations each, giving \(40(10)=400\) observations. There are 12 events and 388 non-events, so $$ \hat{p}=\frac{12}{400}=0.0300. $$ Only 9 clusters contain any events: 8 clusters have 1 event each and 1 cluster has 4 events; the remaining 31 clusters have none. The event count is therefore not only small overall but unevenly distributed across clusters.
Suppose PQL gives a focal coefficient of \(-1.20\), while likelihood-based integration gives \(-0.72\), with corresponding standard errors 0.55 and 0.50. The coefficient difference is \(-1.20-(-0.72)=-0.48\), nearly one standard error on either scale. If the fitted probabilities also range from 0.004 to 0.69 and the estimated random-intercept standard deviation is 1.10, the boundary probabilities and substantial cluster heterogeneity add to the concern.
These facts do not prove which estimate is closer to the model's target. They do show that PQL is not robust to the estimation approach for this example. The appropriate decision is to report the sensitivity, avoid presenting the PQL estimate as settled, and consider whether the data support the intended inference at all. The low event fraction alone would be an incomplete diagnosis; the cluster distribution and method comparison make the warning more specific.
Worked Example: A Healthy Overall Rate Hides Cluster Imbalance
Suppose a hypothetical data set has 6 small clusters of size 4 and 24 large clusters of size 30. Its total size is $$ 6(4)+24(30)=24+720=744. $$ The small clusters have no events, while the large clusters have 15 events each, giving \(24(15)=360\) events. The overall event fraction is \(360/744\approx0.4839\), which might appear reassuring if viewed alone.
However, all 360 events occur in the 24 large clusters, while all 24 observations in the small clusters are non-events. Suppose PQL estimates the focal fixed effect as \(0.30\), and likelihood-based integration estimates it as \(0.28\), with standard errors 0.12 and 0.13. Those fixed-effect results are close. Yet suppose the estimated random-intercept standard deviations are 0.90 under PQL and 0.52 under integration.
The event fraction and fixed-effect comparison give some reassurance for that coefficient, but the unequal cluster sizes, separation of event patterns by cluster-size group, and discrepancy in the variability estimate are important. The decision should be outcome-specific: the fixed effect may be reasonably stable in this comparison, while the random-effect variability is not. PQL would therefore need a clear caveat if the variance estimate is substantively important.
Comparing the Main Choices and Recognizing Failure Modes
The practical alternatives are not simply “PQL” and “no analysis.” The main comparison is between PQL and a method that evaluates the random-effects contribution more directly, such as likelihood-based integration. These methods can differ in computational demands and approximation behavior. A comparison is informative only when the fixed- and random-effects structure, data, and coding are held constant; otherwise, differences cannot be attributed to the estimation approach alone.
| Evidence | More reassuring pattern | Reason for concern |
|---|---|---|
| Events | Events and non-events occur across many clusters | Few events, many event-free clusters, or events concentrated in a few clusters |
| Cluster design | Information is reasonably spread across clusters | Very uneven sizes or a few clusters dominating the observations |
| Fitted probabilities | Few probabilities near zero or one; no major fitting warnings | Many boundary probabilities, extreme working quantities, or convergence problems |
| Random-effect variability | Estimate is stable and not evidently boundary-sensitive | Large, near-boundary, or method-sensitive variance estimate |
| Sensitivity | Target estimates and conclusions are similar across methods | Material changes in estimates, uncertainty, or substantive conclusion |
Diagnostics can fail in less obvious ways. A smooth convergence message is not a check of approximation bias. A moderate overall event fraction can disguise sparse clusters, and a stable fixed-effect estimate can coexist with an unstable variance estimate. Conversely, one warning sign does not automatically invalidate all PQL results. The checklist supports a reasoned assessment rather than a mechanical pass/fail label.
PQL is most defensible when the event information is distributed across a useful number of clusters, cluster sizes do not create severe concentration, fitted probabilities do not show pervasive boundary behavior, and relevant estimates are stable under a suitable alternative. When these conditions are doubtful, report the concern and sensitivity rather than implying that a single fitted model resolves it. The limitation remains that any comparison is conditional on the chosen model and available data.
Common Mistake
Correction: The overall fraction ignores how events are distributed across clusters, and convergence only indicates that an algorithm met its stopping rule. Check cluster-level event counts and sizes, fitted probabilities, random-effect variability, and sensitivity of the particular estimates you plan to interpret.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A data set has 500 outcomes and 250 events, but all events occur in 10 of 50 clusters. Is the overall event fraction enough to support PQL?
- At a current fitted probability \(p=0.02\), calculate the working weight \(p(1-p)\). What does this value suggest you should inspect?
- PQL and likelihood-based integration give similar fixed-effect coefficients, but substantially different random-intercept standard deviations. What can and cannot be concluded?
- Why should cluster-size distribution be reported alongside total sample size?
- Give two examples of sensitivity evidence that would make a PQL conclusion more concerning.
Solutions. (1) No. The event fraction is \(250/500=0.500\), but concentration in 10 of 50 clusters is a warning to examine cluster-level information and sensitivity. (2) \(0.02(1-0.02)=0.02(0.98)=0.0196\). This small working weight, particularly if boundary probabilities are common, motivates inspection of fitted probabilities and working quantities; it is not a standalone failure criterion. (3) The fixed effect appears stable in that comparison, but the variance estimate is method-sensitive. Do not claim that both parts of the model are equally reliable. (4) The same total number of observations can represent many informative clusters or observations concentrated in a few clusters; these designs need not provide the same evidence for cluster-level variation. (5) Examples include a material shift in the target coefficient, a meaningful change in its standard error or interval, a changed substantive conclusion, or a large difference in the random-effect variance across methods.