4 Pillars Toolkit for Adolescent Vaccination: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for 4 Pillars Toolkit for Adolescent Vaccination (NCT02165722), a randomized na study. The primary outcome is Change in Percentage of Cumulative HPV Series Initiation Vaccination Rates for Adolescents 11-17 Years of Age.

2. Clinical Question

The study is listed as a randomized, parallel cluster trial with no masking. The intervention was directed at clinical practices, while the primary outcome concerns adolescents’ HPV series initiation. This makes the practice setting central to interpreting the comparison: outcomes from adolescents are grouped according to the practices assigned to each study arm. The registry does not describe the number of practices in each arm beyond identifying 11 control practices, nor does it clarify how the enrollment count of 22 relates to the clusters.

The study’s broader stated purpose was to test whether the Toolkit increased adolescent vaccination rates, particularly for influenza and HPV, and also for Tdap and MCV. The posted primary outcome is narrower: change in cumulative HPV series initiation rates by age group. The registry record does not report a primary question about completion of the HPV series or a combined measure across the different vaccines.

3. Trial Design

The registry identifies the study as a randomized controlled cluster trial in pediatrics and family medicine. Allocation was randomized, the model was parallel, and masking was listed as none. The intervention was the 4 Pillars Toolkit; the comparator was standard practice. The record describes the intervention as behavioral and says the practices used electronic medical records.

Because this is a cluster trial, assignment is at the practice level rather than at the level of each adolescent. That feature matters statistically: outcomes from adolescents treated in the same practice may be more alike than outcomes from adolescents treated in different practices. General statistical principle: analyses of cluster-randomized studies need to account for the way assignment was made and for possible within-practice similarity when estimating uncertainty. The registry names Chi-squared tests for the reported comparisons, but whether those tests account for clustering is not reported in the registry record.

The control-arm description identifies 11 clinical practices as control sites. The registry enrollment count is 22, but the unit represented by that count is not stated. The number of intervention practices is not reported in the registry record. The study is listed as completed, with a start date of 2013-09 and a primary completion date of 2016-06. The registry lists phase as NA.

The Toolkit was evaluated in the context of routine clinical practice. The registry summary says that the vaccines were FDA licensed and were to be used according to national guidelines. It does not detail which Toolkit components were implemented at each practice, how consistently they were implemented, or what standard practice entailed. Those details are important for understanding how similar or different the care environments were, but are not reported in the registry record.

The design is suitable for asking whether outcomes differed between practices assigned to the Toolkit and those assigned to standard practice, provided the analysis reflects cluster assignment. The registry record does not provide enough detail to determine how the analysis handled practice-level clustering, whether the randomized assignment was preserved in the analysis, or whether any adjustments were made for baseline differences.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

Analysis population for the primary outcome, as defined in the registry: HPV series initiation stratified by age of 11-13 years and 14-17 years of age. Baseline period was 7/1/2013 to 6/30/2014. Intervention period was 7/1/2014 to 3/31/2015.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Change in Percentage of Cumulative HPV Series Initiation Vaccination Rates for Adolescents 11-17 Years of Age, measured over Baseline period to post-intervention.

The primary endpoint is the change in cumulative HPV series initiation vaccination rates, stratified by age group: 11-13 years and 14-17 years. Series initiation means receipt of the first dose of HPV vaccine, as described in the registry. The endpoint is framed as a change from the baseline period to the post-intervention period rather than as a measure of completing the full HPV vaccine series.

The registry describes the baseline period as 7/1/2013 to 6/30/2014 and the intervention period as 7/1/2014 to 3/31/2015. It also describes the primary outcome as change from baseline to the end of year 1. The recorded unit is percent change in vaccination rates. No confidence interval or standard error is reported for the arm-level values.

Series initiation is relevant to the stated study aim because it captures whether adolescents received a first HPV vaccine dose. It does not, on its own, indicate whether later doses were received. Separating the results by age group also avoids treating adolescents aged 11-13 years and 14-17 years as a single group, though the registry does not explain the rationale for those strata.

The registry reports values of 13.7 and 10.1 for the intervention and control groups, respectively, among adolescents aged 11-13 years. For adolescents aged 14-17 years, it reports 7.8 and 5.7, respectively. These are the posted arm-level outcome values; the record does not provide uncertainty intervals around them or a single adjusted estimate of the Toolkit-versus-standard-practice difference.

7. Why No Re-analysis Is Shown

8. Statistical Interpretation

9. Statistical Methods Explained

The registry lists Chi-squared tests for the reported analyses. In general, a Chi-squared test evaluates whether observed counts differ from counts expected under a specified null hypothesis. It is often used with categorical outcomes, such as whether an event occurred. The registry does not state the count construction, the exact null hypothesis for each test, or whether the tests used any adjustment. It reports p < 0.001 for each of the four analyses.

The four reported pairings address different questions. One compares the two age groups within the intervention arm, and another makes that age-group comparison within the control arm. The other two compare intervention and control within each age group. Results from these tests should not be conflated: an age-group comparison within one arm is not the same estimand as a comparison between trial arms. None of the four listed tests is identified as a formal test of whether the intervention effect differs by age.

General statistical principle: a p-value measures how incompatible the observed data are with a specified null model, assuming that model and its assumptions are appropriate. It is not a measure of effect magnitude, and a small p-value does not supply an uncertainty interval. The registry’s p-values are reported as thresholds rather than exact values. Since no confidence intervals or standard errors accompany the endpoint values, the precision of the arm-level changes and the between-arm contrasts cannot be judged from those values alone.

Cluster randomization changes the structure of the data. Adolescents treated at the same practice share a setting and may have similar opportunities for vaccination. General statistical principle: such within-cluster correlation reduces the amount of independent information relative to a study in which every participant is independently assigned. Valid uncertainty estimates usually need to reflect both cluster assignment and any within-cluster correlation. The registry reports the cluster design, but does not describe the statistical treatment of clustering, give a cluster-adjusted effect estimate, or report a cluster-level analysis. The status of cluster adjustment is not reported in the registry record.

The endpoint is described as change in cumulative vaccination rates from baseline to the end of year 1, with the baseline and intervention periods specified in the record. The reported arm values are not accompanied by a definition of the calculation used to obtain the change, beyond the endpoint description and unit. The record does not say whether the analysis compared changes directly, adjusted for baseline rates, or used another model. It also does not report whether the posted tests used individual-level or practice-level data.

For a cluster-randomized study, the appropriate analysis can depend on the number of clusters, how allocation was conducted, and how outcomes were aggregated or modeled. General statistical principle: if a method assumes independent observations while the design creates correlated observations, its standard errors and p-values may not reflect the design. This is a reason to look for a stated cluster-adjusted method; it is not evidence that the analysis was or was not adjusted. The registry does not provide the information needed to resolve that issue.

The record does not report a confidence level or interval method for the primary endpoint, and it gives no interval estimates for the reported values. A confidence interval, in general, is a range of values generated by a procedure that reflects sampling uncertainty under stated assumptions. Its width helps communicate precision, while its location conveys the range of effect sizes compatible with the data and method. Without intervals or another measure of uncertainty, the p-values cannot show whether the estimated differences are tightly or broadly determined.

Multiplicity is also relevant because the registry lists four tests across age groups and arms, and the study’s broader purpose mentions several vaccines. General statistical principle: conducting multiple tests creates more opportunities for a nominally small p-value to occur by chance, so an analysis plan may specify primary comparisons or an adjustment strategy. Whether a multiplicity procedure was planned or used is not reported in the registry record. The record also does not state whether these four tests were prespecified as a family or whether other analyses were conducted.

Missing-data methods and analysis populations are not described. The record does not report how missing vaccination records, incomplete follow-up, or changes in practice participation were handled, nor does it identify an intention-to-treat or other analysis population. These omissions matter because the outcome is derived from vaccination rates over periods of observation, and the relationship between the listed arm denominators and the number of adolescents with complete outcome data is not specified.

The sample-size justification and any interim analyses are not reported in the registry record. This means the record does not explain the assumptions used in designing the study or whether the analyses were monitored during conduct. No claims about those design choices can be made from the posted endpoint results alone.

10. Analyses Reported by the Sponsor

The registry lists the University of Pittsburgh as sponsor and reports four Chi-squared analyses, each with p < 0.001. The registry’s reported tests are the inferential results available for the primary HPV initiation outcome. The arm-level values are also reported for each age group, but no confidence intervals or adjusted intervention effect estimates accompany them.

The discrepancy indicator is false, so there is no flagged disagreement between the sponsor-reported analysis and any recalculation. The registry record does not report a separate recalculated estimate or provide enough detail to reconstruct the analysis method, including whether it accounts for clustering. Accordingly, the reported Chi-squared results remain the relevant posted inference; they should not be replaced with an unadjusted re-analysis that does not match the trial’s design or stated methods.

Even with no flagged discrepancy, the limits of the reported analysis remain important. The registry does not specify the tests’ exact hypotheses, how the cluster design was handled, or how multiple comparisons were considered. Nor does it provide confidence intervals for the observed values. The reported p-values should therefore be read as the registry’s analysis results, without treating them as a complete measure of effect size, precision, or between-age-group differences.

11. Multiplicity, Missing Data and Analysis Populations

The registry reports four Chi-squared tests for the primary outcome: age-group comparisons within each arm and intervention-versus-control comparisons within each age group. Each is reported as p < 0.001. The record does not report whether these tests were prespecified as a family, whether one comparison was designated as primary for inference, or whether a multiplicity adjustment was planned or applied. It also does not describe how the broader study aims involving influenza, Tdap, and MCV relate to the testing strategy.

General statistical principle: when multiple hypotheses are tested, the chance of at least one false-positive result can increase if each test is interpreted independently at the same threshold. A prespecified hierarchy or adjustment can help control that issue. No such approach is described in the registry record, so the p-values should be understood as reported for their individual tests, without assuming that they account for the full set of analyses.

Missing-data handling is not reported in the registry record. The outcome depends on cumulative vaccination rates over baseline and intervention periods, but the record does not describe how unavailable vaccination information, incomplete observation, or missing practice-level data were addressed. The relationship between the listed group denominators and the analysis population is also not explained.

The analysis population is not identified as intention-to-treat, per-protocol, or another defined group. It is not reported whether all randomized practices were included in the analyses or whether participants were analyzed according to the practice’s assigned arm. These details are especially relevant in a cluster-randomized design because the practice is the unit of assignment.

The registry does not report a sample-size justification, an interim-analysis plan, or a detailed statistical analysis plan. It also does not specify whether the Chi-squared tests account for clustering or whether any covariate adjustment was used. These unreported items limit assessment of prespecification, precision, and robustness; they do not establish that a particular method was absent from the full study conduct.

12. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The registry lists serious adverse events and other adverse events for intervention sites and control sites. For intervention sites, 4942 were listed as at risk, with 0 serious events and 0 other events reported. For control sites, 5919 were listed as at risk, with 0 serious events and 0 other events reported. The record does not list deaths for either group, and no top serious-event terms are reported.

These entries describe what was recorded in the adverse-event fields; they do not establish that no adverse events occurred beyond the scope of what was collected or entered. The registry does not explain the adverse-event collection period, definitions, ascertainment procedures, or whether these figures refer to individuals, visits, or another unit. It also does not explain how the adverse-event groups relate to the primary-outcome denominators.

General statistical principle: observing no reported events does not by itself establish that the underlying event risk is zero. Interpretation depends on how events were sought, the duration and completeness of observation, and the population included. Those details are not reported in the registry record, so the safety figures are best read narrowly as zero events entered in the listed fields, rather than as proof of absence of all harms.

13. What the Result Does, and Does Not, Mean

The posted results describe changes in cumulative HPV series initiation rates for adolescents in two age groups. The registry reports higher arm-level change values in the intervention group than in the control group for both age strata: 13.7 versus 10.1 for ages 11-13 years, and 7.8 versus 5.7 for ages 14-17 years. It also reports p < 0.001 for each intervention-versus-control Chi-squared comparison. These are the registry’s reported outcome values and tests.

The results do not provide a confidence interval or a single adjusted estimate of the difference between arms. Therefore, they do not show how precisely the between-arm differences are estimated. The reported p-values do not substitute for such estimates and do not communicate the magnitude of the difference on their own.

The record does not report a formal test comparing the intervention effects across age strata. The within-arm comparisons between age groups do not answer that question. It also does not clarify whether the reported tests account for practice-level clustering, a defining feature of the randomized design. These limitations constrain the strength of conclusions that can be drawn from the posted statistical results.

The endpoint is HPV series initiation, defined as receipt of the first dose, rather than completion of the series. Although the study’s broader purpose includes adolescent influenza, Tdap, and MCV vaccination, results for those outcomes are not reported in the registry record used. The reported primary outcome should not be generalized to those other vaccines or to outcomes that were not posted.

The findings are specific to the practices, period, intervention, comparator, and outcome described in the registry. The record does not provide enough detail to assess implementation consistency, missing data, analysis populations, or cluster-adjusted precision. The results therefore describe the posted pattern of arm-level changes and tests, while leaving important questions about uncertainty and analysis methods unresolved. They do not, by themselves, establish a causal effect with a known degree of precision.

14. Why This Trial Matters Statistically

This study illustrates why a cluster-randomized design requires attention to more than the number of adolescents recorded in each arm. Practices, rather than individual adolescents, were assigned to the Toolkit or standard practice. Outcomes from people in the same practice may be correlated, so the way the analysis accounts for clusters is central to interpreting the reported uncertainty. The registry identifies the cluster design but does not specify whether clustering was incorporated into the Chi-squared analyses.

The endpoint also demonstrates how a study’s broad public-health question can differ from its posted primary outcome. The summary names several adolescent vaccines, while the primary result concerns HPV series initiation, stratified by age. That distinction helps keep interpretation tied to the outcome actually reported rather than extending a finding to other vaccines or to series completion.

Finally, the record reports small p-value thresholds but no confidence intervals or adjusted effect estimates. This is a useful reminder that statistical evidence against a null hypothesis is not the same as a precise estimate of the size of a difference. A reader needs both the outcome scale and an account of uncertainty, with methods that reflect randomization and data structure, to assess what an analysis can support.

The result is therefore informative as a registry-reported comparison, while also showing why transparency about cluster adjustment, multiplicity, missing data, and analysis populations is important. Those details are needed to evaluate how robustly the posted tests represent the design and the outcome.

15. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

A central limitation in interpreting this trial-specific record is that the analysis method is only partly described. The registry names Chi-squared tests but does not state whether they account for practice-level clustering. Because assignment occurred by cluster, this omission prevents a clear assessment of whether the reported p-values reflect the study design.

The outcome values are reported without confidence intervals, standard errors, or an adjusted arm contrast. The registry therefore gives limited information about the precision of the changes and does not provide a direct interval estimate for the intervention-versus-control effect. The stated unit is percent change in vaccination rates, but the calculation underlying the arm-level change is not described in further detail.

The registry gives an enrollment count of 22 but does not identify the unit represented by that count. It identifies 11 clinical practices in the control-arm description, but does not state the number of intervention practices. It also does not clarify the relationship between those numbers and the outcome denominators. This makes it difficult to assess the cluster structure from the registry record alone.

The record does not describe missing-data methods, the analysis population, baseline adjustment, implementation fidelity, or the details of standard practice. It also does not report whether the reported tests were prespecified or whether multiplicity was addressed. These items limit interpretation of the statistical plan and of how faithfully the analysis represents the randomized comparison.

The primary result covers HPV series initiation, not completion, and the registry entries do not include results for the other vaccines in the study’s broader aim. The study was unmasked, as recorded, and the registry does not describe whether outcome assessment could be influenced by knowledge of assignment. These limitations are about what can be assessed from the registry record; they do not prove that the full study lacked additional procedures or analyses.

16. Related Tutorials

17. Related Statistical Calculators

18. Sources

• ClinicalTrials.gov: 4 Pillars Toolkit for Adolescent Vaccination — NCT02165722.
• Registry record last updated 2018-08-28; status last verified 2018-07.
• Results first posted 2018-07-30.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

19. Record Summary

NCT02165722, titled “4 Pillars Toolkit for Adolescent Vaccination,” is listed as a completed randomized, parallel cluster trial sponsored by the University of Pittsburgh. The registry identifies no masking and describes a behavioral intervention, the 4 Pillars Toolkit, compared with standard practice. The study summary names adolescent influenza, HPV, Tdap, and MCV vaccination as areas of interest.

The posted primary outcome is change in cumulative HPV series initiation vaccination rates, reported separately for adolescents aged 11-13 years and 14-17 years. The registry reports arm-level values and four Chi-squared tests, each with p < 0.001. Confidence intervals, cluster-adjustment details, missing-data methods, multiplicity procedures, and the analysis population are not reported in the registry record.

The trial is listed with a start date of 2013-09 and a primary completion date of 2016-06. The enrollment field lists 22, while the unit represented by that count is not stated. The control description identifies 11 clinical practices.