Implementing a Skills-Based Caregiver Training Program (Caregivers FIRST): Function QUERI 2.0: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Implementing a Skills-Based Caregiver Training Program (Caregivers FIRST): Function QUERI 2.0 (NCT05319535), a randomized na study. The primary outcome is Penetration 1.

2. Clinical Question

3. Trial Design

The registry describes a randomized, parallel-group study with no masking. It compares two implementation strategies: Foundational REP and Enhanced REP. The listed phase is not applicable. The study title identifies the Caregivers FIRST program and Function QUERI 2.0; the brief summary describes a type III effectiveness-implementation hybrid design intended to compare implementation strategies for broader spread of the caregiver training program.

The registry describes a focus on 24 sites that did not meet implementation adoption benchmarks. The registry enrollment is 25, but the primary outcome description states that facilities were the unit of analysis and that the overall number of participants analyzed is not available. These are different record entries, and the record does not explain their relationship. The arm-level outcome summaries list 12 facilities for Foundational REP and 13 for Enhanced REP.

Foundational REP is described as a set of implementation activities developed and tested in prior Function QUERI work. Enhanced REP retains those activities and adds higher-intensity support for approximately 4 months, including facilitation. Details beyond the partial description are not reported in the registry record. The sponsor is the VA Office of Research and Development. The study is recorded as completed; the listed start date is 2022-04-25, and primary completion is listed as 2024-09-30.

Randomization can support a comparison between assigned strategies, but the registry information does not specify how assignment was generated or whether it was stratified. The assignment procedure, allocation concealment, and analysis plan are not reported in the registry record. Because the unit of analysis is the facility, the results should be understood as site-level summaries rather than as an individual-caregiver randomized comparison.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: The Protocol Enrollment reflect the number of VA Facilities (sites) enrolled.

Overall Study

Analysis population for the primary outcome, as defined in the registry: Unit of analysis for this study was VA Facilities (site level); therefore, the Overall Number of Participants Analyzed is not available.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

Population: Primary and secondary outcomes were site level; therefore, the unit of analysis for this study was VA Facilities (sites). Since outcome data was collected at the site level, these characteristics are not relevant and were not collected.

6. Primary Endpoint

The registry lists the primary outcome as Penetration 1, measured over 12 months (cumulative).

The primary endpoint, Penetration 1, is the mean percentage of caregivers who received consults for VA caregiver education and training services and attended at least one class. It is cumulative over 12 months. In practical terms, this measure describes how widely the training reached among the caregiver group defined by the endpoint; attendance at one or more classes is the threshold for counting toward penetration.

The endpoint is relevant to an implementation study because the question is whether an implementation strategy is associated with broader uptake of a service. It focuses on participation in caregiver training rather than on a caregiver's health, a Veteran's health, or the effectiveness of the training content itself. The registry does not give further operational details about how the percentage was calculated at each facility, such as the precise handling of unavailable denominators or repeated attendance.

The reported outcome is a mean across facilities, with a standard deviation in each arm. The registry explicitly identifies facilities as the unit of analysis and says the overall number of participants analyzed is unavailable. Therefore, the result should not be read as a direct estimate of the percentage of all individual caregivers across the study who attended. It is a comparison of facility-level endpoint summaries over the cumulative 12-month period.

7. Results Posted in the Registry

• Enhanced REP: mean 8.30 versus 6.10 on control; difference 2.20 (95% CI -1.33 to 5.73), standardized effect d = 0.49; z-test p = 0.222.

8. Secondary Efficacy Endpoints

The registry also posts results for secondary outcomes. For each outcome below, the endpoint type, the test applied to the posted summary statistics, the result and its interpretation are given in turn. Secondary endpoints are usually not adjusted for multiple testing, so they support rather than confirm the primary finding.

Fidelity 1

• Endpoint type: Continuous: a mean value per participant, compared as a difference in means.
• Test: Unadjusted two-sample z-test for the difference in means, using the posted standard deviations.
• Result:
  • Enhanced REP: mean 104.80 versus 68.80 on control; difference 36.00 (95% CI -15.65 to 87.65), standardized effect d = 0.54; z-test p = 0.172.
• Interpretation:
  • The 95% confidence interval includes no difference, so these data do not show a statistically significant difference between Enhanced REP and the control. The interval gives the range of effects that remain compatible with the data.

Fidelity 2

• Endpoint type: Continuous: a mean value per participant, compared as a difference in means.
• Test: Unadjusted two-sample z-test for the difference in means, using the posted standard deviations.
• Result:
  • Enhanced REP: mean 2.40 versus 1.90 on control; difference 0.50 (95% CI -0.18 to 1.18), standardized effect d = 0.58; z-test p = 0.151.
• Interpretation:
  • The 95% confidence interval includes no difference, so these data do not show a statistically significant difference between Enhanced REP and the control. The interval gives the range of effects that remain compatible with the data.

Fidelity 3

• Endpoint type: Continuous: a mean value per participant, compared as a difference in means.
• Test: Unadjusted two-sample z-test for the difference in means, using the posted standard deviations.
• Result:
  • Enhanced REP: mean 13.20 versus 11.30 on control; difference 1.90 (95% CI -4.93 to 8.73), standardized effect d = 0.22; z-test p = 0.586.
• Interpretation:
  • The 95% confidence interval includes no difference, so these data do not show a statistically significant difference between Enhanced REP and the control. The interval gives the range of effects that remain compatible with the data.

Across the secondary outcomes with posted comparative results, the Enhanced REP means tend to be higher than those for Foundational REP. The uncertainty intervals, however, extend across values compatible with no difference, and the endpoints do not present a uniform pattern of strong statistical evidence. These outcomes concern implementation fidelity and caregiver participation in classes, so they add context about implementation processes but do not replace the primary measure of penetration.

The primary result is also an uncertain comparison. The secondary findings therefore do not change the main reading: the registry's data do not establish a clear difference between the strategies on the primary endpoint. Nor should directional estimates across multiple outcomes be treated as independent confirmation of a common effect. The measures are related to implementation, and the registry does not report how their correlations were handled.

Unadjusted secondary results carry less weight because multiple outcomes create multiple opportunities for apparently notable findings, even when no underlying difference exists. A prespecified multiplicity strategy can address this concern, but one is not reported in the registry record. The registry identifies Adoption as a secondary outcome, but a comparative estimate for it is not reported in the available outcome details. Without a complete analysis plan and results for each prespecified measure, secondary patterns are best viewed as descriptive rather than confirmatory.

Secondary outcomes registered for this trial, for reference (only those above were recalculated):

• Fidelity 1
• Fidelity 2
• Fidelity 3
• Adoption

9. Statistical Interpretation

10. Statistical Methods Explained

The primary endpoint is continuous and summarized as a mean with a standard deviation. A mean gives the average facility-level value in an arm. A standard deviation describes the spread of facility values around that arm's mean; it is not the uncertainty in the estimated mean difference. In this study, the two arm summaries are compared through a difference in means, with Enhanced REP as the arm of interest and Foundational REP as the comparator.

What does the confidence interval communicate?

A confidence interval describes the statistical uncertainty around an estimated contrast under the assumptions of the analysis. The registry's primary 95% interval extends from -1.8 to 6.1 for the mean difference. Because it includes zero, the data are compatible with no mean difference as well as with differences in either direction. The interval does not say that there is a 95% probability that the true effect lies within those particular limits. Its interpretation relies on the model and data-generating assumptions being suitable.

How should the p-value be read?

The registry reports p = 0.26 for the primary comparison. A p-value measures how unusual results at least as incompatible with a null hypothesis would be under that hypothesis and the analysis assumptions. It is not the probability that the null hypothesis is true, the probability that the result occurred by chance, or a measure of practical importance. The registry does not report a threshold selected in advance for this analysis, so the reported value should be read alongside the estimate and its interval rather than as a stand-alone decision.

Why distinguish the reported model from the recalculation?

The registry reports a linear regression analysis of the mean difference in final values. The recalculation uses the posted arm means, standard deviations, and arm-level sample sizes to form a simple unadjusted comparison. It does not account for covariates or other modeling choices that may be part of the regression. Small differences between the reported and recalculated intervals or p-values can result from differences in methods; the registry does not supply enough analysis detail to identify the exact reason for every difference. The recalculation is therefore a descriptive cross-check, not a substitute for the reported analysis.

What role does the unit of analysis play?

The registry says facilities were the unit of analysis. This matters because facilities, not individual caregivers, are the observations represented in the arm summaries. Caregivers are the people whose attendance defines the endpoint, but their outcomes are aggregated or otherwise represented at the facility level for this study. The registry does not report the total number of caregivers contributing to the analysis, the facility-level denominator construction, or the handling of missing facility data. It would be inappropriate to interpret the arm-level sample sizes as counts of caregivers.

Why can over-dispersion matter?

General statistical explanation: for count outcomes, a simple model may assume that the variability is determined by the mean. When observed variability exceeds that assumption, over-dispersion can make a model's uncertainty estimates too small if it is not accommodated. The registry reports a negative binomial regression for Fidelity 3, a count-like endpoint. This model family can accommodate extra variability compared with a basic Poisson model. The record does not report diagnostic checks, model covariates, or other details, so the adequacy of the model cannot be assessed from the listed result alone.

The other reported secondary analyses use linear regression and mean differences. The registry gives each endpoint's estimate, confidence interval, and p-value, but does not describe covariate adjustment or model diagnostics. Such details affect how precisely estimates are interpreted and how fully another analyst could reproduce them.

What can planning calculations tell a reader?

Planning calculations translate an assumed effect size and statistical design into a prospective sample-size illustration. They are conditional on the stated assumptions and are not evidence about the probability that the completed study detected an effect. The primary planning illustrations use fractions of the observed effect and show the numbers of participants per arm corresponding to the specified power targets. Because the analysis unit in the trial is the facility, a future design would need to ensure its planning unit and clustering structure match the intended analysis; the registry's illustration does not report such design details.

The sample-size illustrations use a two-sided significance level of 0.05, equal allocation, and a normal approximation. The original trial's design assumptions and sample-size justification are not reported in the registry record. No interim analysis plan, missing-data method, or detailed statistical analysis plan is reported either. Those omissions limit the ability to determine whether the posted analyses followed a prespecified approach or how sensitive they are to alternative assumptions.

11. Confidence Intervals and Precision

The primary estimate is a mean difference of 2.2. The registry's 95% confidence interval, from -1.8 to 6.1, is wide enough to include a range of plausible differences, including a small difference favoring Foundational REP and a larger difference favoring Enhanced REP. The interval is therefore not precise enough to pin down the direction or magnitude of the underlying contrast. Its endpoints should be interpreted in the units of the endpoint, which is a percentage of caregivers.

The simple unadjusted recalculation gives a 95% confidence interval from -1.331 to 5.731. It likewise spans zero and permits effects in both directions. The slight difference between this interval and the registry's regression interval reflects that these are not identical analyses. The recalculated interval is based on posted summary statistics and does not recreate all possible features of the regression analysis.

Confidence intervals express uncertainty under an analysis model; they do not capture every source of uncertainty. In particular, the registry does not report enough information to evaluate missing outcomes, facility-level denominator variation, model diagnostics, or the impact of any analysis choices. The number of caregivers analyzed is not available, and the facility is the stated unit of analysis. This distinction matters when judging how much information supports the estimate.

The interval's inclusion of zero should not be interpreted as proof of no difference, and its inclusion of positive values should not be interpreted as proof that Enhanced REP increases penetration. The evidence is compatible with more than one direction and size of effect. Precision is best described as limited: the point estimate is positive, but the data do not tightly constrain the comparison.

12. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The planning display gives participants per arm for 80% power and participants per arm for 90% power. These are per-arm figures, not total event counts. The endpoint is continuous, so the event-count convention for time-to-event outcomes does not apply. The illustrations are based on the observed effect and are not the trial's own design targets.

The stated planning assumptions are a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. The effect to detect is set to the stated fraction of the observed effect. The display includes planning for 100% of the observed effect, planning for 75% of the observed effect, and planning for 50% of the observed effect. Smaller effects require more participants per arm to achieve the same power target, because they are harder to distinguish from random variation.

The 80% power column represents a planning target under which a study would have an 80% chance of detecting the assumed effect at the stated significance level if that effect were real and the assumptions held. The 90% column uses a higher chance of detection and therefore generally requires more participants per arm for the same assumed effect. The figures are illustrations based on the observed estimate, not a report of the trial's original design assumptions. The original sample-size justification and design assumptions are not reported in the registry record.

These calculations are not observed power. They do not tell whether the completed trial had a particular probability of finding a statistically significant result, and they should not be used to reinterpret the p-value after the study. They show how prospective sample size would vary under the stated assumptions and effect scenarios. Because the study's analysis unit is facilities, a future planning exercise would also need to align the planned sample and statistical model with facility-level allocation and analysis.

13. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, Δ is the difference in means to detect and σ is the common standard deviation.

Worked calculation for this trial

1. Observed difference Δ = 2.200; pooled standard deviation σ = 4.500.
2. n = 2 × (1.96 + 0.842)² × 4.500² / 2.200² = 65.7, rounded up to 66 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
delta <- 2.2                      # observed difference in means
sigma <- 4.5                     # pooled standard deviation
n_per_arm <- function(power, frac = 1) {
  ceiling(round(2 * (z_a + qnorm(power))^2 * sigma^2 / (delta * frac)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (t-based, so very slightly larger)
cat("power.t.test:", round(power.t.test(delta = delta, sd = sigma, power = 0.80)$n, 1), "\n")

# Simulation: exact two-sample t-test using simulated means and variances
set.seed(5319535)
n    <- n_per_arm(0.80)
reps <- 20000
m_diff <- rnorm(reps, delta, sigma * sqrt(2 / n))             # difference in sample means
s2     <- sigma^2 * rchisq(reps, 2 * (n - 1)) / (2 * (n - 1))   # pooled sample variance
tstat  <- m_diff / sqrt(s2 * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(tstat) > qt(0.975, 2 * (n - 1))), reps))

Output from running the code

100% of observed effect: n per arm = 66 (80% power), 88 (90% power)
 75% of observed effect: n per arm = 117 (80% power), 157 (90% power)
 50% of observed effect: n per arm = 263 (80% power), 352 (90% power)
power.t.test: 66.7 
Simulated power at n = 66 per arm: 0.793 (20000 replicates)

The simulation draws many sets of group means and variances at the planned sample size and counts how often the two-sample t-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

14. Analyses Reported by the Sponsor

The registry's own primary analysis is a linear regression comparison of final values. It reports a mean difference of 2.2, a 95% confidence interval from -1.8 to 6.1, and p = 0.26. The simple recalculation from the posted arm summaries also estimates a mean difference of 2.2, but gives a 95% confidence interval from -1.331 to 5.731 and an unadjusted p-value of 0.222.

The central estimate is the same in these summaries, while the interval limits and p-values differ modestly. This is expected to be possible when a regression analysis and a summary-statistic calculation are not identical procedures. The registry does not report enough model detail to reconstruct the regression, so the recalculation should be treated as an unadjusted cross-check rather than a replacement or correction.

The registry does not flag a discrepancy. Interpretation should therefore retain the registry's analysis as the reported result and identify the recalculation as a separate, transparent calculation from arm summaries. Both intervals include zero, and both leave substantial uncertainty about the direction and size of the comparison. Neither analysis establishes equivalence between the implementation strategies.

15. Multiplicity, Missing Data and Analysis Populations

The registry lists several secondary outcomes in addition to the primary endpoint. It does not report a multiplicity adjustment, a hierarchy for testing outcomes, or another method for controlling the overall chance of false-positive findings across the outcome family. Without such information, the secondary p-values should be interpreted individually and cautiously; they do not necessarily support confirmatory claims.

The registry also does not report the number of missing observations by arm, how missing data were handled, whether all randomized facilities were included in the analysis, or whether sensitivity analyses were conducted. The primary endpoint description notes that the overall number of participants analyzed is unavailable and identifies facilities as the unit of analysis. It does not state how missing caregiver or facility-level information affected the endpoint.

Analysis-population details are not reported in the registry record. In particular, it is not clear whether the analysis followed assignment regardless of implementation received, whether facilities were excluded, or whether any per-protocol or other population was used. The assignment mechanism, allocation concealment, covariate adjustment, and detailed model specifications are also not reported. These omissions do not show that the analyses were inappropriate; they limit the ability to assess risk of bias, reproduce the results, and understand how robust the estimates are to missingness or analytic choices.

Interim analyses and stopping rules are not reported. The original sample-size assumptions and justification are also not reported. These features are important for evaluating whether the final analyses and uncertainty measures correspond to the study's planned design, but the registry information does not permit that evaluation.

16. What the Result Does, and Does Not, Mean

The posted primary result describes a higher mean penetration measure in the Enhanced REP arm than in the Foundational REP arm, but the estimated difference is uncertain. The registry's confidence interval includes zero and spans values in both directions. The result therefore does not establish that one implementation strategy produced greater penetration, and it does not establish that the strategies were equivalent.

The endpoint is a facility-level implementation measure concerning attendance at caregiver training. It does not measure caregiver well-being, Veteran health, or the clinical effectiveness of Caregivers FIRST. Nor does it show whether caregivers who attended classes benefited from them. Those questions are outside the primary outcome described in the registry record.

The comparison concerns the implementation strategy as assigned, but the registry does not report enough information about allocation, analysis populations, missing data, or model details to fully assess causal interpretation. The results also arise from a site-level analysis; they should not be translated into an individual caregiver effect without information that supports such a translation.

In practical statistical terms, the estimate is compatible with a range of outcomes: Enhanced REP could be associated with greater penetration, the difference could be close to none, or the comparison could favor Foundational REP. The data do not narrow that range enough to make a definitive statement about the direction or practical importance of the contrast. The secondary outcomes provide implementation context but, without reported multiplicity handling and complete details, do not resolve that uncertainty.

17. Why This Trial Matters Statistically

This study is statistically distinctive because the intervention contrast is between implementation strategies rather than between clinical treatments. The central question concerns whether additional implementation support is associated with broader uptake of a caregiver training program. That makes the unit of analysis especially important: the registry identifies facilities as the analyzed units, even though the endpoint is defined through caregiver attendance.

The study also illustrates why an estimate should be read together with its uncertainty. The primary point estimate favors Enhanced REP, but the confidence interval includes zero and allows differences in either direction. A directional point estimate alone is not enough to establish an effect. The same discipline applies to secondary outcomes, particularly when several measures are assessed and multiplicity handling is not reported.

For implementation research, clear reporting of site-level denominators, missing data, allocation methods, analysis populations, and model specifications helps readers judge how well results apply to other settings. Those details are not all available in this registry record. The posted results are useful for understanding the observed comparison, but the uncertainty and reporting limitations remain central to interpretation.

18. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry record leaves important design and analysis details unresolved. It does not describe the randomization procedure, allocation concealment, the original sample-size justification, or a full statistical analysis plan. It also does not report the analysis population, missing-data handling, interim analyses, or sensitivity analyses. These omissions limit assessment of how the reported estimates relate to the intended design.

The number analyzed is presented at the facility level for the arm summaries, while the total number of caregivers contributing to the outcome is unavailable. The endpoint depends on caregivers who received consults, but the record does not explain the construction of facility-level denominators or how unavailable data were handled. This makes it difficult to assess the stability and comparability of the percentages across sites.

Details of the implementation strategies are incomplete in the registry descriptions. Enhanced REP is described as adding higher-intensity support for approximately 4 months, including facilitation, but the process description is partial. The available record therefore does not fully specify what support was delivered at each site or how adherence to the strategy was assessed.

The outcome is cumulative over 12 months, and the record does not give further detail about timing or repeated observations within that period. Secondary outcomes are listed, but comparative information is not available for every named outcome in the listed results. Finally, this site-level comparison concerns implementation measures; it does not directly address clinical outcomes for caregivers or Veterans.

19. Related Tutorials

20. Related Statistical Calculators

21. Sources

• ClinicalTrials.gov: Implementing a Skills-Based Caregiver Training Program (Caregivers FIRST): Function QUERI 2.0 — NCT05319535.
• Registry record last updated 2025-04-16; status last verified 2025-03.
• Results first posted 2025-04-16.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

22. Record Summary

The registry describes a completed, randomized, parallel-group study with no masking comparing Foundational REP with Enhanced REP for implementing Caregivers FIRST. The sponsor is the VA Office of Research and Development. The listed study start is 2022-04-25, and primary completion is 2024-09-30.

The primary endpoint is the cumulative 12-month mean percentage of caregivers receiving consults who attended at least one class. Facilities were the unit of analysis; the overall number of participants analyzed is not available. The registry reports a primary mean difference of 2.2, with a 95% confidence interval from -1.8 to 6.1 and p = 0.26. The interval is compatible with differences in either direction, so the result does not establish a clear between-strategy difference.

Key information about allocation, missing data, analysis populations, multiplicity, and the original sample-size rationale is not reported in the registry record. Secondary results offer implementation context but do not remove the uncertainty in the primary comparison.