An Effectiveness Trial of Project ALERT: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for An Effectiveness Trial of Project ALERT (NCT00650585), a randomized na study. The primary outcome is Self-reported 30-day Use of Alcohol.

2. Clinical Question

The outcome concerns self-reported use, not a clinical diagnosis or a measure of the amount consumed. Students were asked how many days they had used alcohol in the preceding 30 days. Responses were collapsed into two categories: none, or at least one day. The posted group counts indicate 330 participants with the outcome in the control group and 332 in the treatment group.

The registry describes a randomized, parallel-group study with single masking. It does not identify who was masked. The primary comparison is an unadjusted risk difference for the treatment group relative to control. Its estimate is positive, indicating a slightly higher observed proportion in the treatment group, while the reported interval includes values on either side of no difference. The registry does not report an adjusted analysis for this comparison.

This is a question about the outcome as recorded at the specified assessment, rather than about every possible effect of the program or later substance use. The registry record does not report longer-term results for this primary comparison. It also does not report the original statistical plan, so the posted result should be read as a registry summary rather than a complete account of every planned analysis.

3. Trial Design

The registry describes an effectiveness trial of Project ALERT, intended to replicate a longitudinal evaluation of a substance-abuse prevention program targeting middle school students. The study is listed as randomized, with a parallel model and single masking. The two recorded groups were a treatment group that received Project ALERT and a control group that did not receive it. The identity of the person or people masked is not reported in the registry record.

The sponsor was the Pacific Institute for Research and Evaluation. The recorded start was 2004-05, the primary completion was 2008-06, and the study status is completed. The listed enrollment is 8338. That overall enrollment is distinct from the outcome-specific denominators reported for the primary alcohol measure: 2358 in the control group and 2324 in the treatment group. The registry does not explain in detail how the overall enrollment relates to the analytic participants for this measure.

The primary outcome was self-reported 30-day alcohol use, measured approximately 30 days after intervention completion. Students completed an 81-item self-report questionnaire and indicated how many days they had used alcohol during the previous 30 days. The response categories were dichotomized into none and at least one day. The posted outcome is therefore binary: whether a participant reported any use in the period, rather than the number of days of use.

The registry lists Alcohol Drinking, Smoking, and Marijuana Smoking as conditions. It records eight outcome measures, but no secondary results are posted. The trial phase is listed as NA. The registry record does not report the original sample-size justification, the allocation procedure beyond randomization, details of masking, or the trial’s full statistical analysis plan. It also does not report how participants were recruited, how treatment delivery was monitored, or whether there were interim analyses.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

Reasons for not completing this period:

Analysis population for the primary outcome, as defined in the registry: Data were cleaned to remove cases with logical inconsistencies which decreased our sample by 3.6% of respondents at pretest and 7.5% at post-test. Missing covariate data were imputed using the Expectation Maximization (EM) algorithm. Missing data were imputed using the EM algorithms implemented in the Missing Value Analysis module in SPSS 13.0.

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Self-reported 30-day Use of Alcohol, measured over measured approximately 30 days after intervention completed.

The primary endpoint is the number of participants reporting any alcohol use during the 30 days before the assessment. The measure was collected approximately 30 days after the intervention was completed. Students answered an 81-item self-report questionnaire about how many days they had used alcohol in the previous 30 days. The response options ranged from none to 20 or more days, and the registry says these responses were grouped as none versus at least one day.

This dichotomy makes the endpoint a binary indicator: a participant either reported no use or reported use on at least one day. The posted results give the number with the outcome and the outcome-specific group denominator. The comparison is consequently about the proportion of participants who reported any use, not about frequency among those who reported use, total consumption, or change from an earlier assessment.

As a general statistical point, a binary endpoint can provide a clear group-level summary for an event defined in advance, but collapsing categories discards information about variation within each category. Here, the endpoint does not distinguish a participant reporting use on one day from a participant reporting use on many days. Self-report also means the endpoint reflects questionnaire responses; the registry does not report an independent verification method.

The registry record does not report whether the questionnaire or dichotomization was validated for this specific study, or whether there were additional rules for handling an incomplete response to this item. The endpoint should therefore be interpreted narrowly as the reported binary outcome at the stated follow-up assessment.

7. Results Posted in the Registry

• Treatment Group: 14.3% (332/2324) versus 14.0% (330/2358) on control; risk difference 0.3 percentage points (95% CI -1.7 to 2.3), risk ratio 1.02 (0.89–1.18); unadjusted two-proportion z-test p = 0.775.

8. Statistical Interpretation

9. Statistical Methods Explained

The registry reports a binary primary outcome and an unadjusted risk-difference comparison. These statistics answer related but distinct questions: the outcome counts describe how many participants had the event, the risk difference describes the contrast between groups, and the confidence interval and p-value describe uncertainty under the analysis used. The registry does not provide a full statistical analysis plan, so the explanation below distinguishes the reported quantities from general statistical interpretation.

What does the risk difference represent?

A risk difference compares the proportion with an outcome in one group with the proportion in another group. It is expressed on an absolute scale, so it describes how far apart the group proportions are, rather than how many times one proportion is the other. Here, the comparison is treatment relative to control. The estimate is positive, meaning the observed proportion reporting any alcohol use was higher in the treatment group. The estimate is close to zero, and the confidence interval covers zero.

As a general concept, absolute differences can be useful because they keep the comparison anchored to the observed event proportions. They do not, by themselves, explain why a difference occurred. In a randomized study, random allocation is intended to support comparisons between groups, but interpretation still depends on how participants were followed, how outcomes were measured, and how missing data were handled. The registry record does not report enough detail to assess all of those features.

How should a confidence interval be read?

A confidence interval gives a range of values that are compatible with the data under a stated statistical procedure and its assumptions. It communicates the uncertainty around the estimated risk difference. The reported interval extends across zero, which is the value representing no difference between the groups. The posted estimate alone points in one direction, but the interval shows that the data are also compatible with the opposite direction and with no difference.

In general, a wider interval indicates less precision than a narrower one, although interval width should be interpreted in relation to the scale and the question. The exact confidence level and method used to calculate this interval are not reported in the registry record. It is therefore appropriate to describe its range and whether it includes zero, but not to assign an unreported confidence level or infer details of the calculation.

What does the p-value add?

The unadjusted p-value is 0.775. In general, a p-value evaluates how unusual the observed data, or data more extreme, would be under a specified null hypothesis and analysis model. It is not the probability that the treatment has no effect. It is also not an estimate of effect size, an assessment of bias, or a measure of the probability that a finding will replicate.

The p-value and interval are complementary summaries. The interval shows the range of effect values represented by the estimate and its uncertainty; the p-value gives a test-based summary under a null model. The registry does not name the test or model used for the unadjusted p-value. The planning illustration lists a two-sided significance level of 0.05, but that planning assumption does not establish the threshold used in the trial’s original analysis.

What information does the binary endpoint retain?

The questionnaire offered response categories for days of alcohol use, but the primary outcome grouped them into none versus at least one day. This creates a binary endpoint that can be summarized with event counts and proportions. As a general statistical matter, dichotomization can simplify interpretation, while discarding differences within each category. A response indicating use on one day is classified with a response indicating use on 20 or more days. The primary result therefore addresses any reported use, not frequency or intensity.

The event counts in the registry are 330 for control and 332 for treatment. The corresponding outcome-specific denominators are 2358 and 2324. Because the endpoint is based on self-report, the recorded outcome reflects students’ questionnaire responses. The registry does not report an independent method of confirming those responses or detail how unanswered endpoint items were treated.

What can be said about other statistical features?

The record describes randomized allocation and a parallel-group structure, but gives no detailed allocation procedure or identification of who was masked. It does not report an adjusted model, covariate set, handling of clustering, or analysis population. The missing-covariate description says the Expectation Maximization algorithm was used for imputation. It also says data were cleaned to remove cases with logical inconsistencies, with reductions of 3.6% of respondents at pretest and 7.5% at post-test. The record does not fully describe which variables were subject to each procedure or how those procedures affected the primary endpoint analysis.

As a general concept, over-dispersion matters when observed variation is greater than a model assumes; if that assumption is wrong, standard errors and uncertainty estimates can be misleading. This endpoint is binary, and no dispersion measure or model diagnostic is reported. There is no basis in the registry record to claim over-dispersion was assessed or to specify a model adjustment. Similarly, the presence of randomized groups does not reveal whether any unreported analysis adjustments were planned or performed.

10. Confidence Intervals and Precision

The primary comparison is reported as a risk difference with a confidence interval that runs from -0.017 into positive values. Its point estimate is positive and close to zero, while the interval includes zero. The lower endpoint is negative; the upper endpoint is positive. This range makes clear that the direction of the underlying difference is not pinned down by the posted estimate with high precision.

Precision is about how narrowly the data locate an effect estimate, not about whether the intervention is important or whether the measurement is free of bias. A confidence interval is a range of estimates compatible with the data under the method’s assumptions. Because this interval includes zero, the result is compatible with no risk difference. It is also compatible with a difference in either direction. The exact confidence level and interval calculation method are not reported in the registry record.

The event counts and outcome-specific denominators provide context for the estimate: 330 of 2358 control participants and 332 of 2324 treatment participants had the reported outcome. The unadjusted p-value is 0.775. Neither the p-value nor the point estimate removes the uncertainty expressed by the interval. The registry does not report an adjusted estimate or a separate precision analysis.

The interval pertains to this particular dichotomized, self-reported outcome and assessment time. It does not convey precision for the number of days of use, other substance outcomes, or later follow-up. Nor does it account automatically for possible measurement or missing-data limitations. The registry record does not provide enough information to assess those sources of uncertainty fully.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The sample-size figures are planning illustrations, not a report of the trial’s original design calculation. They give participants per arm for 80% power and participants per arm for 90% power. The assumptions listed for the illustration are a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. The effect to detect is set as a stated fraction of the observed effect. The original trial’s own sample-size assumptions are not reported in the registry record.

For planning for 100% of the observed effect, the illustration lists 225,365 participants per arm for 80% power and 301,699 participants per arm for 90% power. For planning for 75% of the observed effect, it lists 399,788 participants per arm for 80% power and 535,202 participants per arm for 90% power. For planning for 50% of the observed effect, it lists 897,585 participants per arm for 80% power and 1,201,612 participants per arm for 90% power.

In plain terms, the 80% column describes a planning target under which a study of the stated size would have an 80% chance of detecting the selected effect, if the assumptions used for the illustration hold. The 90% column uses a higher chance of detection and therefore gives a larger planned group size for the same effect assumption. These are prospective planning quantities, not evidence that the completed trial had a particular chance of detecting its result. This is not observed power.

The rows vary the effect size used for planning. A smaller effect is harder to distinguish from random variation, so the illustration requires more participants per arm to reach the same power target. These figures should not be read as estimates of the number of participants actually needed for every future study: they depend on the listed assumptions and on using a fraction of the observed effect as the target.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, p₀ and p₁ are the control and treatment proportions, and p̄ is their average.

Worked calculation for this trial

1. Control proportion p₀ = 0.140; treatment proportion p₁ = 0.143; average p̄ = 0.141.
2. First term: z_1−α/2 × √[2p̄(1−p̄)] = 1.96 × 0.4928 = 0.9658.
3. Second term: z_1−β × √[p₀(1−p₀) + p₁(1−p₁)] = 0.842 × 0.4928 = 0.4147.
4. n = (0.9658 + 0.4147)² / (0.0029)² = 225,364.2, rounded up to 225,365 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
p0 <- 0.139949109414758                          # control proportion
p1 <- 0.142857142857143                          # treatment proportion
n_per_arm <- function(power, frac = 1) {
  q1 <- p0 + (p1 - p0) * frac               # smaller effect = fraction of the observed difference
  pbar <- (p0 + q1) / 2
  z_b <- qnorm(power)
  ceiling(round((z_a * sqrt(2 * pbar * (1 - pbar)) + z_b * sqrt(p0 * (1 - p0) + q1 * (1 - q1)))^2 / (q1 - p0)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (unrounded n per arm)
cat("power.prop.test:", round(power.prop.test(p1 = p0, p2 = p1, power = 0.80)$n, 1), "\n")

# Simulation: how often does the z-test reject at the 80%-power sample size?
set.seed(650585)
n    <- n_per_arm(0.80)
reps <- 20000
x0 <- rbinom(reps, n, p0); x1 <- rbinom(reps, n, p1)
ph <- (x0 + x1) / (2 * n)
z  <- ((x1 - x0) / n) / sqrt(ph * (1 - ph) * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(z) > z_a, na.rm = TRUE), reps))

Output from running the code

100% of observed effect: n per arm = 225,365 (80% power), 301,699 (90% power)
 75% of observed effect: n per arm = 399,788 (80% power), 535,202 (90% power)
 50% of observed effect: n per arm = 897,585 (80% power), 1,201,612 (90% power)
power.prop.test: 225364.2 
Simulated power at n = 225,365 per arm: 0.798 (20000 replicates)

The simulation draws many pairs of event counts at the planned sample size and counts how often the two-proportion z-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Multiplicity, Missing Data and Analysis Populations

The registry lists eight outcome measures, but reports no secondary results. It does not report a multiplicity adjustment or explain how the family of outcomes was handled when interpreting results. When several outcomes or comparisons are tested, the chance of at least one apparently unusual result can increase; methods for addressing this depend on the prespecified analysis plan. The registry record does not provide that plan, so it is not possible to determine whether or how multiplicity was addressed.

The record’s primary outcome description says cases with logical inconsistencies were removed from the data. It reports reductions of 3.6% of respondents at pretest and 7.5% at post-test. It also says missing covariate data were imputed using the Expectation Maximization algorithm. The record does not report enough detail to determine which variables were included, the exact imputation implementation, or how imputed covariates entered the primary comparison.

How missing primary-outcome responses were handled is not reported in the registry record. Neither is the complete analysis population definition. The group denominators for the posted primary result are given, but the record does not fully explain how those participants were selected from the overall enrollment, or whether any participants were excluded from this particular analysis for reasons beyond the described data cleaning.

The registry also does not report whether the analysis followed an intention-to-treat principle, whether participants were analyzed according to their randomized group, or how any deviations from assigned intervention were addressed. These details matter because analysis choices can affect the interpretation of randomized comparisons. Without them, the posted counts and unadjusted comparison can be described, but a complete evaluation of missing-data and population-handling decisions is not possible.

14. What the Result Does, and Does Not, Mean

The posted result describes the primary outcome as a self-reported binary measure of any alcohol use in the prior 30 days, assessed approximately 30 days after the intervention ended. The observed outcome proportion was slightly higher in the treatment group than in the control group. The estimated risk difference is close to zero, the reported confidence interval crosses zero, and the unadjusted p-value is 0.775. These summaries indicate substantial uncertainty about the direction and size of the group difference.

This result does not establish that the groups were exactly alike. A confidence interval crossing zero is compatible with no difference, but it also includes nonzero differences. It also does not establish that a meaningful difference exists. The posted estimate and its uncertainty should be considered together, and the registry does not report the clinical or policy threshold that would define a difference as important.

The result is limited to the specified outcome and assessment. It does not describe the number of days participants used alcohol, changes over time, or outcomes for smoking and marijuana smoking. The registry lists those conditions and eight outcome measures, but no secondary results are posted. The available primary result therefore cannot stand in for a full assessment of all outcomes associated with the program.

The endpoint is based on self-report, and response categories were collapsed into none versus at least one day. The result does not distinguish occasional from more frequent reported use. The registry does not report independent confirmation of questionnaire responses or enough detail about missing outcome data to evaluate their possible influence.

The trial is recorded as randomized, but the registry record does not provide the full analysis plan, masking details, or a complete account of the analysis population. The posted comparison is explicitly unadjusted. These limitations do not erase the reported result; they define what it can support: a descriptive, unadjusted comparison for the stated outcome, at the stated follow-up, in the participants included in that result.

15. Why This Trial Matters Statistically

This trial illustrates why an effect estimate, uncertainty interval, and p-value should be read together. The point estimate for the risk difference is positive but close to zero, the interval spans negative and positive values, and the unadjusted p-value is 0.775. A single statistic cannot convey all of that information. The direction of the estimate is not the same as a precise conclusion about the direction of an underlying difference.

The endpoint also highlights how outcome definitions shape interpretation. The questionnaire asked about days of alcohol use, but the primary outcome classified responses simply as none or at least one day. That makes the result straightforward to express as a count and proportion, while leaving out differences in frequency. Statistical interpretation must therefore stay aligned with what was actually measured.

The registry reports overall enrollment separately from the denominators for this outcome. Recognizing that distinction helps prevent readers from treating the trial enrollment as the number contributing to a particular endpoint. The record also describes data cleaning and imputation for missing covariates, yet leaves important details about the analysis population and missing endpoint responses unreported. This is a useful example of why outcome-specific denominators and transparent handling of missing data matter.

Finally, the sample-size illustration is based on the observed effect and stated planning assumptions, rather than the original trial design. It demonstrates the general relationship between effect size and required group size: smaller effects require more participants to detect under the same assumptions. It should not be mistaken for observed power or a record of the trial’s original planning. These distinctions help readers separate what the registry reports from what statistical concepts imply in general.

16. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry record gives only limited information about trial conduct and analysis. Although it records randomization, parallel groups, and single masking, it does not identify who was masked or describe the allocation procedure. The original statistical plan, sample-size justification, analysis model, and any interim analyses are not reported. The posted primary result is unadjusted, and there are no adjusted analyses recorded.

The primary endpoint is self-reported and dichotomizes a range of response categories into no use versus any use. As a result, it does not preserve information about frequency within the any-use group. The record does not describe independent verification of responses, the handling of incomplete questionnaire items, or the impact of the data-cleaning steps on this endpoint.

The record reports that logical inconsistencies led to reductions of 3.6% of respondents at pretest and 7.5% at post-test, and that missing covariate data were imputed using the Expectation Maximization algorithm. It does not fully describe the variables, implementation, or analysis population. Missing primary-outcome handling is not reported in the registry record. These gaps limit assessment of potential selection or missing-data effects.

The adverse-event entries list zero at-risk and affected counts for serious and other events in both groups, while death counts are not reported in the registry record. No serious-event terms are listed. The record does not explain the scope or completeness of adverse-event collection, so these entries should not be treated as a detailed safety evaluation.

Finally, no secondary results are posted despite the record listing eight outcome measures. The primary comparison pertains only to the stated self-reported alcohol measure and follow-up. It cannot answer questions about other outcomes or later assessments that are not reported.

17. Related Tutorials

18. Related Statistical Calculators

19. Sources

• ClinicalTrials.gov: An Effectiveness Trial of Project ALERT — NCT00650585.
• Registry record last updated 2013-06-13; status last verified 2013-06.
• Results first posted 2009-10-21.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

20. Record Summary

The completed registry record describes a randomized, parallel-group, single-masked effectiveness trial of Project ALERT, a behavioral substance-abuse prevention program for middle school students. The control group did not receive the program. The sponsor was the Pacific Institute for Research and Evaluation. The recorded study start was 2004-05, primary completion was 2008-06, and overall enrollment was 8338.

The primary endpoint was self-reported alcohol use during the previous 30 days, assessed approximately 30 days after intervention completion and dichotomized as none versus any use. The posted outcome counts were 330 among 2358 control participants and 332 among 2324 treatment participants. The unadjusted risk-difference estimate was positive and close to zero; its reported interval includes zero, and the unadjusted p-value was 0.775.

The registry lists eight outcome measures but no secondary results, and it reports no adjusted analyses. It describes data cleaning and imputation of missing covariates, but several details of analysis populations and missing outcome handling are not reported in the registry record.