← Clinical Trials
Cognitive Dysfunction NA Binary Endpoint NCT03512301

CAMCI: Advancing the Use of Computerized Screening in Healthcare: Complete Statistical Analysis

The registry record describes a single-group validation study of CAMCI, with the primary analysis comparing CAMCI classifications with clinical adjudication. The record specifies an agreement-focused endpoint but does not report the agreement estimates or their confidence intervals.

Trial start: 2019-08-08  ·  Enrollment: 773  ·  Sponsor: Psychology Software Tools, Inc.
Scope of this record

This page provides an independent statistical analysis and educational interpretation of publicly reported results. ClinicalTrials.gov provides the official trial registry record. Numerical trial results on this page are restricted to the results posted in the ClinicalTrials.gov record; calculations added by Clinical Biostats are labelled as such.

1. Trial at a Glance

This page covers the registry results for CAMCI: Advancing the Use of Computerized Screening in Healthcare (NCT03512301), a na na study. The primary outcome is Agreement Analysis (Agreement to Reference Standard) - Test Group.

773
Enrollment
NA
n/a
Arms
In the primary outcome
n/a
Effect
No re-analysis shown
n/a
95% CI
See limitations
FeatureRegistry value
ClinicalTrials.gov identifierNCT03512301
PhaseNA
ConditionCognitive Dysfunction, Cognitive Impairment, Cognitive Decline
AllocationNot applicable (single group)
Design modelSingle Group
MaskingNone (open label)
Enrollment773
Arms in the primary outcomen/a
Lead sponsorPsychology Software Tools, Inc.
Results postedYes

2. Clinical Question

Population

The study concerns participants enrolled in the CAMCI Validation group. The registry states that 773 participants were enrolled and that 9 were excluded, leaving 764 for analysis. It also describes a baseline subset of participants who completed CAMCI version A (N=680), from which Training and Test groups were formed. Further participant characteristics are not reported in the registry record.

Intervention

The main assessment of interest is CAMCI, described as computerized tasks for assessing memory and cognitive impairment. The intervention list also names MoCA and Clinical Adjudication, and the arm description notes traditional paper-pencil neuropsychological tests administered by a trained administrator. The primary endpoint specifically concerns CAMCI classifications.

Comparator

For the primary agreement analysis, CAMCI classifications are compared with classifications from clinical adjudication. This is a reference-standard comparison within a single-group validation study, not a randomized comparison between treatment arms. MoCA is listed in the registry, but its role in the primary analysis is not reported in the registry record.

Primary question

At baseline, how closely do CAMCI and clinical adjudication classifications agree in the Test group? The planned agreement description includes positive and negative percent agreement, confidence interval estimates, and quadratic weighted kappa. The registry record does not report the resulting estimates, so the observed degree and precision of agreement are not reported in the registry record.

3. Trial Design

CAMCI: Advancing the Use of Computerized Screening in Healthcare (NCT03512301) is registered as a completed, single-group study. Its allocation is listed as NA, masking as none, and phase as NA. The sponsor is Psychology Software Tools, Inc. The recorded study start date is 2019-08-08, and the primary completion date is 2023-05-31.

The arm is named CAMCI Validation. Its description combines completion of computerized CAMCI tasks with traditional neuropsychological tests administered using paper and pencil by a trained administrator. The intervention list also includes MoCA and Clinical Adjudication. The primary analysis described in the registry compares CAMCI and clinical adjudication classifications in a Test group at baseline.

The registry says 773 participants were enrolled and 9 were excluded, leaving 764 for analysis. Among participants who completed CAMCI version A at baseline (N=680), stratified random sampling assigned participants to Training (75%) or Test (25%) groups. The listed balancing characteristics are age, education, race, ethnicity, gender, and cognitive characteristics; the final word of the registry description is truncated, so the specific characteristic is not reported in the registry record.

The Training and Test groups are analytic subsets, not randomized intervention and control arms. The registry explicitly flags that this is not a randomized comparison and that arms are not a like-for-like control comparison. The record does not report further details about the assignment procedure beyond stratified random sampling, nor does it report a separate control group.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Overall Study

StageCAMCI Validation
STARTED773
COMPLETED764
NOT COMPLETED9

Reasons for not completing this period:

ReasonCAMCI Validation
Protocol Violation2
Did Not Qualify3
Age-ineligible1
Inadequate fluency with the English language3

Analysis population for the primary outcome, as defined in the registry: Test group. Of 773 participants enrolled, 9 were excluded, leaving 764 for analysis. Participants who completed CAMCI version A at baseline (N=680) were assigned through stratified random sampling to Training (75%) or Test (25%) groups, balanced by age, education, race, ethnicity, gender, and cognitive classification. The groups were further adjusted to match the racial and ethnic distribution in the U.S., resulting in a Training group (N=374) and a Test group (N=126).

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

CharacteristicUnitCAMCI Validation
Participants analysed764
Age, Continuousyears71.0 (60 to 93)
Sex: Female, Male: FemaleParticipants550
Sex: Female, Male: MaleParticipants214
Ethnicity (NIH/OMB): Hispanic or LatinoParticipants116
Ethnicity (NIH/OMB): Not Hispanic or LatinoParticipants619
Ethnicity (NIH/OMB): Unknown or Not ReportedParticipants29
Race (NIH/OMB): American Indian or Alaska NativeParticipants0
Race (NIH/OMB): AsianParticipants21
Race (NIH/OMB): Native Hawaiian or Other Pacific IslanderParticipants0
Race (NIH/OMB): Black or African AmericanParticipants266
Race (NIH/OMB): WhiteParticipants434
Race (NIH/OMB): More than one raceParticipants43
Race (NIH/OMB): Unknown or Not ReportedParticipants0
Region of Enrollment: United Statesparticipants764
Neuropsychological classification: NormalParticipants521
Neuropsychological classification: IndeterminateParticipants176
Neuropsychological classification: ImpairedParticipants67
Montreal Cognitive Assessment (MoCA): Normal (Score Range 26-30)Participants337
Montreal Cognitive Assessment (MoCA): Indeterminate (Score Range 19-25)Participants388
Montreal Cognitive Assessment (MoCA): Impaired (Score Range 10-18)Participants39
CAMCI: Normal (Score Range 21-50)Participants596
CAMCI: Indeterminate (Score Range 16-20)Participants96
CAMCI: Impaired (Score Range 0-15)Participants72
WRAT5 - Word Readingpoints59.59 (SD 8.33)
WAIS-IV Digit Span Forward: Max Span (0-9) - number of digits recalled on the final correct trialpoints6.23 (SD 1.30)
WAIS-IV Digit Span Forward: Score (0-16) - total of points obtained on each trial/span (0,1,2)points9.42 (SD 2.23)
WAIS-IV Digit Span Backward: Max Span (0-8) - number of digits recalled on the final correct trialpoints4.40 (SD 1.34)
WAIS-IV Digit Span Backward: Score (0-16) - total of points obtained on each trial/span (0,1,2)points7.77 (SD 2.36)
WMS-IV Logical Memory I - Immediate Recall: Story A 1st Immediate Recall (older adult), Score Range (0-14)points8.59 (SD 2.72)
WMS-IV Logical Memory I - Immediate Recall: Story A 2nd Immediate Recall (older adult), Score Range (0-14)points11.16 (SD 2.94)
WMS-IV Logical Memory I - Immediate Recall: Story B Immediate Recall (older adult), Score Range (0-25)points11.52 (SD 4.68)
WMS-IV Logical Memory I - Immediate Recall: Immediate Recall Total (older adult), Score Range (0-53)points31.27 (SD 8.69)
WMS-IV Logical Memory I - Immediate Recall: Story B Immediate Recall (adult), Score Range (0-25)points13.48 (SD 4.17)
WMS-IV Logical Memory I - Immediate Recall: Story C Immediate Recall (adult), Score Range (0-25)points11.84 (SD 3.86)
WMS-IV Logical Memory I - Immediate Recall: Immediate Recall Total (adult), Score Range (0-50)points25.32 (SD 7.36)
WMS-IV Logical Memory II - Delayed Recall: Story A Delayed Recall (older adult), Score Range (0-14)points7.94 (SD 3.72)
WMS-IV Logical Memory II - Delayed Recall: Story B Delayed Recall (older adult), Score Range (0-25)points9.99 (SD 5.09)
WMS-IV Logical Memory II - Delayed Recall: Delayed Recall Total (older adult), Score Range (0-39)points17.93 (SD 8.10)
WMS-IV Logical Memory II - Delayed Recall: Story B Delayed Recall (adult), Score Range (0-25)points11.57 (SD 4.69)
WMS-IV Logical Memory II - Delayed Recall: Story C Delayed Recall (adult), Score Range (0-25)points10.37 (SD 4.28)
WMS-IV Logical Memory II - Delayed Recall: Delayed Recall Total (adult), Score Range (0-50)points21.93 (SD 8.27)
Rey-Osterrieth Complex Figure Test (ROCF): Figure Copypoints28.31 (SD 5.61)
Rey-Osterrieth Complex Figure Test (ROCF): Immediate Recall - Short Delay (3 minutes)points13.24 (SD 5.99)
Rey-Osterrieth Complex Figure Test (ROCF): Delayed Recall - Long Delay (30 minutes)points12.99 (SD 5.98)

6. Primary Endpoint

The registry lists the primary outcome as Agreement Analysis (Agreement to Reference Standard) - Test Group, measured over baseline.

ElementRegistry entry
Result typeCount Of Participants
Measure of spreadnot applicable
UnitParticipants
Endpoint classificationBinary Endpoint

The primary endpoint is titled “Agreement Analysis (Agreement to Reference Standard) - Test Group.” It is measured in participants and is specified as a count of participants. Its timeframe is baseline. The accompanying description identifies the central quantities as positive and negative percent agreement, confidence interval estimates, and quadratic weighted kappa for the comparison of CAMCI and clinical adjudication classifications.

Agreement measures describe how often or how closely classifications from an index assessment correspond to classifications from a reference assessment. Positive and negative percent agreement separately describe concordance for the positive and negative classification categories. Kappa expresses agreement while accounting for agreement expected by chance; a weighted version gives different penalties to different degrees of disagreement when categories have an order. These are general statistical descriptions, not reported findings from this study.

Using more than one agreement measure can make the nature of concordance clearer than relying on a single summary. Confidence intervals are used to show the uncertainty around an estimate. The registry record describes these measures but does not report their observed values or interval limits. The endpoint therefore defines what the analysis was intended to assess, but the endpoint description alone cannot establish the observed agreement or its precision.

7. Why No Re-analysis Is Shown

Reported as posted, not recalculated. Not a randomized comparison (allocation: NA); arms are not a like-for-like control comparison. A result is recalculated only when the posted data support a clean two-arm comparison, so no result is shown beyond what the sponsor posted.

8. Statistical Interpretation

Clinical Biostats interpretation

The primary analysis is an agreement assessment, not a test of whether an intervention changes a clinical outcome. It compares CAMCI classifications with clinical adjudication classifications in the Test group at baseline. The registry identifies the reference-standard comparison and the planned agreement measures, but the observed positive and negative percent agreement, confidence intervals, and quadratic weighted kappa are not reported in the registry record. Without those estimates, the strength of agreement and the uncertainty around it cannot be characterized from this record.

Agreement and association are different ideas. Two methods can tend to classify participants similarly while still disagreeing in clinically meaningful cases; conversely, a summary association alone would not show how often classifications match. This is a general statistical distinction. The listed agreement measures are intended to describe concordance more directly, but interpretation depends on their observed values, the category definitions, and the relevant uncertainty estimates. Those details are not reported in the registry record.

The study is registered as single-group, with no masking and no randomized allocation. Its Training and Test groups arose through stratified random sampling among participants who completed CAMCI version A at baseline. That split supports an analytic validation structure, but it does not create a randomized treatment comparison. The Test group is used for the stated agreement analysis; no causal treatment contrast follows from that design.

The registry reports enrollment of 773 participants, 9 exclusions, and 764 participants remaining for analysis. It separately describes the baseline version A subset (N=680) used to form Training and Test groups. The reason for exclusion, the handling of missing observations, and the number contributing to each reported agreement calculation are not reported in the registry record. Thus, the described analysis population does not by itself establish whether all eligible records contributed to every endpoint estimate.

The registry includes a planning statement for agreement: power calculations for Cohen Kappa assumed at least 75% Normals, 80% power, and .05 significance, and the description says the study was sufficiently powered with at least 120 individuals in the Test set for a Kappa of at least 0.3. This is a design-planning statement, not an observed agreement result or an estimate of observed power. The record does not report the actual Test group size for the primary analysis or the observed Kappa.

Overall, the registry specifies a diagnostic agreement question and a planned set of agreement statistics. It does not provide the numerical endpoint results in the registry record recorded for this analysis. Consequently, conclusions about the observed concordance, its precision, or whether any threshold was met are not reported in the registry record.

9. Statistical Methods Explained

The primary method is an agreement analysis comparing CAMCI classifications with clinical adjudication classifications. In a general agreement analysis, each participant has a classification from each assessment, and the analysis describes how often those classifications coincide. The registry names positive and negative percent agreement and quadratic weighted kappa, with confidence interval estimates. It does not report the resulting estimates or the category-level data needed to reconstruct them.

What do positive and negative percent agreement describe?

Positive percent agreement focuses on concordance among classifications in the positive category, while negative percent agreement focuses on concordance in the negative category. Reporting them separately can reveal whether concordance differs by classification category. These quantities describe agreement with the reference assessment; they should not be casually relabeled as sensitivity or specificity unless the analysis definition and assumptions support that interpretation. The exact definitions used for the registry calculations are not reported in the registry record.

What does weighted kappa add?

Kappa adjusts observed agreement for the amount of agreement expected by chance under its calculation framework. A weighted kappa extends this idea by assigning penalties to disagreements according to their degree. In general, quadratic weights assign a penalty that increases with the distance between categories. This can be useful when categories are ordered, because a nearby disagreement can be treated differently from a more distant disagreement. The registry names quadratic weighted kappa, but does not report the classification scale, weighting implementation, or observed value.

Kappa is not an absolute measure of clinical importance, and it can be influenced by the distribution of classifications. For that reason, it is useful to interpret kappa alongside category-specific agreement measures and the underlying classification context. This is general statistical guidance. The registry describes the planned agreement measures but does not report enough numerical results to assess their relationship or the distribution of classifications.

Why report confidence intervals?

A confidence interval communicates uncertainty in an estimated quantity. Narrower intervals generally indicate greater precision, while wider intervals indicate more uncertainty, subject to the method and assumptions used. An interval is not a guarantee that the true value lies within its limits. The registry states that confidence interval estimates were part of the comparison, but the limits, confidence level, and interval method are not reported in the registry record.

How should the planning statement be read?

The registry describes power calculations for agreement using Cohen Kappa, assuming at least 75% Normals, 80% power, and .05 significance. It states that the study was sufficiently powered with at least 120 individuals in the Test set for a Kappa of at least 0.3. This describes a planning calculation based on assumptions. It does not establish that the observed Kappa reached that value, that the estimate was precise, or that observed power was 80%. The actual Test set size used for this endpoint is not reported in the registry record.

The planning statement also depends on the assumed classification distribution and the target Kappa. If those assumptions differ from the observed study conditions, the relationship between the planning calculation and the analysis may differ. The registry record does not provide the details needed to evaluate that correspondence. This is a general limitation of interpreting power statements without the underlying assumptions and observed data.

What does the sampling structure imply?

Participants who completed CAMCI version A at baseline (N=680) were assigned by stratified random sampling to Training (75%) or Test (25%) groups, balanced by age, education, race, ethnicity, gender, and cognitive characteristics as far as the available description specifies. This allocation describes a data-splitting procedure for the validation study, not random allocation to different treatments. The primary agreement endpoint is specified for the Test group. Further details of the sampling implementation, including any selection or exclusion rules beyond those stated, are not reported in the registry record.

The record identifies 773 enrolled participants, 9 exclusions, and 764 participants for analysis. It does not explain why participants were excluded or how missing measurements were handled. These details matter because agreement estimates may depend on which participants have usable paired classifications. The exact number contributing paired CAMCI and clinical adjudication classifications is not reported in the registry record.

Finally, the record lists three outcome measures and 29 analyses. Those counts alone do not identify which analyses were confirmatory, how they relate to each other, or whether any multiplicity adjustment was planned. The statistical analysis plan, including any adjustment, is not reported in the registry record.

10. Multiplicity, Missing Data and Analysis Populations

The registry lists three outcome measures and 29 analyses. Those counts indicate that the record includes multiple analytic entries, but do not establish which analyses were primary, secondary, exploratory, or confirmatory beyond the primary endpoint description. A multiplicity strategy, such as adjustment of significance thresholds or a prespecified testing hierarchy, is not reported in the registry record. The primary endpoint description includes confidence interval estimates and an agreement planning statement, but does not provide a complete statistical analysis plan.

The record states that 773 participants were enrolled, 9 were excluded, and 764 remained for analysis. It also identifies a baseline CAMCI version A subset (N=680) used to form Training and Test groups. Reasons for exclusions, missing-data definitions, imputation methods, and rules for handling incomplete or discordant assessments are not reported in the registry record. The number of participants with paired classifications contributing to each agreement statistic is also not reported in the registry record.

The record does not state whether the analysis followed an intention-to-diagnose, complete-case, or other analysis-population definition. It does not report whether exclusions occurred before or after assessment, or whether missingness differed by classification or participant characteristics. These omissions limit assessment of how the analysis population relates to the enrolled group and how missing information may affect agreement estimates.

11. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

ArmParticipants with serious adverse eventsParticipants at riskPercentage
CAMCI Validation67730.8%

Deaths

ArmDeaths (participants affected of at risk)
CAMCI Validation5 of 773

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

ArmParticipants affected of at risk
CAMCI Validation1 of 773

Most frequent serious adverse event terms

Event termOrgan systemCAMCI Validation
Death unrelated to intervention reported by caregiver or test siteGeneral disorders5 of 773
Withdrawal due to health problems unrelated to interventionGeneral disorders1 of 773

Most frequent non-serious adverse event terms

Event termOrgan systemCAMCI Validation
StrainGeneral disorders1 of 773

Serious events with the highest arm-level rate: death unrelated to intervention reported by caregiver or test site (0.6%), withdrawal due to health problems unrelated to intervention (0.1%).

Do not overextend the safety comparison. Serious-adverse-event counts are safety observations, not efficacy outcomes, and should not be combined with the primary result into a single benefit-risk number.

For the CAMCI Validation group, the registry lists 773 participants at risk for serious events and 6 participants with a serious event. It separately lists 773 at risk for other events and 1 participant with an other event, along with 5 deaths. These are reported counts; the registry does not establish from these figures alone whether event categories overlap, so they should not be added together.

The serious-event terms listed include “Death unrelated to intervention reported by caregiver or test site” at 0.6% and “Withdrawal due to health problems unrelated to intervention” at 0.1%. These are registry descriptions of reported event terms. They do not provide detail about event timing, individual circumstances, or the method used to assess relatedness. The deaths are listed separately in the record, and the registry entries do not establish how they correspond to the serious-event term count.

Because this is a single-group study without a randomized control group, these safety counts do not provide a comparative estimate of risk attributable to CAMCI. The registry record does not report further safety-analysis details in the registry record for this analysis.

12. What the Result Does, and Does Not, Mean

The record defines a question about agreement between CAMCI and clinical adjudication classifications at baseline in a Test group. It names positive and negative percent agreement, confidence interval estimates, and quadratic weighted kappa as the planned measures. This identifies the type of evidence the study was intended to produce: concordance information for a computerized assessment relative to clinical adjudication.

The record does not report the observed agreement estimates, confidence interval limits, or observed Kappa. It is therefore not possible from the registry record to describe how closely the classifications agreed, how precise the estimates were, or whether a particular agreement threshold was reached. The planning statement about a Kappa of at least 0.3 is a target used in the stated power calculation, not a reported study result.

The design also does not support a randomized treatment-effect interpretation. The registry lists a single group, no masking, and allocation as NA. The Training and Test groups are described as sampling-based analytic groups, not like-for-like treatment arms. Thus, the study design as recorded addresses validation agreement rather than a causal comparison of treatment outcomes.

Agreement with a reference classification is not, by itself, a complete assessment of diagnostic usefulness. Interpretation can depend on the definition and quality of the reference, the classification categories, participant characteristics, and the consequences of different types of disagreement. Those details and the observed results are not reported in the registry record. The available information defines the analytic aim, but does not establish the performance of CAMCI in practice or the effect of using it in healthcare.

13. Why This Trial Matters Statistically

Diagnostic agreement studies ask whether an assessment produces classifications that correspond with a reference assessment. For computerized screening, this is a distinct statistical question from whether screening changes patient outcomes or whether a treatment is effective. The CAMCI record is framed around agreement with clinical adjudication, making the choice and description of agreement measures central to interpretation.

The planned combination of positive and negative percent agreement, confidence intervals, and quadratic weighted kappa recognizes that a single summary may not convey every feature of concordance. Separate category-specific agreement can help expose differences between positive and negative classifications, while a weighted kappa can account for the degree of disagreement where categories have an order. These are general reasons such measures can be informative; the actual estimates are not reported in the registry record.

The distinction between planning and observed results is also important. The record includes assumptions for a Kappa-based power calculation, including at least 75% Normals, 80% power, .05 significance, and a target Kappa of at least 0.3 with at least 120 individuals in the Test set. Those assumptions describe a planned adequacy assessment, not the observed agreement or its precision. Keeping these concepts separate prevents design targets from being mistaken for findings.

Finally, the study’s sampling-based Training and Test groups should not be mistaken for randomized treatment groups. Recognizing that structure helps keep conclusions aligned with the question the record specifies: agreement in the Test group, rather than a comparative treatment effect.

14. Limitations

The registry entries describe the primary endpoint and planning rationale but do not include the observed agreement results. Positive and negative percent agreement, confidence interval limits, quadratic weighted kappa, and the number of paired classifications contributing to those estimates are not reported in the registry record. This prevents evaluation of effect magnitude and precision.

The population description is partly truncated after listing characteristics used for balancing. The record reports enrollment of 773, 9 exclusions, and 764 participants for analysis, then describes a baseline CAMCI version A subset (N=680) used for sampling into Training and Test groups. Reasons for exclusion and the relationship between the broader analysis group and the endpoint-specific set are not reported in the registry record.

The registry identifies clinical adjudication as the reference standard but does not report how adjudication was conducted, how disagreements among adjudicators were handled, or whether assessors were blinded to other assessment results. Masking is listed as none. The consequences of that design detail for classification agreement cannot be quantified from the record.

The classification definitions, data completeness rules, missing-data handling, analysis-population rules, and full statistical analysis plan are not reported in the registry record. The record also lists MoCA without specifying its role in the primary endpoint. These limitations restrict assessment of the analysis and its broader interpretation. The single-group design does not provide a randomized treatment comparison.

15. Related Tutorials

16. Related Statistical Calculators

17. Sources

Continue through the Clinical Biostats statistical pathway

Use this trial's endpoint as a starting point for deeper study of effect measures, confidence intervals, and sample-size planning.

18. Record Summary

NCT03512301, CAMCI: Advancing the Use of Computerized Screening in Healthcare, is registered as a completed, single-group study sponsored by Psychology Software Tools, Inc. The record lists a start date of 2019-08-08 and a primary completion date of 2023-05-31, with phase and allocation listed as NA and masking listed as none.

The primary endpoint concerns baseline agreement between CAMCI and clinical adjudication classifications in the Test group. Planned measures include positive and negative percent agreement, confidence interval estimates, and quadratic weighted kappa. The record describes a Kappa-based planning calculation but does not report the observed agreement estimates or their confidence intervals.

The registry lists 773 enrolled participants, 9 exclusions, and 764 participants for analysis. It describes a baseline CAMCI version A subset (N=680) sampled into Training and Test groups. Reasons for exclusions, missing-data procedures, and the endpoint-specific analysis count are not reported in the registry record. The record lists three outcome measures and 29 analyses.

Clinical Biostats