Virtual Reality Job Interview Training in Severe Mental Illness: Complete Statistical Analysis

1. Trial at a Glance

This page covers the registry results for Virtual Reality Job Interview Training in Severe Mental Illness (NCT03049813), a randomized na study. The primary outcome is Employment Rates.

2. Clinical Question

3. Trial Design

This was a randomized, parallel-group trial with single masking. The registry lists an enrollment of 168 and gives the status as completed. The trial dates listed are 2017-06-01 for start and 2021-06-30 for primary completion. The phase is recorded as NA. The registry record does not explain who was masked or how masking was maintained.

The two arms were Virtual Reality Job Interview Training and Supported Employment (Services as Usual). The trial summary describes VR-JIT as an enhancement to supported employment and identifies a community-based evaluation. Northwestern University is described as partnering with Thresholds Inc. to evaluate effectiveness, cost effectiveness, and implementation at Thresholds. The sponsor listed in the registry is the University of Michigan.

For the primary employment outcome, the posted arm counts are 23 participants in the VR-JIT group, with an arm size of 54, and 10 participants in the supported-employment group, with an arm size of 36. These outcome groups are not the same as the registry’s overall enrollment figure. The reason for that difference, including whether it reflects follow-up availability, analysis eligibility, or another process, is not reported in the registry record.

Random allocation supports a comparison between assigned groups, but the registry entry alone does not establish how allocation was generated or concealed. Those details, as well as the original design assumptions and any departures from the planned design, are not reported in the registry record. The record lists five secondary outcomes: job interview skills, interview self-confidence, interview anxiety, social competence, and psychological distress measured with the Brief Psychiatric Rating Scale.

4. Participant Flow and Accounting

The registry reports how participants moved through each stage of the study. All counts below are exactly as posted; missing counts are not inferred from other stages.

Before assignment: Randomized Control Trial Phase: 105 people were enrolled and signed consent forms, but 6 people dropped out of the study before randomization and were not assigned to a group. Therefore, the Randomized Control Trial Phase includes 99 people who signed consent forms and were actually randomized into the study. Focus Group Phase: This includes both Thresholds staff focus groups as well as focus groups completed by those who received the intervention in the Randomized Control Trial Phase.

Randomized Control Trial Phase

Reasons for not completing this period:

Focus Group Phase

5. Baseline Characteristics

The table reproduces the baseline characteristics posted in the registry, which describe the participants analysed at baseline and help show whether the randomized groups were comparable.

6. Primary Endpoint

The registry lists the primary outcome as Employment Rates, measured over 9 month Follow-Up from Randomization Date.

The primary endpoint is employment rates at the 9-month follow-up from randomization, measured with a questionnaire. The registry records this as a count of participants. In practical terms, the posted result reports how many participants in each arm had the employment outcome at that follow-up, alongside the arm sizes used for that outcome.

Employment is relevant to the trial’s stated aim of assessing whether VR-JIT, as an enhancement to supported employment, is associated with access to employment. The endpoint therefore addresses an outcome distinct from interview skills or confidence: it records employment status at a specified follow-up rather than performance during interview practice. The registry lists job interview skills, interview self-confidence, interview anxiety, social competence, and psychological distress as secondary outcomes.

A questionnaire-based measure depends on the information collected and recorded at follow-up. The exact questionnaire items, scoring rules, handling of uncertain responses, and definition of employment for this endpoint are not reported in the registry record. The record also does not state how missing questionnaire responses were handled. Those details matter when interpreting what a recorded employment count represents and how completely it captures the intended outcome.

7. Results Posted in the Registry

• Virtual Reality Job Interview Training: 42.6% (23/54) versus 27.8% (10/36) on control; risk difference 14.8 percentage points (95% CI -4.9 to 34.5), risk ratio 1.53 (0.83–2.82); unadjusted two-proportion z-test p = 0.153.

8. Statistical Interpretation

9. Statistical Methods Explained

What does the risk difference describe?

A risk difference compares the proportion with an outcome in one group with the proportion in the other group by subtracting one group’s proportion from the other. It is an absolute measure: it expresses the difference in outcome frequency on the same scale as the underlying proportions. For this trial, the unadjusted recalculation compares employment counts relative to the arm sizes posted for the 9-month outcome. The estimated risk difference is positive, indicating a higher observed proportion in the VR-JIT arm, but its confidence interval spans zero. That interval means the data are compatible with no difference as well as with differences in either direction under the recalculation’s assumptions.

What does an unadjusted recalculation mean?

An unadjusted recalculation uses the posted group counts and arm sizes without accounting for other variables. It is a straightforward way to describe the observed group contrast, but it need not reproduce a model-based analysis. Logistic regression can account for covariates if they are included in the model, while a chi-squared test evaluates association between group and outcome using a different calculation. The registry record does not state which covariates, if any, were included in the logistic regression, nor does it report the regression coefficient or its uncertainty. Consequently, the exact source of the difference between the sponsor’s analyses and the unadjusted recalculation is not reported in the registry record.

How should the p-values be read?

A p-value is calculated under a statistical model that assumes a null hypothesis, commonly no group difference. It describes how unusual the observed data, or data more extreme, would be under that model. It does not measure the probability that the intervention has an effect, quantify the clinical importance of an effect, or establish that an effect is absent when a result is not conventionally statistically significant. Different methods can yield different p-values because they may use different assumptions or calculations. The registry lists a logistic-regression p-value of 0.027 and a chi-squared p-value of 0.076. The unadjusted recalculation is also different. The discrepancy flag indicates that the sponsor’s own analysis takes precedence over the recalculation; the registry does not explain the specific reason for the divergence.

Why can the analyses differ?

General statistical considerations include differences in model specification, adjustment for covariates, treatment of the data, or the particular test used. These are possible reasons in general, not explanations established for this trial. The record names logistic regression and a chi-squared method but does not provide sufficient detail to attribute the difference to any one cause. It also does not report whether analyses were prespecified, whether assumptions were checked, or whether any alternative analyses were conducted.

What can the confidence interval add?

A confidence interval communicates uncertainty around an estimate. A narrow interval indicates greater precision under the stated method; a wide interval indicates that a broader range of effects remains compatible with the observed data. The recalculated risk-difference interval includes zero and extends to values on both sides of it. This makes the unadjusted estimate imprecise for distinguishing a positive difference from no difference or a difference in the other direction. The sponsor analyses are reported without estimates or confidence intervals, limiting assessment of their precision.

What does randomization contribute?

In general, random assignment is intended to make groups comparable with respect to measured and unmeasured factors at the start of a trial, on average. It helps support a causal comparison when allocation, follow-up, outcome measurement, and analysis are conducted appropriately. The registry identifies the allocation as randomized, but detailed methods for sequence generation and concealment are not reported in the registry record. The posted information also does not describe adherence to assigned groups or whether the analyses followed an intention-to-treat approach. Randomization is relevant to interpretation, but the registry entry alone cannot resolve these design and analysis questions.

What does the outcome count represent?

The primary result is a count of participants with the outcome, not a time-to-event measure. The registry describes employment rates at the 9-month follow-up measured by questionnaire. The posted count does not reveal the timing of employment during follow-up, how long employment lasted, or whether multiple jobs were considered. Those definitions are not reported in the registry record. As a general point, a single follow-up status and a measure of employment duration answer different questions; the posted endpoint should be interpreted according to its stated follow-up measure.

10. Confidence Intervals and Precision

The unadjusted recalculation estimates a positive risk difference, but the confidence interval spans zero. This means the estimate is not precise enough to rule out no difference under the recalculation’s method. It also allows values on either side of zero, so the data do not pin down the direction of the underlying difference with high precision. The interval’s breadth is important: the point estimate alone gives an incomplete picture of uncertainty.

A confidence interval is a range generated by a statistical procedure that, under its assumptions, has a specified long-run coverage property. It is not a guarantee that the true effect lies within the particular interval, and it does not mean that every value in the interval is equally plausible. Its interpretation depends on the calculation and its assumptions. Here, the interval is for an unadjusted recalculation from the posted summary counts. It should not be treated as the interval for the sponsor’s logistic-regression analysis, for which the registry does not report an estimate or confidence interval.

The outcome groups have arm sizes of 54 and 36. With limited outcome information, estimates can be sensitive to individual observations, and confidence intervals may remain broad. This is a general statistical consideration, not a statement about any particular participant’s influence. The registry’s overall enrollment is 168, while the posted primary outcome groups have different arm sizes; the explanation for this difference is not reported in the registry record. Missing outcome data or analysis exclusions can affect precision and interpretation, but the registry does not describe their extent or handling.

Precision is separate from whether a result crosses a conventional significance threshold. A small p-value cannot by itself convey the size of an effect, and a p-value that does not meet a threshold does not show that effects are absent. The sponsor analyses lack reported effect estimates and intervals, so their precision cannot be judged from the registry values alone.

11. Planning a Similar Trial: Sample Size

This is a planning illustration, not a re-evaluation of the original trial. It asks: if a new study wanted to detect a difference like the one observed, or a smaller one, how large would it need to be? It is deliberately not “observed power” of the finished trial, which is not a meaningful quantity.

The planning figures are participants per arm, not total participants across both arms. They are illustrative calculations based on the observed effect, rather than the trial’s original design calculation. The assumptions listed for these planning illustrations are a two-sided significance level of 0.05, equal allocation between the two arms, and a normal approximation. The effect to detect is the stated fraction of the observed effect. The trial’s own original sample-size assumptions are not reported in the registry record.

Rows describing planning for the full observed effect, 75% of the observed effect, and 50% of the observed effect represent progressively smaller target effects. Smaller effects are harder to distinguish from random variation, so the planning requirement increases as the target effect becomes smaller. This pattern is a general feature of statistical planning, assuming the other planning conditions remain the same.

The 80% power column gives the planned number of participants per arm for a design with an 80% chance of detecting the specified effect under the stated assumptions, if that effect and those assumptions are correct. The 90% power column gives the corresponding per-arm planning figure for a 90% chance. Power is a property of a planned design under assumptions, not a measure of whether the observed result is reliable or whether the intervention has an effect. These figures are not observed power.

The listed planning values should not be read as a recommendation for a future study without considering its population, outcome definition, analysis plan, and expected effect. They are mathematical illustrations anchored to the observed effect in this trial. Because that observed estimate is uncertain, planning around it inherits that uncertainty. The registry does not report the assumptions used for the original trial’s design, so the planning illustrations do not explain why the trial enrolled the number listed in the record.

12. How the Sample Size Is Derived

Power is the probability that a study of a given size rejects the null hypothesis when a specific true effect exists. It depends on four things: the effect to detect, the variability of the outcome, the significance level (here a two-sided α of 0.05, so z_1−α/2 = 1.960) and the desired power (z_1−β = 0.842 for 80% and 1.282 for 90%).

Formula

n is the number of participants in each arm, p₀ and p₁ are the control and treatment proportions, and p̄ is their average.

Worked calculation for this trial

1. Control proportion p₀ = 0.278; treatment proportion p₁ = 0.426; average p̄ = 0.352.
2. First term: z_1−α/2 × √[2p̄(1−p̄)] = 1.96 × 0.6754 = 1.3237.
3. Second term: z_1−β × √[p₀(1−p₀) + p₁(1−p₁)] = 0.842 × 0.6672 = 0.5615.
4. n = (1.3237 + 0.5615)² / (0.1481)² = 161.9, rounded up to 162 participants per arm for 80% power.

The smaller-effect rows in the table above repeat the same calculation with the effect multiplied by 0.75 and 0.5 (for ratios, applied to the logarithm of the ratio).

R code

alpha <- 0.05
z_a   <- qnorm(1 - alpha / 2)          # two-sided
p0 <- 0.277777777777778                          # control proportion
p1 <- 0.425925925925926                          # treatment proportion
n_per_arm <- function(power, frac = 1) {
  q1 <- p0 + (p1 - p0) * frac               # smaller effect = fraction of the observed difference
  pbar <- (p0 + q1) / 2
  z_b <- qnorm(power)
  ceiling(round((z_a * sqrt(2 * pbar * (1 - pbar)) + z_b * sqrt(p0 * (1 - p0) + q1 * (1 - q1)))^2 / (q1 - p0)^2, 6))
}
for (f in c(1, 0.75, 0.5))
  cat(sprintf("%3.0f%% of observed effect: n per arm = %s (80%% power), %s (90%% power)\n", 100 * f,
              format(n_per_arm(0.80, f), big.mark = ","), format(n_per_arm(0.90, f), big.mark = ",")))

# Check against R's built-in function (unrounded n per arm)
cat("power.prop.test:", round(power.prop.test(p1 = p0, p2 = p1, power = 0.80)$n, 1), "\n")

# Simulation: how often does the z-test reject at the 80%-power sample size?
set.seed(3049813)
n    <- n_per_arm(0.80)
reps <- 20000
x0 <- rbinom(reps, n, p0); x1 <- rbinom(reps, n, p1)
ph <- (x0 + x1) / (2 * n)
z  <- ((x1 - x0) / n) / sqrt(ph * (1 - ph) * 2 / n)
cat(sprintf("Simulated power at n = %s per arm: %.3f (%d replicates)\n", format(n, big.mark = ","), mean(abs(z) > z_a, na.rm = TRUE), reps))

Output from running the code

100% of observed effect: n per arm = 162 (80% power), 217 (90% power)
 75% of observed effect: n per arm = 282 (80% power), 377 (90% power)
 50% of observed effect: n per arm = 616 (80% power), 825 (90% power)
power.prop.test: 161.9 
Simulated power at n = 162 per arm: 0.803 (20000 replicates)

The simulation draws many pairs of event counts at the planned sample size and counts how often the two-proportion z-test rejects the null. It should land close to the target power; at very small required sample sizes it can fall below it, because the normal approximation is rough there.

13. Analyses Reported by the Sponsor

The registry reports two sponsor analyses of the primary comparison: logistic regression, with a p-value of 0.027, and a chi-squared analysis, with a p-value of 0.076. It does not report estimates or confidence intervals for either analysis. A separate unadjusted recalculation from the posted outcome counts produces a different result. The discrepancy flag is true, so the sponsor’s own analysis takes precedence over the recalculation when describing the registry’s inferential analysis.

These results should not be combined or treated as though they came from one model. Logistic regression and a chi-squared test use different statistical procedures, and an unadjusted comparison is not necessarily equivalent to either. The registry does not report enough information about model specification, adjustment variables, or analysis populations to explain the difference. It also does not identify which sponsor analysis was designated as primary when the results were reported.

The sponsor’s p-values indicate the outputs of the named analyses, but without effect estimates and confidence intervals they do not show the magnitude or precision of the model-based group contrast. The count-based recalculation offers an effect estimate and interval, but it is not a substitute for the sponsor’s analysis. Its role is descriptive and supplemental; the discrepancy means its result should not override the sponsor-reported analysis.

14. Multiplicity, Missing Data and Analysis Populations

The registry lists five secondary outcomes: job interview skills, interview self-confidence, interview anxiety, social competence, and psychological distress measured with the Brief Psychiatric Rating Scale. It also records seven outcome measures and seven analyses. The relationship between those counts, the listed secondary outcomes, and the analyses is not explained in the registry record. Results for the secondary outcomes are not included in the registry record for this statistical interpretation, so no conclusions about them can be drawn.

When a study examines multiple outcomes or performs multiple analyses, the chance of obtaining at least one apparently notable result can increase. Statistical methods that account for multiplicity can help address that issue, but the registry does not report whether any adjustment was planned or used. It also does not report a hierarchy of outcomes, a prespecified testing sequence, or how the sponsor selected the analyses presented. This absence does not establish that multiplicity was ignored; it means the approach is not reported in the registry record.

The registry identifies a primary employment endpoint and reports two analyses for the comparison, but the analysis plan and the designation of a primary statistical test are not reported. Further details about covariate adjustment, model diagnostics, and the treatment of any deviations from the intended analysis are also not reported in the registry record.

The primary outcome groups have arm sizes of 54 and 36, whereas the registry lists enrollment of 168. The reason for the difference is not reported. Missing-data rates, reasons for missingness, imputation procedures, and sensitivity analyses are not reported in the registry record. It is therefore not possible to determine whether the posted outcome counts represent all randomized participants, only those with available follow-up, or another analysis population.

The registry record does not say whether participants were analyzed according to their assigned group, whether participants who changed or did not receive their assigned intervention were handled differently, or whether exclusions were made. The analysis population and missing-data approach matter because they can affect both the comparison and its uncertainty. No conclusion about those procedures should be inferred from the posted arm counts alone.

15. Safety Results

The registry reports serious adverse events as affected participants divided by participants at risk.

Deaths

Other (non-serious) adverse events

Participants with at least one non-serious event at or above the registry's 0% reporting threshold.

The serious adverse event entry reports zero affected participants in the supported-employment group, with 38 at risk, and zero in the VR-JIT group, with 59 at risk. The entry also records zero deaths in each group. No serious adverse event terms are listed.

These figures describe what the registry records for the reported safety fields. They should not be interpreted as proof that no adverse events of any kind occurred. The entry distinguishes serious adverse events from other adverse events, and the “other” adverse event fields also record zero affected participants, with 38 at risk in the supported-employment group and 59 at risk in the VR-JIT group. The registry record does not provide additional event descriptions or details about the collection and reporting process.

The safety counts are separate from the primary employment outcome and do not resolve the statistical uncertainty in that outcome. The registry does not explain why the safety group sizes differ from the primary outcome arm sizes or from overall enrollment. As with any registry safety summary, the recorded counts should be read within the scope of the fields and information reported.

16. What the Result Does, and Does Not, Mean

The registry’s primary outcome is employment at the 9-month follow-up, measured by questionnaire. The posted counts are 23 participants with the outcome in the VR-JIT arm and 10 in the supported-employment arm. The corresponding arm sizes are 54 and 36. The unadjusted recalculation estimates a positive difference, but its confidence interval spans zero, indicating substantial uncertainty about the size and direction of the underlying contrast.

The sponsor reports a logistic-regression analysis with a p-value of 0.027 and a chi-squared analysis with a p-value of 0.076. The simple recalculation differs from those analyses, and the discrepancy flag means the sponsor’s own analysis takes precedence. Because effect estimates and confidence intervals for the sponsor analyses are not reported, their magnitude and precision cannot be evaluated from the registry record. The different p-values also should not be collapsed into a single conclusion.

The result does not establish that VR-JIT caused the observed group difference. Randomization is relevant to causal interpretation, but details on allocation procedures, adherence, missing outcome data, and the analysis population are not reported. The registry also does not describe the employment definition or the questionnaire’s scoring rules. These omissions limit assessment of how the reported endpoint was constructed and who contributed to the comparison.

The findings do not answer whether any difference is meaningful to participants, whether it persists beyond the specified follow-up, or whether it generalizes to settings beyond the community-based evaluation described in the trial summary. The listed secondary outcomes may address related skills and experiences, but their results are not available in the registry record used for this interpretation. No conclusion about those outcomes follows from the primary employment count.

In short, the record shows different observed employment counts by arm and reports sponsor analyses with differing p-values. It also leaves important details about estimates, precision, missing data, multiplicity, and analysis populations unreported. The result is best read as a registry-reported comparison with limits on how firmly its size, certainty, and broader meaning can be characterized.

17. Why This Trial Matters Statistically

This trial is statistically interesting because it compares a behavioral training enhancement within a supported-employment context and uses employment at follow-up as its primary outcome. The endpoint moves beyond interview practice itself to a reported employment outcome, while the listed secondary outcomes concern skills, confidence, anxiety, social competence, and psychological distress. These outcomes represent different dimensions and should not be treated as interchangeable measures of the same construct.

The record also illustrates why an effect estimate, its uncertainty, and the analysis method all matter. The posted counts yield an unadjusted recalculation whose interval spans zero, while the registry reports two sponsor analyses with different p-values. The methods are not interchangeable, and the record does not give enough detail to resolve why they differ. The discrepancy flag appropriately makes the sponsor’s own analysis the primary reference for reported inference, without making its p-values a complete account of effect size or precision.

Another important issue is the gap between overall enrollment and the arm sizes in the posted primary outcome. Without an account of follow-up completeness and the analysis population, readers cannot tell how the outcome groups relate to everyone enrolled. This is not merely a reporting detail: missingness and exclusions can influence both the estimate and its precision.

Finally, the planning illustrations emphasize the sensitivity of sample-size requirements to the effect targeted. Smaller effects require more participants per arm under the stated assumptions. Because the calculations use the observed effect and the original design assumptions are not reported, they serve as planning context rather than an explanation of the trial’s enrollment. Together, these features underscore the value of clear reporting of methods, outcome definitions, analysis populations, and uncertainty.

18. Limitations

• The posted summary statistics are the only data these results rest on; patient-level data, full protocols and statistical analysis plans are not available.
• Recalculated results are simple, unadjusted calculations from posted summaries. They do not replace the sponsor's prespecified analysis.

The registry record leaves several details unavailable for interpreting this specific trial. It does not describe eligibility criteria in detail, participant characteristics, or how participants were recruited. The outcome groups are smaller than the overall enrollment figure, but the reason is not reported. Without a participant flow description or missing-data account, it is unclear how the posted primary outcome groups relate to the full enrolled population.

The employment endpoint is described as a questionnaire measure at the 9-month follow-up, but the questionnaire, employment definition, scoring rules, and handling of ambiguous responses are not reported. The record does not describe how missing questionnaire data were handled or whether sensitivity analyses were conducted. It also does not say whether participants were analyzed according to their randomized assignment.

Although randomization and single masking are recorded, details of allocation generation, concealment, and masking are not reported. The registry does not identify who was masked. It names logistic regression and chi-squared analyses, but gives no model estimate or confidence interval and does not explain the differing results. Adjustment variables, diagnostics, and the prespecified analysis plan are not reported.

The record lists multiple secondary outcomes and seven outcome measures and analyses, but does not explain their relationship or describe any approach to multiplicity. Results for the secondary outcomes are not included in the registry entries. The registry also does not provide a detailed account of the intervention’s delivery, amount of training, adherence, or variation in implementation. These are specific limits on interpreting the posted results, not evidence that any particular method or procedure was absent.

19. Related Tutorials

20. Related Statistical Calculators

21. Sources

• ClinicalTrials.gov: Virtual Reality Job Interview Training in Severe Mental Illness — NCT03049813.
• Registry record last updated 2023-03-09; status last verified 2023-03.
• Results first posted 2022-11-21.
• Data retrieved from ClinicalTrials.gov on 2026-11-09.
• Group sizes, means, event counts and sponsor-reported p-values are reproduced as posted. Effect estimates, confidence intervals and p-values not attributed to the sponsor are independently calculated from the posted summary statistics.

22. Record Summary

NCT03049813 is a completed randomized, single-masked, parallel-group trial of Virtual Reality Job Interview Training and Supported Employment (Services as Usual) for people with severe mental illness. The registry lists an enrollment of 168, a start date of 2017-06-01, and a primary completion date of 2021-06-30. The sponsor is the University of Michigan.

The primary endpoint is employment at the 9-month follow-up from randomization, measured by questionnaire. The posted outcome counts are 23 of 54 in the VR-JIT arm and 10 of 36 in the supported-employment arm. The registry reports logistic-regression and chi-squared analyses with different p-values. An unadjusted recalculation is also different; because the discrepancy flag is true, the sponsor’s own analysis takes precedence.

The record lists five secondary outcomes, but their results are not reported in the registry record for this interpretation. Reasons for the difference between overall enrollment and primary outcome group sizes, as well as details about missing data and analysis populations, are not reported in the registry record.