Using a protocol approved by the University of Pittsburgh Institutional Review Board, our single-center trial recruited patients from 26 primary care offices that shared a common electronic medical record (EMR) (Epic). Informed written consent was obtained from all participants.

We exposed PCPs to an EMR “Best Practice Alert” reminder about our study at the time of the clinical encounter.¹⁵ It launched automatically for all patients aged 18 to 75 years whenever anxiety, generalized anxiety, panic, or depression was entered as an encounter diagnosis. If the patient agreed to a referral, the PCP electronically “signed” the alert, which forwarded the patient’s name to a study recruiter who then called the patient by telephone to review protocol eligibility.

Eligible patients needed to have internet and email access; a score of 10 or greater on either the 7-Item Generalized Anxiety Disorder scale (GAD-7)¹⁶ or the 9-Item Patient Health Questionnaire (PHQ-9)¹⁷; and no alcohol dependence as determined by the Alcohol Use Disorders Identification Test,¹⁸ active suicidality, or other serious mental illness for which our interventions may be inappropriate. If confirmed, the recruiter reviewed a mailed consent form and obtained the patients’ signed consent on a recorded telephone line. Afterwards, they administered the 12-Item Short-Form Health Survey (SF-12) to determine health-related quality of life,¹⁹ the fixed-length Patient-Reported Outcomes Measurement Information System (PROMIS) depression and anxiety measures to assess mood and anxiety symptoms,²⁰ and the Primary Care Evaluation of Mental Disorders to provide an anxiety and mood disorder diagnosis,²¹ and collected information on patients’ self-reported race/ethnicity, sex, and other sociodemographic characteristics.

Following the baseline assessment, we randomized patients in a 3:3:1 ratio to (1) care manager–guided CCBT (CCBT alone), (2) care manager–guided access to both CCBT and our ISG (CCBT+ISG), or (3) usual care (UC) under their PCP. We stratified randomization by practice size and age group using randomly permuted blocks according to a computer-generated assignment sequence prepared in advance by our study statistician and concealed until after the baseline assessment. Afterwards, we informed all patients of their treatment assignment and notified their referring PCP.

For ethical reasons,²² we informed patients receiving UC of their mood and anxiety symptoms and their referring PCP. However, we provided no treatment advice unless we detected suicidality or a 25% worsening of symptoms from baseline on a follow-up assessment.

We employed college graduates with mental health research experience as care managers and assigned each exclusively to one intervention arm. We first prepared them in a basic understanding of mood and anxiety disorders, our pharmacotherapy algorithm,²³ CCBT program, and tracking registry and later reinforced this training in our weekly case review sessions.

We used the Beating the Blues CCBT program, which has been proven to be effective.^24,25 It consists of a 10-minute introductory video followed by eight 50-minute interactive sessions that our care managers encouraged patients to complete every 1 to 2 weeks. Each session used easily understood text, audiovisual clips, and “homework” assignments to impart basic CBT techniques, and patients completed the GAD-7 and PHQ-9 at the start of each CCBT session to self-track their symptoms (eMethods in Supplement 2).

We used WordPress software to create our password-protected ISG that patients could access via computer or smartphone (eFigure 1 in Supplement 2). In addition to a variety of discussion boards created by the care manager moderator and study patients, the ISG curated links to external resources, including local $4 generic pharmacy programs, “find-a-therapist” and various crisis hotlines, and brief YouTube videos on insomnia, nutrition, exercise, and other topics, and we embedded links to our EMR’s patient portal to integrate its use into routine care. To enhance patient engagement, we featured (1) status indicators on members’ profiles and comments (eg, stars and “likes”), (2) email notifications of new ISG activities, (3) automated highlighting of recent comments on members’ home pages personalized to their ISG profile and past activities, (4) invited member-guest moderators, and (5) various contests to encourage log-ins and comments.

To preserve confidentiality, we assigned members’ user names, encouraged them to select a representative avatar (eg, a sunrise or animal), and sent reminders not to post self-identifying information. For additional safety, an investigator logged into the ISG daily to review new posts for suicidal thoughts and other potentially inappropriate content, and we allowed members to flag comments for potential removal.

Care managers emailed their assigned patients a web link to the CCBT program and, if applicable, the ISG and requested a time to schedule an introductory telephone call to review the program(s) and establish rapport. Later, they logged into the CCBT program’s clinical helper portal to monitor their patients’ progress (eg, sessions completed, self-reported symptoms, and problems they chose to address), sent personalized feedback and encouragement via email, and contacted patients via telephone who either had not improved or failed to log in regularly.

Care managers presented their patients to the study PCP, psychiatrist, and project coordinator in weekly 60-minute case review sessions split by intervention arm. To efficiently focus our time, we developed an electronic registry that could sort patients by randomization date, last contact, and highest PHQ-9 or GAD-7 score (eMethods in Supplement 2). In addition to conveying general lifestyle adjustments, including exercise and social engagement, we recommended antidepressant/anxiolytic pharmacotherapy based on patients’ treatment preferences and response to CCBT as well as referrals to mental health specialists when they did not improve or had complex psychosocial issues.²³

Depending on a patient’s symptoms and level of engagement, the care manager emailed or telephoned biweekly for approximately 2 months, and these contacts lasted approximately 15 to 30 minutes. Afterwards, the patient transitioned to the continuation phase of care, during which the care manager contacted the patient approximately monthly until the end of our 6-month intervention. Given our collaborative care framework, we provided PCPs with our treatment recommendations and regular updates of their patients’ progress via EMR.

Patients, PCPs, and care managers were not blinded to their treatment assignment. Therefore, we employed several blinded assessors to determine the effectiveness of our interventions. They contacted patients by telephone to administer our assessment battery at 3-month, 6-month, and 12-month follow-up and later sent patients $15 after each completed assessment for their time (up to $60).

We trained our assessors using audiotapes, manuals, and practice interviews and used a computer-assisted telephone interview system to guide them through each assessment. We digitally recorded these calls and conducted periodic spot checks to confirm responses were rated accurately and corresponded with those entered into our study database, reviewed interactions with suicidal patients, and provided staff with feedback on their performance. Later, we abstracted data from the EMR to collect information on patients’ medical conditions and health services use, our server logs to measure engagement with the CCBT and ISG programs, and our care managers’ electronic registry to document the number of email and telephone contacts.

We programmed our computer-assisted telephone interview system to identify patients receiving UC whose blinded PROMIS score increased by 25% or more above baseline. Following a review, we notified their PCP via EMR and offered treatment advice. Whenever our care managers or assessors encountered suicidality, either expressed spontaneously or on routine administration of our measures, our computer-assisted telephone interview system automatically launched our Suicide Risk Management Protocol that provided triage advice.²⁶ The CCBT program also notified the care manager whenever a patient endorsed suicidality on the PHQ-9 administered at each session. Finally, an independent external data and safety monitoring board appointed by our funding agency monitored the progress and safety of our trial.

We powered our trial to test the primary hypothesis that patients receiving CCBT+ISG will report 0.30 or greater effect size (ES) improvement from baseline at 6 months on the SF-12 Mental Health Composite Scale (MCS) vs CCBT alone. Assuming a 2-sample t test to compare between-arm differences in 6-month improvements and 2-tailed α = .05, we needed 300 patients per arm to have 90% or greater power to detect a 0.30 ES difference in our primary outcome measure.

We compared baseline sociodemographic and clinical characteristics by randomization status using t tests for continuous data and χ² analyses for categorical data. Our primary intent-to-treat analyses included all randomized participants regardless of adherence to their assigned treatment. We used linear mixed models²⁷ that included fixed effects for time, study arm, time-by-study arm, age strata, practice size, and random effects for patients. We considered time as a categorical variable because of the assumption of nonlinearity over time. To test our hypotheses, we used contrasts to estimate the adjusted mean difference between study arms in the 6-month improvement of SF-12 MCS and PROMIS measures (secondary outcomes) and 6 months later to assess treatment durability. Additionally, we calculated ESs for 6-month changes in SF-12 with 95% CIs by (1) prespecified subgroups of age group, sex, race/ethnicity, baseline symptom severity, and practice size and (2) unplanned subgroups of education and living alone status. We considered a significant 3-way interaction between time, study arm, and the potential covariate as a significant subgroup effect and used Poisson regression to compare rates of PCP contacts, emergency department visits, and hospitalizations between study arms.

We investigated a potential dose response between the number of CCBT sessions completed within our CCBT alone and UC study arms using the same linear mixed model described earlier but parametrizing the CCBT alone arm by assigning each patient a value equal to the proportion completed of the 8-session program. Finally, we conducted exploratory post hoc per-protocol analyses restricted to those who completed 4 or more and all 8 CCBT sessions.

Every effort was made to identify the mechanism of missing data. We compared participants who withdrew from study participation by baseline covariates and analyzed time until withdrawal by study arm using Kaplan-Meier curves.²⁸ Our linear mixed models for our primary analyses assumed that data were missing at random and were robust to ignorable missingness assumptions.²⁹ All reported P values are 2-tailed with significance levels at P ≤ .05, and all analyses were performed with SAS version 9.4 (SAS Institute).

From August 2012 to September 2014, PCPs referred 2884 patients in response to our EMR prompt. Of these, 704 (24.4%) met all eligibility criteria, provided informed consent, and were randomized (Figure 1). Their baseline sociodemographic and clinical characteristics (Table 1) and completion rate of follow-up assessments at both 6 months (604 [85.8%]) and 12 months (593 [84.2%]) were similar by randomization status (Figure 1), and we found no differences in the sociodemographic and clinical characteristics between participants who withdrew and those who did not.

By 6 months, 504 of 603 patients (83.6%) with CCBT access had completed at least 1 session and 221 (36.7%) had completed all 8, and the mean sessions completed was 5.4, which was similar by randomization status (Table 2), sex, race/ethnicity, and age strata (eTable 1 in Supplement 2). Overall, 228 of 302 patients (75.5%) in the CCBT+ISG arm logged into the ISG at least once, of whom 141 (61.8%) made at least 1 online comment or post (mean, 10.5; median, 3; range, 1-306) (Table 2) (eFigure 2 in Supplement 2).

At 6-month follow-up, patients in the CCBT+ISG and CCBT alone arms reported similar improvements on our primary outcome measure (SF-12 MCS: ES, 0.02; 95% CI, −0.17 to 0.13) and on the PROMIS Depression and Anxiety scales that continued 6 months later (Figure 2). We also identified a significant treatment interaction favoring CCBT+ISG for patients aged 60 to 75 years on the SF-12 MCS (Figure 3) (eFigure 3 in Supplement 2) and CCBT alone for patients aged 35 to 59 years on the PROMIS Depression and Anxiety scales (eFigure 4 in Supplement 2).

Compared with patients in the UC arm, patients in the CCBT alone arm reported significant 6-month improvements on the PROMIS Depression and Anxiety scales (eFigure 4 in Supplement 2) but not the SF-12 MCS scale (Figure 2). However, these differences resolved 6 months later, as patients’ symptoms in the UC arm improved. Again, we observed significant treatment interactions favoring CCBT for patients aged 35 to 59 years on the SF-12 MCS (eFigure 3 in Supplement 2) and PROMIS Depression and Anxiety scales (eFigure 4 in Supplement 2), for patients living alone on the PROMIS Depression and Anxiety scales, and for nonwhite patients on the PROMIS Depression scale (eFigure 4 in Supplement 2). Moreover, patients reported improved 6-month SF-12 MCS (mean points, 0.80; 95% CI, 0.37-1.22), PROMIS Depression (mean points, 0.48; 95% CI, −0.76 to −0.19), and PROMIS Anxiety (mean points, 0.48; 95% CI, −0.79 to −0.17) scores for each additional CCBT session completed, and per-protocol analyses revealed a similar pattern (PROMIS mood symptoms: patients who completed ≥4 sessions: ES, 0.41; 95% CI, 0.17-0.65; patients who completed all 8 sessions: ES, 0.52; 95% CI, 0.26-0.78) (eTable 2 in Supplement 2).

Primary care physicians and care manager contacts with patients via telephone and email were similar by intervention arm (Table 2). Moreover, patients receiving UC and intervention had similar rates of PCP contacts, use of antidepressant and anxiolytic pharmacotherapy, and visits to mental health specialists, emergency departments, and hospitals (Table 2).

To our knowledge, this is the first trial to examine the effectiveness of incorporating either CCBT or an ISG into a collaborative care program for treating depression or anxiety in primary care. Our report confirms the effectiveness of guided CCBT, highlights the critical importance of patient engagement with online interventions, and provides high-quality evidence about the limits and potential benefits of these emerging technologies.

Few trials have evaluated the psychologic benefits of ISGs, and none were linked to patients’ usual source of primary care as ours was.^7,30– 32 Perhaps most comparable with our trial is the trial by Griffiths et al,³¹ who randomized 478 adults with elevated depressive symptoms to either a moderated ISG, an online psychotherapy program, both interventions, or to a UC control. They too found their combined ISG and online psychotherapy interventions improved mood symptoms vs UC and no benefit from their ISG beyond UC. However, engagement was low, as only 62% of patients in the ISG group logged into the site, only 15% created 1 or more ISG posts, and those assigned to the ISG arm were more likely than patients in the UC arm to miss a blinded telephone assessment (52% vs 34% at 12 months).³¹ These findings as well as other reports³³ add to our understanding of the challenges in sustaining patient engagement with online interventions to improve clinical outcomes.^34,35

Unlike our earlier collaborative care trials, where care managers assigned patients homework lessons in printed workbooks,^36– 38 the CCBT program enabled our care managers to unobtrusively monitor their patients’ engagement with treatment while providing similarly effective care to a doubled caseload (90 to 100 patients).^36,38 Although the ES improvements we obtained were smaller than those described in meta-analyses of “supported” CCBT (major depression: ES, 0.78; 95% CI, 0.59-0.96⁴), we identified a dose effect that confirms the importance of patient engagement.³⁹ Indeed, Gilbody et al⁴⁰ reported in 2015 no differences in mood symptoms among 691 patients with depression in primary care they randomized to either Beating the Blues or MoodGYM (HealthMed) CCBT programs⁴¹or usual care from their PCPs. Similar to our protocol, their study staff contacted patients by telephone to promote use of their programs; however, they did not monitor patients’ symptoms or send recommendations to PCPs as we did, and patient adherence with both CCBT programs was low (median sessions completed, <2).⁴⁰

Given our findings and other recent reports,^3– 5 we anticipate more engaging and powerful CCBT programs better tailored to patients’ specific needs, sociodemographic characteristics, medical conditions, and cultural and linguistic preferences that are integrated into the EMR for documentation and billing purposes will become widely deployed over the next decade. Finally, while we were unable to demonstrate a measurable benefit from our ISG, we remain optimistic that more engaging ISGs that apply machine learning algorithms to EMR and claims data to present patients with more personalized information in real time will soon be tested by health care organizations experimenting with social media.^42,43

Our study has limitations, several of which potentially affect the generalizability of our findings. First, our use of EMR-generated prompts to promote identification of patients for study participation is limited to settings with systems capable of generating these alerts, clinician recognition of targeted conditions, and entry of the proper diagnostic codes into the EMR. Second, we relied on 1 CCBT and ISG, and others using different programs and levels of human support to promote adherence may obtain different outcomes. Third, because we lacked information on symptom duration, patients with long-term mild mood and anxiety symptoms may have similar outcomes as those with severe acute symptoms. Fourth, given the nature of our interventions, patients knew their treatment assignment, which could have biased their responses to our blinded assessors. Finally, study sites were not cluster randomized, and the same physicians cared for patients in all study arms. Although this could have diminished outcome differences between treatment arms, we observed similar ES improvements as previous collaborative care trials.¹

In summary, although our ISG did not produce any measurable benefit over CCBT alone, providing online CCBT to patients with depression and anxiety receiving primary care via a centralized collaborative care program is an effective strategy for delivering mental health care at scale. Our study findings have important implications for transforming the way mental health care is delivered in primary care and focus further attention to the emerging field of e–mental health.