Email updates

Keep up to date with the latest news and content from Implementation Science and BioMed Central.

Open Access Study protocol

A randomized controlled dismantling trial of post-workshop consultation strategies to increase effectiveness and fidelity to an evidence-based psychotherapy for Posttraumatic stress disorder

Shannon Wiltsey Stirman1, Norman Shields2, Josh Deloriea3, Meredith SH Landy3, Jennifer M Belus4, Marta M Maslej3 and Candice M Monson3*

Author Affiliations

1 Women’s Health Sciences Division, National Center for PTSD, VA Boston Healthcare System, Boston University, 150 South Huntington Ave (116B3), Boston, MA 02130, USA

2 Veterans Affairs Canada, Operational Stress Injuries National Network (OSINN), 305 Boul des Anciens-Combattants, Sainte-Anne-de-Bellevue, QC H9X 1Y9, Canada

3 Department of Psychology, Ryerson University, 350 Victoria Street, Toronto, ON M5B 2K3, Canada

4 Department of Psychology, University of North Carolina, 209 South Rd., Chapel Hill, NC 27599, USA

For all author emails, please log on.

Implementation Science 2013, 8:82  doi:10.1186/1748-5908-8-82


The electronic version of this article is the complete one and can be found online at: http://www.implementationscience.com/content/8/1/82


Received:12 April 2013
Accepted:26 July 2013
Published:1 August 2013

© 2013 Wiltsey Stirman et al.; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Posttraumatic Stress Disorder (PTSD) is a serious mental health condition with substantial costs to individuals and society. Among military veterans, the lifetime prevalence of PTSD has been estimated to be as high as 20%. Numerous research studies have demonstrated that short-term cognitive-behavioral psychotherapies, such as Cognitive Processing Therapy (CPT), lead to substantial and sustained improvements in PTSD symptoms. Despite known benefits, only a minority of clinicians provide these therapies. Transferring this research knowledge into clinical settings remains one of the largest hurdles to improving the health of veterans with PTSD. Attending a workshop alone is insufficient to promote adequate knowledge transfer and sustained skill; however, relatively little research has been conducted to identify effective post-training support strategies.

Methods

The current study investigates whether clinicians receiving post-workshop support (six-month duration) will deliver CPT with greater fidelity (i.e., psychotherapy adherence and competence) and have improved patient outcomes compared with clinicians receiving no formal post-workshop support. The study conditions are: technology-enhanced group tele-consultation; standard group tele-consultation; and fidelity assessment with no consultation. The primary outcome is independent assessment (via audio-recordings) of the clinicians’ adherence and competence in delivering CPT. The secondary outcome is observed changes in patient symptoms during and following treatment as a function of clinician fidelity. Post-consultation interviews with clinicians will help identify facilitators and barriers to psychotherapy skill acquisition. The study results will inform how best to implement and transfer evidence-based psychotherapy (e.g., CPT) to clinical settings to attain comparable outcomes to those observed in research settings.

Discussion

Findings will deepen our understanding of how much and what type of support is needed following a workshop to help clinicians become proficient in delivering a new protocol. Several influences on clinician learning and patient outcomes will be discussed. An evidence-based model of clinical consultation will be developed, with the ultimate goal of informing policy and influencing best practice in clinical consultation.

Trial registration

ClinicalTrials.gov: NCT01861769

Keywords:
Training; Implementation; Consultation; Psychotherapy; PTSD; Cognitive processing therapy

Background

The lifetime prevalence of PTSD is approximately 9% in Canada’s general population [1], 6.8% in the United States (U.S.) [2], and 15% to 19% among veterans of the recent conflicts in Iraq and Afghanistan [3,4]. PTSD is associated with substantial functional impairment, suicide risk [5], numerous physical diseases [6-9], higher rates of mortality, high healthcare utilization and costs, and annual productivity loss of over $3 billion in the U.S. [10-15]. Studies of chronic PTSD indicate little change over time without active intervention [16].

Cognitive-behavioral therapies (CBT) for PTSD have been shown to be efficacious and are recommended as first-line therapies in treatment guidelines [17,18]. Cognitive Processing Therapy (CPT) is a 12-session, trauma-focused cognitive therapy that targets symptoms of PTSD and its comorbidities [19] that results in sustained, significant reductions in symptoms and increases in quality of life for diverse populations [20-34]. Despite substantial efficacy data, clinicians do not routinely deliver CPT or other evidence-based psychotherapies (EBPs) [35].

In Canada, the U.S., and the United Kingdom, policymakers have prioritized investigating the implementation of EBPs to address this critical gap between research and practice [36]. In response to the shortage of adequately trained clinicians [37], policymakers have issued mandates, provided incentives, and invested billions of dollars in EBP training of clinicians in public mental health settings [35,38]. However, many of the utilized training strategies are not empirically supported, and much remains to be learned about how best to disseminate and implement EBPs in routine clinical practice. Without proven strategies to increase clinician use, there is little assurance that the investment that individuals and organizations make to develop EBPs will ultimately benefit patients. Both the Canadian Institute of Health Research (CIHR)’s Health Research Roadmap (2009 to 2014; [39]) and the U.S. National Institute of Mental Health [40] note the growing need for implementation science to better understand the determinants and uptake of knowledge. The use of evidence-based implementation strategies will ensure that money spent on treatment research has maximum impact.

Didactic workshops are a cornerstone of existing large-scale EBP implementation efforts [35,38,41]. However, workshops alone are insufficient to promote high-quality implementation and sustainment of EBPs in clinical settings. After a single workshop, most clinicians are unable to deliver EBPs with fidelity (adherence to the protocol and competence in delivery of the interventions [42,43]). Fidelity to EBPs has been shown to predict patient outcomes for a number of mental health disorders [44-49], and has been associated with PTSD symptom outcomes in our preliminary research on CPT. To adequately prepare clinicians to deliver EBPs with fidelity, training packages that include post-workshop consultation and feedback have emerged as best practices, but little is understood about the relative benefits of different consultation strategies [41,42].

Previous comparisons of post-workshop consultation strategies have lacked external validity (e.g., have been conducted under controlled conditions with highly motivated therapists in training clinics), rarely assessed fidelity objectively and in real-world practice conditions [50], had poor follow-up rates, rarely addressed barriers to implementation, and have demonstrated less than optimal results [41-43]. Thus, there is little empirical evidence to identify optimal, cost-efficient consultation and feedback strategies. Feedback based on observation is considered the ‘gold standard’ for training and is assumed to be necessary for optimal fidelity [42]. Without more direct observation of therapist behavior, there is limited opportunity for the expert to provide the type of feedback required to promote fidelity and high-quality service delivery [51]. The one randomized study that tested this assumption found that coaching based on review of session material resulted in greater change talk among clients [52]. This study evaluated individual-level consultation under controlled conditions with limited external validity, and had no comparison to group consultation strategies. Individual feedback and consultation is too costly and resource-intensive to be feasible in large healthcare delivery systems [53]. One strategy used in several training initiatives is expert-delivered group tele-consultation, in which clinicians receive feedback from an expert on their treatment provision based solely on their verbal reports of their sessions [35]. However, the association between clinician report of their behavior and objective assessment of therapy fidelity is unknown.

To address this limitation of the standard consultation strategy, the present study also tests a technology-enhanced group tele-consultation. Our approach involves an expert and group members listening to, and providing feedback on, portions of group members’ audio-recorded therapy sessions on a weekly basis. This strategy provides exposure to a broad sample of case material for vicarious learning, and also allows the expert to provide more specific and accurate feedback than in standard consultation. However, while it provides the opportunity for more in-depth discussion of work samples for the clinicians who present in a given week, the time spent on the audio review may result in less time for other clinicians in the group to discuss their own cases when not presenting audio. Thus, each consultation strategy may have unique advantages over the other. The two consultation conditions will be compared to a no consultation (fidelity monitoring only) condition in order to assess the incremental benefit of post-workshop consultation. The study is funded by the Canadian Institutes of Health Research and represents the collaborative efforts of investigators at Ryerson University, Veterans Affairs Canada’s Operational Stress Injury National Network (OSINN), the U.S. National Center for PTSD, and Boston University. This article describes the study’s aims, method and proposed analytic strategy.

Research questions and hypotheses

Question 1: The principal research question to be addressed is whether different types of post-workshop support for mental health clinicians have differential effects on clinician fidelity to the CPT protocol.

Hypothesis 1: We hypothesize that the technology-enhanced group tele-consultation condition will evidence the highest levels of fidelity, the standard group tele-consultation condition will evidence intermediate levels of fidelity, and the no consultation condition will evidence the lowest fidelity.

Question 2: Is clinician fidelity to the CPT protocol is related to client outcomes?

Hypothesis 2: We hypothesize that fidelity to the CPT protocol, irrespective of consultation condition, will be positively associated with improved client outcomes.

Question 3: What are contextual barriers or facilitators to the skilled use of CPT in their routine practice?

Hypothesis 3: We hypothesize that organizational contextual variables such as local leadership support, organizational structure, and organizational climate, as well as clinician attitudes toward EBPs will influence the uptake of CPT.

Methods

This randomized controlled dismantling trial has independent fidelity rating as the primary outcome and patient outcomes as the secondary outcomes. Clinicians are randomly assigned to six months of post-workshop support following an intensive two-day CPT workshop delivered by the last author. The conditions are: standard group tele-consultation, technology-enhanced group tele-consultation, or no consultation/fidelity monitoring only. The post-workshop consultation conditions vary as to whether audio-recorded therapy sessions and other work samples, such as CPT worksheets, are reviewed in the group. This design permits the evaluation of the additive value of each of the elements (i.e., expert consultation, technology-assisted expert consultation). Follow-up assessments of fidelity will be conducted at six months post-workshop for clinicians, and follow-up symptom assessments will be conducted at three months post-treatment for patients.

Participation in the study requires that all clinicians consent at least two clients to have all of their CPT sessions audio-recorded for use in consultation and/or fidelity rating. Four therapy sessions will be randomly selected across the six-month period for fidelity rating, and fidelity to treatment will be assessed by trained graduate students and post-doctoral fidelity assessors. Consistent with recommended clinical practice and research trials, patient outcomes will be assessed pre-treatment, at alternate therapy sessions, and at three months post-treatment. A secure web-based application is used to manage the continuous uploading of encrypted audio-recorded therapy sessions to the study server and to protect the privacy and confidentiality of patients and clinicians. Study-provided digital audio-recorders (USB-equipped) are used to record, store and upload CPT sessions to the secure server for fidelity ratings. Study staff retrieve the randomly-selected audio-recordings and assign them to the CPT fidelity raters, who are blinded to study condition. To check if rater blinding has been achieved, fidelity raters will be asked to guess the study condition for each session they rated. Patient self-report data is collected by the clinicians, who then upload the data directly to the server and clinician CPT activity outside of study-related procedures is also recorded.

Interventions

Participants in all conditions attend the workshop, receive the CPT manual and related materials, and have access to resources available through the free CPT-web online training (https://cpt.musc.edu/index webcite). All participants who upload the required number of session recordings and patient symptom measures will be eligible to receive ‘CPT Provider’ status. This status is not tied to any change in remuneration for services. Provider status will be awarded to clinicians who demonstrate an acceptable level of fidelity to CPT based on session recordings randomly selected from the last two weeks of the six-month post-workshop period. Workshop participants who provide informed consent are randomized to one of the three post-workshop consultation conditions after completing the workshop. Participants in the two tele-consultation conditions receive six months of weekly one-hour group tele-consultation with a CPT expert. Each call includes discussions of the CPT protocol, challenging cases, treatment obstacles, and specific issues raised by participants within each group. Both tele-consultation conditions include a maximum of six to eight clinicians per call. CPT experts provide consultation across both consultation conditions to control for any expert-related effects. The prescribed and proscribed activities for each consultation condition are described in manuals, and study team members (SWS, CMM, ML) review consultant calls on a monthly basis to ensure that fidelity to the consultation condition is maintained. Consultants are trained by study team members prior to beginning consultation calls and receive consultation and feedback every four to six weeks or as needed for the first two waves of consultation, and as needed during the third consultation wave.

Inclusion and exclusion criteria

Mental health clinicians from the Operational Stress Injury clinics, Canadian Forces mental health services, and the broader Canadian community are eligible to participate in the study if they: attend a CPT workshop; are licensed as a mental health clinician and are able to practice independently; currently provide psychotherapy to patients with PTSD; consent to be randomized to one of the study conditions; and are willing to solicit patient participation. Eligible patient participants are: diagnosed with PTSD; and consent to have their sessions audio-recorded and listened to by a fidelity rater and potentially other clinicians teaching and learning CPT. Patients are permitted to continue other psychotherapeutic interventions if they are not specifically focused on treating PTSD symptoms. Ineligible patients include those not eligible for CPT based on the state of research evidence, including those with: current uncontrolled psychotic or bipolar disorder; substance dependence; imminent suicidality or homicidality that requires acute care; and significant cognitive impairment (although mild to moderate traumatic brain injury is permitted).

Outcome measures and assessment procedures

Fidelity ratings and assessment of CPT modification from audio-recorded CPT sessions

This study uses a modified version of the CPT fidelity measure [54] that has been used in previous clinical trials [55]. The CPT fidelity measure examines clinicians’ adherence to, and competence in, delivering specific CPT interventions prescribed in each session. Clinicians are rated on their adherence and competence in delivering these elements (rated on a 7-point, Likert-type scale). Two studies have assessed the reliability of the measure and found 97% agreement between two raters across all items for adherence in one study (agreement was not reported for competence scores; [55]) and 100% agreement for adherence and overall competence ratings in another [56]. Pilot research has been conducted to assess the predictive validity of the unique and essential scales for a small sample of veterans. CPT competence was highly correlated across all sessions with competence in cognitive therapy as measured by a validated cognitive therapy fidelity-assessment instrument, r = 0.82, p = 0.001 [44,57]. Therapist competence in the first session of CPT was positively correlated with self-reported symptom improvement among patients from baseline to post-treatment, r = 0.79, p = 0.02. Additionally, we use a framework of modifications to identify any deviations from or adaptations to CPT that occur in the sessions [58]. At least 10% of the sessions will be rated by all raters to independently assess inter-rater agreement.

Patient outcome measures

Patients are asked to complete measures related to their PTSD symptoms, functional impairment and overall quality of life at baseline (pre-treatment), at every alternate CPT treatment session, and at post-treatment follow-up (three months post-treatment). The assessment measures include: ‘The Posttraumatic Stress Disorder Checklist’ (PCL-S; [59]), a 17-item questionnaire that reliably measures the severity of distress related to PTSD symptoms (Chronbach’s alpha = 0.94). Each item is measured on a 5-point Likert scale, which maps on to the DSM-IV criteria for PTSD; ‘The Outcome Questionnaire – 45’ (OQ- 45; [60]), a 45-item self-report measure of three dimensions of client functioning in the past week: symptom distress (e.g., anxiety, depression), interpersonal problems, and social role adjustment (Chronbach’s alpha = 0.71 to 0.93). The OQ-45 has established clinical cut-offs and a reliable change indices to signal meaningful improvements or deterioration; and ‘The Short-Form Quality of Life Health Survey’ (SF-12; [61]), a 12-item survey that measures mental components (vitality, social functioning, role-emotional, and mental health) and physical components (physical functioning, role-physical, bodily pain, and general health) of health related to quality of life. All questionnaires are available in English and French, and take clients around 15 to 20 minutes to complete. Therapists upload summary scores to a secure website at the same time that they upload their audio-recordings.

Clinician demographic characteristics and experience

A pre-workshop questionnaire is administered to assess relevant clinician demographic information (i.e., age, gender, education), clinician experience (years of licensed practice, hours of formal training in CBT, hours of supervised post-graduate CBT training), prior CPT training experience (workshop hours and supervision hours), experience treating clients with PTSD (number of clients), and volume of current clinical activity (caseload and hours devoted to psychotherapy).

Clinician monthly CPT activity reporting

During the six-month active study period, all clinicians complete a monthly report via a web-based survey on CPT activity. Questions include: the number of CPT sessions provided over the past month (irrespective of patients’ enrollment in the study); cumulative number of new CPT clients seen; frequency, duration, and satisfaction with consultation over the past month (as applicable); and a rating of clinician confidence in their ability to adhere to adherence to the CPT protocol.

Quantitative and qualitative measurement of potential barriers and facilitators to CPT skills uptake and implementation outcomes

The Organizational Readiness to Change Questionnaire [62] is a pre-workshop self-report questionnaire that assesses various elements of a clinician’s perceived support for training implementation. A total of 18 scales (Chronbach alphas = 0.56 to 0.92) are grouped within four dimensions of readiness (i.e., motivation for change, adequacy of resources, staff attributes, organizational climate). The Dimensions of Organizational Readiness-Revised (DOOR-R; [63]) scale assesses perspectives on intra- and extra-organizational implementation readiness. The Evidence-Based Practice Attitudes-50 (EBPAS-50; [64]) is a 50-item, self-report measure of clinician attitudes toward EBPs, consisting of 12 factors, with moderate to large factor loadings and fair to excellent internal consistency reliabilities.

We also interview clinicians at baseline, post-consultation, and three months post-consultation to further assess for barriers and facilitators to the use of CPT in their practice. The interviews query effective tactics and methods that may have been used outside of the consultation methods (i.e., positive deviance; [65]) that the clinicians deemed helpful in acquiring and maintaining CPT skills. We based our interview questions on those used to assess influences on training and implementation in previous research [66], which are based in part on the Consolidated Framework for Implementation Research (CFIR [67]).

Sample size and justification for power calculations

In conducting power calculations, we considered assessment attrition, within-clinician correlation, and within-setting effects. Based on a power curve calculation with the following assumptions: 30 clinicians per condition; 4 sessions rated for fidelity per clinician, randomly selected over time; with an alpha-level of 0.05, within provider correlation estimated at 0.38, 10% proportion of variance accounted for by the random effect associated with agency, based on a two-sided test, the detectable effect size for at least 80% power will be medium to large, per Cohen’s [68] classifications, ranging from d = 0.41 to 0.55.

Recruitment and randomization

We recruited OSI clinicians directly. Recruitment took place at CPT workshops in Canada from 2012 to 2013. Three months before the first workshop, the OSINN invited clinicians and clinic coordinators from the OSI clinic network, Canadian Forces health services, and approximately 800 community-based clinicians who provide services to veterans with PTSD. The clinicians include psychologists, social workers, psychiatric nurses, and psychiatrists. This invitation was accompanied by a letter soliciting participation in the study. Prior to enrolling clinicians, ethics approval for the study was obtained from the clinic or institution where the clinician worked. Community clinicians were informed that the CHIR-supported workshop was free, and 24 clinicians received up to $500 reimbursing travel and accommodation. Partial reimbursement for the community clinicians was intended to offset the barrier of cost associated with a two-day workshop. Following the initial CIHR-funded workshop, two provincial health authorities and one hospital hosted workshops. One of the authors (CMM) was the primary CPT trainer. Interested clinicians consented prior to, or within the week following the workshop. Consent forms outlined the purpose and requirements of the study, as well as potential risks. A random allotment of participants was generated by the study statistician for each condition. To avoid any biases in condition assignment, condition assignment was communicated by participant number to the study coordinator after participant numbers had been assigned to the consented participants. Clinicians received a notification e-mail from the study coordinator within one week after the workshop regarding their randomized condition. The e-mail also included consultant contact information and a schedule of consultation sessions (if applicable to their condition). As in other studies of this nature (e.g., [52]), the clinicians were asked to provide an information sheet about the study and obtain verbal consent from patients who agreed to participate in the study, then document patient consent in his/her medical records. Upon returning to their respective work places, participants began audio-recording CPT sessions with consented patients.

Proposed analytic strategy

Trial reporting will follow the CONSORT 2010 statement for reporting randomized controlled trials [69]. Analyses will be preceded by careful examination of the descriptive statistics and distributional form of all variables. The effectiveness of randomization will be evaluated by comparing the different conditions on baseline measures, demographics, and key outcome variables using Chi-squared tests and t-tests. The primary analyses will be conducted using random effects regression modeling. These models provide more flexibility in modeling missing data than general linear models and are appropriate for nested data. ‘Intention-to-train’ principles will be used in the analyses of clinicians who are assigned to a consultation modality but either fail to engage or are unable to fulfill the post-workshop participation requirements (e.g., do not attend consultation sessions). Analyses of both the intention-to-train and completer samples will be conducted.

To explore the contribution of organization and clinician-level factors to the outcomes of interest, in addition to random effects regression modeling, we will conduct qualitative analyses on interview data. Qualitative and quantitative data will be integrated to facilitate a fine-grained understanding of processes and characteristics that influence training outcomes. Interviews will be analyzed in line with goals, objectives and key research questions. Using content analysis, we will identify analytical categories to describe and explain our observations. Our work will occur in five stages outlined in Pope, Ziebland, and Mays’ framework approach to qualitative analysis [70]: Familiarization; Identifying a Thematic Framework; Indexing; Charting; and Mapping and Interpretation. In the second stage, codes will be derived deductively by identifying categories at the beginning of the research (e.g., elements of the CFIR), and inductively by identifying those that emerge gradually from the data. In subsequent phases, survey data on contextual factors will be examined in conjunction with fidelity scores for the purposes of validation, hypothesis generation, and expansion [71,72] to understand ways in which contextual factors influence fidelity to CPT.

Trial status

To date, six workshops have been completed, and, through efforts to enroll clinicians over three waves of recruitment, the recruitment goal of 90 clinicians was exceeded. The first and second waves of consultation have been completed, and the third wave is underway. Pre- and post-training interviews with providers from waves one and two are complete. Thus far, our retention rate and compliance with study procedures (e.g., uploading of sessions) has been high, and a high proportion of clinicians uploaded recordings at baseline and over the course of the consultation period. We anticipate that data collection will be complete in early 2014.

Discussion

There are four significant innovations that the current project offers the field of Implementation Science. First, this will be the first study to examine psychotherapy training and consultation within a broader implementation context. Previous randomized training studies have not examined the organizational context. Our use of mixed-methods research strategies in the context of a randomized controlled trial [72] will allow us to examine a broad range of potential influences on the outcomes that we are assessing. This approach will extend our understanding of interactions between these factors and training outcomes.

A second major advancement is the empirical investigation of the optimal (efficacious, scalable and replicable) methods of post-workshop consultation. To adequately test the incremental value of consultation, we are conducting a randomized, dismantling research design, which includes manipulation of the type of post-workshop consultation while holding other variables constant. Results will help inform policy on post-workshop support. The inclusion of the fidelity monitoring only condition allows for testing an as yet unevaluated and potentially efficient strategy for increasing fidelity to EBPs: recording sessions with the knowledge that they will be reviewed and evaluated for fidelity. Our control condition allows for the formal testing of this possibility while allowing us to examine the incremental value of different types of expert consultation.

Although fidelity is a key implementation outcome in the model by Proctor and colleagues [73] and in this study, fidelity to EBPs has not been adequately assessed by independent evaluators in prior implementation research. It has either not been assessed at all, or has been assessed through self-report, standardized patient protocols [74], or a single session selected by providers (c.f., [52,75]). Our third innovation is to improve upon this methodological limitation of prior studies by randomly selecting audio-recorded sessions for fidelity monitoring. We will also conduct the only investigation of the effect of clinician fidelity to an EBP in routine care settings on PTSD outcomes to date. Because this investigation will examine sessions from novice clinicians who received varying intensities of consultation, we expect to see a greater range of fidelity than that seen in previous investigations of the impact of fidelity on symptom outcomes, as most studies to date have used clinical trial data with sessions from closely supervised clinicians. Finally, no evidence-based guidance exists regarding how to facilitate fidelity to the CPT protocol and ultimately patient outcomes. Using transcripts from consultation sessions, members of our research team will develop and test a theory of evidence-based consultation.

Ethical

The study has been approved by the Ryerson University Research Ethics Board, and the Boston VA and Boston University Institutional Review Boards.

Abbreviations

PTSD: Posttraumatic stress disorder; CPT: Cognitive processing therapy; CBT: Cognitive behavioral therapy; EBP: Evidence based psychotherapy.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

CMM, SWS, and NS designed the study and each wrote sections of the proposal. ML, JD, MMM, and JB provided a critical review of the manuscript, contributed to the development and description of study procedures and compiled information on the current status of the research. All authors had the opportunity to review the final draft of this manuscript.

References

  1. Van Ameringen M, et al.: Post-traumatic stress disorder in Canada.

    CNS Neurosci Ther 2008, 14(3):171-181. PubMed Abstract | Publisher Full Text OpenURL

  2. Kessler RC, et al.: Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the national comorbidity survey replication.

    Arch Gen Psychiatry 2005, 62(6):617. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  3. Tanielian T, Jaycox L: Invisible wounds of war : psychological and cognitive injuries, their consequences, and services to assist recovery. Santa Monica: RAND Corporation; 2008. OpenURL

  4. Schell TL, Marshall GN: Survey of individuals previously deployed for OEF/OIF, in invisible wounds of war: psychological and cognitive injuries, their consequences, and services to assist recovery. Edited by Tanielian T, Jaycox LH. Santa Monica: RAND Center for Military Health Policy Research; 2008:87-115. OpenURL

  5. Sareen J, et al.: Physical and mental comorbidity, disability, and suicidal behavior associated with posttraumatic stress disorder in a large community sample.

    Psychosom Med 2007, 69(3):242-248. PubMed Abstract | Publisher Full Text OpenURL

  6. Sareen J, et al.: The relationship between anxiety disorders and physical disorders in the U.S. National Comorbidity Survey.

    Depress Anxiety 2005, 21(4):193-202. PubMed Abstract | Publisher Full Text OpenURL

  7. Schnurr PP, et al.: Predictors and outcomes of PTSD in World War II veterans exposed to mustard gas.

    J Consult Clin Psychol 2000, 68(2):258-268. PubMed Abstract | Publisher Full Text OpenURL

  8. Sibai AM, Fletcher A, Armenian HK: Variations in the impact of long-term wartime stressors on mortality among the middle-aged and older population in Beirut, Lebanon, 1983–1993.

    Am J Epidemiol 2001, 154(3):128-137. PubMed Abstract | Publisher Full Text OpenURL

  9. Schnurr P, Green BL: Understanding relationships among trauma, post-traumatic stress disorder, and health outcomes.

    Adv Mind Body Med 2004, 20(1):18-29. PubMed Abstract OpenURL

  10. Kessler RC: Posttraumatic Stress Disorder: The burden to the individual and to society.

    J Clin Psychiatry 2000, 61(suppl 5):4-12. PubMed Abstract OpenURL

  11. Hoge CW, Auchterlonie JL, Milliken CS: Mental health problems, use of mental health services, and attrition from military service after returning from deployment to Iraq or Afghanistan.

    J Am Med Assoc 2006, 295:1023-1032. Publisher Full Text OpenURL

  12. Wolfe J, et al.: Posttraumatic stress disorder and war-zone exposure as correlates of perceived health in female Vietnam War veterans.

    J Consult Clin Psychol 1994, 62(6):1235-1240. PubMed Abstract | Publisher Full Text OpenURL

  13. Schnurr PP, et al.: PTSD and utilization of medical treatment services among male Vietnam veterans.

    J Nerv Ment Dis 2000, 188(8):496-504. PubMed Abstract | Publisher Full Text OpenURL

  14. Walker EA, Newman E, Koss M: Costs and health care utilization associated with traumatic experiences, in Trauma and health: Physical health consequences of exposure to extreme stress. Edited by Schnurr P, Green BL. Washington, DC: American Psychological Association; 2004:43-69.

  15. Dobie JD, et al.: Posttraumatic stress disorder in female veterans: association with self-reported health problems and functional impairment.

    Arch Intern Med 2004, 164(4):394-400. PubMed Abstract | Publisher Full Text OpenURL

  16. Resick PA, Monson CM, Gutner C: Psychosocial treatments for PTSD., in handbook of PTSD: science and practice. Edited by Friedman MJ, Keane TM, Resick PA. NY: Guilford; 2007:330-358.

  17. Foa EB, et al.: Effective treatments for PTSD: Practice guidelines from the International Society for Traumatic Stress Studies (2nd ed.). 2nd edition. New York, NY US: Guilford Press; 2009. [Effective treatments for PTSD: Practice guidelines from the International Society for Traumatic Stress Studies] OpenURL

  18. Group, V.A.D.o.D.T.M.o.P.-T.S.W:

    VA/DoD Clinical Practice Guideline for Management of Post-Traumatic Stress. 2010. OpenURL

  19. Resick MC, Chard KM:

    Cognitive processing therapy veteran/military version: Therapist and patient materials manual. 2008. OpenURL

  20. Resick PA, et al.: A randomized clinical trial to dismantle components of cognitive processing therapy for posttraumatic stress disorder in female victims of interpersonal violence.

    J Consult Clin Psychol 2008, 76(2):243-258. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  21. Resick PA, et al.: Long term outcomes of cognitive behavioral treatments for posttraumatic stress disorder among female rape survivors.

    J Consult Clin Psychol 2012, 80(2):210-212. OpenURL

  22. Schulz PM, et al.: The effectiveness of cognitive processing therapy for PTSD with refugees in a community setting.

    Cognitive and Behavioral Practice 2006, 13:322-331. Publisher Full Text OpenURL

  23. Resick PA, et al.: A comparison of cognitive processing therapy, prolonged exposure and a waiting condition for the treatment of posttraumatic stress disorder in female rape victims.

    J Consult Clin Psychol 2002, 70(4):867-879. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  24. Monson CM, et al.: Cognitive processing therapy for veterans with military-related posttraumatic stress disorder.

    J Consult Clin Psychol 2006, 74(5):898. PubMed Abstract | Publisher Full Text OpenURL

  25. Thomas TL, Kang HK, Dalager NA: Mortality among women Vietnam veterans, 1973–1987.

    Am J Epidemiol 1991, 134:973-980. PubMed Abstract | Publisher Full Text OpenURL

  26. Chard KM: Cognitive processing therapy for sexual abuse: a treatment outcome study.

    J Consult Clin Psychol 2005, 73:965-971. PubMed Abstract | Publisher Full Text OpenURL

  27. Galovski TE, et al.: Does cognitive-behavioral therapy for PTSD improve perceived health and sleep impairment?

    J Trauma Stress 2009, 22(3):197-204. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  28. Alvarez J, et al.: The comparative effectiveness of cognitive processing therapy for male veterans treated in a VHA posttraumatic stress disorder residential rehabilitation program.

    J Consult Clin Psychol 2011, 79(5):590-599. PubMed Abstract | Publisher Full Text OpenURL

  29. Resick PA, Schnicke MK: Cognitive processing therapy for sexual assault victims.

    J Consult Clin Psychol 1992, 60(5):748-756. PubMed Abstract | Publisher Full Text OpenURL

  30. Chard KM, et al.: A comparison of OEF and OIF veterans and vietnam veterans receiving cognitive processing therapy.

    J Trauma Stress 2010, 23(1):25-32. PubMed Abstract | Publisher Full Text OpenURL

  31. Chard KM, et al.: Dissemination and experience with cognitive processing therapy.

    JARD 2012, 49(5):667-678. OpenURL

  32. Ahrens J, Rexford L: Cognitive processing therapy for incarcerated adolescents with PTSD.

    Journal of Aggression, Maltreatment & Trauma 2002, 6(1):201-216. PubMed Abstract | Publisher Full Text OpenURL

  33. Schulz PM, et al.: The effectiveness of cognitive processing therapy for PTSD with refugees in a community setting.

    Cognitive and Behavioral Practice 2006, 13(4):322. Publisher Full Text OpenURL

  34. Chard KM: An evaluation of cognitive processing therapy for the treatment of posttraumatic stress disorder related to childhood sexual abuse.

    J Consult Clin Psychol 2005, 73(5):965-971. PubMed Abstract | Publisher Full Text OpenURL

  35. McHugh RK, Barlow DH: The dissemination and implementation of evidence-based psychological treatments. A review of current efforts.

    American Psychology 2010, 65(2):73-84. OpenURL

  36. Insel TR: Devising prevention and treatment strategies for the nation’s diverse populations with mental illness.

    Psychiatr Serv 2007, 58(3):395. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  37. Weissman MM, et al.: National survey of psychotherapy training in psychiatry, psychology, and social work.

    Arch Gen Psychiatry 2006, 63(8):925-934. PubMed Abstract | Publisher Full Text OpenURL

  38. Karlin BE, et al.: Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the veterans health administration.

    J Trauma Stress 2010, 23(6):663-673. PubMed Abstract | Publisher Full Text OpenURL

  39. Canadian Institutes of Health Research:

    Health Research Roadmap: Creating innovative research for better health and health care. 2013.

    http://www.cihr-irsc.gc.ca/e/documents/strat_plan_2009_e.pdf webcite

    OpenURL

  40. National Institute of Mental Health:

    Dissemination and Implementation Research Workshop: Harnessing Science to Maximize Health.

    Available from: http://www.nimh.nih.gov/research-priorities/scientific-meetings/2007/dissemination-and-implementation-research-workshop-harnessing-science-to-maximize-health.shtml webcite

    OpenURL

  41. Rakovshik SG, McManus F: Establishing evidence-based training in cognitive behavioral therapy: a review of current empirical findings and theoretical guidance.

    Clin Psychol Rev 2010, 30(5):496-516. PubMed Abstract | Publisher Full Text OpenURL

  42. Beidas RS, Kendall PC: Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective.

    Clin Psychol Sci Pract 2010, 17(1):1. Publisher Full Text OpenURL

  43. Herschell AD, et al.: The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations.

    Clin Psychol Rev 2010, 30(4):448-466. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  44. Shaw BF, et al.: Therapist competence ratings in relation to clinical outcome in cognitive therapy of depression.

    J Consult Clin Psychol 1999, 67(6):837-846. PubMed Abstract | Publisher Full Text OpenURL

  45. Hogue A, et al.: Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems.

    J Consult Clin Psychol 2008, 76(4):544-555. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  46. DeRubeis RJ, et al.: Medications versus cognitive behavior therapy for severely depressed outpatients: mega-analysis of four randomized comparisons.

    Am J Psychiatry 1999, 156(7):1007-1013. PubMed Abstract | Publisher Full Text OpenURL

  47. Strunk DR, Brotman MA, DeRubeis RJ: The process of change in cognitive therapy for depression: predictors of early inter-session symptom gains.

    Behav Res Ther 2010, 48(7):599-606. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  48. Strunk DR, et al.: Therapist competence in cognitive therapy for depression: predicting subsequent symptom change.

    J Consult Clin Psychol 2010, 78(3):429-437. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  49. Schoenwald SK, et al.: Therapist adherence and organizational effects on change in youth behavior problems one year after multisystemic therapy.

    Adm Policy Ment Health 2008, 35(5):379-394. PubMed Abstract | Publisher Full Text OpenURL

  50. Decker SE, Jameson MT, Naugle AE: Therapist training in empirically supported treatments: a review of evaluation methods for short- and long-term outcomes.

    Adm Policy Ment Heath 2011, 38:254-286. Publisher Full Text OpenURL

  51. Brosan L, Reynolds S, Moore RG: Self evaluation of cognitive therapy performance: Do therapists know how competent they are?

    Behav Cogn Psychother 2008, 36(5):581-587. Publisher Full Text OpenURL

  52. Miller WR, et al.: A randomized trial of methods to help clinicians learn motivational interviewing.

    J Consult Clin Psychol 2004, 72(6):1050-1062. PubMed Abstract | Publisher Full Text OpenURL

  53. Ruzek JI, Rosen RC: Disseminating evidence-based treatments for PTSD in organizational settings: a high priority focus area.

    Behav Res Ther 2009, 47(11):980-989. PubMed Abstract | Publisher Full Text OpenURL

  54. Resick PA, Nishith, and M.C. Astin: Empirically based therapy for posttraumatic stress disorder. , in Effective Interventions in Mental Health: Cognitive Behavior Therapy Introduced. Edited by Bilsen HV. Wellington, New Zealand: Central Institute of Technology Press; 1999. OpenURL

  55. Resick PA, et al.: A randomized clinical trial to dismantle components of cognitive processing therapy for posttraumatic stress disorder in female victims of interpersonal violence.

    J Consult Clin Psychol 2008, 76(2):243. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  56. Resick PA, et al.: A comparison of cognitive-processing therapy with prolonged exposure and a waiting condition for the treatment of chronic posttraumatic stress disorder in female rape victims.

    J Consult Clin Psychol 2002, 70(4):867. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  57. Vallis TM, Shaw BF, Dobson K: The cognitive therapy scale: psychometric properties.

    J Consult Clin Psychol 1986, 54:381-385. PubMed Abstract | Publisher Full Text OpenURL

  58. Stirman S, et al.: Development of a framework and coding system for modifications made to evidence-based programs and interventions.

    Implement Sci 2013, 8:65. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  59. Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA: Psychometric properties of the PTSD checklist (PCL).

    Behavioral Research & Therapy 1996, 34:669-673. PubMed Abstract | Publisher Full Text OpenURL

  60. Lambert MJ, et al.: The reliability and validity of the outcome questionnaire.

    Clin Psychol Psychother 1996, 3:249-258. Publisher Full Text OpenURL

  61. Ware JE, Kosinski M, Keller SD: A 12-item short-form health survey: construction of scales and preliminary tests of reliability and validity.

    Med Care 1996, 34(3):220-233. PubMed Abstract | Publisher Full Text OpenURL

  62. Lehman WE, Greener JM, Simpson DD: Assessing organizational readiness for change.

    J Subst Abuse Treat 2002, 22(4):197-209. PubMed Abstract | Publisher Full Text OpenURL

  63. Hoagwood K, Schoenwald SK, Chapman JE:

    Dimensions of Organizational Readiness-Revised (DOOR-R). 2003.

    Unpublished instrument

    OpenURL

  64. Aarons G, et al.:

    Expanding the Domains of Attitudes Towards Evidence-Based Practice: The Evidence Based Practice Attitude Scale-50. 2010., 1 [Administration and policy in mental health and mental health services research] OpenURL

  65. Marsh DR, et al.: The power of positive deviance.

    BMJ 2004, 329(7475):1177. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  66. Stirman SW, et al.: Perspectives on training and consultation in a cognitive therapy implementation program: implications for implementation.

    Depression Research and Treatment 2012, 2012:11. OpenURL

  67. Damschroder LJ, et al.: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.

    Implement Sci 2009, 4:50. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  68. Cohen J: Statistical Power Analysis for the Behavioral Sciences (Second ed.). Hillsdale, NJ: Erlbaum; 1988. OpenURL

  69. Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials.

    Ann Intern Med 2010, 152(11):726-W.293. OpenURL

  70. Pope C, Ziebland S, Mays N: Qualitative research in health care. Analysing qualitative data.

    BMJ Research 2000, 320(7227):114-116. Publisher Full Text OpenURL

  71. Palinkas L, et al.: Mixed Method Designs in Implementation Research.

    Administration and Policy in Mental Health and Mental Health Services Research 2010, 38(1):44-53. OpenURL

  72. Spillane J, et al.: Mixing methods in randomized controlled trials (RCTs): validation, contextualization, triangulation, and control.

    Educ Assess Eval Account 2010, 22(1):5-28. Publisher Full Text OpenURL

  73. Proctor EK, et al.: Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges.

    Administration and Policy in Mental Health and Mental Health Services Research 2009, 36(1):24-35. PubMed Abstract | Publisher Full Text OpenURL

  74. Baer JS, et al.: An evaluation of workshop training in motivational interviewing for addiction and mental health clinicians.

    Drug Alcohol Depend 2004, 73(1):99-106. PubMed Abstract | Publisher Full Text OpenURL

  75. Simons AD, et al.: Training and dissemination of cognitive behavior therapy for depression in adults: a preliminary examination of therapist competence and client outcomes.

    J Consult Clin Psychol 2010, 78(5):751-756. PubMed Abstract | Publisher Full Text OpenURL