Email updates

Keep up to date with the latest news and content from Implementation Science and BioMed Central.

Journal App

google play app store
Open Access Study protocol

Testing use of payers to facilitate evidence-based practice adoption: protocol for a cluster-randomized trial

Todd Molfenter1*, Jee-Seon Kim1, Andrew Quanbeck1, Terry Patel-Porter2, Sandy Starr3 and Dennis McCarty4

Author Affiliations

1 Department of Industrial and Systems Engineering, University of Wisconsin-Madison, 1513 University Avenue, Madison, Wisconsin 53706, USA

2 Ohio Dept. of Alcohol & Drug Addiction Services, 30 W. Spring Street, 6th Floor, Columbus, Ohio 43215-2256, USA

3 Outcomes, and Research, Ohio Dept. of Alcohol & Drug Addiction Services, 30 W. Spring Street, 6th Floor, Columbus, Ohio 43215-2256, USA

4 Oregon Health Sciences University, 3181 S.W. Sam Jackson Park Road, Portland, Oregon 97239-3098, USA

For all author emails, please log on.

Implementation Science 2013, 8:50  doi:10.1186/1748-5908-8-50

The electronic version of this article is the complete one and can be found online at: http://www.implementationscience.com/content/8/1/50


Received:8 March 2013
Accepted:29 April 2013
Published:10 May 2013

© 2013 Molfenter et al.; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

More effective methods are needed to implement evidence-based findings into practice. The Advancing Recovery Framework offers a multi-level approach to evidence-based practice implementation by aligning purchasing and regulatory policies at the payer level with organizational change strategies at the organizational level.

Methods

The Advancing Recovery Buprenorphine Implementation Study is a cluster-randomized controlled trial designed to increase use of the evidence-based practice buprenorphine medication to treat opiate addiction. Ohio Alcohol, Drug Addiction, and Mental Health Services Boards (ADAMHS), who are payers, and their addiction treatment organizations were recruited for a trial to assess the effects of payer and treatment organization changes (using the Advancing Recovery Framework) versus treatment organization changes alone on the use of buprenorphine. A matched-pair randomization, based on county characteristics, was applied, resulting in seven county ADAMHS boards and twenty-five treatment organizations in each arm. Opioid dependent patients are nested within cluster (treatment organization), and treatment organization clusters are nested within ADAMHS county board. The primary outcome is the percentage of individuals with an opioid dependence diagnosis who use buprenorphine during the 24-month intervention period and the 12-month sustainability period. The trial is currently in the baseline data collection stage.

Discussion

Although addiction treatment providers are under increasing pressure to implement evidence-based practices that have been proven to improve patient outcomes, adoption of these practices lags, compared to other areas of healthcare. Reasons frequently cited for the slow adoption of EBPs in addiction treatment include, regulatory issues, staff, or client resistance and lack of resources. Yet the way addiction treatment is funded, the payer’s role—has not received a lot of attention in research on EBP adoption.

This research is unique because it investigates the role of payers in evidence-based practice implementation using a randomized controlled design instead of case examples. The testing of the Advancing Recovery Framework is designed to broaden the understanding of the impact payers have on evidence-based practice (EBP) adoption.

Trial registration

http://NCT01702142 webcite (ClinicalTrials.gov registry, USA)

Keywords:
Evidence-based practice implementation; Buprenorphine; Addiction treatment; Innovation

Background

The use of evidence-based clinical practices in health care is suboptimal [1,2]. The addiction treatment field adopts evidence-based practices (EBPs) at an even lower rate than general health care [3-5]. For example, use of medication-assisted therapy for addiction disorders is not standard practice, despite the evidence that medication improves clinical, societal and financial outcomes through increased abstinence rates [6], reduced relapse rates [7], and reduced criminal costs [8]. A survey of 354 private addiction treatment centers reported that 34 percent of programs provided medication for opioid dependence and 24% used medication for treatment of alcohol dependence [9]. Within treatment programs that do offer medication, only 40% of the treatment plans for opioid dependence included medication [10].

Buprenorphine received Food and Drug Administration approval for treating opioid dependence in 2002 but remains underutilized. Less than one-fifth of patients with an opioid dependence diagnosis are offered buprenorphine [11,12]. Its adoption has been impeded by behavioral resistance, lack of or low reimbursement, insufficient access to personnel needed to prescribe the medication, licensure and contractual rules that prohibit the use of any pharmacotherapies in some settings, and insufficient organizational systems to encourage its use. Buprenorphine was selected as a candidate for EBP implementation and dissemination research because the broad set of behavioral and systemic barriers that limit its use incorporates factors that affect other EBPs with poor use rates.

Payer policy and organizational interventions can address barriers to the adoption of medication (and evidence-based practices more generally) [13,14]. Policy makers and local organizations collaborate to disseminate school reform, alternative energy use, and third-world agricultural practices [15,16]. Few trials in healthcare have tested the impact of aligning payer policy and treatment organization interventions to enhance implementation of treatment innovations [17]. However, from 2006 through 2010, NIATx (formerly the Network for the Improvement of Addiction Treatment), a process improvement research center based at the University of Wisconsin–Madison, conducted a demonstration project titled ‘Advancing Recovery: State and Treatment Organization Partnerships for Quality Addiction Care’. Funded by the Robert Wood Johnson Foundation, Advancing Recovery promoted the use of evidence-based practices such as medication-assisted treatment by building relationships between treatment organizations and the payers that fund their services. These partnerships made system changes to regulatory, clinical and administrative practices to increase the implementation of EBPs [18]. Advancing Recovery involved payer-treatment organization partnerships in 12 states, but was not a randomized controlled trial. The Advancing Recovery demonstration project did develop a framework for systems change to be tested in this randomized trial, calling it the Advancing Recovery Framework.

Objectives

The primary research question is to determine if the Advancing Recovery Framework increases the use of buprenorphine in addiction treatment organizations. The secondary research question is to determine if county characteristics (e.g., percentage of opioid abusers, existing addiction treatment medication availability, and payment policy) and organizational characteristics (e.g., organization size, access to physicians, number of nurses, percentage of clinicians with graduate degrees, and patient demographic characteristics) can moderate the delivery of buprenorphine.

Methods

Trial design

The participating ADAMHS boards have been randomized into two study conditions. In the Control Arm (Arm 1), participants will use the NIATx model for organizational change only. In Arm 2, participants will use the Advancing Recovery Framework that includes the payer intervention and NIATx models. Payers will not play a direct role in the control arm. The NIATx organizational change model (Figure 1) was chosen as the control condition because research exists that suggests organizational change models alone may be sufficient for EBP implementation [19,20]. A cluster randomized controlled design was used to study the effects that payers (or county ADAMHS boards) and treatment organizations have on buprenorphine adoption and also to control for ‘contamination’ across the treatment and control arms. A 1:1 allocation ratio of counties was utilized in the randomization.

thumbnailFigure 1. The NIATx organizational change model.

Participants

Ohio Alcohol, Drug Addiction and Mental Health Boards (ADAMHS) coordinate the public behavioral health system in Ohio. Each board covers one to five of Ohio’s 88 total counties and disperses Substance Abuse and Prevention Treatment (SAPT) block grant funds, as well as funds from local levies directed to addiction services. These Boards are responsible for 55% of addiction treatment services funding and often have responsibility for the quality and cost effectiveness of addiction treatment services within their jurisdictions [21]. The boards contract with provider organizations to deliver addiction and mental health treatment services. All 46 of Ohio’s ADAMHS boards were eligible for the study. The ADAMHS boards were recruited through an invitation from the Ohio Association of County Behavioral Health Authorities from May to September, 2012. The study team also actively recruited ADAMHS boards needed to meet the study’s matching criteria. A total of 14 boards representing 23 counties agreed to participate (Figure 2). Within each board jurisdiction, treatment agencies with 75 or more annual admission were invited to participate in the study. Fifty treatment agencies volunteered. Two to eight treatment agencies are participating per board.

thumbnailFigure 2. Consort Diagram (Enrollment & Intervention Phases).

Interventions

The Advancing Recovery Framework builds upon the multi-organizational learning collaborative method used in the NIATx 200 national trial to implement process improvement strategies in over 200 treatment organizations [22,23]. The Advancing Recovery Framework delivers technical assistance through learning sessions and coaching. Learning sessions are face-to-face meetings that occur once in the NIATx-only arm and three times in the Advancing Recovery arm to facilitate partnering between the ADAMHS boards and their treatment organizations. In these sessions, change teams from the ADAMHS board area (Arm 2 only) and each of the treatment organizations convene in face-to-face meetings to support and encourage each other. Outside experts offer advice on changes to make and ways to make them. Coaching assigns an expert in medication-assisted treatment adoption and process improvement to help the ADAMHS board and its treatment organizations make, sustain and spread buprenorphine adoption efforts. Coaches help the ADAMHS boards’ counties in Arm 2, and organizations in both arms think through key issues, broker relationships with other organizations, offer process improvement training, and suggest changes to make and how to do so. Participants in both arms receive coaching through monthly group teleconference calls, through e-mail, and at in-person learning sessions.

The payer intervention portion of the Advancing Recovery Framework has four phases: ‘Pre-Work for Leadership’ identifies the individuals who will lead and execute the change project; ‘Preparing the Change Leader & Team’ provides training in change management techniques; ‘Planning’ uses a five-lever policy analysis to catalog barriers to buprenorphine use and pilot tests potential tactics to remedy; and ‘Implementation’ takes tactics to apply from the planning phase, implements them, and focuses on sustaining the gains. During the learning sessions and monthly coach calls between the payer and treatment organizations, the payer has the opportunity to partner and collaborate with treatment organizations. Table 1 outlines the steps, tools, and training methods for each phase of the payer intervention model.

Table 1. Payer intervention model of the advancing recovery framework (applied in Arm 2 only)

The five levers described in the planning phase provide a mechanism for payers to identify systemic barriers to EBP use and promote change. The ‘customer impact analysis’ examines how the EBP benefits the customer, affects the number or type (i.e., demographic) of customers involved in treatment, and which customers are needed to advocate for the change. A ‘financing and purchasing analysis’ determines how financing, reimbursement and payment provide incentives for EBP use. The ‘regulatory and policy analysis’ uncovers regulatory barriers. The ‘operations analysis’ assesses the training and skill-building payers and treatment organizations need for EBP implementation. Finally, an ‘inter-organizational analysis’ identifies what other organizations need to be involved to remove regulatory or financing barriers.

Timeline

During project months 1 to 5, the study team will identify and recruit counties (through the Ohio Association of County Behavioral Health Authorities) and treatment organizations. In months 6 to 11, the study team will collect baseline data on buprenorphine use rates, conduct buprenorphine training workshops for treatment organizations, and update NIATx training materials to reflect the study conditions. In months 12 to 36, the study team will implement the study arm interventions and collect outcome data. During months 37 to 50, the study team will collect quantitative data on the sustainability of the changes. Data analysis, publication development, and dissemination of findings will occur at several points, but primarily during months 51 to 60.

Sample size

To determine sample size, we fit a linear mixed-effects model to the monthly results for the performance measure of percent opioid dependent patients receiving buprenorphine from each of the participating organizations and estimate the ‘payer-treatment organization’ effect. A mixed-effect model was selected instead of conventional regression models because, first, the mean outcomes for a given treatment organization are expected to be correlated from month to month. A mixed-effect model allows for auto-correlated error terms. Second, a number of unobservable treatment organization characteristics are likely to influence the key measures. Random effects at the treatment organization level can be employed within the mixed-effect model to capture the correlation introduced by such characteristics. Proper reflection of auto-correlation and random effects allows for more reliable estimates of the fixed effects, starting in months 6 to 11 (for baseline), then during months 12 to 36 (intervention stage), and through months 37 to 50 (follow-up stage). The power of the study design is determined by the anticipated standardized effect size based on effects experienced in the Advancing Recovery pilot project. The Advancing Recovery pilot data found that the payer-treatment organization intervention increased buprenorphine use rates from 6% to 43%, with an estimated standard deviation of 23.5% based on a sample size of 1,016. This yields an effect size of 0.935. Intraclass correlation (ICC) among counties affects the power of cluster-randomized trials [24]. An estimate of ICC is around 0.05. For this study, a dropout rate of 20% is being proposed, giving a total of 40 organizations available for analysis. With a total of 40 treatment organizations, each with an average of 91 patients, the study will achieve at least a power of 0.82 with a one-tailed type I error rate of 0.05. Power and sample size calculations were performed using Optimal Design Software by Raudenbush (Version 3.01) [25].

Randomization & implementation

Matched-pair blocks were created on the basis of the county characteristics of: percentage of opiate admissions; population covered; and ratio of opiate addiction treatment admissions to physicians licensed to prescribe buprenorphine [26]. The cut-off criteria used for matching, based on the median measures from the 2011 Ohio Department of Alcohol and Drug Addiction Services treatment organization dataset, were: 20% opiate admissions (above/below); 200,00 people covered (above/below); and 50 treatment admissions per physicians in the county registered to prescribe buprenorphine per annum (above/below). As a result, seven county board matched-pairs were identified. A random number generator was utilized within the seven county board pairs to assign organizations into one of the two study conditions. Study staff completed enrollment and received consent approval from the county boards as well as providers. No visible concealment was applied, and blinding of participants and researchers was considered infeasible due to logistical reasons.

Data analysis

Because the number of treatment organizations within an ADAMHS Board is small (n = 2 to 8) we intend to use an optimal design in which treatment arms are allocated following a multi-site cluster randomization procedure [27,28]. The initial exploratory analyses of both outcomes and covariates will assess standard summary statistics and graphical presentations. Scatter plots and correlation analyses identify possible associations between outcome and predictor variables. The analysis will represent the multi-site cluster randomized trial as a mixed-effects model; percentage of buprenorphine use by opioid dependent patients is nested within cluster (treatment organization); and clusters are nested within site (ADAMHS board). In this mixed-effects model, there will be random effects due to county board and organization; fixed effect due to study arm and time. For the percent of opioid admissions receiving buprenorphine, a logistic regression analysis will be performed. We will examine the Advancing Recovery effect using a separate model at each time point and in an omnibus growth curve model that includes the main effects of Advancing Recovery and Time as well as the interaction between Advancing Recovery and Time. The percent of opioid patients receiving buprenorphine will be measured monthly for each organization, and it is expected that these values will be correlated over time. Therefore, instead of assuming an independence (i.e., zero correlation) or a compound symmetry covariance structure (i.e., a constant correlation regardless of the proximity of measurement time points), we will allow errors to be first-order auto-correlated, denoted as AR (1). This can be done by allowing an additional parameter in the mixed-effects model that represents the correlation between the two most adjacent measures of Opioid Patients Receiving Buprenorphine. This correlation is reduced exponentially as the measures become further apart.

For the ADAMHS board and organizational covariates, at each time point we will assess moderation by including each potential moderator as an independent variable and as an interaction with the Advancing Recovery effect. Statistical significance of the interaction effect will indicate the presence of moderation. The estimate for the effect size of moderation, which is cross-level interaction, is between 0.17 and 0.19.

Ethics

The study received approval from the institutional review boards at the University of Wisconsin–Madison, Ohio Department of Health’s Human Subjects Institutional Review Board, and Oregon Health and Science University.

Trial status

The recruitment phase of the trial has just been completed and baseline data collection has begun.

Discussion

Conceptual multi-level models of EBP adoption are present in the literature and are based on correlates of EBP adoption discovered at the larger systems/environment, organizational, and individual levels [13,17,29]. These models identify which factors EBP implementation efforts should attempt to influence at the different levels in the healthcare system, but do not address how to change these factors. Accordingly, this research represents a movement from correlate identification to implementation model testing. This progression is described in McGovern’s ‘Implementation Research Flow’ paradigm, which explains how the purpose of implementation models is to positively influence the implementation and maintenance trajectories of the EBP adoption [30].

At present, the implementation research on how administrative practices at the payer level promote use of clinical practices at the treatment organization level primarily relies on case examples. The described research protocol has the unique aspect of randomizing by payer. The use of implementation research randomizing by county has been limited [31,32], and no instances of randomization at the state level were found. Research studying use of payer administrative practices to promote clinical evidence-based practice implementation should begin to use randomized trials to augment and validate findings found in payer case-examples.

This trial is taking lessons from past implementation research through case examples from the Advancing Recovery pilot [33-39], and packages them into an implementation model [18]. The research trial testing the Advancing Recovery Framework will build on several implementation research questions: What are the roles of payers with planning and stakeholder engagement in the provision of EBPs? What policy levers have an impact on EBP implementation and sustainability rate? What are the synergistic effects of having multiple levels focused on implementing a targeted EBP? What is the role of leadership at the payer and provider levels in the broader use of EBPs? What training and technical assistance offerings in the learning collaborative model are most helpful in a multi-level design?

Ohio was selected as a research site because it was considered to be a representative governmental healthcare environment to test this implementation model. A total of 34% of the states in the United States have county or region boards to distribute SAPT block grant funds. Each Ohio ADAMHS board represents a unique payer environment. The state has a mix of urban and rural counties. Within these counties, buprenorphine use varies from 7% to 58%. Limitations could be present due to the research environment. This trial addresses governmental payers securing healthcare services. Governmental healthcare is a considerable enterprise, where an effective framework for evidence-based practice implementation could prove to be extremely helpful. While this trial may not be generalizable to governmental payers for other community services, the Advancing Recovery framework could give these sectors interventions to consider in their efforts to improve services.

Conclusion

In conclusion, this trial will contribute to the evolving implementation science field over the next five years. Future results of this multi-level approach will be published as they become available.

Abbreviations

ADAMHS: Alcohol, Drug Addiction and Mental Health Services; EBP: Evidence-based practice; ICC: Intraclass correlation; NIATx: Network for the Improvement of Addiction Treatment; SAPT: Substance Abuse and Prevention Treatment.

Competing interests

The authors declare no competing interests.

Authors’ contributions

All authors made substantial contributions to the study design as well as the development and editing of the manuscript. All authors read and approved the final manuscript.

Acknowledgements

The study is funded by the National Institutes on Drug Abuse (NIDA) (R01 DA030431-01A1). NIDA has played no role in the study design or preparation of this manuscript.

References

  1. Baker TB, McFall RM, Shoham V: Current status and future prospects of clinical psychology: toward a scientifically principled approach to mental and behavioral health care.

    Psychol Sci Public Interest 2008, 9(2):67-103. OpenURL

  2. Bodenheimer T: The American health care system–the movement for improved quality in health care.

    N Engl J Med 1999, 340(6):488-492. PubMed Abstract | Publisher Full Text OpenURL

  3. Institute of Medicine: Bridging the gap between practice and research: forging partnerships with community based drug and alcohol treatment. Washington, DC: National Academy Press; 1998. OpenURL

  4. Willinbring ML, Kivlahan D, Kenny M, Grillo M, Hagedorn H, Postier A: Beliefs about evidence-based practices in addiction treatment: a survey of veterans administration program leaders.

    J Subst Abuse Treat 2004, 26(2):79-85. PubMed Abstract | Publisher Full Text OpenURL

  5. Garner B: Research on the diffusion of evidence-based treatments within substance abuse treatment: a systematic review.

    J Subst Abuse Treat 2009, 36(4):376-399. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  6. Moore TJ, Cohen MR, Furberg CD: Serious adverse drug events reported to the food and drug administration, 1998–2005.

    Arch Intern Med 2007, 167(16):1752. PubMed Abstract | Publisher Full Text OpenURL

  7. O’Brien CP: Recent developments in the pharmacotherapy of substance abuse.

    J Consult Clin Psychol 1996, 64(4):677-686. PubMed Abstract | Publisher Full Text OpenURL

  8. Harris AH, Gospodarevskaya E, Ritter AJ: A randomised trial of the cost effectiveness of buprenorphine as an alternative to methadone maintenance treatment for heroin dependence in a primary care setting.

    PharmacoEconomics 2005, 23(1):77-91. PubMed Abstract | Publisher Full Text OpenURL

  9. Knudsen HK, Abraham AJ, Roman PM: Adoption and implementation of medications in addiction treatment programs.

    J Addict Med 2011, 5:21-27. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  10. Knudsen HK, Roman PM: Financial factors and the implementation of medications for treating opioid use disorders.

    J Addict Med 2012, 6(4):280-286. PubMed Abstract | Publisher Full Text OpenURL

  11. Wallack S, Thomas C, Martin T, Chilingerian J, Reif S: Substance abuse treatment organizations as mediators of social policy: slowing the adoption of a congressionally approved medication.

    J Behav Health Serv Res 2010, 37(1):64-78. PubMed Abstract | Publisher Full Text OpenURL

  12. Stein BD, Gordon AJ, Sorbero M, Dick AW, Schuster J, Farmer C: The impact of buprenorphine on treatment of opioid dependence in a Medicaid population: recent service utilization trends in the use of buprenorphine and methadone.

    Drug Alcohol Depend 2012, 123(1–3):72-78. PubMed Abstract | Publisher Full Text OpenURL

  13. Ferlie EW, Shortell SH: Improving the quality of health care in the United Kingdom and the United States: a framework for change.

    Milbank Q 2001, 79(2):281-315. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  14. Grol R: Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction.

    JAMA 2001, 286(20):2578-2585. PubMed Abstract | Publisher Full Text OpenURL

  15. Slavin RE: Research on achievement outcomes of success for all.

    Phi Delta Kappan 2000, 82(1):38. OpenURL

  16. Rogers EM: Diffusion of innovations. New York: Free Press; 2003. OpenURL

  17. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations.

    Milbank Q 2004, 82(4):581-629. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  18. Schmidt LA, Rieckmann T, Abraham A, Molfenter T, Capoccia V, Roman P, Gustafson DH, McCarty D: Advancing Recovery: implementing evidence-based treatment for substance use disorders at the systems level.

    J Stud Alcohol Drugs 2012, 73(3):413-422. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  19. Simpson DD: A conceptual framework for transferring research to practice.

    J Subst Abuse Treat 2002, 22(4):171-182. PubMed Abstract | Publisher Full Text OpenURL

  20. Proctor EK: Leverage points for the implementation of evidence-based practice.

    Brief Treat Crisis Interv 2004, 4(3):227-242. Publisher Full Text OpenURL

  21. Mark TL, Levit KR, Coffey RM, McKusick DR, Harwood HJ, King EC, Bouchery E, Genuardi JS, Vandivort-Warren R, Buck JA, Ryan K: National expenditures for mental health services and substance abuse treatment: 1993-2003.Vol SAMHSA Publication No. SMA 07-4227. Rockville, MD: Substance Abuse and Mental Health ServicesAdministration; 2007. OpenURL

  22. Quanbeck AR, Gustafson DH, Ford JH II, Pulvermacher A, French MT, McConnell KJ, McCarty D: Disseminating quality improvement: study protocol for a large cluster randomized trial.

    Implement Sci 2011, 6(1):44. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  23. Gustafson DH, Quanbeck AR, Robinson JM, Ford JH II, Pulvermacher A, French MT, McConnell KJ, Batalden PB, Hoffman KA, McCarty D: Which elements of improvement collaboratives are most effective? a cluster-randomized trial.

    Addiction

    in press

    OpenURL

  24. Hsieh FY, Lavori PW, Cohen HJ, Feussner JR: An overview of variance inflation factors for sample-size calculation.

    Eval Health Prof 2003, 26(3):239-257. PubMed Abstract | Publisher Full Text OpenURL

  25. Raudenbush SW, Spybrook J, Congdon R, Liu X, Martinez A, Bloom H, Hill C:

    Optimal design software for multi-level and longitudinal research (Version 3.0).

    [http://www.wtgrantfoundation.org/resources/consultation-service-and-optimal-design webcite]

    OpenURL

  26. Knudsen HK, Ducharme LJ, Roman PM: Early adoption of buprenorphine in substance abuse treatment centers: data from the private and public sectors.

    J Subst Abuse Treat 2006, 30(4):363-373. PubMed Abstract | Publisher Full Text OpenURL

  27. Spybrook J, Raudenbush SW, Liu X, Congdon R, Martinez A:

    Building capacity to evaluate group-level interventions.

    http://sitemaker.umich.edu/group-based webcite

    OpenURL

  28. Raudenbush SW, Liu X: Statistical power and optimal design for multisite randomized trials.

    Psychol Methods 2000, 5(2):199-213. PubMed Abstract | Publisher Full Text OpenURL

  29. Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors.

    Adm Policy Ment Health 2011, 38(1):4-23. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  30. McGovern MP, Saunders EC, Kim E: Substance abuse treatment implementation research.

    J Subst Abuse Treat 2013, 44(1):1-3. PubMed Abstract | Publisher Full Text OpenURL

  31. Gilbert CM, O’Donnell E: The hospital library’s role in recruitment and retention of medical staff.

    J Hosp Librariansh 2004, 4(4):7-30. Publisher Full Text OpenURL

  32. Chamberlain P, Brown CH, Saldana L, Reid J, Wang W, Marsenich L, Sosna T, Padgett C, Bouwman G: Engaging and recruiting counties in an experiment on implementing evidence-based practice in California.

    Adm Policy Ment Health 2008, 35(4):250-260. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  33. Magnabosco JL: Innovations in mental health services implementation: a report on state-level data from the National Evidence-Based Practices Project.

    Implement Sci 2006, 1:13. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  34. Bruns EJ, Hoagwood KE, Rivard JC, Wotring J, Marsenich L, Carter B: State implementation of evidence-based practice for youths, Part II: recommendations for research and policy.

    J Am Acad Child Adolesc Psychiatry 2008, 47:499-504. PubMed Abstract | Publisher Full Text OpenURL

  35. Shea MA, Callis BP, Cassidy-Stewart H, Cranston K, Tomoyasu N: Diffusion of effective HIV prevention interventions--lessons from Maryland and Massachusetts.

    AIDS Educ Prev 2006, 18:96-107. PubMed Abstract | Publisher Full Text OpenURL

  36. Raghavan R, Bright CL, Shadoin A: Toward a policy ecology of implementation of evidence-based practices in public mental health settings.

    Implement Sci 2008, 3(1):26. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  37. Ganju V: Implementation of evidence-based practices in state mental health systems.

    Schizophr Bull 2003, 29:125-131. PubMed Abstract | Publisher Full Text OpenURL

  38. Rapp CA, Bond GR, Becker DR, Carpinello SE, Nikkel RE, Gintoli G: The role of state mental health authorities in promoting improved client outcomes through evidence-based practice.

    Community Ment Health J 2005, 41:347-363. PubMed Abstract | Publisher Full Text OpenURL

  39. Isett KR, Burnam MA, Coleman-Beattie B, Hyde PS, Morrissey JP, Magnabosco JL, Rapp C, Ganju V, Goldman HH: The role of state mental health authorities in managing change for the implementation of evidence-based practices.

    Community Ment Health J 2008, 44:195-211. PubMed Abstract | Publisher Full Text OpenURL