Skip to main content
Download PDF
Download ePub

Factors influencing implementation dose and fidelity thereof and related student outcomes of an evidence-based national HIV prevention program

Implementation Science volume 10, Article number: 44 (2015) Cite this article

Abstract

Background

Teachers’ implementation of evidence-based prevention programs in schools is inconsistent. Using data gathered from the national implementation among grade six students in The Bahamas of an evidence-based HIV intervention [Focus on Youth in the Caribbean (FOYC)], this study examines differences in the degree of implementation (“dose”) and adherence to the core activities (“fidelity of implementation”) by teachers according to theoretically and historically relevant teachers’ characteristics, attitudes, and experiences pre-intervention and post-intervention. The relationship of implementation dose and implementation fidelity is assessed according to student outcomes.

Methods

Beginning in 2008, the Bahamian Ministry of Education (MOE) included FOYC in the grade six curriculum nationwide. Consistent with standard practice, teachers were offered MOE training workshops in FOYC prior to delivery. The MOE conducted an anonymous curricular assessment among the grade six students at the beginning and end of the school year. Teachers agreeing to participate in the research component were asked to complete a pre-implementation and post-implementation assessment of attitudes and prior experiences.

Results

Teachers taught 15.6 out of 30 core activities, 24 out of the 46 total activities, and 4.6 out of 8 sessions on average. Three teachers’ implementation groups were identified: 1) High Implementation Group (31.7% of the teachers), characterized by high levels of implementation dose and fidelity of implementation; 2) Moderate Implementation Group (52.8%), showing moderate levels of implementation dose but high levels of fidelity of implementation; and 3) Low Implementation Group (15.6%), with low levels of implementation dose and fidelity of implementation. Low Implementation Group teachers compared to teachers in the two higher performing groups had less training in interactive teaching, limited prior exposure to the FOYC curriculum, incomplete attendance at FOYC training workshops, and low levels of comfort in teaching FOYC lessons. Students taught by teachers in the Low Implementation Group demonstrated poorer outcomes relevant to the four student outcomes (HIV/AIDS knowledge, preventive reproductive health skills, self-efficacy, and intention to use protection if they were to have sex).

Conclusions

Both implementation dose and implementation fidelity are related to student outcomes. Teachers at risk for limited implementation can be identified pre-intervention, thus opening the possibility for focused pre-intervention training.

Peer Review reports

Introduction

Evidence-based prevention and health promotion interventions offer great promise for improving the well-being of a population. Accordingly, much progress has been made in developing and evaluating health-promoting interventions addressing a range of disorders yielding a substantial portfolio of effective public-health programs. By contrast, less progress has been made in assuring their introduction and sustained use in an effective manner in the community [1]. The past decades have witnessed the development of robust HIV prevention intervention portfolios as illustrated by the Center for Disease Control’s identification of 44 “best evidence” HIV prevention programs through its “Prevention Synthesis Project” [2]. Despite significant declines in AIDS-related deaths and lower rates of new HIV infection, the potential impact of evidence-based behavioral prevention programs has been muted because of the difficulties in sustaining effective delivery of these interventions in real-life settings as opposed to effectiveness trials. For interventions addressing children and adolescents, school-based programs are especially appealing given their wide reach to a “captive” audience. Initiating implementation of effective programs within a school system is a critically important and complex first step and sustaining these efforts even more so, and sustaining them with fidelity has proven to be a daunting task [3-6].

Sustainability of evidence-based interventions has been identified as a major research priority area for many years, producing some instructive findings [7,8]. The extant literature indicates that sustained implementation of evidence-based behavioral interventions is low [6]. Factors reported to influence sustainability of implementation within a school system include the following: program’s acceptability and perceived relative importance to school administrators and teachers, perceived effectiveness of the intervention, feasibility to implement on an ongoing basis, and flexibility and adaptability [9-13].

Studies assessing fidelity of implementation of effective programs (using a range of definitions, criteria, and cutoffs) likewise find “low” levels of fidelity [14]. Factors associated with fidelity of implementation include ease of integration of the program into their existing curricular approach/routine [8,15] and teacher training in the curriculum [16-18]. Teacher characteristics found to be related to fidelity include the following: teacher’s positive attitude towards prevention programs [19,20], shorter duration of time as a teacher [6], and confidence in one’s ability to teach interactive methods [6]. Teacher perceptions that are important include the following: the intervention reflects the values of and is “from” their community [21], students’ are engaged in the intervention curriculum, and students are positively impacted by the intervention [15,22].

There is evidence that the more similar the intervention delivery is to that which was demonstrated to be effective, the more likely it is that positive outcomes will be realized. A recent systematic review reveals that a majority of studies have found that higher implementation dose and/or implementation fidelity are associated with better program outcomes [23]. Blakely and colleagues conducted a comprehensive examination of fidelity and effectiveness assessing seven nationally disseminated education and criminal justice projects and found that implementations conducted with high-fidelity were more effective than low-fidelity implementations [24]. Other studies have found that suboptimal quality delivery of evidence-based programs can result in minimal or null effects [14]. An evaluation of 14 school-based anti-bullying programs found that the majority of programs yielded non-significant outcomes on measures of self-reported victimization and bullying [25]. Derzon et al. assessed intervention effectiveness of drug prevention programs and found no significant effect on drug use among program participants [26]. These studies suggest that inadequate program delivery may overshadow the potential impact of prevention programs [25,26]. However, Blakely’s research noted above [24] did suggest that if adaptation of the curriculum occurs in the form of local additions to the program, effectiveness appeared to be increased or at least not decreased, supporting the view that some adaptation may be important in enabling appropriate fit of the intervention in a new setting.

The literature systematically identifying implementation patterns by teachers of effective HIV prevention programs in school settings is sparse, but some relevant studies do exist. Klingner et al. categorized teachers who participated in the implementation of four research-based programs into three implementation groups [27], and Shin et al. identified five teachers’ delivery profiles based on teacher engagement and delivery techniques [28]. While informative, these studies are limited by the qualitative nature of data, the modest sample sizes, and the non-systematic methods for identification of the implementation clusters. The current study seeks to fill this research gap by identifying teachers’ implementation patterns using cluster analysis and relating implementation patterns to student outcomes.

Understanding whether and why teachers did and did not implement an intervention as it was designed is important. Critical to improving outcomes is the ability to identify teachers who are at increased risk of not implementing an intervention in a manner likely to bring greatest benefit before teaching the curriculum. Therefore, in the present analyses, we sought to address four questions relevant to implementation of an effective HIV risk prevention program. First, do teachers vary in their degree of implementation of an HIV risk reduction intervention in terms of implementation dose (e.g., how much of the intervention do they deliver) and integrity of implementation (e.g., do they adhere to the activities and presentations as described in the intervention manual)? Second, are groups of teachers who vary in their degree of implementation identifiable prior to intervention delivery? If so, what are the characteristics of these groups? Third, do these characteristics change in response to intervention delivery? Fourth, do student outcomes vary according to either or both implementation dose and fidelity of implementation?

Methods

Study site

From 2011 through 2013, all 80 government elementary schools in the Commonwealth of The Bahamas participated in national implementation of Focus on Youth in the Caribbean (FOYC), including the research component described herein. The 80 schools are located on 14 of the major islands constituting The Bahamas, where more than 98% of the population resides. Forty-six schools were located on the three most populated islands: 25 schools on Island #1 (I-1), 11 schools on Island #2 (I-2), and 10 schools on Island #3 (I-3). The remaining 34 schools were located on eleven small islands. The 80 participating schools housed 208 grade six classes and teachers: I-1 housed 122 (58.7% of the total) teachers; I-2, 27 (13%) teachers; I-3, 16 (7.7%) teachers; and the six small islands housed 43 (20.7%) teachers. The research protocol was approved by the Wayne State University Human Investigation Committee and the Institutional Review Board of the Bahamian Princess Margaret Hospital, Public Hospitals Authority.

HIV prevention program

FOYC was adapted from Focus on Youth (FOY), a ten-session sexual risk reduction program targeting mid-adolescents. FOY and Informed Parents and Children Together (CImPACT) are currently part of the Centers for Disease Control and Prevention’s “Diffusion of Effective Behavioral Interventions (DEBI)” Portfolio. FOYC and FOY are based on a social cognitive model, the Protection Motivation Theory [29]. The intervention includes games, interactive discussions, stories, and exercises to reinforce main messages and increase knowledge and skills regarding sexual risk avoidance, effective communication, negotiation, and condom use [30]. Multiple exercises in communication and a decision-making model is applied in most of the sessions illustrating the steps that should guide a child in selecting a course of action. Intervention evaluations showed that the intervention significantly increased Bahamian youth’s HIV/AIDS knowledge, perceptions of their ability to use condoms, and condom use intention [31,32] with evidence of increased condom use [33].

Teacher training

The Bahamian Ministry of Education (MOE) launched 11 teacher training workshops following the protocol used by the DEBI program to train future interventionists in the delivery of FOY including two 4-day-, eight 2-day-, and one 1-day-long workshops. Duration was based on available time in the MOE workshop training schedules. Regardless of the length of the workshop, the teacher training covered the following: 1) review of the need for HIV prevention in The Bahamas, 2) overview of FOYC including the research showing its effectiveness, 3) a walk-through of each of the sessions of FOYC with participation and “role-play” of the core activities considered to be critical to the success of FOYC, and 4) a didactic question-and-answer period regarding contraception and condom use. The longer workshops (4 days compared to 2 days compared to 1 day) provided a more detailed curriculum experience, longer didactic session, and more teacher involvement in role-plays and discussions. All 208 teachers (regardless of attendance at a workshop) were given a copy of the FOYC teacher training manual.

Nearly half of the teachers (49.2%) attended some or all of a FOYC training workshop in 2011/2012, 30.6% completed the workshop, and 23 (11%) teachers reported that they had attended a FOYC workshop before 2010 when they participated in the original FOYC intervention effectiveness trial. Overall, 60.2% of the teachers received training supporting their delivery of the FOYC curriculum, with the remainder receiving no training.

Measures

Implementation dose and fidelity of implementation

To assess implementation, all teachers were asked to complete a Teacher Implementation Checklist specific for each of the eight sessions of FOYC after they had taught the session. The checklist includes all 46 activities in the FOYC curriculum, 30 of which were identified by the developers as “core elements”. The teachers indicated which activities they had and had not taught in each session; implementation dose was defined as the number of the 30 core activities that were taught. For those core activities that they taught, the teachers recorded whether they had modified the format of the activity as outlined in the manual to determine fidelity of implementation. Teachers also recorded the total number of all activities (from among 46) that they taught. The number of sessions taught was calculated by summing the sessions in which a teacher taught one or more core activities; if teachers taught any core activity within a session, it counted as “teaching that session”.

Factors associated with implementation: teachers’ perceptions of the FOYC curriculum content, their prior training experience, and other factors shown to impact intervention implementation and fidelity

At pre- and post-intervention delivery, all participating teachers were asked to complete questionnaires assessing factors described in the prior research summarized in the “Introduction” section as influencing fidelity of intervention implementation. Both pre-implementation and post-implementation questionnaires assessed the following: teachers’ perceptions of the importance of prevention programs, HIV prevention, and FOYC intervention (very important, somewhat important, or not important) for grade six students in their community or schools; teacher’s confidence in teaching the FOYC intervention; teacher’s sense of “ownership” of the curriculum (i.e., perceiving it as a “Bahamian intervention”); and the relative importance of their time spent in teaching FOYC compared to the time spent teaching reading skills in grade six (less important, about the same, more important) [6,14,19,21].

The pre-implementation questionnaire also collected information on the teacher’s level of formal education, years as a teacher/guidance counselor, teacher’s attendance at FOYC training workshop, training in interactive teaching, and prior experience of teaching FOYC or other HIV prevention programs. The post-implementation questionnaire further assessed the role of competing priorities as a reason for not being able to complete delivering the FOYC curriculum and perceived student benefits from FOYC curriculum (very much, somewhat/not at all). Hypothetical relationships among these variables are discussed in a prior publication [34].

In bivariate analyses, responses were grouped into two or three collapsed categories due to low frequencies in some categories: years as a teacher/guidance counselor (1–5 years, 6–10 years, >10 years), training in interactive teaching (a lot/some, a little/none), perceptions of the importance of HIV prevention (very important and somewhat/not important), and confidence in teaching FOYC (very comfortable and somewhat/not comfortable). The Teacher Implementation Checklist described above asked teachers to record their degree of comfort in teaching the lesson (very comfortable, rather comfortable, or not comfortable) and how many students (most, some, few) appeared to be engaged in the lesson.

Student outcomes

An anonymous curricular assessment instrument (with identifying information only at the level of the school and teacher), adapted by the MOE from a version of the Bahamian Youth Health Risk Behavioral Inventory (BYHRBI) [30], was administered by the MOE to grade six students at the beginning of the grade six before receipt of FOYC and at the end of grade six. The instrument assessed HIV/AIDS knowledge and preventive reproductive health skills, as well as some perceptions, intentions, and self-reported behaviors. A 15-item scale including true and false statements was used to assess the level of HIV/AIDS knowledge. The internal consistency of the scale was high (Cronbach’s α = 0.85). Correct responses were scored 1 and incorrect 0, resulting in a summary score of 0 to 15 for each participant. Preventive reproductive health skills were assessed through an adaptation of the Condom-use Skills Checklist [35]. The validated scale includes true and false statements describing the steps of correct condom use from opening a condom pack for use to disposal after use. This six-item scale demonstrated adequate internal consistency (Cronbach’s α = 0.83). Correct responses were scored 1 and incorrect 0, resulting in a summary score of 0 to 6 for each participant. A three-item scale was used to assess self-efficacy for using pregnancy/STI prevention methods. All three items employed a yes/no response scale, with one point assigned for each “yes” response. Individual item scores were added to yield a summary score (range 0 to 3). The internal consistency of the scale was 0.81. Intention to use protection was measured using the question, “if you were to have sex in the next six months, how likely is it that you would use a condom (to prevent yourself from getting HIV)?” Youth rated the likelihood on a five-point Likert scale ranging from 1 (very unlikely) through 5 (very likely).

Analysis

Cluster analysis was used to identify teachers who were at high risk for inadequate implementation FOYC before they taught the course. Cluster analysis is a person-oriented approach which identifies homogenous groups within a sample of diverse individuals based on similarities across a set of attributes [36]. A two-step cluster analysis in SPSS 22 based on the two variables was used to define implementation dose (i.e., number of core activities completed) and fidelity of implementation (i.e., percentage of core activities being changed during the implementation). The optimal number of clusters was determined automatically by the statistical software based on Schwarz’s Bayesian information criterion, and the overall quality of the cluster solution was investigated based on the Silhouette measure of cohesion and separation and each variable’s level of importance to the cluster solution [37].

The differences in teachers’ education, total years as a teacher, attendance of training workshop, training in interactive training, prior experience of teaching FOYC or other HIV prevention programs, and perceptions of importance of HIV prevention or FOYC to grade six youth between the clusters were examined using chi-square test. The number of all activities completed and number of sessions taught in the classroom were compared across the clusters using ANOVA test.

Teachers’ post-implementation perceptions of the importance of HIV prevention, program ownership, student benefits from FOYC intervention, comfort level in teaching intervention curriculum, relative importance of the time spent in teaching FOYC and teaching reading skills, competing priorities, and student engagement were compared across the clusters, using chi-square test (for categorical variables) and ANOVA test (for continuous variables). Differences in teachers’ perceptions of the importance of HIV prevention, program ownership, and comfort level in teaching FOYC at pre-intervention and post-implementation were assessed to determine if these factors changed during the course of teaching the curriculum using McNemar’s test.

To assess differential intervention effects on student outcomes (HIV/AIDS knowledge, preventive reproductive health skills, self-efficacy, intention to use protection) by teachers’ implementation cluster membership, the change over time within each group of students, as well as the differences in change scores between the clusters, were compared using the Student t test and ANOVA with the Tukey honestly significant difference post hoc tests. The association of teachers’ patterns of implementation (High, Moderate, and Low Implementation Groups, with the Low Implementation Group serving as the reference group) with student outcomes was examined using mixed-effects modeling, adjusting for clustering effects of classroom and/or school. Independent variables included teachers’ implementation clusters (high/moderate/low implementers), student’s age, sex, and baseline student outcomes. School and class were included as random effect variables in the model. Regression coefficients were calculated for all variables. All the analyses except the two-step cluster analysis were performed using SAS 9.3 statistical software package (SAS Institute Inc., Cary, NC, USA).

Results

Implementation dose and fidelity of implementation of FOYC intervention

The number of core activities, “all activities” (including core and non-core activities), and sessions taught by 208 teachers are displayed in Figure 1. On average, teachers taught 15.6 core activities (SD = 8.2) from among the 30 core activities, 24 of the 46 total activities (“all activities”) (SD = 12.8), and 4.6 sessions (SD = 2.3) from among the total eight sessions. Only two (1%) teachers completed all core activities and covered all eight sessions while six (2.9%) teachers did not teach any activities in their classes. Sixteen (7.7%) teachers taught ≥28 core activities; 28 (13.5%) taught ≥40 core and non-core activities; 56 (27%) taught seven or eight sessions of FOYC curriculum. Overall, 38 (18.3%) teachers taught less than eight core activities (and less than two sessions). Teachers changed 3.1 core activities (SD = 3.7) on average. Nearly one third of teachers did not change any core activities while one third changed one to three core activities. Approximately 10% of teachers changed ≥10 core activities.

Figure 1
figure 1

Number of core activities, all activities, and sessions in the FOYC curriculum taught by 208 teachers.

Full size image

Teachers’ implementation groups

Results of the two-step cluster analysis indicated three distinct clusters or groups of teachers. The Silhouette measure of cohesion and separation was above 0.5, indicating good cluster quality [38]. The predictor importance indicated that both the number of core activities completed (i.e., implementation dose) and percentage of core activities being changed (i.e., fidelity of implementation) are very important for the clustering solution (score range: 0.9–1.0). The first cluster of teachers, which includes 31.7% of the teachers, is characterized by the highest probabilities of teaching core activities in the classroom and thus is labelled the “High Implementation Group”. Teachers in this cluster taught over 80% of core activities (25 out of 30 core activities) on average and changed only 14% of the core activities. The second cluster, the “Moderate implementation Group”, is the largest cluster (52.8% of the teachers) and is distinguished by moderate probabilities of teaching core activities when compared with the High implementation Group. Teachers in the Moderate Implementation Group taught less than half of core activities (12.3 out of 30 core activities) on average and changed 13% of core activities that they taught in the classroom, similar to the rate of change of activities in the High Implementation Group. The third cluster, the “Low Implementation Group”, is the smallest cluster (15.6%) and is characterized by a combination of the lowest probabilities of teaching core activities and higher probabilities of modifying core activities compared to the other two groups; these taught less than one third of core activities (8.6 out of 30 core activities) on average and changed three quarters of core activities that they taught in the classroom.

Teachers’ implementation group membership was also related to numbers of all activities completed in the classroom. Teachers in the High Implementation Group taught the greatest numbers of all activities, followed by teachers in the Moderate Implementation Group, with teachers in the Low Implementation Group having taught the lowest numbers of all activities (38.9 vs. 18.9 vs. 12.6, p < 0.001). Teachers in the High Implementation Group also taught more sessions than teachers in the Moderate and Low Implementation Groups (7.2 vs. 3.6 vs. 3.0, p < 0.001) (Table 1).

Table 1 Teacher patterns of implementation and pre-implementation characteristics, teaching and training experiences and attitudes towards HIV prevention
Full size table

Association of teachers’ implementation group membership with teachers’ characteristics and pre- and post-implementation perceptions

Table 1 shows the differences between the three groups of teachers in terms of pre-implementation characteristics, teaching and training experiences, and attitudes towards HIV prevention. Duration of experience as a teacher, training in interactive teaching, attendance of the FOYC training workshop, comfort level in teaching the FOYC lessons, and islands where the teachers worked significantly differed among the High, Moderate, and Low Implementation Groups. More teachers in the Moderate or Low Implementation Groups than teachers in the High Implementation Group had worked as a teacher or guidance counselor for over 10 years (66% vs. 58% vs. 44%, χ2 = 15.98, p < 0.01). In contrast, more teachers in the High Implementation Group had served as a teacher for 6 to 10 years. Higher proportions of teachers in the High and Moderate Implementation Groups compared to teachers in the Low Implementation Group received extensive training in interactive teaching (73% vs. 57% vs. 21%, χ2 = 16.10, p < 0.001), indicated high levels of comfort in teaching FOYC lessons (66% vs. 58% vs. 22%, χ2 = 10.43, p < 0.01), and attended all aspects of their training workshops (40% vs. 33% vs. 3%, χ2 = 4.92, p < 0.05). Although the workshops differed by length, teachers completing their assigned training did not differ significantly in terms of implementation dose or fidelity of implementation. More teachers in the High Implementation Group were teaching in Island #1 (the most populous Bahamian island and the island in which FOYC was originally adapted for The Bahamas and tested for efficacy) (73% vs. 58% vs. 45%, χ2 = 7.45, p < 0.05) while more teachers in the Low Implementation Group worked in the other islands constituting The Bahamas (the “Family Islands”). Teachers’ level of education, prior experiences of teaching FOYC or other HIV prevention programs, program ownership, and perceptions of importance of HIV prevention for grade six youth were not significantly different among the three implementation groups.

Of the eight perceptions assessed post-intervention, five differed significantly between the three implementation groups: perceived importance of FOYC, program priority, perceived student benefits, ownership of the FOYC curriculum, and comfort level in teaching core activities. More teachers in the High Implementation Group perceived that the FOYC intervention was very important for grade six students in their schools, their students benefited very much from the intervention, and the FOYC curriculum was a Bahamian intervention. More teachers in the Moderate and Low Implementation Groups than teachers in the High Implementation Group reported that they had other priorities than teaching FOYC in their daily work. Teachers in the High and Moderate Implementation Groups reported higher levels of comfort in teaching core activities in the FOYC curriculum than teachers in the Low Implementation Group. Teachers’ perceptions regarding the importance of HIV prevention for grade six youth in general and student engagement in FOYC lessons were not associated with teachers’ implementation group membership (Table 2).

Table 2 Teacher patterns of implementation and post-implementation perceptions, program ownership, perceived student benefit, and comfort level in teaching intervention curriculum
Full size table

Changes in teachers’ perceptions between pre-intervention and post-intervention

Almost all teachers (97%–98.8%) perceived that prevention programs or HIV prevention programs were very important for youth pre-intervention. Higher proportions of teachers in the High and Moderate Implementation Groups than teachers in the Low Implementation Group perceived that prevention programs were very important for youth post-intervention (100% vs. 98% vs 89%, χ2 = 8.54, p < 0.05) while the rates were comparable pre-intervention. The majority of teachers indicated that HIV prevention was very important for grade six youth in general or FOYC was very important for grade six youth in their school. Comparable at baseline, the proportion of teachers who post-intervention perceived the FOYC to be of great importance for grade six youth in their school was highest in the High Implementation Group, followed in order by the Moderate Implementation Group and the Low Implementation Group (97% vs. 79% vs 60%, χ2 = 17.20, p < 0.001). Within-group comparison indicated that the proportion of teachers who perceived the importance of FOYC in the Low Implementation Group was reduced from 90% pre-intervention to 60% post-intervention (χ2 = 4.73, p < 0.05). Over 80% of the teachers perceived that their time spent teaching FOYC is about the same or more important than time spent teaching reading skills in grade six; this perception did not differ significantly pre-intervention and post-intervention or across the three groups. Nearly 60% of the teachers perceived the FOYC curriculum as a Bahamian intervention (“ownership”). Higher proportions of teachers in the High and Moderate Implementation Groups than teachers in the Low Implementation Group perceived program ownership post-intervention (72% vs. 59% vs. 28%, χ2 = 14.25, p < 0.01) while the rates had been comparable pre-intervention. The proportion of teachers who perceived program ownership in the Low Implementation Group reduced from 59% pre-intervention to 28% post-intervention (χ2 = 3.99, p < 0.05) while it increased from 54% pre-intervention to 72% post-intervention for teachers in the High Implementation Group (χ2 = 3.90, p < 0.05). Comfort level in teaching FOYC increased from pre-intervention to post-intervention for teachers in all three groups. Comfort level in teaching FOYC for teachers in the Low Implementation Group was lower than that of teachers in the High Implementation Group post-intervention (2.9 vs. 2.8 vs. 2.7, F = 3.13, p < 0.05) while the rates were comparable pre-intervention.

Teachers’ implementation group membership and student outcomes

Table 3 presents the change in HIV/AIDS knowledge, preventive reproductive health skills, self-efficacy, and intention to use protection from baseline to follow-up among students according to the implementation group membership of their teacher. All student outcomes increased significantly among all three teacher implementation groups over the 12 months. At baseline, knowledge and reproductive health skills were higher among students whose teachers belonged to the High or Moderate Implementation Groups. At follow-up, knowledge and reproductive health skills were highest among students whose teachers belonged to the High Implementation Group, followed in order by students whose teachers were in the Moderate Implementation Group and students whose teachers were in the Low Implementation Group (knowledge: 10.5 vs. 10.0 vs. 9.2, F = 58.21, p < 0.001; skills: 4.0 vs. 3.7 vs. 3.6, F = 27.13, p < 0.001). At baseline, self-efficacy and intention to use protection were comparable across the three implementation groups. At follow-up, self-efficacy and intention were significantly higher among students whose teachers were in the High or Moderate Implementation Groups compared to students whose teachers were in the Low Implementation Group (self-efficacy: 1.3 vs. 1.3 vs. 1.1, F = 9.17, p < 0.001; intention: 3.3 vs. 3.2 vs. 2.9, F = 10.43, p < 0.001); students whose teachers were in the High or Moderate Implementation Groups demonstrated greater increases in self-efficacy and intention than students whose teachers were in the Low Implementation Group.

Table 3 Teacher patterns of implementation and student outcomes (n = 4,411)
Full size table

The results of the mixed-effects models indicate that teachers’ patterns of implementation were significantly related to improvement in all four student outcome measures. At follow-up, compared to students whose teachers belonged to the Low Implementation Group, students whose teachers were in the High and/or Moderate Implementation Groups demonstrated higher levels of HIV/AIDS knowledge, reproductive health skills, self-efficacy, and intention to use protection if they were to engage in sex after controlling for age, gender, baseline difference, and clustering effects of school and classroom. Older age was associated with improvement in condom use self-efficacy. Male gender was associated with increased reproductive health skills, self-efficacy, and intention to use protection. Classroom random effects were significant in all four models, indicating significant variation among classrooms with regard to students’ knowledge of HIV/AIDS, reproductive health skills, self-efficacy, and intention to use protection. School random effects were significant for self-efficacy only (Table 4). The mixed-effects models were rerun using the Moderate Implementation Group as the reference group. Compared to students whose teachers belonged to the Moderate Implementation Group, students whose teachers were in the High Implementation Group demonstrated higher levels of HIV/AIDS knowledge (β = 0.48, SE = 0.18, t = 2.64, p = 0.008) and reproductive health skills (β = 0.24, SE = 0.07, t = 3.46, p = 0.0005).

Table 4 Mixed-effects models assessing the association between teacher’s pattern of implementation and students’ outcomes
Full size table

Discussion

That teachers’ cluster into three identifiable groups according to their levels of implementation dose and implementation fidelity and that these groups are significantly associated with student performance outcomes are important findings. Nearly one third of the teachers were identified as high implementers. These teachers taught over 80% of core activities in their classroom and adhered to the format for 86% of the activities outlined in the manual. Low implementers, representing approximately one sixth of the teachers, taught less than one third of core activities and modified most of these taught. About half of the teachers were identified as moderate implementers; these teachers taught less than half of all core activities but they adhered to the format of most of the activities that they taught.

Teachers’ implementation group membership is associated with both prior teaching experience and intervention training experience. Teachers in the High Implementation Group were less likely to have been teaching for more than 10 years than teachers in the other two groups and were more likely to have received training in interactive teaching. These findings are consistent with previous research suggesting that fewer years of teaching experience and confidence in using interactive methods in intervention delivery are positively associated with fidelity of implementation [6,39].

Teachers’ full attendance in the FOYC training workshop is associated with High (and Moderate) Implementation Group membership. By contrast, no training or attending only part of the assigned workshop is associated with Low Implementation Group membership. Duration of training was not in and of itself the important factor. This observation is not consistent with previous research suggesting that longer training is associated with higher quality implementation [18]. Rather, full attendance was the critical component, suggesting that a workshop can be streamlined provided that the critical components of the intervention are retained in the training. This is important in long-term implementation, as multi-day training programs are expensive for government-supported educational systems and difficult for teachers to accommodate in their schedules.

Teachers’ High Implementation Group membership is associated with teachers’ perceptions of the importance of the FOYC intervention for grade six students, student benefits from the intervention, and program ownership at post-intervention, which are consistent with prior studies [5,14,19,21]. Pre-intervention perceptions regarding the importance of FOYC and program ownership did not differ by implementation group membership; pre-intervention, all three groups expressed endorsement of the importance of such educational material for students. Teachers in the High Implementation Group retained this perspective, while those in the Moderate Implementation Group experienced some erosion and those in the Low Implementation Group experienced a marked decrease of this endorsement. This finding suggests that something occurred in the latter two groups to undermine their confidence in the curriculum. Whether this decrease resulted in the sub-optimal implementation or their sub-optimal implementation resulted in their changed perception is not known but requires further exploration as potential remedies will vary. It is noteworthy that teachers in the High Implementation Group did not perceive a conflict in terms of other priorities competing with teaching FOYC. Whether this reflects actual differences in the scheduled courses or other obligations of teachers in the High and Low Implementation Groups or it reflects differing perspectives requires further evaluation as implications for intervening will differ.

Teachers in the Low Implementation Group exhibited several distinguishing characteristics pre-intervention compared to teachers in the two higher performing clusters: less training in interactive teaching, limited prior exposure to the FOYC curriculum, failure to attend the full training FOYC workshop, and low levels of comfort in teaching FOYC lessons. Lack of prior training experience and low levels of comfort in teaching intervention activities may increase teachers’ anxiety which in turn has an impact on implementation quality, particularly when prevention programs are perceived as competing with other priorities [40]. Missing part of a training workshop may contribute to low-quality implementation and/or may reflect disinterest or discomfort (or both) with the curriculum. These warning signs are easily identifiable pre-intervention or during early implementation and should alert the need for additional training or support. More investigation is needed to better understand the reasons for and resolutions to inadequacies in implementation of FOYC. In the interim, strategies could include increasing teachers’ awareness of adolescent risk behaviors and health consequences, engaging in discussions with teachers about the importance of HIV prevention programs in their schools/communities, reviewing the data with teachers regarding curriculum impact on student outcomes, and enhancing teachers’ competency in teaching the intervention curriculum.

Regardless of teachers’ implementation clusters, students’ HIV/AIDS knowledge, reproductive health skills, self-efficacy, and intention to use protection significantly increased from baseline to follow-up. These changes are similar to those found in previous randomized controlled trials [31,32]. Our study reveals that quality of implementation was significantly related to student outcomes (better implementation leads to better outcomes). Students whose teachers were in the High and/or Moderate Implementation Groups demonstrated greater improvements in three student outcomes (knowledge, skills, self-efficacy, and intention) compared to students whose teachers belonged to the Low Implementation Group. Students whose teachers were in the High Implementation Group demonstrated greater improvements in HIV/AIDS knowledge and reproductive health skills compared to students whose teachers belonged to the Moderate Implementation Group. These findings are consistent with previous research suggesting that implementation dose and implementation fidelity influences program outcomes [23].

There are several potential limitations in this study. First, our findings are based on teachers’ self-reports of their extent of implementation of the FOYC intervention. It is possible that teachers over-reported their level of implementation and provided responses that they thought would be more appropriate. In the current study, trained observers independently observed and assessed approximately 20% of each teacher’s classes. The teacher and observer reports on activities covered in these sessions were compared to determine the level of agreement; in general, we found that the observer-teacher agreement was high (over 80%), indicating that teachers’ self-reports of their implementation of the intervention curriculum in their classrooms are reliable. Second, school-level factors such as support by the principal and school administrator’s perception of importance of HIV intervention were only collected from about half of the participating schools. Thus, these data were not included in the present analysis.

Our study adds to the sparse but emerging literature on the implementation of evidence-based interventions in school settings. Findings regarding inconsistent implementation, three teachers’ implementation groups, and differences in teacher characteristics, teaching, and training experiences and pre- and post-implementation perceptions by implementation groups provide a greater understanding of the barriers and facilitators impacting large-scale implementation of effective intervention programs in schools. The results suggest that additional training or assistance should be provided to those teachers who received minimum training in interactive teaching and in teaching the intervention curriculum, who were not confident in teaching intervention curriculum or held less favorable attitudes towards intervention programs at pre-intervention. Finally, while the questions asked pre-intervention clearly identify at-risk teachers, additional information is needed to identify remedies to the sub-optimal teaching performance. Findings from this study can help interventionists or health practitioners develop better approaches to promote the implementation of effective HIV prevention interventions in schools.

References

  1. Spoth R, Rohrbach LA, Greenberg M, Leaf P, Brown CH, Fagan A, et al. Addressing core challenges for the next generation of type 2 translation research and systems: the translation science to population impact (TSci Impact) framework. Prev Sci. 2013;14(4):319–51.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Center for Disease Control and Prevention (CDC). Compendium of evidence-based HIV behavioral interventions. 2011. http://www.cdc.gov/hiv/topics/research/prs/compendium-evidence-based-interventions.htm.

  3. Gottfredson DC, Gottfredson GD. Quality of school-based prevention programs: results from a national survey. J Res Crime Delinquency. 2002;39:3–35.

    Article  Google Scholar 

  4. Lyles CM, Kay LS, Crepaz N, Herbst JH, Passin WF, Kim AS, et al. Best-evidence interventions: findings from a systematic review of HIV behavioral interventions for US populations at high risk, 2000–2004. Am J Public Health. 2007;97(1):133–43.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Ringwalt CL, Ennett S, Johnson R, Rohrbach LA, Simons-Rudolph A, Vincus A, et al. Factors associated with fidelity to substance use prevention curriculum guides in the nation’s middle schools. Health Educ Behav. 2003;30(3):375–91.

    Article  PubMed  Google Scholar 

  6. Rohrbach LA, Graham JW, Hansen WB. Diffusion of a school-based substance abuse prevention program: predictors of program implementation. Prev Med. 1993;22(2):237–60.

    Article  CAS  PubMed  Google Scholar 

  7. Adelman HS, Taylor L. On sustainability of project innovations as systemic change. J Educ Psychol Consultation. 2003;14(1):1–25.

    Article  Google Scholar 

  8. Gersten R, Chard D, Baker S. Factors enhancing sustained use of research-based instructional practices. J Learn Disabil. 2000;33(5):445–57.

    Article  CAS  PubMed  Google Scholar 

  9. Boardman AG, Argüelles ME, Vaughn S, Hughes MT, Klingner J. Special education teachers’ views of research-based practices. J Special Educ. 2005;39(3):168–80.

    Article  Google Scholar 

  10. Han SS, Weiss B. Sustainability of teacher implementation of school-based mental health programs. J Abnorm Child Psychol. 2005;33(6):665–79.

    Article  PubMed  Google Scholar 

  11. Kincaid D, Childs K, Blase KA, Wallace F. Identifying barriers and facilitators in implementing schoolwide positive behavior support. J Positive Behav Interv. 2007;9(3):174–84.

    Article  Google Scholar 

  12. McIntosh K, Horner RH, Sugai G. Sustainability of systems-level evidence-based practices in schools: current knowledge and future directions. In: Sailor W, Dunlap G, Sugai G, Horner RH, editors. Handbook of positive behavior support. New York: Springer; 2009. p. 327–52.

    Chapter  Google Scholar 

  13. McIntosh K, Predy LK, Upreti G, Hume AE, Turri MG, Mathews S. Perceptions of contextual features related to implementation and sustainability of school-wide positive behavior support. J Positive Behav Interv. 2013. doi:10.1177/1098300712470723

  14. Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18(2):237–56.

    Article  PubMed  Google Scholar 

  15. McIntosh K, Mercer SH, Humes AE, Frank JL, Turri MG, Matthews S. Factors related to sustained implementation of school-wide positive behavior support. Except Child. 2013;79(3):293–311.

    Google Scholar 

  16. Basen-Engquist K, O’Hara-Tompkins N, Lovato CY, Lewis MJ, Parcel GS, Gingiss P. The effect of two types of teacher training on implementation of Smart Choices: a tobacco prevention curriculum. J School Health. 1994;64(8):334–9.

    Article  CAS  PubMed  Google Scholar 

  17. Dusenbury L, Falco M. Eleven components of effective drug abuse prevention curricula. J School Health. 1995;65(10):420–5.

    Article  CAS  PubMed  Google Scholar 

  18. Perry CL, Murray DM, Griffin G. Evaluating the statewide dissemination of smoking prevention curricula: factors in teacher compliance. J School Health. 1990;60:501–4.

    Article  CAS  PubMed  Google Scholar 

  19. Beets MW, Flay BR, Vuchinich S, Acock AC, Li K-K, Allred C. School climate and teachers’ beliefs and attitudes associated with implementation of the positive action program: a diffusion of innovations model. Prev Sci. 2008;9(4):264–75.

    Article  PubMed  Google Scholar 

  20. Parcel GS, O’Hara-Tompkins NM, Harrist RB, Basen-Engquist KM, McCormick LK, Gottlieb NH, et al. Diffusion of an effective tobacco prevention program. Part II: evaluation of the adoption phase Health Educ Res. 1995;10(3):297–307.

    CAS  PubMed  Google Scholar 

  21. Draper CE, Nemutandani SM, Grimsrud AT, Rudolph M, Kolbe-Alexander TL, de Kock L, et al. Qualitative evaluation of a physical activity-based chronic disease prevention program in a low-income, rural South African setting. Rural Remote Health. 2010;10(3):1467.

    PubMed  Google Scholar 

  22. Mihalic SF, Fagan AA, Argamaso S. Implementing the LifeSkills Training drug prevention program: factors related to implementation fidelity. Implementation Science. 2008;3(5):1–16.

    Google Scholar 

  23. Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41(3–4):327–50.

    Article  PubMed  Google Scholar 

  24. Blakely CH, Mayer JP, Gottschalk RG, Schmitt N, Davidson WS, Roitman DB, et al. The fidelity-adaptation debate: implications for the implementation of public sector social programs. Am J Community Psychol. 1987;15(3):253–68.

    Article  Google Scholar 

  25. Smith DJ, Schneider BH, Smith PK, Ananiadou K. The effectiveness of whole-school antibullying programs: a synthesis of evaluation research. School Psychol Rev. 2004;33(4):547–60.

    Google Scholar 

  26. Derzon JH, Sale E, Springer JF, Brounstein P. Estimating intervention effectiveness: synthetic projection of field evaluation results. J Prim Prev. 2005;26(4):321–43.

    Article  PubMed  Google Scholar 

  27. Klingner JK, Ahwee S, Pilonieta P, Menendez R. Barriers and facilitators in scaling up research-based practices. Except Child. 2003;69(4):411–29.

    Google Scholar 

  28. Shin Y, Miller-Day M, Pettigrew J, Hecht ML, Krieger JL. Typology of delivery quality: latent profile analysis of teacher engagement and delivery techniques in a school-based prevention intervention, keepin’ it REAL curriculum. Health Educ Res. 2014;29(6):897–905.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Rodgers R. Cognitive and physiological processes in fear appeals and attitude change: a revised theory of protection motivation. In: Caaioppi T, Petty R, editors. Social psychology. New York, NY: Guilford Press; 1983. p. 153–76.

    Google Scholar 

  30. Deveaux L, Stanton B, Lunn S, Cottrell L, Yu S, Brathwaite N, et al. Reduction in human immunodeficiency virus risk among youth in developing countries. Arch Pediatr Adolesc Med. 2007;161(12):1130–9.

    Article  PubMed  Google Scholar 

  31. Gong J, Stanton B, Lunn S, Deveaux L, Li X, Marshall S, et al. Effects through 24 months of an HIV/AIDS prevention intervention program based on protection motivation theory among preadolescents in the Bahamas. Pediatrics. 2009;123:e917–28.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Dinaj-Koci V, Brathwaite N, Deveaux L, Lunn S, Cottrell L, Harris C, et al. When things are not as they appear: assessing the adequacy of cluster randomization when outcome events are rare at baseline. AIDS Res Treat. 2012. doi:10.1155/2012/806384

  33. Chen X, Lunn S, Deveaux L, Li X, Brathwaite N, Cottrell L, et al. A cluster randomized controlled trial of an adolescent HIV prevention program among Bahamian youth: effect at 12 months post-intervention. AIDS Behav. 2009;13:499–508.

    Article  PubMed  Google Scholar 

  34. Wang B, Deveaux L, Knowles V, Koci V, Rolle G, Lunn S, et al. Fidelity of implementation of an evidence-based HIV prevention program among Bahamian sixth grade students. Prev Sci. 2015;16:110–21.

    Article  PubMed  Google Scholar 

  35. Stanton B, Deveaux L, Lunn S, Yu S, Brathwaite N, Li X, et al. Condom-use skills checklist: a proxy for assessing condom-use knowledge and skills when direct observation is not possible. J Health Popul Nutr. 2009;27:406–13.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Rothrauff TC, Eby LT. Substance abuse counselors’ implementation of tobacco cessation guidelines. J Psychoactive Drugs. 2011;43(1):6–13.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Sinyor M, Schaffer A, Streiner DL. Characterizing suicide in Toronto: an observational study and cluster analysis. Can J Psychiatry. 2014;59(1):26–33.

    PubMed  PubMed Central  Google Scholar 

  38. Swaminathan S, Thomas T, Yusuf S, Vaz M. Clustering of diet, physical activity and overweight in parents and offspring in South India. Eur J Clin Nutr. 2013;67:128–34.

    Article  CAS  PubMed  Google Scholar 

  39. Ennett ST, Ringwalt CL, Thorne J, Rohrbach LA, Vincus A, Simons-Rudolph A, et al. A comparison of current practice in school-based substance use prevention programs with meta-analysis findings. Prev Sci. 2003;4(1):1–14.

    Article  PubMed  Google Scholar 

  40. Domitrovich CE, Bradshaw CP, Poduska JM, Hoagwood K, Buckley JA, Olin S, et al. Maximizing the implementation quality of evidence-based preventive interventions in schools: a conceptual framework. Adv School Mental Health Promot. 2008;1(3):6–28.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Pediatrics, Wayne State University School of Medicine, 4707 Saint Antoine Street, Detroit, MI, 48201, USA

    Bo Wang, Bonita Stanton, Veronica Koci, Linda Kaljee, Sharon Marshall & Xiaoming Li

  2. Office of HIV/AIDS, Ministry of Health, Shirley Street, Nassau, Bahamas

    Lynette Deveaux, Maxwell Poitier, Sonja Lunn & Richard Adderley

  3. Department of Education, Ministry of Education, Thompson Boulevard, PO Box N-3913, Nassau, Bahamas

    Glenda Rolle

Authors
  1. Bo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bonita Stanton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lynette Deveaux
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maxwell Poitier
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sonja Lunn
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Veronica Koci
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Richard Adderley
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Linda Kaljee
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sharon Marshall
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Xiaoming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Glenda Rolle
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bonita Stanton.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

BW, BS, LD, and XL conceived the initial research questions. BW and BS analyzed the data and drafted the manuscript. LD, SL, LK, SM, XL, and GR helped in interpreting the data. MP, VK, and RA were fully involved in data acquisition, data entry, and maintaining the data base. BS obtained the funding and overall oversight of study. All authors were involved in the critical revision of the paper for intellectual content. All authors read and approved the final manuscript.

Rights and permissions

Open Access  This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/.

The Creative Commons Public Domain Dedication waiver (https://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, B., Stanton, B., Deveaux, L. et al. Factors influencing implementation dose and fidelity thereof and related student outcomes of an evidence-based national HIV prevention program. Implementation Sci 10, 44 (2015). https://doi.org/10.1186/s13012-015-0236-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13012-015-0236-y

Keywords

Implementation Science

ISSN: 1748-5908