Health Promotion International, 2017;32:79–90
doi: 10.1093/heapro/dav055
Advance Access Publication Date: 17 June 2015
Article
Sustainability of the good behaviour game
in Dutch primary schools
Marieke A. M. Dijkman1,*, Janneke Harting2, Lenneke van Tol1,
and Marcel F. van der Wal1
1
Cluster E&G, Public Health Service Amsterdam (GGD), PO Box 2200, Amsterdam 1000 CE, The Netherlands,
and 2Department of Social Medicine, Academic Medical Centre, University of Amsterdam, Amsterdam 1100
DD, The Netherlands
*Corresponding author. E-mail: [email protected]
Summary
Sustainability of health promotion programs is essential to maintain their positive effects. However, few
studies have examined the extent of program sustainability and the factors influencing it. We examined
these issues through the Good Behaviour Game (GBG), a classroom-based program in primary schools
with beneficial behavioural and health-related effects that was implemented in 2008. GBG coordinators
of 17 participating schools were invited in the study 2 years after the initial program implementation.
Sustainability was measured using a 20-item checklist comprised of four dimensions of routinization
including: memory, adaptation, values and rules. A semi-structured interview was then completed
with 16 of the GBG coordinators to discuss the checklist scores and to probe in more depth the current
level of sustainability. Based on the checklist scores, sustainability of the GBG was considered ‘high’ in
five schools, ‘medium’ in another five and ‘weak’ in six. Factors influencing sustainability identified by
GBG coordinators were organizational strength, strong leadership, program championship and the perceived modifiability and effectiveness of the GBG. Also, different factors were related to different
dimensions of routinization. The combination of a sustainability checklist and an interview about influential factors may help to further clarify the sustainability construct and reveal which implementation
sites, routinization dimensions and influential factors should be explored to further facilitate the sustaining of programs with proven effectiveness.
Key words: sustainability, evidence-based school programs, behaviour problems
BACKGROUND
Sustainability of evidence-based health programs is essential to maintain their positive effects in target populations.
However, many initially successfully implemented programs fail to become part of the habits and routines of
the host organizations, fading away as time progresses
(Scheirer et al., 2008; Stirman et al., 2012). Not surprisingly,
policy-makers, funders and other stakeholders are increasingly concerned about the long-term impact of their
investments but little is known about the extent of program sustainability and about the factors influencing it
(Scheirer and Dearing, 2011; Stirman et al., 2012).
Although more research is needed, this has been hampered
by a lack of a clear and uniform definition of sustainability
and validated instruments to measure sustainability and
© The Author 2015. Published by Oxford University Press. All rights reserved. For Permissions, please email: [email protected]
80
M. A. M. Dijkman et al.
its influencing factors (Shediac-Rizkallah and Bone, 1998;
Scheirer, 2005; St. Leger, 2005; Savaya et al., 2008;
Scheirer and Dearing, 2011; Stirman et al., 2012).
The large-scale implementation of the evidence-based
Good Behaviour Game (GBG) in primary schools in
Amsterdam (the Netherlands) was an excellent opportunity to make a contribution to understanding the complexities of sustainability, as requested in the literature
(Scheirer, 2005; Stirman et al., 2012). The GBG is a
classroom-based intervention that aims to reinforce prosocial behaviour and reduce aggressive and disruptive behaviour (Barrish et al., 1969; Dolan et al., 1989) among
primary school children. The GBG has been shown effective in preventing and reducing behavioural problems in
the classroom (Embry, 2002; Van Lier et al., 2004;
Tingstrom et al., 2006) and also has positive long-term effects on, for example, smoking, drug and alcohol abuse,
antisocial personality disorder and violent and criminal
behaviour (Kellam et al., 2008; Petras et al., 2008;
Poduska et al., 2008; van Lier et al., 2009). Because of
its strong beneficial short- and long-term effects, the
GBG is able to deliver a substantial positive contribution
to population health; however, to achieve this, the program has to be sustained for longer periods of time (see
Box 1 for a detailed description of the GBG program
and its implementation).
To guide our research on the sustainability of the
GBG, we based our theoretical framework (Figure 1) on
Box 1: Case description of the Good Behaviour Game
Good Behaviour Game
The Good Behaviour Game (GBG) is a classroom-based intervention in primary schools that aims to
reinforce pro-social behaviour and reduce aggressive and disruptive behaviour. In the GBG, teachers
are taught to explicitly define and systematically reward appropriate behaviour, ignore inappropriate
behaviour and facilitate the interaction between disruptive and non-disruptive children. Teachers and
children establish positively formulated class rules, which are represented by pictograms. After observing children in terms of carefully pre-defined appropriate behaviours in the classroom, teachers assign
the children to small teams containing equal numbers of disruptive and non-disruptive children. Each
team receives a number of cards. During the game, which is always played during regular educational
activities like maths or reading, teachers take away a card when a pupil violates one of the rules. Teams
are rewarded when at least one card remains on their desk at the end of the playing time. Playing the
GBG encourages children to manage their own and their team mates’ behaviour through a process of
group reinforcement and mutual self-interest.
GBG implementation protocol
The implementation of the GBG is carried out by an external trainers from School Advisory Services and
takes about one school year. During this period, teachers receive three afternoons of training and additional
coaching by the external trainers during ten 60-min classroom observations with feedback. At the beginning of the year, the GBG is played three times a week for 15 min. Over the course of the year, the amount of
time spent playing the GBG is gradually increased. By the end of the year, the GBG is played at different
times throughout the day, during different educational activities, and it is incorporated in existing working
methods. The teachers receive a GBG certificate when they meet the predefined quality criteria.
Continuation of the GBG
By the end of the first implementation year, the external trainers from the School Advisory Service convey their responsibility for the implementation of the GBG to the internal school advisors, who are
responsible for the social emotional learning of the students. By the train-the-trainer method, these
internal school advisors learn how to play the GBG, as well as how to coach the teachers in the structural
implementation of the GBG. After accomplishing the required capabilities, the internal school advisors
receive a certificate and while being assigned the role of GBG coordinator in the school. This transfer of
responsibilities and capabilities is assumed to warrant the continued implementation of the GBG once
the external GBG trainers have left. The GBG protocol does not include additional sustainability strategies and it is currently not compulsory for Dutch schools to have a school health policy plan in
which programs like the GBG can be incorporated.
Sustainability of the GBG in Dutch primary schools
81
Fig. 1: Theoretical framework for sustainability of health programs.
previous studies (Shediac-Rizkallah and Bone, 1998;
Pluye et al., 2004a,b; Scheirer, 2005; Scheirer and
Dearing, 2011; Stirman et al., 2012). In the framework,
sustainability means that the program is incorporated
into the organization and has become a stable and regular
part of organizational procedures and behaviour (Yin,
1979; Steckler and Goodman, 1989; Shediac-Rizkallah
and Bone, 1998; Pluye et al., 2004a,b). This manifests itself in the presence of organizational routines, which can
be divided into four dimensions: memory, adaptation, values and rules (Pluye et al., 2004a,b). ‘Memory’ is defined
as the conditions needed to perform the program, such as
qualified people, materials and time. ‘Adaptation’ refers to
program-related activities that take place in accordance
with a local context. ‘Values’ refer to the collective ideals
and beliefs associated with the program within the organization. Finally, ‘rules’ are routinized program-related
activities that guide decision-making and action. In addition, our framework distinguishes three groups of factors
that affect the sustainability of health programs (ShediacRizkallah and Bone, 1998; Scheirer and Dearing, 2011).
First, sustainability depends on ‘characteristics of the
intervention and its implementation’, including variables
such as project effectiveness and program complexity
(Shediac-Rizkallah and Bone, 1998; Scheirer, 2005;
Forman et al., 2009). Second, it depends on ‘characteristics in the organizational setting’, like the presence and
role of the program champion or coordinator and organizational strength (Shediac-Rizkallah and Bone, 1998;
Han and Weiss, 2005; Scheirer, 2005; Forman et al.,
2009; Savaya and Spiro, 2011). Lastly, sustainability is influenced by ‘characteristics in the broader community environment’, which include support provided by other
organizations, together with the socioeconomic and political environment (Shediac-Rizkallah and Bone, 1998;
Scheirer, 2005; Savaya and Spiro, 2011).
The aim of this study was to examine the extent to
which the GBG was sustained in the schools 2 years
after initial implementation, as well as the factors influencing this process. Guided by the theoretical framework,
the study was also expected to contribute to understanding the complexities of sustainability.
METHODS
Design and population
Sustainability of the GBG was measured in a nonexperimental mixed-methods study, in which, guided by
the theoretical sustainability framework, quantitative and
qualitative data were collected simultaneously (Creswell
and Plano Clark, 2011). Two years after initial implementation, we invited GBG coordinators of 17 primary schools
in Amsterdam that had adopted and implemented the program in the school year 2008–2009.
Procedure
Data were collected in spring 2011 at the schools by two
researchers (M.A.M.D. and L.T.). First, we assessed the
extent to which the GBG was sustained using a 20-item
checklist that was personally presented to the respondents.
This allowed the researcher to explain and clarify items if
necessary. The answers were noted by the researcher.
After completing the checklist, the researcher estimated
the general level of sustainability: that is, as weak, medium
or high (Pluye et al., 2004a). The respondents were then
asked to explain the outcome of this general assessment,
in order to gain a first impression of the factors influencing
the sustainability of the GBG. Next, a semi-structured
interview guide was used to discuss in-depth the influential
factors from the questionnaire in relation to the scores on
the individual items of the checklist. The interview guide
82
ensured that all potential influencing factors were discussed with each of the respondents in a similar way.
Completing the checklist and the semi-structured interview took on average 30–45 min.
Instruments
Sustainability checklist
We developed a 20-item checklist to measure the presence
of organizational routines of the GBG as indicators of program sustainability (Table 1). For this, we adapted a
15-item checklist, previously developed by Goodman
et al. (Goodman et al., 1993) and Pluye et al. (Pluye
et al., 2004b) to the Dutch school environment and characteristics of the GBG. Memory was operationalized with
six items; these addressed the structural availability of
money, materials, people and time to carry out the program in a sustainable way, as well as the availability of
structural support and training facilities for the teachers
involved. Adaptation was measured with two items concerning the way the program was locally adjusted and enjoyed. Values were operationalized with four items; these
asked whether the program fitted into the school’s policy
aims, whether program-related rituals were present, and
whether the program’s logo and specific program jargon
were being used. Rules were operationalized with eight
items. First, these asked whether a formal GBG coordinator had been appointed and whether the program was
subject of regular deliberation meetings. Second, it was
asked whether the program was mentioned on the school’s
website as well as in the school guidebook, the school’s annual plans and the school’s policy plan. Third, it was
asked whether the program was included in the job specifications of the GBG teachers and/or the GBG coordinators. Answering categories for each of the items were
‘Yes’ (1 point), ‘No’ (0 point) or ‘Don’t know’ (0 point).
Semi-structured interview guide
The development of the interview guide explored the three
previously identified categories of factors influencing sustainability (Table 2). Questions about the program and
implementation factors addressed, e.g. project effectiveness, program complexity and modifiability and training
of professionals including feedback. Questions regarding
the organizational setting asked, for example, about the
presence and role of the program champion or coordinator, organizational strength and strong leadership.
Questions about the factors within the broader community environment investigated the support received from
other organizations, the socioeconomic and political
environment and the cooperation between different
stakeholders.
M. A. M. Dijkman et al.
Data analysis
Quantitative data
Item scores were entered into Excel. The checklist scores
were calculated as overall measures and then as subtotals
by each of the four dimensions of organizational routines.
The total score indicated the sustainability of the GBG
program in the primary school organization, divided
into three groups (tertiles) of high (≥12), medium (8–11)
and weak (≤7) sustainability (Pluye et al., 2004a).
Qualitative data
Interviews were recorded and transcribed verbatim. Each
interview was subject to content analysis using the computer program MaxQda 10.0. We developed a coding
structure that included all potential influential factors addressed by the interview guide (Table 2). To ensure internal reliability (Polit and Beck, 2004), five interviews were
coded and extensively discussed by two researchers
(M.A.M.D. and L.T.). When needed, definitions of codes
were refined and text fragments were recoded. The remaining interviews were coded by M.A.M.D. alone.
Data integration
To determine the factors that had influenced the sustainability of the GBG, we used typology development as the
integrative strategy for mixed-method data analysis
(Caracelli and Green, 1993). First, we used the classification of schools (checklist data) to identify differences in
sustainability dimensions and items within and between
the three sustainability groups. This provided information
on which dimensions and items had, or had not, been
important for each of the three sustainability groups in
general. Second, within each sustainability group, we
arranged the qualitative interview data in accordance
with the four sustainability dimensions and, subsequently,
in accordance with the items used to operationalize these
dimensions (items 1–20). Next, we used the method of
constant comparison (Polit and Beck, 2004) to search
for commonalities within and differences between schools
that were classified as high, medium or weak sustainability
(Caracelli and Green, 1993). This iterative process allowed
us to identify factors that, overall, per dimension and per
item, could explain differences in the checklist scores
between sustainability groups. All comparisons were discussed with a senior researcher (J.H.) until consensus
was reached.
Ethical considerations
This study was approved by the medical ethical committee
of the VU Medical Centre in Amsterdam.
1
4
1
5
2
1
2
1
Subtotal values
1
1
1
1
1
1
1
2
1
1
1
6
1
1
5
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Values items
9. Does the GBG fit into the aims formulated in the school’s policy plan?
10. Are there certain rituals with regard to the GBG? (e.g. around
awarding GBG teacher licences or GBG student diplomas)
11. Does your school make use of the GBG logo?
12. Is GBG jargon or GBG terminology used at your school?
Adaptation items
7. Did you modify the GBG to circumstances in your school?
(e.g. to existing teaching methods/ways of working or to specific
groups of pupils)
8. Is the GBG carried out each year just because teachers find it a pleasure,
apart from the results of the GBG?
Subtotal adaptation
Memory items
1. Is [structurally] enough money available to sustainably carry out the
GBG? (e.g. for support by a GBG coordinator, to hire external
supervision for the GBG trainer etc.)
2. Are [structurally] enough complete GBG boxes available to sustainably
carry out the GBG?
3. Has the school appointed enough teachers with a GBG licence to carry
out the GBG year after year?
4. Do most of the teachers with a GBG licence allocate enough time to
carry out the GBG?
5. Do coordinators structurally allocate enough time for the supervision of
teachers with a GBG licence?
6. Is training every new teacher in the use of the GBG the standard
procedure?
Subtotal memory
D
1
1
1
1
5
1
1
1
1
1
E
2
1
1
2
1
1
4
1
1
1
1
1
1
1
1
4
1
1
1
1
G
2
1
1
1
1
6
1
1
1
1
1
1
H
1
1
1
1
4
1
1
1
1
I
1
1
0
5
1
1
1
1
1
J
F
C
A
B
Medium sustainability [MS]
High sustainability [HS]
1
1
1
1
2
1
1
K
1
1
1
1
3
1
1
1
L
2
1
1
1
1
2
1
1
M
1
1
0
2
1
1
N
1
1
0
2
1
1
O
0
0
2
1
1
P
Continued
Weak sustainability [WS]
Table 1: Checklist scores on routinization dimensions: participating schools classified in high, medium and weak sustainability based on their total checklist score
Sustainability of the GBG in Dutch primary schools
83
84
0
2
1
4
Response
1
4
1
1
L
1
1
6
1
1
1
3
7
0
5
M
N
RESULTS
K
Weak sustainability [WS]
O
P
M. A. M. Dijkman et al.
Of the 17 GBG coordinators, 16 (14 women and 2 men)
agreed to take part in the study. One coordinator refused
to participate due to limited time and disinterest in the
study. The length of education experience of the respondents ranged from 3 to 35 years, and their age from 26
to 62 years. All respondents worked at public schools.
1
10
1
3
9
2
8
1
1
1
1
4
10
1
2
10
1
5
12
3
12
5
13
1
1
1
1
1
1
1
6
13
1
1
1
1
1
1
7
14
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
J
I
H
G
F
A
B
C
D
E
Rules items
13. Does the school have coordinator who is in charge of the GBG?
14. Is the GBG regularly subject of deliberation meetings?
15. Is the GBG mentioned on the school’s website?
16. Is the GBG mentioned in the school guidebook?
17. Is the GBG mentioned in the annual plans of the school?
18. Is the GBG mention in the school’s policy plan?
19. Is the GBG part of the job description of the GBG teachers?
20. Is the GBG part of the job description of the GBG coordinators?
Subtotal rules
Total score
Table 1: Continued
High sustainability [HS]
Medium sustainability [MS]
Level of sustainability
Total scores on the checklist (maximum score 20) ranged
from 2 to 14 (Table 1). Evident differences between the
three sustainability subgroups (weak, medium or high) existed for the memory and rules components, while there
were less clear differences between subgroups for adaptation and no differences for values. In the following sections, we compare the checklist scores by subgroups and
elaborate on the factors influencing the notable differences
in sustainability levels by incorporating the results from
the semi-structured interviews.
Memory
Scores on memory (maximum score 6) ranged from 2 to 6.
With regard to ‘project financing’, three respondents from
the schools with high sustainability explained that the
GBG had been integrated into their range of duties, for
which enough money was available (Table 1; item 1).
Moreover, two respondents of the same subgroup found
the GBG to be of such importance that, irrespective of municipal funding, they would in any case continue the program. One said:
There’s no extra money available for the GBG. It’s a task
that was allocated to me. It’s part of the pedagogical climate and class management, which is also part of my responsibility. So that’s how it is simply part of my normal
tasks. [C; HS]
In all three sustainability groups, the GBG materials were
structurally available (item 2). Consequently, no distinctive influential factors could be identified between the subgroups. All schools with medium and high sustainability
levels reported having appointed a sufficient amount of
teachers with a GBG certificate (item 3). On the other
hand, five of six respondents of the weak schools answered
this question in the negative. Having an insufficient number of qualified teachers was explained by a high level of
staff turnover, associated with limited ‘organizational
strength’. One of them said:
I think, this is mostly because of the enormous staff turnover that we had. Teachers with a GBG certificate left the
Sustainability of the GBG in Dutch primary schools
85
Table 2: Interview guide regarding factors influencing sustainability of the GBG
Program and implementation factors
Specific needs or problems
Project negotiation process
Project financing
Project effectiveness
Project duration
Project type
Training and performance feedback
Modifiable at local circumstances
Complexity
Factors within the organizational setting
Program champion/coordinator
Leadership actions
Institutional strength
Fit of the program in the organization
Factors within the broader community environment
Support from other organizations
Socioeconomic and political considerations
Cooperation between stakeholders
How does the GBG meet the needs or problems of users?
How was the GBG adopted in the school? Who was involved in the
decision-making?
Is project funding available for the GBG?
In what way is the GBG perceived as effective? Is it visible? What are the
advantages of the GBG?
What is the GBG’s grant period?
Is the GBG a preventive or curative program?
What was the quality of training and consultations?
In what way is the GBG modifiable to local circumstances?
Is the GBG easy or complex to use?
What is the presence and role of a GBG program champion/coordinator?
Does management support and facilitate the GBG?
Is management committed to the GBG?
What is the strength of the organization (including maturity, staff turnover)?
Does the GBG fit within the existing organizational mission and procedures?
Do experts from other organizations support or give advice?
How favourable is the general socioeconomic and political environment for the
sustainability of the GBG?
Is there collaboration between the program developers, organizational decision
makers, implementers and evaluators?
school. Other people with no certificate came and they
were not taken into this procedure. [N; WS]
Schools with weak sustainability levels less often confirmed that teachers with a GBG certificate were allocated
adequate time to deliver the GBG program compared with
schools with higher levels (item 4). One determining factor
in this respect was the schools’ ‘perceived needs or problems’. Schools with high and medium program sustainability explained that the GBG was needed to continue
to solve children’s behavioural problems and therefore
met a certain need. One respondent from the high sustainability group said:
It gives the team power. And, especially now, with more
children with behavioral problems in the classroom.
When you stay on the positive side, almost all children
will get along. [A; HS]
Compared with weakly sustained schools, in highly and
medium sustained schools, allocating adequate time to deliver the GBG (item 4) was also related to the program’s
‘perceived effectiveness’. That is, noticing the positive results in terms of better academic skills and a more positive
class climate, made teachers willing to continue delivering
the GBG.
Of the schools with weak sustainability, the GBG coordinators had not set aside adequate time for the supervision of the teachers with GBG certificates (item 5),
whereas in the other two groups this was more often
agreed upon. Important in this respect was ‘organizational
strength’. Weak sustainability schools, for instance,
pointed to long-lasting illnesses of GBG coordinators
and a powerless school management to explain these
shortcomings. One said:
In contrast, respondents of the weakly sustained schools
explained that the GBG was no longer needed at the
school. One of these coordinators explained:
We had a very chaotic year. The school director was set
aside and that caused a lot of problems. At that point,
these kind of things, like the GBG, move to a second,
third or fourth place. [N; WS]
The most important reason for not doing it anymore is
that the necessity is gone. A lot of teachers left and new
teachers came. These new ones are another type of teacher,
they don′t need it anymore. [P; WS]
Weak sustainability schools more often reported that
training new teachers in the use of the GBG was not the
norm, whereas it was the standard procedure in the high
and medium subgroups (item 6). Such standardization of
86
M. A. M. Dijkman et al.
training procedures was related to strong ‘leadership actions’, reflected by a school management that considered
the GBG their schools’ core component. The training of
new teachers had come to a halt in cases where the management was less actively involved. A respondent from a
highly sustained school said:
These GBG coordinators also emphasized that they perceived the GBG as being fundamental to their responsibilities. One said:
Besides, when new teachers apply, the management always asks: ‘It is custom to play the GBG in this school.
Do you agree with that?’ When management doesn′t support that, it will fade away. [D; HS]
The ‘role of the coordinator’ seemed to be less or not formalized in the medium and weakly sustained schools.
Although the GBG coordinators acknowledged the importance of their roles, they were not always able to sufficiently fulfil their responsibilities, due to a lack of time and
other competing priorities.
In more than half of the schools, the GBG was subject
to regular deliberation meetings (item 14), with no clear
differences between subgroups. Only one school mentioned the GBG on its website (item 15). The GBG was
more often mentioned in the school guidebooks (item 16)
in highly sustainability schools. In three of them, the GBG
was also mentioned in their annual plan (item 17). When
asked, some respondents of medium and weakly sustainability schools said they did not make mention of the GBG, but
appreciated the suggestion for the future. It made them aware
of their possibilities to better sustain the GBG.
The GBG was mentioned in the policy plans (item 18)
in all of the highly sustained schools, but only in two of the
five medium sustained schools. For two additional ones,
the respondents explained that new school policy plans
were being developed and that the GBG was likely to be
included. None of the weakly sustained schools included
the GBG in their policy plans. ‘Leadership actions’ undertaken by strong and committed school managements were
suggested to play an essential role in this respect, given that
they write the policy plans. One respondent from the
weakly sustained group said:
Adaptation
Scores on adaptation (maximum score 2) ranged from 0 to
2. Overall, the GBG was found to be less frequently modified to local circumstances in the weakly sustained schools
(item 7). Although many schools perceived either the
method of working or the principle of positive rewarding
of the GBG as less suitable for children aged ≤6 years and
≥10 years, highly and medium sustained schools perceived
the program as ‘modifiable’ and searched for ways to solve
apparent mismatches. One said:
The teachers were not satisfied about the GBG for the
older children (over 10 years) because it became too childish for them. However, together with their GBG trainer,
they found a way to deal with that. [B; HS]
As a result, local adaptations enabled teachers to continue
the use of the GBG, whereas teachers in the schools with
weaker sustainability stopped using the program altogether
the moment they encountered a problem.
Only three respondents confirmed the question about
carrying out the GBG each year because of simply finding
it a pleasure (item 8). Therefore, no relation could be made
with influential factors from the interviews.
Values
Scores on values (maximum score 4) ranged from 0 to
2. All but one respondent confirmed that the GBG fit
with the policy aims of the school (item 9). The other
items of this dimension (items 10, 11 and 12) were endorsed only once or twice. No additional information
about influential factors in this respect was mentioned in
the interviews.
Rules
Scores on rules (maximum score 8) ranged from 0 to 7. All
highly and medium sustained schools had appointed a formal GBG coordinator, which was only the case for two of
the six weakly sustained schools (item 13). In fact, in the
highly sustained schools, the ‘role of the coordinator’ was
regarded as influencing the continuation of the program.
Honestly, I think that if it was not one of my tasks, the
GBG would have just fallen over. No coordination – no
GBG in the school. [B; HS]
The management has to think it’s important enough to
make it school policy. At the moment that’s not really
the case, because they see it from a distance and are not
really committed to it. [L, WS]
The GBG’s inclusion in the school’s policy plan also depended on ‘project effectiveness’. In highly sustained
schools, the GBG was used more frequently and with
greater program fidelity. These schools experienced more
positive effects, which further enhanced the program’s inclusion in the schools’ policy plan.
It’s called ‘evidence proved’, it has proven itself . . . and
people noticed very quickly that it really works. So, it
was obvious to make it school policy. [E, HS]
Although teachers with a GBG certificate and GBG coordinators did not have formal job descriptions (items 19
Sustainability of the GBG in Dutch primary schools
and 20), more respondents of highly sustainable schools
recognized the GBG as informally being part of that.
DISCUSSION
To assess the sustainability of the GBG, a school-based
prevention program with short-term behavioural and
long-term health-related effects, we developed a sustainability checklist that operationalized the four dimensions
of organizational routines (Pluye et al., 2004b), and an
interview guide that addressed three categories of factors
influencing sustainability (Shediac-Rizkallah and Bone,
1998; Scheirer and Dearing, 2011).
An important finding is that none of the schools in our
study came close to achieve all sustainability items. We
suggest this namely had to do with validation issues of
the measurement instrument, which will be described
later on.
The degree of sustainability of the program varied considerably between the schools. High and medium sustainability schools scored higher on items concerning memory
(e.g. the structural availability of materials and trained
staff to perform the GBG). Factors most influential in
this respect were strong leadership (e.g. management
showing robust commitment and providing strong support) and organizational strength (e.g. strong management,
a mature organizational structure and low staff turnover).
Especially high sustainability schools also scored higher
on items concerning rules (e.g. a formally appointed GBG
coordinator and the incorporation of the GBG in the school
policy plan). Additional influential factors in this respect
were a program champion role for the GBG coordinator
and the perceived effectiveness of the GBG. In addition,
high and medium sustainability schools tended to score
slightly higher on adaptation (e.g. adapting the GBG to existing working methods or to local circumstances). An important associated factor was perceiving the GBG as open
to modification. We observed no differences for the dimension ‘value’.
Our theory-based procedure to measure sustainability
may contribute to a further clarification of the sustainability construct, which is considered as a requirement for theory building and practical performance (Pluye et al.,
2004a; Scheirer, 2005; Stirman et al., 2012).
First, measuring sustainability by operationalizing the
four dimensions of organizational routines provided us
with a detailed understanding of the extent to which the
specific program was sustained. Such insights offer an advantage over dichotomous measures that merely assess
whether or not a program is still being carried out at a certain point of time (Pluye et al., 2004b; Scheirer, 2005).
Second, simultaneous assessment of the factors influencing
87
sustainability revealed which factors were of actual importance to the sustainability process under study. Although the
categories of influential factors we identified (e.g. organizational strength and leadership support) were similar to
those reported earlier, our approach also revealed that
different factors were related to different dimensions of
organizational routines.
Our research findings have conceptual and practical
implications. Conceptually, our study confirms the assumption that sustainability should preferably be regarded
as a multidimensional construct rather than a singular outcome or a momentary state (Mancini and Marek, 2004;
Scheirer, 2005). We infer this from the substantial
between-school differences in the checklist scores, which
reflects that sustainability is not simply a matter of still carrying out a program, or not. Instead, the sustainability of
the GBG program appeared to take shape in various organizational dimensions. The between-school differences
in scores on the various routinization dimensions also suggest that these dimensions may reflect successive stages in
the sustainability process (Pluye et al., 2005; Scheirer,
2005). That is, the memory dimension (the conditions;
achieved in all three sustainability groups) seems to precede the rules dimension (activities that guide decisionmaking and action; mainly achieved in the high sustainability group), whereas the value dimension may need
even more time to become established (apparently not
yet achieved by any of the sustainability groups; see also
the discussion below on). However, this tentative finding
needs to be tested in larger studies. Practically, a more
detailed view on the sustainability process may promote
the development of segmented and tailored sustainability-enhancing strategies (Pluye et al., 2004a). That is,
approaches like ours may reveal which implementation
sites, routinization dimensions and influential factors
should be addressed to further facilitate sustainment of
programs with proven effectiveness. For example, in
some organizations, the primary focus should be on formally assigning a program coordinator, whereas in other
organizations, commitment of the management to the program should be facilitated to enhance sustainability. This
proposed segmentation and tailoring of strategies may increase the chances of success (Glasgow et al., 2003).
Although showing face validity (Pluye et al., 2004b),
the sustainability checklist should be further developed
and validated to serve as a scientific measurement instrument. The content validity of the various routinization dimensions requires further questioning (Polit and Beck,
2004). Although we based our operationalization on previous theory-based and empirical work (Goodman et al.,
1993; Pluye et al., 2004a,b), we are not sure whether the
items of the checklist sufficiently represent different
88
dimensions of routinization. Especially problematic in this
respect may be the value dimension. Most of the value
items were not endorsed by our respondents, and the remaining scores on this dimension did not differentiate between the schools. As the original checklist from which we
adopted the value items was developed in Canada, our
findings may indicate that values is a culturally sensitive
dimension that needs to be operationalized to a specific
context (Eccles et al., 2005). Another explanation may
be that values related to the GBG should be operationalized in closer connection to the program’s core values;
in the case of the GBG, this might mean that questions specifically address the systematically reinforcement of positive behaviour in children (Barrish et al., 1969). A final
explanation may be that, in the schools participating in
the present study, the sustainability process of the GBG
was not far enough along for the value dimension to sufficiently express itself. Further qualitative inquires could
shed light on these content validity matters (Polit and
Beck, 2004).
Another validation issue is the use of cut-off points to
discriminate between schools in which the GBG was highly, medium or weakly sustained. Although we adopted the
tertile division of the overall checklist scores from a previous study (Pluye et al., 2004a), this procedure has not yet
been validated. Nevertheless, our approach showed some
indications of concurrent validity (Polit and Beck, 2004).
That is, our respondents obviously ascribed different
scores to the different dimensions of routinization, and
these within-school differences appeared to be partly associated with different sustainability influencing factors.
Thus, to a certain extent, our checklist was able to distinguish between different routinization dimensions and to
discriminate between highly, medium and weakly sustained schools. However, our cut-off points should be reconsidered in relation to the relatively low total scores
obtained in our study for all schools (i.e. 14 of 20). This
raises the question as to what an optimal sustainability
score would be and what it means for appropriate categorization and cut-off points. Therefore, future studies should
explore the checklist’s concurrent as well as predictive validity (Polit and Beck, 2004), by comparing present sustainability scores with either short-term behavioural
outcomes or long-term health-related outcomes through
sustained implementation of the GBG.
Besides being a valuable measurement instrument, the
way we combined checklists and interviews had practical
advantages and implications. First, the checklist functioned as a useful tool to create respondents’ awareness
of the degree of sustainability of the program. While answering the checklist’s questions, the respondents better
understood how their school scored on the different
M. A. M. Dijkman et al.
dimensions and on which specific factors the school
needed to focus to better sustain the program. Thus,
while using the checklist as a measurement instrument, it
also served as a sustainability-enhancing strategy, while
respondents made plans to tackle certain points. This fits
with the recommendations described in a recent review on
sustainability (Stirman et al., 2012). In this latter review,
the use of checklists is mentioned to sustain interventions
that require a high degree of fidelity to produce the
intended health benefits. Second, the checklist can be
used to guide the organization throughout the entire
adoption–implementation–sustainability cycle (Scheirer,
2005), while planning sustainability begins as soon as
the program starts (Fagen and Flay, 2009). For example,
when using a checklist that has been adapted to the specific program in the adoption phase, key predictors of program sustainability, like the commitment of management
and their intention to formalize the coordinator’s role,
could already be tackled at an early stage (Dijkman
et al., 2014). This might promote and reinforce sustainability of evidence-based programs and ensure the desired
long-term positive effects. A third benefit of the checklist
was that it was easy to use and took only ∼10 min to complete. Respondents were enthusiastic about the checklist
and some asked for a copy in order to discuss the results
with the school management.
Apart from these validity issues, three other limitations
should be mentioned. First, the routinization dimensions
we used are only one way of defining sustainability. This
equivalent of organizational routines merely reflects structural changes in the organization in which the program is
implemented. Other types of sustainability indicators that
have been suggested are: continued consumer benefits or
outcomes and diffusion of programs and replication in
other sites (Scheirer, 2005). Second, our study consisted,
in part, of qualitative methods. However, because no
new information emerged during our last few interviews
we feel confident that the saturation of information
(Polit and Beck, 2004) was sufficient to warrant the internal validity of our study. Third, our findings are based on a
single case study that examined the sustainment of one
specific program in a limited number of schools within
one Dutch city. Although we succeeded in recruiting the
majority of schools in which the program was implemented, we acknowledge that the generalizability of our findings may be limited. Future studies should consider the
additional application of quantitative measurement methods. For example, the checklist sustainability could be
used in larger samples to validate it as an instrument to
measure the sustainability of the GBG specifically and/or
comparable school-based prevention programs in general.
In addition, a quantitative survey to assess the factors
Sustainability of the GBG in Dutch primary schools
influencing sustainability of the GBG should be considered. That is, in the case where sufficient qualitative information on the sustainability process is available, the four
routinization dimensions as well as the sustainability influencing factors can be measured quantitatively in large
samples. Such quantitative measurements may further elucidate the construct of sustainability as well as the factors
influencing the sustainment of effective programs.
CONCLUSION
None of the schools in this study came close to achieve all
sustainability items, which namely had to do with validation issues of the instrument used to measure sustainability. The degree of sustainability of the GBG varied
considerably between the participating schools. Factors
that influenced sustainability most were organizational
strength, strong leadership, program championship and
the perceived modifiability and effectiveness of the GBG,
while different factors were related to different dimensions
of routinization. The combination of a sustainability
checklist and an interview addressing influential factors
may help to further clarify the sustainability construct
and reveal which implementation sites, routinization dimensions and influential factors should be addressed to
further facilitate the sustainment of programs with proven
effectiveness.
FUNDING
This work was supported by the Netherlands Organisation
for Health Research and Development (ZonMw, grant
number: 120710001).
REFERENCES
Barrish H. H., Saunders M., Wolf M. M. (1969) Good behavior
game: effects of individual contingencies for group consequences on disruptive behavior in a classroom. Journal of
Applied Behavior Analysis, 2, 119–124.
Caracelli V. J., Green J. C. (1993) Data analysis strategies for
mixed methods evaluation designs. Educational Evaluation
and Policy Analysis, 15, 195–207.
Creswell J. W., Plano Clark V. L. (2011) Designing and
Conducting Mixed Methods Research, 2nd edition. Sage,
Thousand Oaks, CA.
Dijkman M. A. M., Harting J., Van der Wal M. F. (2014)
Adoption of the Good Behaviour Game: an evidence-based
intervention for the prevention of behaviour problems.
Health Education Journal, 74, 168–182.
Dolan L. J., Jaylan T., Werthamer L., Kellam S. (1989) The Good
Behavior Game Manual. The John Hopkins Prevention
Research Center, Baltimore.
89
Eccles M., Grimshaw J., Walker A., Johnston M., Pitts N. (2005)
Changing the behavior of healthcare professionals: the use of
theory in promoting the uptake of research findings. Journal
of Clinical Epidemiology, 58, 107–112.
Embry D. D. (2002) The Good Behavior Game: a best practice
candidate as a universal behavioral vaccine. Clinical Child
and Family Psychology Review, 5, 273–297.
Fagen M. C., Flay B. R. (2009) Sustaining a school-based prevention program: results from the Aban Aya Sustainability
Project. Health Education and Behavior, 36, 9–23.
Forman G. A., Olin S. S., Hoagwood K., Crowe M., Saka N.
(2009) Evidence-based interventions in schools: developers’
views of implementation barriers and facilitators. School
Mental Health, 1, 26–36.
Glasgow R. E., Lichtenstein E., Marcus A. C. (2003) Why don’t
we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition.
American Journal of Public Health, 93, 1261–1267.
Goodman R. M., McLeroy K. R., Steckler A. B., Hoyle R. H.
(1993) Development of level of institutionalization scales for
health promotion programs. Health Education Quarterly, 20,
161–178.
Han S. S., Weiss B. (2005) Sustainability of teacher implementation of school-based mental health programs. Journal of
Abnormal Child Psychology, 33, 665–679.
Kellam S. G., Brown C. H., Poduska J. M., Ialongo N. S.,
Wang W., Toyinbo P., et al. (2008) Effects of a universal classroom behavior management program in first and second
grades on young adult behavioral, psychiatric, and social outcomes. Drug and Alcohol Dependence, 95(Suppl 1), S5–S28.
Mancini J. A., Marek L. I. (2004) Sustaining community-based
programs for families: conceptualization and measurement.
Family Relations, 53, 339–347.
Petras H., Kellam S. G., Brown C. H., Muthen B. O., Ialongo N.
S., Poduska J. M. (2008) Developmental epidemiological
courses leading to antisocial personality disorder and violent and criminal behavior: effects by young adulthood of
a universal preventive intervention in first- and second-grade
classrooms. Drug and Alcohol Dependence, 95(Suppl 1),
S45–S59.
Pluye P., Potvin L., Denis J. L. (2004a) Making public health programs last: conceptualizing sustainability. Evaluation and
Program Planning, 27, 121–133.
Pluye P., Potvin L., Denis J. L., Pelletier J. (2004b) Program sustainability: focus on organizational routines. Health
Promotion International, 19, 489–500.
Pluye P., Potvin L., Denis J. L., Pelletier J., Mannoni C. (2005)
Program sustainability begins with the first events. Evaluation
and Program Planning, 28, 123–137
Poduska J. M., Kellam S. G., Wang W., Brown C. H., Ialongo N.
S., Toyinbo P. (2008) Impact of the Good Behavior Game, a
universal classroom-based behavior intervention, on young
adult service use for problems with emotions, behavior,
drugs or alcohol. Drug and Alcohol Dependence, 95(Suppl
1), S29–S44.
Polit D. F., Beck C. T. (2004) Nursing Research, Principles and
Methods. JB Lippincott Co, Philadelphia.
90
Savaya R., Spiro S. (2011) Predictors of sustainability of social
programs. American Journal of Evaluation, 33, 26–43.
Savaya R., Spiro S., Elran-Barak R. (2008) Sustainability
of Social Programs. American Journal of Evaluation, 29,
478–493.
Scheirer M. A. (2005) Is sustainability possible? A review and
commentary on empirical studies of program sustainability.
American Journal of Evaluation, 26, 320–347.
Scheirer M. A., Dearing J. W. (2011) An agenda for research on
the sustainability of public health programs. American
Journal of Public Health, 101, 2059–2067.
Scheirer M. A., Hartling G., Hagerman D. (2008) Defining sustainability outcomes of health programs: illustrations from
an on-line survey. Evaluation and Program Planning, 31,
335–346.
Shediac-Rizkallah M. C., Bone L. R. (1998) Planning for the sustainability of community-based health programs: conceptual
frameworks and future directions for research, practice and
policy. Health Education Research, 13, 87–108.
St. Leger L. (2005) Questioning sustainability in health promotion
projects and programs. Health Promotion International, 20,
317–319.
M. A. M. Dijkman et al.
Steckler A., Goodman R. M. (1989) How to institutionalize
health promotion programs. American Journal of Health
Promotion., 3, 34–43.
Stirman S. W., Kimberly J., Cook N., Calloway A., Castro F.,
Charns M. (2012) The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science, 7. http://
www.implementationscience.com/content/7/1/17
Tingstrom D. H., Sterling-Turner H. E., Wilczynski S. M. (2006)
The Good Behavior Game: 1969–2002. Behavior Modification,
30, 225–253.
Van Lier P. A., Muthen B. O., van der Sar R. M., Crijnen A. A.
(2004) Preventing disruptive behavior in elementary schoolchildren: impact of a universal classroom-based intervention. Journal of Consulting and Clinical Psychology, 72,
467–478.
Van Lier P. A., Huizink A., Crijnen A. (2009) Impact of a preventive intervention targeting childhood disruptive behavior problems on tobacco and alcohol initiation from age 10 to 13
years. Drug and Alcohol Dependence, 100, 228–233.
Yin R. K. (1979) Changing Urban Bureaucracies: How New
Practices Become Routinized. Lexington Books, Lexington.