Qualita?ve Evalua?on in Health Care Developing your protocol 29 October 2013 09:00-­‐10:00 (Sweden Stockholm ?me) Arrie Odendaal Medical Research Council of South Africa All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐
Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org project website: h/p://www.arcade-­‐project.org/ @ARCADEprojects 1. 
2. 
Lecture readings It is not necessary to read beforehand but please bring to the lecture Please prepare these ar?cles for the wri/en exam Prior to dra@ing the protocol Bryson, J. M., PaHon, M. Q. and Bowman, R. A. (2011). Working with evaluaQon stakeholders: A raQonale, step-­‐wise approach and toolkit. Evalua2on and Program Planning, 34 (1), 1-­‐12. Visualising the programme Hasson, H. (2010). SystemaQc evaluaQon of implementaQon fidelity of complex intervenQons in health and social care. Implementa2on Science, 5(67), 1-­‐9. Asking, Answering and Analysing the quesQons Saunders, R. P., Evans, M. H. and Joshi, P. (2005). Developing a process-­‐evaluaQon plan for assessing health promoQon program implementaQon: A How-­‐To Guide. Health Promo2on Prac2ce, 6(2), 134-­‐147 1. 
Please bring to the lecture Doc 1: Proposal content page; Doc 2: Example of Protocol; Doc 3: Example of field notes All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐
Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org Lecture outline 1.  Principles and three key elements of your protocol •  Review examples from the literature 2.  PracQse quesQon formulaQon 3.  Reflect on your Content page All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐
Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org project website: h/p://www.arcade-­‐project.org/ @ARCADEprojects The learning objec?ves are: •  To introduce you to the issues to keep in mind when dra@ing your evaluaQon protocol. •  To criQcally review examples from the literature on protocol development. •  To help you reflect on a protocol for the research study that you submiHed beforehand. All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org Discussion: Protocol development Proposal (for commissioning agency/ethics) = Protocol + literature review + budget + references [see Doc 1: Proposal content page/Doc 2: Example of Protocol] •  EvaluaQon = Comparison = Asking quesQons •  Protocol = a detailed ‘road map’ to compare the ‘paper’ of the intervenQon/
programme with the ‘pracQse’ thereof. Principles 1.  Involvement of evaluaQon stakeholders 1.  Those who commission and those to become co-­‐evaluators 2.  Awareness of apprehension towards the evaluaQon 2.  Phrasing of evaluaQon: not judgement (fail or pass) but to learn how it works and “demysQfying the black box”. All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org Discussion: Protocol development Principles 3.  TheoreQcal underpinning/jusQficaQon for the ‘how’ of the evaluaQon 4.  IteraQve and flexible process 5.  As detailed as possible 6.  Journaling … journaling … journaling: nothing to trivial [See Doc 3: Example of field notes] 7.  Peer review throughout All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org Protocol checklist Step 1: Scoping the programme 1a: Analysis of evaluaQon stakeholders (Bryson et al., 2011) •  All who can affect and/or are affected by the evaluaQon and/
or its findings •  Diverse, therefore needs focus: “primary users” (see p. 2) •  Power and Interest: Fig 2 •  Programme theory (see Astbury and Leew, 2010) •  Fig 4: pracQcal and note their note in the capQon “Cycle back and revise …” All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org Protocol checklist Step 1: Scoping the programme 1a: Analysis of evaluaQon stakeholders (Bryson et al., 2011) •  NB: Throughout all of the evaluaQon phases •  AmbiQous …RealWorld Evalua2on (Bamberger et al., 2012): dealing with constraints Prac2cal example: MulQsite RCT: a home visitaQon intervenQon to reduce risks to unintenQonal childhood injuries •  Who will be the evaluaQon stakeholders? •  Omnipresence of evaluator: An ‘Eureka’ moment when they began volunteering informaQon All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org Protocol checklist 1b: Visual display of the programme (Hasson, 2010) •  Flowcharts, tables, text to connect the dots •  Moderators shaping each other; moderators shaping fidelity … not a linear relaQonship •  The jargon of scienQsts can be confusing: Ø  Context/Mechanisms/Outcome paHern configuraQons (Pawson and Tilley, 2004) Ø  Figure 1 and Table 1 (Saunders et al., 2005) •  … but the essence stays the same: it is all about the art of formulaQng quesQons Exercise: Evalua2ng lay community health workers’ use of mobile phones (mHealth) to record their services across 3 sites •  What are the quesQons you would want to answer through your qualitaQve evaluaQon? All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org Protocol checklist Step 2: How are you going to answer the ques?ons and what are you going to do with the answers •  Table 4 (Saunders et al., 2005) to plot your evaluaQon Exercise: mHealth programme -­‐ ques2ons around the training Let’s assume it is the same trainer, the training is done per site and one of the quesQons was: ‘Are there differences in how the training is conducted across the sites?’ •  How and from whom will you collect the answers? Ø  Structuring the observaQons (see Bouffard et al. 2003; Doc 3) Ø How do you propose to analyse observaQons? •  Can we do without mixing quanQtaQve data collecQon into the pot? All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org To be included in your proposal Step 3: Ensure that the other components are detailed in your proposal a.  A theoreQcal framework for the evaluaQon •  4th GeneraQon (Guba and Lincoln) •  UQlisaQon-­‐focused (PaHon) •  Realist (Pawson and Tilley) •  RealWorld (Bamberger) •  And others b.  ReporQng and disseminaQon c.  Ethics d.  Budget e.  Time frame All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org The integrity of the profession of evaluaQon “rests firmly on the quality of criQcal thinking exhibited by its pracQQoners” (PaHon, M.Q. (1987). CreaQve evaluaQon (2nd ed.), SAGE, California, p. 35) “Never expect to know “what works,” just keep trying to find out” Pawson, R., & Tilley, N. (2001). RealisQc evaluaQon bloodlines. American Journal of Evalua2on, 22, 317-­‐324 All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org Recommended readings Astbury, B. and Leeuw, F. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluaQon. American Journal of Evalua2on, 31(3), 363-­‐381 Bamberger, M., Rugh, J. and Mabry, L. RealWorld evaluaQon: working under budget, Qme, data and poliQcal constraints. 2nd ed., 2012, Thousand Oaks, California: Sage Bouffard, J. A., Taxman, F. S. and Silverman R. (2003). Improving process evaluaQons of correcQonal programs by using a comprehensive evaluaQon methodology. Evalua2on and Program Planning, 26(2), 149-­‐161 Guba, E. G. and Lincoln, Y. S. (2001). Guidelines for construcQvist (a.k.a.) Fourth GeneraQon) evaluaQon. Available from: hHp://dmeforpeace.org/sites/default/files/Guba%20and
%20Lincoln_ConstrucQvist%20EvaluaQon.pdf Pawson, R., & Tilley, N. (2004). Realist evaluaQon. Available from: hHp://www.communitymaHers.com.au/RE_chapter.pdf All course materials are published by the ARCADE Projects under the Crea?ve Commons A/ribu?on Non-­‐Commercial‑ShareAlike 3.0 Unported license. h/p://www.arcade-­‐project.org