The National Community Inclusion Initiative: Building the Capacity of Organizations to Evaluate Systems Change Jason Newberry, PhD Senior Researcher Centre for Community Based Research Kitchener, ON Jaime Lee Brown PhD Candidate Department of Psychology University of Guelph Guelph, ON Canadian Evaluation Society Quebec City, May 13th, 2008 www.communitybasedresearch.ca We are a non-profit and independent communitybased research organization located in Kitchener, Ontario, Canada. We engage in diverse streams of work, including applied research and evaluation, systems change initiatives, training & education, facilitation & planning, and international consulting. Our values emphasize community participation, action, and relevance in the work we do. Our focus areas include: • Mental Health & Addictions, Disabilities, Family Support & Early Child Development, Cultural Diversity, Organizational Change. Our evaluation experience is extensive: • small, local, program-focused evaluations. • Large, national, multi-sector evaluations concerned with systems change and complex interventions. • Organization evaluation capacity building and education. • Many different types of interventions and initiatives – individualized programs, group programs, educational initiatives, policy and systems change, community development. The Purpose of Today’s Presentation Introduce the National Community Inclusion Initiative and its goals. Describe the evaluation needs of the Canadian Association for Community Living and its national partners. Review our approach to evaluating systems change. Discuss our findings to date, future work, and general implications for evaluating complex, systems-level interventions. What is the Community Inclusion Initiative? A national initiative composed of P/T Associations for Community Living and People First of Canada, from 1998 to the present. Funded by HRSDC. Across the country ACLs have been partnering with communities and other organizations to address fundamental barriers to inclusion of people with intellectual. The Goal: “To build capacity that communities require to successfully include people with intellectual disabilities in ways that promote their roles as full citizens in society” A Major Learning and a Major Transition The initiative has reached many people – thousands of families in 600 communities – but the partners recognized that it was a constant uphill battle. CI helped people on an individual or limited organizational level, but systems remained the same – they perpetuated exclusion. The initiative engaged in a transformation so that their work focused on changing systems. Evaluating Programs vs. Systems Single, small, local program complexity Multiple programs, sites, focus on systems • focus on benefits / changes to individual people • focuses on changes to systems to in turn benefit people • direct control over activities & infl. on outcomes • less control over activities; far less control over the way people will eventually influenced. • theory of change is specific & targeted • most often, outcomes are known in advance • implementation errs on side of consistency • theory of change is general • outcomes are often not known in specific sense • implementation is strategic but may deviate from plan: opportunistic & flexible These differences reflect different levels of ecological systems theory. Systems change intervenes at a higher ecological level. Prevention-focus, community or population-level target. An Example in Community Inclusion – Inclusive Education Individual Model Working with individual families to teach them how to navigate the system, know their rights, etc. Advocating with individual teachers and schools to address exclusion. Systems Model Partnering with teachers unions, boards, principals to promote inclusion. Developing best practices and policy positions and working with government to implement them. The challenge of evaluating systems change Systems change evolves – its difficult to specify many activities and outcomes in advance. Activities are many steps removed from actual benefits to people. Little control over specific impact. Measurement & design is challenging b/c theory of change evolves over time. Gov’t & other funders priviledge pre-specified interventions and a priori evaluations. At outset, limited to making connections between general forms of interventions and general outcomes. “If we do A, B, and C, good things will tend to happen”. It’s really the same rationale as to why we go to conferences like CES. How do we address these difficulties? Create an overarching systems-level logic model that describes what the systems change theory looks like in a general sense. Apply this general model to local settings & initiatives. Get specific and concrete by developing very detailed “implementation chains”. Detail the theory of change – what worked, what did not – “on the fly”. Collect supporting data. Focus on tracking implementation and short-term changes. For example… “Effective Partnerships” are viewed as key mechanisms for systems change. But the term is vague and so is practice. We must track what actually happens in partnerships to see if best practices emerge that go beyond basic understandings. We have to know how partnerships link to other aspects of the systems change initiative – to the community, to policy, to research, etc. Building a theory of systems change requires ongoing, flexible, developmental evaluation. The Role CCBR – Part I 1. 2. 3. Revisit and review past learnings of Community Inclusion initiatives & community priorities. Develop a framework that captures systemslevel change and helps drive project planning, monitoring and evaluation. Support national partners to develop systemsfocused community inclusion projects consistent with the framewok. The Challenge: Building a Theory of Social Innovation & Systems Change We needed to develop a framework that: created clarity about how projects are intervening in order to change systems. described long “implementation chains” – simply put, lots has to happen, and in order, to get to systems change. What needed to track the sequence of systems change activities showed commonalities and differences between P/Ts. demonstrate how CI can be viewed as a unified national initiative (how P/Ts and local projects share common goals and work toward them in similar ways). generate and share innovative strategies in systems change to improve practice and evaluation. COMMUNITY INCLUSION Guiding Principle s CI Mission To foster citizenship engagement in development and implementation of public policies, practices, and community systems to strengthen and sustain the capacities of local communities… Self-determination Equality & Equity Priority Areas Autonomy & decision-making Community-based interventions Deinstitutionalization Education Support for personal relationships Participation & Full Citizenship Income & Employment •Public Awareness & Dialogue (e.g. community forums, plain Common Mechanisms Housing Social & Economic Integration Accessible communities Family & Disability Supports Community Associational Life & Capacity •Research & Information (e.g., research studies) •Knowledge Networking (e.g., conferences) •Policy Development (e.g. policy position papers) language) •Community Capacity Building (e.g. developing partnerships, leadership, training, strengthening PF chapters) Changes to resources, practices, relationships, etc. Outputs & Short-term Outcomes Long-term Outcomes Goal Resources • Creation, access to, and use of new learning & action resources • Sustainable finances • Community space Policy/practice Creation of or changes to policies supportive of inclusion; changes in practice Collaboration/N etworks Positive change & maintenance in participation, group capacity, operations, climate, etc. Knowledge creation & sharing Strategies for research & evaluation, knowledge dissemination, mobilization, networks, awareness Changes to Individuals Increases/improvements in… • knowledge & skills, learning & sharing • motivation, participation, engagement • advocacy & leadership • inclusive practices • giving and receiving support More INCLUSIVE ENVIRONMENTS and corresponding improvements to… • Safety • Education • Family & Social support • Personal Relationships • Economic security Community & Civic Participation • Health & Quality of Life • • Local communities are fully inclusive of people with intellectual disabilities and their families. COMMUNITY INCLUSION Guiding Principle s CI Mission To foster citizenship engagement in development and implementation of public policies, practices, and community systems to strengthen and sustain the capacities of local communities… Self-determination Equality & Equity Priority Areas Autonomy & decision-making Community-based interventions Deinstitutionalization Education Support for personal relationships Participation & Full Citizenship Income & Employment •Public Awareness & Dialogue (e.g. community forums, plain Common Mechanisms language) •Community Capacity Building (e.g. developing partnerships, Housing Social & Economic Integration Accessible communities Family & Disability Supports Community Associational Life & Capacity •Research & Information (e.g., research studies) •Knowledge Networking (e.g., conferences) •Policy Development (e.g. policy position papers) leadership, training, strengthening PF chapters) Individuals changes… Changes to structures and processes… Outputs & Short-term Outcomes Community Outcomes (changes experienced by individuals & families) Goal Resources • Creation, access to, and use of new learning & action resources • Sustainable finances • Community space Policy/practice Creation of or changes to policies supportive of inclusion; changes in practice Collaboration/N etworks Positive change & maintenance in participation, group capacity, operations, climate, etc. Knowledge creation & sharing Strategies for research & evaluation, knowledge dissemination, mobilization, networks, awareness Increases/improvements in… • knowledge & skills, learning & sharing • motivation, participation, engagement • advocacy & leadership • inclusive practices • giving and receiving support More INCLUSIVE ENVIRONMENTS and corresponding improvements to… • Safety • Education • Family & Social support • Personal Relationships • Economic security Community & Civic Participation • Health & Quality of Life • • Local communities are fully inclusive of people with intellectual disabilities and their families. There are six Priority Areas (or Sectors) that make up the National Community Inclusion Project. Priority Areas Deinstitutionalization Education Income & Employment Housing Family & Disability Supports Community Associational Life & Capacity • Multiple P/Ts are developing projects within each of the six priority areas. Each P/T is engaging in at least one priority, but often more than one. • The six areas share the same overall mission/goal of CI, but also have their own goals as well. • Each priority area has its own visual model (which we will get to) that is consistent with (and very similar to) the overall framework. COMMUNITY INCLUSION Guiding Principle s CI Mission To foster citizenship engagement in development and implementation of public policies, practices, and community systems to strengthen and sustain the capacities of local communities… Self-determination Equality & Equity Priority Areas Autonomy & decision-making Community-based interventions Deinstitutionalization Education Support for personal relationships Participation & Full Citizenship Income & Employment •Public Awareness & Dialogue (e.g. community forums, plain Common Mechanisms language) •Community Capacity Building (e.g. developing partnerships, Housing Social & Economic Integration Accessible communities Family & Disability Supports Community Associational Life & Capacity •Research & Information (e.g., research studies) •Knowledge Networking (e.g., conferences) •Policy Development (e.g. policy position papers) leadership, training, strengthening PF chapters) Individuals changes… Changes to structures and processes… System outcomes Community Outcomes (changes experienced by individuals & families) Goal Resources • Creation, access to, and use of new learning & action resources • Sustainable finances • Community space Policy/practice Creation of or changes to policies supportive of inclusion; changes in practice Collaboration/N etworks Positive change & maintenance in participation, group capacity, operations, climate, etc. Knowledge creation & sharing Strategies for research & evaluation, knowledge dissemination, mobilization, networks, awareness Increases/improvements in… • knowledge & skills, learning & sharing • motivation, participation, engagement • advocacy & leadership • inclusive practices • giving and receiving support More INCLUSIVE ENVIRONMENTS and corresponding improvements to… • Safety • Education • Family & Social support • Personal Relationships • Economic security Community & Civic Participation • Health & Quality of Life • • Local communities are fully inclusive of people with intellectual disabilities and their families. What are mechanisms for…? Common Mechanisms •Public Awareness & Dialogue (e.g. •Research & Information (e.g., research community forums, plain language) •Community Capacity Building (e.g. developing partnerships, leadership, training, strengthening PF chapters) studies) •Knowledge Networking (e.g., conferences) •Policy Development (e.g., policy position papers, policy advocacy) Mechanisms provide some guidance to for P/Ts on how and where to focus their efforts – they lead to more specific and concrete activities. They help unify the projects together under general areas of systems intervention They allow us to see where our efforts are being focused in each sector and help us organize information in away that will facilitate sharing, collaboration, proposal writing, and reporting. A main objective of our sector forums is to discuss specific CI activities in the COMMUNITY INCLUSION Guiding Principle s CI Mission To foster citizenship engagement in development and implementation of public policies, practices, and community systems to strengthen and sustain the capacities of local communities… Self-determination Equality & Equity Priority Areas Autonomy & decision-making Community-based interventions Deinstitutionalization Education Support for personal relationships Participation & Full Citizenship Income & Employment •Public Awareness & Dialogue (e.g. community forums, plain Common Mechanisms language) •Community Capacity Building (e.g. developing partnerships, Housing Social & Economic Integration Accessible communities Family & Disability Supports Community Associational Life & Capacity •Research & Information (e.g., research studies) •Knowledge Networking (e.g., conferences) •Policy Development (e.g. policy position papers) leadership, training, strengthening PF chapters) Individuals changes… Changes to structures and processes… Outputs & Short-term Outcomes Community Outcomes (changes experienced by individuals & families) Goal Resources • Creation, access to, and use of new learning & action resources • Sustainable finances • Community space Policy/practice Creation of or changes to policies supportive of inclusion; changes in practice Collaboration/N etworks Positive change & maintenance in participation, group capacity, operations, climate, etc. Knowledge creation & sharing Strategies for research & evaluation, knowledge dissemination, mobilization, networks, awareness Increases/improvements in… • knowledge & skills, learning & sharing • motivation, participation, engagement • advocacy & leadership • inclusive practices • giving and receiving support More INCLUSIVE ENVIRONMENTS and corresponding improvements to… • Safety • Education • Family & Social support • Personal Relationships • Economic security Community & Civic Participation • Health & Quality of Life • • Local communities are fully inclusive of people with intellectual disabilities and their families. Outputs & Short Term Outcomes contributing to Systems Change… •Systems change can be thought of as “changing the way things are done, but systemically”. •In the short-term this means changing the day-to-day functioning of people and the organizations that they are in, creating/providing new resources, strategically bringing people together around an issue, or developing new policies and practices (organizationally, legislatively, etc.). • Another way to describe it is to describe changes to structures (e.g., the creation of a coalition), processes (e.g., who is talking to who and about what). We are also interested in the things new things that are created and used in relation to this change (new resources, policy statements, media, etc.) Changes to structures and processes… Resources • Creation, access to, and use of new learning & action resources • Sustainable finances • Community space Policy/ practice Creation of or changes to policies supportive of inclusion; changes in practices Collaboration/ Knowledge creation Networks & sharing Positive change & maintenance in participation, group capacity, operations, climate, etc. Strategies for research & evaluation, knowledge dissemination, mobilization, networks, awareness These are general categories. Specific initiatives describe specific changes, resources, relationships, etc. Short-term outcomes – benefits or changes to people – follow from changes to structures & processes. For example… a new resource is created… …which leads to increased awareness of barriers to inclusion… …and used by a new coalition Individuals changes… Changes to structures and processes… Resources • Creation, access to, and use of new learning & action resources • Sustainable finances • Community space Policy/practice Creation of or changes to policies supportive of inclusion; changes in practice Collaboration/N etworks Positive change & maintenance in participation, group capacity, operations, climate, etc. Knowledge creation & sharing Strategies for research & evaluation, knowledge dissemination, mobilization, networks, awareness Increases/improvements in… • knowledge & skills, learning & sharing • motivation, participation, engagement • advocacy & leadership • inclusive practices • giving and receiving support •All this work to modify existing systems is in the service of ultimately improving our communities. •These improvements are represented by the experiences of community members – across all people that stand to benefit, are there improvements to safety, education, income security, etc? In our model, these are called “community outcomes”. Community Outcomes (changes experienced by individuals & families) More INCLUSIVE ENVIRONMENTS and corresponding improvements to… • Safety • Education • Family & Social support • Personal Relationships • Economic security • Community & Civic Participation • Health & Quality of Life • The Role CCBR – Part II Once an overall framework was in place, we needed to apply it to the P/T partners’ work. In 5 main sectors, we developed “systemlevel” logic models for each P/T. Thus began the cross-country journey…. Sectors by Provinces & Territories Inc/Emp – 2 Fam/Dis Supp – 4 Inclusive Ed. – 3 Deinstit. – 2 Comm/Assoc - 1 Community Life & Capacity Family & Disability Supports Income & Employment Family & Disability Supports Income & Employment Inclusive Education Family & Disability Supports Inclusive Education Family & Disability Supports* Inclusive Education Deinstitutionalization Running back to Sasktoon! Miles Canyon, Whitehorse Yellowknife, NT St. John’s, NL The Uses of Systems Level Logic Models Promote learning and critical reflection about systems change work. Provide ideas about what to measure, how, and when. Helps communicate complex initiatives within organizations and externally. Helps generate ideas of next steps in pursuing systems change. Mechanisms Project activities Outputs & Actions: (changes to resources, practices, relation-ships) Short Term Outcomes Community Capacity Building Create Inclusive Education Working Group Who: Reps. from Dept. of Ed., CONA, NLACL. When: Monthly meetings (sep-mar. 0607) Actions: Create strategic workplan, guidance for focus group methods & content, forum content. Data: # of meetings, attendance, meeting actions, strategic plan. Create 4 regional partnership committees Who: Itinerants, emp agencies, family networks, local ACLs. When: prior to focus groups. Actions: With NLACL, set purpose & goals of focus groups, organized & promoted, addressed barriers to participation Data: committee membership & actions. Research & Information Public Awareness & Dialogue Conduct regional focus groups with stakeholders Who: Parents & educators When: Fall-Winter 06-07 Actions: • Presentation of past & current special ed. policy, context, challenges. • Discussion of experiences, current practices, ideal policy, gaps, strategies. • Summary report to IE working group NLACL & participants. • Participants added to “electronic learning community”. • Findings used to form policy position paper. Data: # of people, description, participant feedback re: meetings, distribution of report, use of report by IE committee. Increased knowledge of FG group participants re: IE principles, policies, and practice Mechanisms Project activities Outputs & Actions: (changes to resources, practices, relation-ships) Short Term Outcomes Policy Development Draft and circulate policy paper on Inclusive Education Who: NLACL & partners When: completed (date) Actions: • Prepare policy paper w/ vision of IE, experiences in NL & recommendations • Distribute to networks; make available online Data: # of copies sent, requested, downloaded, etc. Create and circulate a National Review of IE policy Who: NLACL When: completed (date) Action: • prepare cross-canada review • distribute @ Nat. IE forum, prov. forum, CACL, all CI projects, NB dept. of ed., IE website. Data: # of copies sent, requested, downloaded, etc. Project activities Outputs & Actions: (changes to resources, practices, relationships) Short Term Outcomes Provincial forum on Inclusive Education Who: NLACL, parents, educators, boards, DofE, HRLE ( & Min), Health Comm serivces, NLTA, Fac of Ed., Educ. students, student support specialists, Itinerants, CLO (tot. 38), When: Mar. 28-29, 2007. Actions: Present history of & current policy, strategies for IE, presentations. Data: # & description of attendees, indicators of future engagement & capacity (e.g., request), resources distributed, forum feedback Increased knowledge / awareness of IE principles & practice among participants Greater engagement in IE issues & practices Sharing of information from forum to others Engage w/ Dept of Education and Boards re: policy development & implementation Who: NLACL, Min. of Ed., ADM When: Upon policy paper completion & after IE forum. Actions: presentation & dialogue w/ minister, ADM. Link issue to FPT group. Data: Record of meeting discussion, policy opportunities, barriers, next steps. Greater engagement & action by decisionmakers on special education related policy Creation & revision of policies that support IE Disseminate special education policy & news Who: website users, CI & IE networks. When: Ongoing. Actions: Data: # of newsletters, distribution, activity on listserv (#, to whom, topics posted, discussed) - e.g, MHAs Increased general awareness of IE issues Increased participation & sharing in electronic learning community Mechanisms Project activities Outputs & Actions: (changes to resources, practices, relationships) Short Term Outcomes Knowledge Networking Provincial forum on Inclusive Education Who: NLACL, parents, educators, boards, DofE, HRLE ( & Min), Health Comm serivces, NLTA, Fac of Ed., Educ. students, student support specialists, Itinerants, CLO (tot. 38), When: Mar. 28-29, 2007. Actions: Present history of & current policy, strategies for IE, presentations. Data: # & description of attendees, indicators of future engagement & capacity (e.g., request), resources distributed, forum feedback Increased knowledge / awareness of IE principles & practice among participants Greater engagement in IE issues & practices Sharing of information from forum to others Policy Development Public Awareness & Dialogue Engage w/ Dept of Education and Boards re: policy development & implementation Disseminate special education policy & news Who: NLACL, Min. of Ed., ADM When: Upon policy paper completion & after IE forum. When: Ongoing. Actions: presentation dialogue w/ minister, ADM. Link issue to FPT group. Data: record of meeting discussion, policy opportunities, barriers, next steps. Who: website users, CI & IE networks. When: Ongoing. Actions: Data: # of newsletters, distribution, activity on listserv (#, to whom, topics posted, discussed) - e.g, MHAs Greater engagement & action by decisionmakers on special education related policy Increased general awareness of IE issues Creation & revision of policies that support IE Increased participation & sharing in electronic learning community Limit of 2007-2008 project activities Long-term outcomes Potential new partnerships, training, & action IE workshops for Student Educators Partnerships with NL Teachers Associaion (e.g, to deliver in-service training) Partnership with Facutly of Eduction for revision of Special Education Curriculum (e.g. course development) Subsequent knowledge & skillbased outcomes Broad public support for inclusive education – early childhood through to postsecondary Partnerships with CONA, promoting supportive services for postsecondary students. Partnerships with all disability organizations to achieve consensus on policy position Subsequent changes to IE policy and practice Elimination of segregation and special education classes as the primary place of instruction for students with disabilities Inclusion is the norm in classrooms, schools, and post-secondary education across the province Theories of System Change: A Selection of Lessons Learned & Promising Practices Policy Development & Engagement Policy development & engagement are strengthened when: Policy research and/or reviews are conducted in a way that is informed by local context. Policy development proceeds with the meaningful input or endorsement of diverse stakeholders. Policy recommendations are applicable to multiple stakeholder groups and/or organizations. Forums for discussion of implications of policy (i.e., it informs practice) follow initial recommendations. Policy documents or positions reach policy makers or other key stakeholders that hold positions of power. Conduct local research with those affected by policy. Having a multi-sector partnership group makes the recommendations more balanced & credible (not “special interest”) Diverse stakeholders can “work with the policy” – improves it, makes it actionable, raises awareness, builds broader consensus Multisector partnerships composed of leaders increase chances of “getting the ear” of policy makers. Partnerships Partnerships are improved when: There is relative consensus on values and principles guiding the partnership Members come from multiple levels of their home organizations (staff and management). They are composed of strong leaders who “have the ear” of their home organizational leadership. Partners are institutionally linked to the goals of the partnership. Diverse partners who have power over different parts of the system is best. Member organizations are linked to other key organizations that exist outside the partnership. Consensus building stage must come first High profile management are often too busy to commit; other effective leaders may exist in the organization. Multi-sector partners can impact different parts of the system. Multiple linkages create new opportunities. Partnerships Partnerships are improved when: They have mutually beneficial goals. Forming partnerships based on “values alone” is not sufficient. The work of the partnership should have practical benefit to all. The activities of the partnership are arranged around an action-oriented, capacity building agenda. For there to be system change, the partnership must be functional – its activities should lead to changes in the capacities and practices of others. The actions of the partnership are (primarily) evidence-based. The credibility and effectiveness of the partnership activities relies on evidence Workshops, Information Sessions, Family Networking: When do they support systems change? Family networking & leadership create a local A source of needs, and provincial base to draw information from. experiences, stories, data In communities that have undeveloped networks, information sharing is a good place to make initial connections with community. for policy work; creates a “political constituency” When workshops or info sessions are linked to provincial policies and practices, it forces the policy/practice “into action”. When workshops, sessions, skill development, and other forms of support are part of larger partnerships, expanded projects. Part of system change involves working from the bottom up – system change has to start somewhere. Policy and leglistation are useless if people don’t use them in practice. Sometimes it doesn’t, but its nonetheless helpful to families. If enough organizations start doing it, practice can precede policy The mandate of community inclusion is not exclusively systems change. Organizational Training – Inclusive Education as an example Teacher training and development appears to work best when: Training of teachers works best when closely linked to the context of the school. If teacher training is disconnected to the school, change organizational culture is difficult. Dialogue and consensus-building precedes and sets the stage for organizational training. Diversity is used as the lens, not disability There is institutional, high level buy-in to the project There is translation of values to practice – practical resources & strategies. New knowledge and skills are not used if they are inconsistent with individual & organizational values. Projects that are seen as “special interest” are seen as narrow. Diversity projects are broadly applicable. High level support increases reach, accountability and is policy directed. Moving from “you should do this” to “this is how you do it”. Supporting Research – What are some best practices? Research is effective – it gets attention, sparks dialogue, moves an agenda – when it is: Participatory and guided by the questions and needs of the communities. If research is grounded in the stated needs & experiences of community members it is more relevant to policy. Endorsed and guided by multiple sectors and organizations. Multiple stakeholders in research (co-ownership) means more opportunities for action. Based on data and information from multiple sources and perspectives. Multiple sources (or indicators) gives a clearer picture of the issue. Able to combine personal experiences with other data (e.g. broader survey data, community or population statistics). Linked to pilot or “demonstration projects”. Experiences add a human face to general data; Supporting data helps generalize the experiences. Policy makers & organizations want best practices and a “solution focus” in research Other Lessons in Pursuing Systems Change Building from the past - many accomplishments of the past year are based on previous partnerships, networking, research, etc. Systems change work relies on flexibility of planning and taking advantage of opportunities. Some of the most important work cannot be planned in advance. Systems change gains and sustains momentum when leadership is passed on and/or shared with other organizations. Train the trainer approaches with other organizations is a common way to build capacity and expand reach.
© Copyright 2025 Paperzz