Measuring outcomes of disability related services and supports Erin Wilson (Deakin University), Nick Hagiliassis and Robert Campain (Scope)
CHASE Symposium, Deakin Prime, Feb 2013, Melbourne
Presentation overview
• Key questions: – Are people with a disability experiencing the lives they want?
– How do we measure this?
• Summarise a body of research by Scope and Deakin that has developed research methods, and collected outcomes data in a range of disability services types and cohorts.
–
–
–
–
Summarise methods thinking
Reflections on ‘what’ to measure
Reflections on ‘how’ to measure
Briefly give some examples from data collection tools
Terminology – key elements
What are ‘outcomes’?
In broadest terms, ‘outcome’ means: what happened
BUT different things are focused on and there is confusion over terms and their relationship:
1. PROCESS: did we receive or deliver a quality service? The ways in which program services and goods are provided.
2. OUTPUT: how much service did we receive or deliver, how many people received it etc? The goods and services produced (quantity, efficiency: e.g., number of people served, speed of response to complaints). 3. OUTCOME: what happened (for the person)? The broader results achieved through the provision of goods and services (Horsch, 2005)
Steps in developing an outcomes measurement approach for human services
1. Decide the desired areas of change – i.e desired outcomes and for whom
2. Define these areas & unpack complex terms – eg. what do we mean by ‘communication competence’, ‘awareness’, ‘community inclusion’, ‘support’, ‘partnership’?
3. Identify the indicators of these changes
– how will we know, what will we see if change occurs?
4. Decide how these indicators can be measured – How to collect data. How to analyse data. Do measures exist? Can existing measures be modified? Do measures need to be developed? 5. Combine and present the resulting information in a clear and informative way. (Adapted from ABS, 2001, ch 1).
Steps of outcomes measurement Task
1. Decide the desired areas of change (what are the outcomes?)
Check if specific outcome areas are identified by funder, government policy, or by others such as individuals who set specific goals for services. Consult stakeholders
2. Define these areas & unpack complex terms Check published literature for concept definition and sub domains of concepts. Consult stakeholders
3. Identify the indicators of these changes
Check published literature to identify
indicators; consult with stakeholders (practitioners and service recipients)
4. Decide how these indicators can be measured Check published literature for existing data collection instruments and methods; consult stakeholders; design methods
5. Combine and present the resulting information Report and publication writing
An example: 1 in 4 Poll Social Inclusion
1. Decide the desired areas of change
Social inclusion
2. Define these areas & unpack complex terms Social exclusion
Service exclusion
3. Identify the indicators
Economic exclusion
Social exclusion e.g., Having social contact with other people
(Saunders et al 2007a, 2007b & 2008)
Service exclusion e.g., Accessing medical services Economic exclusion e.g., Having enough money to get by on
4. Design how to measure
The 1 in 4 Social Inclusion Survey (example items from on line and paper based survey in Easy English)
Reflections on ‘what’ to measure
• Needs to be meaningful => Whole of life rather than focus on single aspects (to enable capture of what is important to people most affected)
• Need to identify ways to improve => Include a focus on what affects outcomes – barriers and enablers – so as to inform future service delivery
Outcomes across whole of life focus: Outcomes and Impacts tool based on the life domains framework (Wilson, 2009) – example items
Reflections on ‘how’ to measure
Methods need to:
• Enable self report – capture the view point of a person with a disability (proxy to report from the viewpoint of person they represent)
• Be responsive – sensitive to small changes
• Have low respondent burden – brief and easy to use
• Be accessible and have alternate formats if needed
• Have low administrative burden – can be administered within time available to service providers, and low amounts of time for analysis
• Be reliable, valid
• Identify different respondent views – distinguish between person with disability (service recipient), staff, and others
Examples of accessibility
•
Achieving accessibility; value in engaging Easy English experts to translate content; checked by people with intellectual disability; – Eg. 1 in 4 poll:
High ‘user friendliness’ ratings (mean rating 8.5);
Examples of self report
•
Ensuring the perspective of the person with a disability is captured;
–
1 in 4 poll: 67.4% of respondents completed the survey independently; 15% had carers complete the survey on their behalf. The survey is being completed by:

A person with disability without assistance
\

A person with a disability with assistance

A carer of person with disability (someone speaking on behalf of a person with a disability who cannot express their views themselves)
Where support is provided, the survey should be completed from the person with a disability’s view point
Examples of self report
•
Ensuring the perspective of the person with a disability is captured;
– MOSS: offer three rating scales in different formats to enable individuals to select which one is meaningful to them. Accessible for some people \
with intellectual disability . Issues we’ve confronted
• Selecting the ‘right’ indices of measurement (eg. Performance, achievement, satisfaction, impact, difficulty, etc)
• Aggregating data – moving from a focus on an individual’s outcomes (highly individualised) to whole of service / population outcomes
• Data analysis – difficulties in quantifiers, thematising
etc – time consuming and often highly subjective
• Administration compliance – getting staff to administer data collection tools (correctly)
• And more…..
Data collection and analysis tools developed and trialed
Scope Outcomes Framework (used for analysis)
• A whole of life outcomes framework across seven life domains
Scope Outcomes and Impacts tool
• A one page survey rating scale to measure impact (positive and negative) of service on life domains of individual or family
Measurement of Services and Supports (MOSS) tool
• A pre and post measure of change resulting from individual goal setting and services based on this goal
Family Stress and Coping tool AND Family Capacity tool
• Two survey scales rating a family’s self perceived level of coping and capacity
Early Childhood Individualised Support Plan tool
• Attachment to Individualised Support Plan to rate outcomes at end of service period, and barriers and enablers to these
Individualised Planning/Review Tool
• A tool based on the categories of a person centred plan to rate achievement against goals
1 in 4 poll Social Inclusion Survey
• A population level survey rating levels of social inclusion related to service exclusion, economic exclusion and social exclusion
Relevant publications by research team
Campain, R.& Wilson, E. (under review) Life is what happens while you’re busy making plans: issues in the development of outcome measurement for individualised plans for people with disability.
Quilliam, C. & Wilson, E. (2011). Literature Review ‐ Outcomes measurement in disability services: a review of policy contexts, measurement approaches and selected measurement tools. Melbourne: Deakin University. Hagiliassis, N., Nicola‐Richmond, K., Mackay, A., & Wilson, E. (2011). Administration Manual ‐
Measuring Outcomes in Services and Supports (MOSS) tool: a means to assess the outcomes of person‐directed goals. Melbourne: Scope.
Nicola‐Richmond, K., Wilson, E., Hagiliassis, N., & Mackay, A. (2011). Summary Sheet ‐ Measuring Outcomes in Services and Supports (MOSS) tool. Melbourne: Scope.
Quilliam, C., Wilson, E., Hagiliassis, N. & Nicola‐Richmond, K. (2010). Measurement of Services and Supports (MOSS): Project Overview and Findings. Melbourne: Deakin University.
Wilson, E. & Campain, R. (2011), Above and Beyond: Exploring outcomes and practices of Scope Southern Region Early Childhood Intervention Service for children with disability, Melbourne: Scope (Vic). Wilson, E. & Campain, R. (2008). Problem solving, people skills, and personalised arrangements. Early results of research into individualised approaches for young adults receiving ‘Futures’ funding at Scope. C.Bigby & C. Fyffe (Eds.). Achieving their own lives. Proceeding from the Third Intellectual Disability Policy Roundtable, La Trobe University, October
Wilson, E. (2006). Defining and measuring the outcomes of inclusive community for people with disability, their families and the communities with whom they engage. C. Bigby; C. Fyffe & J. Mansell (Eds). From Ideology to Reality: Current issues in implementation of intellectual disability policy. Proceedings of the Roundtable on Intellectual Disability Policy. Bundoora: La Trobe University
Wilson, E.; Jenkin, E. & Campain, R. (2011). Outcome Measurement of Community Based Mental Health Services in Western Australia: Literature and Concept Summary. Melbourne: Inclusion Matters PLUS many conferences and workshops and seminars
References
ABS (2001) 4160.0 Measuring Wellbeing: Frameworks for Australian Social Statistics. Viewed at: http://www.abs.gov.au/Ausstats/[email protected]/0/0120F9081822AAEBCA256B5F00804216?Open, on 31 October 2005.
Andreson E 2000, Tools of disability outcomes research: criteria for assessing the tools of disability outcomes research, Archives of physical medicine and rehabilitation, Vol. 81, No. 12, pp. S1‐S4. Horsch, K. (2005). Indicators: Definition and use in a Results‐Based Accountability System. Harvard Family Research Project viewed at http://www.gse.harvard.edu/hfrp/pubs/onlinepubs/rrb/indicators.html on 31 Oct 2005.
Saunders, P., Naidoo, Y. & Griffiths, M. (2008). Left out and missing out: Disability and disadvantage. Social Policy Research Centre, Sydney, Australia.
Saunders, P., Naidoo, Y. & Griffiths, M. (2007a). Deprivation & social exclusion in Australia. Social Policy Research Centre, Sydney, Australia.
Saunders, P., Naidoo, Y. & Griffiths, M. (2007b). Towards new indicators of disadvantage: Deprivation and social exclusion in Australia. Social Policy Research Centre, Sydney, Australia.