Real Time Feedback System Pilot for
People Using Mental Health and
Addiction Services
Second Qualitative Report
Review of Survey, Version 1
June 2014
Provided to the Health and Disability Commission
and Advisory Group
TABLE OF CONTENTS
TABLE OF CONTENTS ..................................................................................................................................... 2
INTRODUCTION AND METHODOLOGY ..................................................................................................... 3
REVIEW OF CORE QUESTIONS .................................................................................................................... 4
ADDITIONAL QUESTIONS ............................................................................................................................ 11
RESPONSE OPTIONS ...................................................................................................................................... 13
MANAGING THE DEVICE ............................................................................................................................. 14
USE OF THE DEVICE IN SERVICES ............................................................................................................ 15
ANALYTICS....................................................................................................................................................... 17
CONCLUDING COMMENTS .......................................................................................................................... 18
SUMMARY OF CURRENT & POSSIBLE REVISED QUESTIONS FOR FURTHER DISCUSSION ... 19
APPENDIX A - SECOND SITE VISIT PLAN ................................................................................................ 21
APPENDIX B – DISCUSSION GUIDE- ADDING IN ADDITIONAL QUESTIONS ................................. 22
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 2
INTRODUCTION AND METHODOLOGY
This is the second qualitative report provided by CBG Health Research to inform how best to modify
questionnaire design and administration for the Real Time Feedback Pilot 2014. The final decisions
will be made in collaboration with the Health and Disability Commission (HDC) and key Stakeholders.
This report follows on from the first qualitative report on documenting consumer, family/ whanau
and provider feedback on question and survey design. This initial feedback is often relevant to issues
raised in this report and reference to the ‘first qualitative report’ are made throughout.
The first version of the survey was made available to all sites from the 28th April 2014. Sites were
visited in June 2014 allowing for at least 6 weeks of data collection. The majority of feedback in this
report was gathered from site visits and other opportunistic feedback. The Malatest evaluation
question has only been published for one week at the time of writing and has not, therefore, been
included. Live reports for this question are available on the public site
www.patientexperiencesurvey.co.nz
The site visit agenda:






Nature of core questions including, other questions or question types that should be
included
Possible additional questions to be added
The way that response options were presented
Exploring the language issues
Managing the device/updating the survey and its use within the service
Possible improvements on usefulness of analytics, including free text analysis
Following discussions with the HDC, advisory group and the independent evaluator Malatest, the
plan for this round of feedback was finalised, data collection then commenced (Appendix A). All sites
were sent a discussion guide (Appendix B) at least 4 weeks prior the scheduled site visit, detailing
the plan, site visit agenda and a bank of possible additional questions. All sites were then contacted
to ensure they understood the purpose of the meeting, discuss who should attend and to encourage
internal discussion on any additional questions prior to the meeting.
Contacting all identified persons attending the meetings was in most cases not possible, as most
sites were not able to supply names of people attending until close to the site visit date. It was
difficult for sites to decide who should be at the meetings due to the short time frame between data
collection commencing and the date of the visit. The pilot site guided who should be present at the
site visits from the following representative groups, with additional input from CBG.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 3

Consumers, family/ whanau



Pilot Site representatives
Providers from service areas where devices have been placed
Research analyst’s
The feedback presented in this report is in a general descriptive feedback form for each of the agenda
items detailed above. All feedback was presented back to pilot sites in a summary report and given
opportunity for validation.
REVIEW OF CORE QUESTIONS
GENERAL FEEDBACK
Feedback on the core questions was high on everyone’s agenda and most sites attached more
importance to getting this right than adding in their own additional questions.
All sites gave strong feedback that the core questions need further review. While some sites gave
individual feedback which differed from other contributing sites, it was completely unanimous that
the questions needed to be simplified due to an overall inconsistent understanding.
Although this had not necessarily resulted in people refusing to complete the survey, in fact all sites
reported a positive overall enthusiasm with few declines reported from people who were asked to
participate. However, many were asking for help and to have the meaning of the questions
explained.
This raised some concerns, highlighted in the following comments:
 “Taking away from the private experience in completing, by involving the therapist too
much”
 “Average time frame of 1-3 min could be a reflection that people are not thinking enough
about it as they are not understanding and simply picking a positive face.”
 “Very little differentiation between responses and ratings too positive, which doesn’t reflect
other feedback from other sources.”
Presenting the survey to people in a face-to-face context, such as at a home visit or therapy session in
itself raises concerns about how safe people feel to be fully truthful in their responses, but adding in
the verbal explanation of the questions, may further bias the ratings. The explanations being given
are also likely to vary between individuals and some providers may not fully comprehend the
questions either. While it is likely that some people will always need or request assistance in
completing the survey, simplifying the questions would help more people to complete the survey
independently.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 4
The lack of differentiation between questions and across sites raised a doubt for some sites, as to
how accurately these questions reflect the true picture. It was suggested that poor overall
comprehension of the questions may result in inadequate consideration of ratings and impulsive
selections being made. The suggestion that short completion time indicates lack of due consideration
is open to debate. In the first qualitative report it was shown a short survey was preferred by
consumers, family / whanau, and is therefore, a successful outcome. For people completing the
survey repeatedly we would expect a faster completion time due to familiarity with the survey layout
and the questions. With this in mind, the average timeframe to complete may not be the best
indicator of question comprehension.
One site had concerns that the survey could become another “meaningless tick box exercise”, and
“currently not a good survey”. The same site suggested that the ideal questions you would use every
session need to be highly functional in style and relate directly to that session. Top of mind examples
were provided below:
-
Do you think your clinician understood what is going on for you at the moment?
Do you feel your culture and beliefs were respected?
Did you talk about the things you wanted to talk about?
Do you feel as though you worked on things that are important to you?
Was today’s session useful for you?
In comparison the current questions were described as being on a larger scale, not so immediate and
functional in nature. Although they recognised them as important, they were not considered
appropriate to ask ,multiple times over short periods, This may warrant further exploration at a
later stage when a longer period of results are available and more people have been asked to
complete the survey multiple times.
Several sites suggested a ‘don’t know’ option, which could be applicable to some or all of the core
questions, this would allow people the option to state this, rather than picking a rating which does
not accurately represent their experiences. We should also bear in mind the questions are currently
compulsory and a rating must be selected in order to move on. A ‘don’t know’ option could be useful
data to evaluate question comprehension. Some specific questions were identified as warranting a
‘don’t know’ for other reasons which is discussed in more detail below. Consideration of this option
would need to discriminate between poor comprehension of the question as opposed to an adequate
understanding, but not knowing the answer. For this reason, two options of ‘don’t know’ or ‘don’t
understand the question’ maybe useful. The potential drawbacks are the danger of crowding the
questions with too many response options and giving people the easy option of a ‘don’t know’ rather
than forcing them to provide a rating and consider the question.
Everyone strongly agreed that the questions were too convoluted. The concept of an overall rating to
rate a domain of patient experience which may contain many elements was too difficult for
consumers to understand and also for providers to comprehend and explain. Particular questions
were identified as being more convoluted than others which are discussed later in this report.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 5
Feedback was that language (vocabulary and/or phrasing) was too complex for the majority, and was
not suitable for the following:




Youth or children
Poor literacy
English a second language
Cognitively/mentally impaired
Youth services may see people as young as 9yrs or even younger and it was suggested a more
appropriate separate survey with ‘youth friendly’ language would be preferred. However, as all adult
services had the same concerns about levels of understanding, simplifying the language is clearly
relevant for all age groups. One of the youth services mentioned they have done satisfaction
surveys in the past with children who have completed them independently, with good results.
Although sites appreciated the difficulty in achieving this, most sites mentioned questions that were
not applicable for some or all services within their organisation. This was a highly significant issue for
primary care and youth services in particular who felt the questions were not therapeutic. They
considered the questions to be more relevant for secondary care and services dealing with ‘illness’
or ‘recovery’ rather than preventive or primary care support services. This issue is discussed in more
detail when looking at feedback below on individual questions.
One site expressed frustration that it seemed as if the initial feedback from the sites, consumers and
family / whanau on question design was not taken into account and was largely ignored. It was felt
that this matter had the potential to impact negatively on the roll out of the pilot and had to be
managed internally. They reiterated it is essential the core questions reflect the experiences of their
services.
One site suggested additional or replacement core questions that looked at general satisfaction. In
keeping with feedback there would need to be an alternative to ‘service’. See discussion on Q5
below for a possible general satisfaction question.



How well is our service meeting your needs?
Overall my level of satisfaction (with the service) is…
In an overall, general sense, how satisfied are you with the service you receive?
QUESTION SPECIFIC FEEDBACK
In order to be applicable to all client groups within each pilot site ‘services or people who support
you’ could be included in an overall statement to introduce each individual question. This could be
stated once on the first page or above each question on each page. This is open to further
discussion.
Thinking about your most recent experience with the service/people who support you, how much do
you agree or disagree with the following statements….
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 6
Q1. The people I see within the service value my opinions and are compassionate, nonjudgemental and respectful of my cultural needs.
Identified by three sites as the worst question, comments included…






“Too confusing”
“Difficult to answer”
“Too many questions within a question to have at the start”
“Conceptually too dense, 4 elements in one question”
“People don’t know which element they are rating”
“Overwhelming as a first question, potential impact on people continuing”
Suggested alternative “staff listen to me and are thoughtful and kind”, but this also contains several
elements and is likely to receive similar criticism. The suggestion highlights the desire for simpler
language, however, and splits out cultural needs. It was stated that cultural needs are a core part of
what services should address and can’t be wrapped into this question with the other elements.
It was discussed in the first qualitative report that a separate cultural core question may not be
relevant to all people completing the survey. One site with a predominant Māori and Pacific
population felt that their services were fully immersed in the predominant culture and as this was
the norm, a cultural question was not required. This warrants further discussion with the advisory
group and HDC as to whether to include an additional core question on culture in the relationship
domain or as an amendment to Q1. Either way CBG would recommend this is a priority question for
review as this domain was given high significance to consumer’s family/ whanau in the first
qualitative report and identified as a highly problematic question in this review.
For Consideration
If an individual site or service considers a cultural question to be required, they have the option to
include as an additional question, but at this stage we need to be mindful of increasing the number
of core questions. Removing this element from this question seems sensible, as well as simplifying
the rest of the question.
Thinking about your most recent experience with the service/people who support you, how much do
you agree or disagree with the following statements….


I felt respected
I feel respected
Q2. I am sufficiently consulted and involved in decisions about services to meet my
identified needs
Similar feedback to Q1, too many elements to the question making it hard to interpret and too
convoluted.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 7
One site was of the opinion that the word ‘service’ may not be understood by everyone. Ideally the
actual name of the service would be preferred, but as a core question that needs to be applicable to
all, they suggested ‘organisation’ instead of ‘service’. There was no specific negative or positive
feedback cited in the first qualitative report on this word, but it was used repeatedly in questions
suggested by the sites. The word ‘services’ could be difficult for people who are involved in more
than one and being seen in a context where more than one service is available.
The word ‘sufficiently’ was criticised as “not simple enough”. A separate site suggested this word
could be redundant, as there is a rating scale which allows them to indicate if the level of
consultation was sufficient or not. CBG would recommend the removal of this word in light of this
feedback and to help simplify the question.
The phrase “meet my identified needs” was identified as not being needed by one site. They
suggested it could be removed and changed to “I am consulted and involved in decisions about
services”. Another site felt the word ‘identified’ was not needed, but didn’t have an issue with
“needs”. The word “needs” was generally well liked in the first round of qualitative research and
frequently appeared in site suggested questions. Initial feedback indicated it was a good alternative
to more illness orientated words such as ‘treatment’ and covered all, including practical issues.
For Consideration
The domain of Communication/Information has many aspects to it from actual information giving, to
collaborative care, services made available etc…and are very different questions. The current core
question is primarily asking about the level of collaboration with services. It may be useful to focus
more on the ideal outcomes from good communication/information and revisit the other initial
suggested question for this domain “I know what to do if I, or the person I am supporting, becomes
unwell”. Predictably based on feedback so far, this question would be criticized for the focus on
“illness” using the phrase “becomes unwell” and therefore this would need to be reviewed so
relevant to all.
Thinking about your most recent experience with the service/people who support you, how much do
you agree or disagree with the following statements….

I am involved in decision making
Q3. The services I use communicate well with each other and are fully informed about my
needs.
One site identified that ‘communicate’ and ‘fully informed’ are two separate questions, they felt the
word ‘services’ was acceptable, contradictory to another site who provided feedback in Q1.
A valid question was raised by another site, “How would they know this? Or know what this should
look like?” They suggested a ‘don’t know’ option could be useful here.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 8
For Consideration
Previous discussion on the word ‘services’ not always being fully understood is very valid and
therefore consideration should be given to alternatives. However, some people found this term
acceptable, in the opening statement, for this reason it has been left in and includes ‘people who
support me’.
Thinking about your most recent experience with the service/people who support you, how much do
you agree or disagree with the following statements….

The people I see know about my needs
(Don’t know option)
Q4. My family / whanau or someone close to me has been given information and support
to be involved in my care and recovery.
One site was of the opinion that Q4 and Q5 were similar and could be perhaps merged into one? But
overall were preferred questions in the survey.
Again, “How would they always know this?” was asked. A provider from another youth organisation
observed incorrect responses on more than one occasion. Positive ratings were given for families
who he knew had not been given any information. A ‘don’t know’ option could be considered, this
could be useful data to inform sites how well they are communicating to their client group on
information giving and support for those close to them.
One site identified the word ‘support’ as an alternative to ‘care’ and another site confirmed as they
believed not everyone would understand what was meant by “care and recovery”. They suggested
the alternative of ‘move forward’ rather than ‘recovery’.
Lastly one site gave feedback on the word ‘information’ as suggesting paper based, leaflet giving
rather than verbal, interactive or other practical advice.
For Consideration
It was identified in the first round of qualitative research as one of the most important issues to
family and whanau. CBG suggest considering a ‘don’t know’ response option.
Thinking about your most recent experience with the service/people who support you, how much do
you agree or disagree with the following statements….


My family / whanau were given the opportunity to be involved
My family / whanau were able to be involved if they wanted
(Don’t know option)
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 9
Q5. The services have supported me and my family move forward in recovery.
The same concerns were raised as cited in other questions containing the word ‘recovery’. One site
was of the opinion that this question needs to be more action orientated, and suggested ‘strategies
to improve wellbeing’ and a change to “I feel more hopeful in my recovery”
“The needs and challenges I am faced with are being addressed” was a suggested question from one
site that maybe relevant here.
For Consideration
It is difficult to make this question specifically action orientated as each service will employ different
strategies and approaches. It was mentioned earlier, that these two questions are quite similar.
Removing this question could be considered, or amending it to enable a greater differentiation from
Q4 and making it more about how they see the future, or general satisfaction.
Thinking about your most recent experience with the service/people who support you, how much do
you agree or disagree with the following statements….

I have the support I need for the future
Q6. I would recommend this service to friends and family if they needed similar care or
treatment.
Mixed opinions on this question, generally most sites felt this was a good question, but the word
‘recommend’ was extremely problematic for some services and made it inappropriate to ask.
Forensic services, for example, where people are there against their will often long term, they felt it
inappropriate to ask. To ‘recommend’ forensic inpatient services to others, “just doesn’t fit”. This
was reflected in another site feedback comments, which stated that “people get to mental health
services when they are feeling bad and the word ‘recommend’ feels wrong”. If re-worded this
question could be more relevant.
The word ‘treatment’ was identified as often inappropriate for primary care or less illness orientated
services, the dislike for this word was also cited in the first qualitative report.
For Consideration
Given this question is being used in other real time feedback surveys, in the NHS and Waitemata
friends and family, it would be sensible to keep this question the same in order to make meaningful
comparisons. An option to skip this question could be discussed.

I would recommend this service to friends and family if they needed similar care or
treatment.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 10
Q7. Is there anything you want to say about your recent experience with the service or
anything you think we can improve on?
The results for this question have not yet been analysed and, therefore, there is no feedback on the
results at this stage. Some sites reported that people generally skip this question. The results show
402 (37%) text entries in 1089 completed surveys.
Very little feedback was given for this question, only one site suggested the following alternatives
“Do you have any suggestions for how the service could be improved?” or “Is there anything about
our service you would like to improve?”
Although not mentioned specifically for this question, the word ‘service’ could be removed. The
current wording makes their response relevant for all episodes of care by asking ‘recent experience’
and is an open question to allow for any feedback.

Is there anything you want to say about your most recent experience or anything you think
can be improved?
Demographic Questions
Ethnicity
•
•
•
Include ‘other Asian’ option
Include a statement so people know they can select more than one (already
implemented in v2)
One youth service has requested a change to younger age bands 0-11, 12-14
All of these requests can be implemented in Version 3
ADDITIONAL QUESTIONS
The qualitative plan attempted to bring all sites within a reasonable distance of each other in the
development of possible additional individual site questions and the site visit would be a final step
towards deciding on one or two draft additional questions for approval. The varied size of pilot sites,
nature of the services involved and differing resources and agendas meant that everyone has
progressed with this at a different pace and not all sites are ready or able to make that decision at
this stage.
Five sites have identified draft questions for further discussion and review with possible
implementation in the next version mid-July.
One site has not yet initiated internal discussions on this and therefore felt it better to exclude the
item from the site visit agenda.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 11
And the last site have decided to hold off identifying any questions until they can see what, if any
changes are made to the core questions. They are also of the opinion that the survey is getting to
long and two more additional questions may be too much.
Limitations to the additional questions were stated in the discussion guide as below, and reiterated
at each site visit. Please note these are pilot limitations and not software limitations.

A maximum of 2 additional questions

All questions will be designed in collaboration with CBG and pilot sites and will ideally take
into account consumer, family/whanau feedback recently provided

All questions will need to be approved by CBG in collaboration with the HDC

Sites can have only 2 surveys running at the same time, it is not possible for the pilot to have
different additional questions on different devices.

-
Core question set only.
-
New survey with the core questions set plus additional questions.
Response options for the additional questions will be the same as the core questions, using
the smiley faces rating scale or selecting from a pre-determined list. Free text responses will
not be available.
All identified questions below were discussed and tailored at the site visit and will need further
review from CBG in collaboration with the HDC. People tended to initially identify questions that
confirmed what they were doing well and were encouraged to think about the usefulness of this, if
not to affirm or provide evidence for other purposes. Questions identified so far…
Question
1. Would you like to receive your mental
health services at your local GP practice?
Rationale
Not Stated
1. Family relationships are improving as a
result of (name of the specific
programme)
This question would allow the organisation to
see what phase they are in the programme
and cross ref this to the other results.
2. How long have you been involved in
(name of the specific programme)?
Less than 6 weeks
6weeks -3mths
3mths-6mths
1. My culture and spiritual beliefs were
respected and acknowledged
Time frame better as they may not know
what phase they are in
Considered that ‘spiritual’ be better than
‘religious’ as covers everyone including those
who may not identify with a culture or
religion and removes the need for a N/A.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 12
1. I am receiving support to remove current
or future challenges to finding a job or
staying in education
2. I am able to access support to help me
find a job or stay in education
1. I am being seen by the services within a
reasonable time frame
2. Our plan is reviewed regularly and our
needs are addressed
Youth organisation so education important as
well as employment. Both versions of this
question being considered with a view to
selecting one.
This question is not making an assumption as
to what is good or not as opinions or
expectation may vary- asking in this way the
actual time doesn’t matter, but persons view
on what was ‘reasonable’
Wanted to take out ‘recovery’ and just say
‘plan’. Confident all would know what ‘plan’
meant, they all have them and are referred
to. ‘Reviewed’ better then ‘referred’-easier to
understand. Word ‘our’ makes it applicable to
family and consumer
RESPONSE OPTIONS
Most sites provided good feedback on the time taken to complete, with people generally finding it
quick. One site noticed how surprised people were that it was so quick and were expecting it to take
longer.
Version 2 recently released, includes an additional question identifying status of person completing
the survey (consumer or family /whanau member), plus an additional evaluation question, which
gives a text option if answered negatively. In addition to a possible two more questions, there is a
danger of making the survey too long.
All sites felt the smiley faces were well liked, served to lighten the mood and generally has a positive
reaction. One team, within a site had feedback from a few individual families that the faces were
“babyish” and one comment that the face on the right was too expressive, but on discussion it was
agreed that this was minority feedback.
A few sites suggested that additional text beneath the faces could be useful to ensure the same
interpretations. We would recommend at this stage that the faces remain the same with no
additional text. There was no compelling feedback to warrant a change at this stage, but it maybe
something to trial at a later date once the more important issues related to core questions have
been addressed.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 13
All sites stated that people were excited and enjoyed the fact that the surveys were on a device, less
intimidating for people and creating a high level of enthusiasm and possibly resulting in a higher
uptake than a paper based survey.
Most sites are keen for language options and this is in the last stages of translation for version 2, but
will need amending to reflect any further changes prior to version 3 being released.
Surrounding introductory and explanatory text were not covered in this round of research, changes
have been made to version 2 to match the wording in the leaflet recently produced by the HDC in
collaboration with CBG. Feedback on this maybe gathered at a later date once version 2 has been in
circulation for a longer period.
Some feedback (two sites) is that images on the splash page need to include more cultures. We can
also look to implement this for the next version if time allows, but other issues will be given higher
priority.
MANAGING THE DEVICE
All sites with the exception of one have had some difficulties to a lesser or greater extent connecting
to Wi-Fi on a regular basis. Many providers have been taking devices home to upload results or
driving to nearby hot spots. Sites seem to be working through these issues and have addressed them
via the following methods:



Purchasing mobile Wi-Fi devices with subscription
Working with IT services to enable Wi-Fi in some service areas
Uploading at regular intervals, every few days or weekly rather than daily
Some sites are still to working through the Wi-Fi availability issues, but have been able to upload
results at regular intervals ranging from immediately to weekly. The issues, therefore, seem to be
manageable, but impacts on the ‘real time’ nature of the results.
All sites are aware of how to upload results completed offline, how to update to new versions of the
survey and how to register new devices. All sites are now finding this process simple and easy to
manage.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 14
USE OF THE DEVICE IN SERVICES
There are 35 devices registered, all of which are returning data. There are another 2-4 expected to
be registered in the month. The service areas where the devices are located are listed in the table
below:
Service Description (where devices located)
Adult Mainstream Community Outpatient
Adult Mainstream Inpatient Consumers Only
Adult Community Addictions
Adult Forensics Inpatient
Adult Mainstream Residential and Community
Youth and Family Therapy, Community
Child Youth and Family Admissions and Assessment
Child Youth and Family School Groups
Number of Devices
9
8
1
1
2
9
1
4
Seventeen of the thirty four devices are regularly offered to the consumer or family / whanau
member to complete during a face-to-face interaction. Ten of these devices are circulated amongst
team members resulting in people being asked to complete the survey approximately every three
weeks. The other seven are either asked during every appointment or when practically possible. A
support worker giving devices to consumers and family / whanau at each home visit described it as a
routine at the end of the visit, with family members passing it around and happy to complete.
At two of the site visits, receptionists came into the meeting to give feedback on how they were
managing the devices…






Found some people playing on the devices and don’t feel they can be left in the waiting
room as theft would be likely. She was also worried about forgetting to keep her eye on
them or asking for them back, as gets so busy.
Needed to be proactive in asking people to complete as otherwise they didn’t look at them.
Arduous task to manage as another thing to think about in a very busy area and thinks will
end up with client saturation.
Those that come in with their support workers sit and complete with them, happy to do and
enthusiastic
Difficult to catch people after an appointment as they leave quickly
One receptionist said some have been too paranoid to complete, started but didn’t finish,
but lots do complete and in another area the receptionist felt people were really keen to
participate.
Inpatient feedback was more positive…



More controlled environment and can catch people as they are being discharged easily.
It is offered to them and taken away to do privately.
Have had a good uptake with a small decline rate.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 15
The concept of ‘episode of care’ remains confusing, with varied interpretations and lack of clarity as
to when people should be approached to complete surveys.
To date, sites have not described an unwillingness to complete the survey more than once, although
there are real concerns that people will become less enthusiastic as time goes on. Given data
collection has only been going for 6-8 weeks, a longer period of data collection is required for this to
be evaluated. The survey is still progressing and further improvements may increase the likelihood
that people will be willing to complete more than once, or on a regular basis. Keeping up provider
interest in administering the survey is also going to be a challenge and sites have already started to
think about moving devices around more often, and advertising results. Advice on long term
management of real time surveys will likely be sought post pilot period and each approach will be
dependent on the organisation, the services and even personalities involved.
The trend data will soon be available. This will allow an overview of the uptake rates week by week,
overall and across services. This information should help inform sites as to where their devices may
be best placed and enable them to continue to improve survey administration and delivery.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 16
ANALYTICS
Feedback on the nature of reporting in the first qualitative report was minimal and the initial analytics
released were a starting point from which to gather more feedback and develop further. All sites have
now put through multiple requests to improve, correct or specify what analytics they require.
It was discussed with all sites the Advisory Group and HDC decision to not provide raw data for the
free text response, but analysed themes. Advised this will be communicated to the public in the
survey and in the leaflet.
The majority of requests have been raised at recent site visits and will be implemented through July.
Requested changes to device labelling/groupings
Show full scale to rating 5, dropping off and harder to see discriminations
Pie chart to include all ethnicities rather than just 'other'
Short label rather than Q1, Q2 etc.
Tab with questions listed for easy reference
Trends-for all sites, pilot sites and per question response overall (and dates if possible)
Totals- 1. Overall number completed per pilot site 2. All sites overall (on public site as well). 3.
Totals per group
Similar view to public site, but for each pilot site overall
Number of results completed weekly (dates recorded) for each group
Graph average experience score by questions and age band (so three per question)
Demographics per group
Show result for Māori Vs all-average score per question (different colour/vertical bar graph-like US
Vs ALL in current form but vertical)
Change to age band for youth services 0-11, 12-14 and so on. CW to implement into next version of
the survey
Drill downs from one analysis to another e.g. click on female and see results for all female
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 17
CONCLUDING COMMENTS
The challenge to develop a set of agreed core questions relevant to both consumers and family/
whanau completing, and evaluating multiple different services and providers is extremely complex.
In starting to develop their own additional questions the sites may have gained a better appreciation
of this task, however, the feedback clearly indicates an overall consensus for a full review of all core
questions with the primary aim to simplify comprehension and ensure the questions are applicable
to all services in the pilot. Although we can certainly improve the questions to reflect these requests,
it may not be possible to develop questions applicable to all, given the variety of services,
approach’s, treatments and differing opinion’s amongst the providers, consumer and family/
whanau involved.
For the majority, the idea of rating an average score based on multiple services they may have seen
that day, or multiple elements within a question, also remains confusing for both provider and
consumer, family /whanau.
Further contradictions make the decisions all the more difficult. Alternative words suggested and
preferred in the first report were rejected in this round of research, however, this time the wording
was being evaluated in the context of live data collection. Significantly less words were identified as
being problematic than in the first report, only a handful still considered too medical and not
applicable to all.
Although the feedback given for this report has been valuable and has informed possible changes to
the survey, this round of feedback has been predominantly from providers representing the
consumer and family/ whanau. The people present at the site visits are a select group and we cannot
conclude that their opinions reliably represent the majority. The evaluation question will allow a
confidential voice for all people completing a survey. Further evaluation work proposed by Malatest,
will allow for a longer data collection period with amended questions, should this be implemented
and any additional questions.
We would hope that the next version of the survey will take into account the feedback in this report
as much as practically possible. This will serve to alleviate current frustrations in delivering questions
or terminology not applicable to their services and assure sites that the feedback is being heard. We
would predict that an improved level of understanding would result in more accurate reporting and
a greater differentiation in the results.
All sites have been passionate in their feedback and are keen to help develop and improve the
survey during this pilot phase. Despite all the constructive criticism, the excitement and enthusiasm
for the real time survey remains high.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 18
SUMMARY OF CURRENT AND POSSIBLE REVISED
QUESTIONS FOR FURTHER DISCUSSION
In order to be applicable to all client groups within each pilot site ‘services or people who support
you’ could be included in an overall statement to introduce each individual question. This could be
stated once on the first page or above each question on each page. This is open to further
discussion.
Thinking about your most recent experience with the service/people who support you, how much do
you agree or disagree with the following statements….
Q1. Relationship/Partnerships
The people I see within the service value my opinions and are compassionate, non-judgemental and
respectful of my cultural needs.


I felt respected
I feel respected
Q2. Communication/Information
I am sufficiently consulted and involved in decisions about services to meet my identified needs

I am involved in decision making
Q3. Continuity of Care/Coordination
The services I use communicate well with each other and are fully informed about my needs.

The people I see know about my needs
(Don’t know option)
Q4. Family involvement
My family / whanau or someone close to me has been given information and support to be involved
in my care and recovery.


My family / whanau were given the opportunity to be involved
My family / whanau were able to be involved if they wanted
(Don’t know option)
Q5. Recovery and Support
The services have supported me and my family move forward in recovery.

I have the support I need for the future
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 19
Q6. Friends and Family question- (no modifications)

I would recommend this service to friends and family if they needed similar care or
treatment.
Q7. Free Form Text Question
Is there anything you want to say about your recent experience with the service or anything you
think we can improve on?

Is there anything you want to say about your most recent experience or anything you think
can be improved?
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 20
APPENDIX A - SECOND SITE VISIT PLAN
1. Obtaining feedback from pilot sites, consumers and their families / whanau
We aim to visit the sites from the week beginning the 26th May 2014 over a 4 week period.
The main purpose of the second site visit is to monitor and inform how best to modify questionnaire
design and administration if required. We propose to do this by exploring…






Nature of the core questions including, any other questions or question types that should
have been included.
Possible additional questions to be added
The way that response options were presented
Exploring the language issues
Managing the device/updating the survey and its use within the service
Possible improvements on usefulness of analytics, including free text analysis
Working with the pilot sites identified liaison person we will arrange a 2-3 hour focus group with up
to 10 people representing the following groups. The pilot site will decide who should be present with
some guidance from CBG.




Consumers, family/ whanau
Pilot Site representatives
Providers from service areas where devices have been placed
Research analyst’s
2. Pre meeting communications
The sites will be supported in identifying additional questions should they choose to do so. The
process of deciding on the additional questions will start as soon possible post data collection
commencing with a view to discussing and deciding on final questions to be submitted for approval
at the 2nd site visits or before. We will also include a bank of possible questions as a guide, this is
provided below.
Once identified CBG will personally contact all attendees to ensure they are fully briefed on the
purpose of the focus group and a follow-up letter and final agenda will be sent as soon as practically
possible prior to the focus group date.
3. Evaluation Question on the Devices
The addition of one evaluation question at the end of the survey asking if the survey represented
their views, or something similar. The results would be displayed as part of the analytics and
available to Maletest (Independent evaluator) to reference as part of their evaluation.
The question would be agreed by both parties, advisory group and approved by the HDC.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 21
APPENDIX B – DISCUSSION GUIDE- ADDING IN ADDITIONAL
QUESTIONS
Introduction
This guide is designed to support pilot sites in the design and implementation of optional additional
questions following on from the core question set, in preparation for the second site visit. It is also
an opportunity to review the core question set with a view to possible amendments being made.
Some pilot sites may not feel confident in deciding on, or designing additional questions and this is
not an expectation. Sites may only wish to identify the domains of patient experience, service or
intervention to be evaluated, CBG will then assist, in collaboration with each site to design the
appropriate question/s.
We have included below a list of possible questions as a guide. These questions have been mostly
selected from two sources:


Discussion guide of sample questions used to facilitate feedback when designing the core
questions
Suggested questions from the first pilot site visits and consumer, family/ whanau groups
In some cases the questions have been omitted or amended to reflect the feedback given so far, and
remove repeat questions. These questions provide a starting point and may need to be amended
further to perfect the wording and ensure relevance to the service. Pilot sites may already have
identified possible additional questions not listed here, that can also be considered.
Where some questions were considered irrelevant or too ‘medical’ to be applied to all pilot sites for
the core question set, we have still included in the list below. For sites wishing to evaluate an
inpatient service or treatment clinic, for example, they may well be useful. The word ‘hospital’ could
be replaced with residential centre, clinic or other service. Likewise a ‘group’ or therapy’ could be
replaced with a specific name.
Although the suggested questions fall into the domains of patient experience already identified in
the core question set, there may be some scope to ask more specific related questions if useful to a
service.
The questions have been written to suit a rating response, but there is scope to have a list of options
to select from, similar to the response options for the demographic questions.
Other language options will be made available for the next version.
2ND QUAL REPORT_REVIEW OF V1 SURVEY_RTF JUNE 2014
Page 22
Restrictions of Pilot

A maximum of 2 additional questions

All questions will be designed in collaboration with CBG and pilot sites and will ideally take
into account consumer, family/whanau feedback recently provided

All questions will need to be approved by CBG in collaboration with the HDC

Sites can have only 2 surveys running at the same time, it is not possible to have different
additional questions on different devices.

-
Core question set only.
-
New survey with the core questions set plus additional questions.
Response options for the additional questions will be the same as the core questions, using
the smiley faces rating scale or selecting from a pre-determined list. Free text responses will
not be available.
Suggested Areas for Discussion

Generic, broad across all services e.g. an additional domain
of patient experience

Service/project specific e.g. a new initiative or group
therapy, change in clinic hours, venue etc…
Relevance of additional
questions

Will the additional questions be relevant to all service areas
where the devices are placed? (remember you can JUST
have the core questions running on some devices)
Questions from other
surveys

Are there any other questions that are being used currently
that work well or you wish to move to this survey?

Any past questions you wish to measure again?

What do you think of the current question set – is there any
important areas missing?
Specific questions that should be included?
Feedback you have received so far?
Type of additional
question/information
to be captured
Core Question Set


CBG HEALTH RESEARCH LIMITED
FIRST CONSUMER & FAMILY/WHANAU REPORT HDC REAL TIME FEEDBACK PILOT APRIL 2014
23
Current Core Question Set
Relationship/Partnerships
The people I see within the service value my opinions and are compassionate, non-judgemental and
respectful of my cultural needs.
Communication/Information
I am sufficiently consulted and involved in decisions about services to meet my identified needs
Continuity of Care/Coordination
The services I use communicate well with each other and are fully informed about my needs.
Family involvement
My family / whanau or someone close to me has been given information and support to be involved
in my care and recovery.
Recovery and Support
The services have supported me and my family move forward in recovery.
Friends and Family question
I would recommend this service to friends and family if they needed similar care or treatment.
Free Form Text Question
Is there anything you want to say about your recent experience with the service or anything you think
we can improve on?
Suggested Additional Questions
Relationship/Partnerships






I was treated with kindness and understanding while I or a family member was in hospital?
Cultural or religious support was available when I needed it
I have confidence and trust in the staff working with me, or a family member.
I have confidence the therapist/provider/service has our family’s best interests at heart
My psychiatrist/therapist/key worker genuinely listens to my or a family members concerns
I was satisfied with the respect and acknowledgement placed upon my cultural needs
Communication/Information




My condition, or that of my family member was explained to me in a way I could understand
I was informed about any treatment side effects in a way I could understand
I was given enough privacy when discussing my condition or treatment when in hospital
My family and I were given information about the consequences of stopping my
medication/treatment and what to do to avoid my readmission to hospital/relapse
CBG HEALTH RESEARCH LIMITED
FIRST CONSUMER & FAMILY/WHANAU REPORT HDC REAL TIME FEEDBACK PILOT APRIL 2014
24


I feel involved in decisions about (service specific treatments or intervention)
The therapy sessions (or group) I attend has improved my knowledge and understanding of my
condition/treatment or that of a family member
Continuity of Care /Coordination




I can access appointments or support with staff when I need to in a timely way


When I was discharged to a GP, I was able to easily access mental health and addictions



How helpful do you find the receptionists at the practice/clinic/hospital?
I was kept informed on how my referral or that of a family member was progressing
I received the help I needed on my or a family members discharge from hospital
My recovery plan is referred to and updated regularly in collaboration with me and/or my family
member
Which of the following methods would you prefer to use to book appointments at the
practice/clinic? (service specific list)
Thinking of times when you want to see a particular person, how quickly do you usually get seen?
How do you rate the ease of booking and the variety of booking times for appointments?
Family Involvement



Everyone in our family, including the children were given age appropriate help
I have received support as a service user to maintain contact with my family/whanau
Family relationships have positively changed/improved because of the service (specific
interventions)
Recovery and Support







I was involved in decisions about discharge from hospital
As a result of the service am I better equipped to deal with my immediate needs
The needs and challenges I am faced with are being addressed.
I am able to access support to find work?
I am able to access support to find a home?
The group therapy sessions I attend are well organised
The group therapy sessions I attend are helping support me and my family
CBG HEALTH RESEARCH LIMITED
FIRST CONSUMER & FAMILY/WHANAU REPORT HDC REAL TIME FEEDBACK PILOT APRIL 2014
25