Qualitative Research and Performance Audit: The Twain Shall Meet
Paper #116 • Mon May 12, 2008 • 1:00-2:45pm
Salle St Louis • Chateau Frontenac • Quebec
Reed Early • BC Office of the Auditor General
1) Evaluation and Performance Audit - related disciplines
a) Ian McPhee (2006), Auditor General of Australia ANAO – spoke on “Evaluation
and PA - close cousins or distant relatives?”
i) Concluded they are close cousins - in most ways
(1) similar aims – improve accountability & program management
(2) may examine process, outcomes, efficiency, effectiveness (sometimes)
(3) share methodologies such as use of interviews
(4) share techniques such as qualitative analysis
(5) are part of good public sector mgmt, promoting best practice
ii) But they are distant cousins …
(1) Audit is independent of auditee and gov; evaluation may not be
(2) Audit reports to head of government; evaluation usually reports lower
(3) Evaluation can look at policy and outcomes directly; Audit can’t/does not
b) PA similarity to impact evaluation - John Owen (2006) Australian evaluator of
high international reputation concludes PA fits into the impact evaluative
approach because it:
(1) objectively obtains and evaluates evidence
(2) makes professional judgements
(3) may assess impacts to determine if program makes a difference
(4) may judge value for money
2) What can evaluation learn from audit and through this improve its reputation:
(audit tends to have a reputation for higher quality than evaluation, in Gov).
a) Sheila Fraser (2006,2007) AG – recommended use of recognized standards for
evidence and conclusions TO 2005
b) John Mayne (2005) –AG TBS CG recommended use of QA to bolster quality of
weak designs (i.e. interviews)
c) Jim McDavid and Huse (2006) – UVic Sch Public Admin – states that evaluation
has lost ground to audit, and needs to reclaim that, esp. in performance
measurement
3) Qualitative research and PA – close cousins compared
Both are used in social services, health, academic, and government,
Empirically based – i.e. rely on observed data/evidence
Uses qualitative data/evidence i.e. spoken and printed words and numbers
Qualitative methods i.e. use of categories or themes to code data/evidence
Use mixed methods i.e. combined interview, focus groups, document analysis and survey
Generates a “story”
Accepts that meaning is contextual
Aims for accurate and verifiable findings
Early, Reed, LastSaved: 05/08/08 01:43 PM
C:\Documents and Settings\rearly\Application Data\Microsoft\Templates\QDA and PA - the Twain Shall Meet.doc
1
4) distant relatives compared
Performance Audit
has an end goal to arrive at facts –
thence truth i.e. disagrees with idea of
multiple truths
generates “the” story
uses findings to conclude against
chosen criteria (must reconcile
opposing facts)
evaluates findings to determine level
of assurance or degree of certainty
uses CICA standards (guidelines
really) of Canadian Institute of
Chartered Accountants
accepts meaning may be socially
constructed but assumes underlying
truth
reduces evidence to themes and then
to facts
expresses an audit opinion
reports are normally public
auditor is independent of the auditee,
usually reports to head of govt, and is
not linked to the client
Qualitative Research
has as end goal to find a truth i.e.
accepts idea of multiple truths
generates “a” story
uses findings to derive themes,
meanings, description, and/or “truths”
(rejects that fact can be in opposition)
accepts findings at face value, ala
post modernist approach
uses The Program Evaluation
Standards
accepts meaning may be socially
constructed and truth is therefore
negotiable
derives themes from data
describes, summarizes or concludes,
but does not venture opinion
reports may or may not be public
not necessarily independent – may be
dependent on funder, stakeholders,
political leaders in some ways
Facts
Findings
corroborate
d into:
· Summaries
· Stories
· Research
papers
reviewed for quality assurance
Findings
Evidence
analyzed into:
· Themes
· Observations
· Descriptions
· Working
papers
corroboration
Evidence
· Interviews
· Documents
· Focus gps
· Surveys
· Emails
· Field notes
· Summaries
· Memos
· Photos
analysis
5) Research process flow
a) Performance audit tends to use same evidence, but goes further in corroborating
facts to reach conclusions and to generate recommendations
b) Qualitative research’s longer history of QDA methods is better documented
Conclusions
Facts compared
to criteria to
generate:
· Conclusions
· Recommendations
Performance Audit – customarily interviews and documents
Qualitative Research – as above plus focus groups, surveys
Early, Reed, LastSaved: 05/08/08 01:43 PM
C:\Documents and Settings\rearly\Application Data\Microsoft\Templates\QDA and PA - the Twain Shall Meet.doc
2
6) Use of categories
a) Audit usually uses criteria – determined from existing standards, best practices, or
other audits – this is used to organize the evidence
b) Qualitative research and data analysis sometimes uses “a priori” categories to
code and organize the data (one study of 72 focus groups (Wiggins 2004) found
less than 50% of focus group analysis used a coding scheme.
c) Goal based evaluation evaluates against existing goals – this is used to make a
judgement of the program
7) Conclusion
a) QR and PA are not kissing cousins – not even desired to be associated with each
others – but more like are like train tracks run parallel and meet in the distance
References
Fraser, Sheila (personal conversation Sept, 2007) Performance Audit Symposium; Ottawa
Fraser, Sheila. (2006) Invited address: The role of the Office of the Auditor General in Canada and the
Concept of Independence. Canadian Journal of Program Evaluation 21(1) p1-10.
Mayne, John (2005) Ensuring Quality for Evaluation: Lessons from Auditors, Canadian Journal of
Program Evaluation 20(1) p37-64.
Mackay, Keith (2004) Two Generations of Performance Evaluation and Management Stems in Australia,
http://unpan1.un.org/intradoc/groups/public/documents/APCITY/UNPAN022995.pdf
McDavid, Jim and Huse, Irene (2006) Will Evaluation Prosper in the Future? Canadian Journal of
Program Evaluation, 21(3) p47-72.
McPhee, Ian. (Unpublished paper Feb. 2006) Evaluation and Performance Audit: Close cousins – or
Distant Relatives. Australian National Audit Office
http://www.anao.gov.au/director/publications/speeches.cfm?pageNumber=2
McStravick, Mike (personal conversation, 2008) Office of the Auditor General of BC
Owen, John.M. (2006) Program Evaluation: Forms and Approaches. Allen and Unwin.
Wiggins, Geoffrey. (2004) The Analysis of focus groups in published research articles. Canadian Journal
of Program Evaluation, 19(2) p143-164.
Early, Reed, LastSaved: 05/08/08 01:43 PM
C:\Documents and Settings\rearly\Application Data\Microsoft\Templates\QDA and PA - the Twain Shall Meet.doc
3
Qualitative Research and
Performance Audit
The Twain Shall Meet
O F F I C E
O F
T H E
Auditor General
of British Columbia
„
Qualitative research
„
„
A tool for evaluation that involves
interviews, focus groups, surveys, field
notes and observations.
Findings tend to be thematic and
descriptive
O F F I C E
O F
T H E
Auditor General
of British Columbia
„
Performance audits assess management
of performance (usually govt) against
defined criteria
„
„
Value for money or Risk audits
Public Performance Reporting –
performance reports for accountability
to the public
„
session #131
O F F I C E
O F
T H E
Auditor General
of British Columbia
Evaluation and Performance
Audit - related disciplines
Ian McPhee (2006), Auditor General of Australia
Evaluation and PA - close cousins or distant relatives?
Concluded they are close cousins
„
„
„
„
„
„
similar aims
examine process, outcomes, efficiency, effectiveness
share methodologies
techniques i.e. qualitative analysis
part of good public sector mgmt
O F F I C E
O F
T H E
Auditor General
of British Columbia
… they are distant cousins …
Audit is independent of auditee and gov; unlike
evaluation
Audit reports to government; evaluation can report
anywhere
Evaluation can look at policy and outcomes directly;
audit can’t/does not
O F F I C E
O F
T H E
Auditor General
of British Columbia
PA similarity to impact
evaluation
„
John Owen, Australia, says PA fits
impact evaluation i.e.
„
„
„
„
objectively obtain and evaluate evidence
make professional judgements
assess impacts to determine if program
makes a difference
judge value for money
O F F I C E
O F
T H E
Auditor General
of British Columbia
What can evaluation learn
from audit?
Evaluation can improve its reputation:
„ Sheila Fraser, AG – recommended evaluation
adopt standards for evidence and conclusions
„ John Mayne, AG TBS CG – use of QA to
improve weak designs
„ Jim McDavid, UVic – evaluation lost to audit
when didn’t go for performance measurement
O F F I C E
O F
T H E
Auditor General
of British Columbia
Qualitative research and PA –
compared
„
close cousins
„
„
„
„
„
„
„
„
Both used in social, health, academic, government
Empirically based
Uses qualitative evidence
Qualitative methods
Use mixed methods
Generates a “story”
Accepts meaning is contextual
Aims for accurate, verifiable findings
O F F I C E
O F
T H E
Auditor General
of British Columbia
…distant relatives…
QR
PA
„ end goal is “a truth”
„ end goal “arrive at facts”
„ generates a story
„ generates “the” story
„ uses findings to derive
„ uses findings to conclude
themes/meanings/
against chosen criteria
description/truths
„ evaluates findings to
determine assurance „ accepts findings at face
value
„ uses CICA standards
„ uses JC Program Eval Stds
O F F I C E
O F
T H E
Auditor General
of British Columbia
distant relatives
PA
„
„
„
„
„
„
assumes underlying truth
reduces to facts
expresses an audit
opinion
reports are normally
public
auditor is independent
con’t
QR
„
„
„
„
„
accepts meaning as social
construct and negotiable
derives themes from data
describes, summarizes or
concludes, but does not give
opinion
reports may not be public
not necessarily independent
O F F I C E
O F
T H E
Auditor General
of British Columbia
Facts
Findings
corroborat
ed into:
• Summaries
• Stories
• Research
papers
reviewed for QA
Findings
Evidence
analyzed
into:
• Themes
• Observations
• Descriptions
• Working
papers
corroboration
Evidence
• Interviews
• Documents
• Focus gps
• Surveys
• Emails
• Field notes
• Summaries
• Memos
• Photos
analysis
Research Process Flow
Conclusions
Facts compared to
criteria to
generate:
• Conclusions
• Recommendations
Performance Audit – customarily interviews and documents
Qualitative Research – interviews, documents, focus groups, surveys, field notes, observations
O F F I C E
O F
T H E
Auditor General
of British Columbia
Flow Con’t
„
„
Performance audit uses same evidence,
pursues further assurance
Qualitative research’s longer history of
QDA methods is better documented
O F F I C E
O F
T H E
Auditor General
of British Columbia
Use of categories
„
„
„
Audit usually uses criteria i.e. to
organize the evidence
Qualitative data analysis sometimes
uses “a priori” categories to code data
goal based evaluation compares to
goals
O F F I C E
O F
T H E
Auditor General
of British Columbia
Conclusion
„
My view - QR and PA run parallel and
meet in the distance
O F F I C E
O F
T H E
Auditor General
of British Columbia
O F F I C E
O F
T H E
Auditor General
of British Columbia