Cognitive
Interviews
For Business surveys
Marcel Levesque
QDRC
Statistics Canada
Structure of the presentation
Cognitive interviewing defined.
 How is an interview constructed?
 How is an interview conducted?
 Cognitive interviewing applied to business
surveys.
 Examples of issues tested in the UES survey.
 Conclusion.

Cognitive interviewing
defined
A process to explore:
a
respondent’s understanding of questions
 their strategies to retrieve the information
requested by a survey
 their judgment of the adequacy of the
retrieved information
 their ability to report the required information
How is the interview
constructed?
Meeting with survey manager to discuss
the issues to be tested
o The appropriateness of questions,
concepts and terminology
o Respondents` willingness and ability to
respond to survey questions
o Respondents’ understanding of the
questions and what to report
o
o
o
Respondents’ use of external sources of
information such as financial or
administrative records and their need to
consult other individuals who can provide
the information requested
The compatibility of questions and
response categories with respondents`
record keeping practices
Difficulties respondents may encounter in
retrieving information and completing the
questionnaire
Designing an interview guide
o
o
Based on the issues discussed with the
survey manager, a guide is designed to
test the questions and issues identified.
Comprehension, information retrieval,
judgment and communication are usually
issues to be explored.
Comprehension
o Difficulties encountered by a respondent in
understanding a question may be
explored:
o Instruction content - inaccurate or conflicting
instructions, instructions
separate from the
question
o Question wording - ambiguous terms, critical
definitions missing
o
o
o
Question structure – several questions in
one, Question
/answer mismatch
Question content - complex topic, topic
carried over to another
section
Navigational instructions - inaccurate
instructions
(move to the
wrong place),
confusing
flow
Information retrieval
o Difficulties encountered by a respondent in
gathering the requested information may
be explored:
o Multiple sources - information may be
distributed in multiple
sections /departments in
the organization
o Record retrieval - records in multiple sources,
record access issues
o
Memory retrieval- reference period too
long, recall problems
and telescoping errors.
Judgment
o Difficulties encountered by a respondent in
assessing the relevance of the retrieved
information to the data requested may be
explored:
o Data incompatible with existing records
o Need to evaluate and synthesize multiple sources of
information
o Potentially sensitive information, strategic or
proprietary issues may prevent revealing certain
information
Communication
o Difficulties encountered by a respondent in
reporting the response to a question may
be explored:
o Missing response categories
o Mismatch with technical language
o Response terminology – critical definitions
may be missing
How is the interview conducted?
o
o
Respondents for cognitive interviews are usually
recruited from Statistics Canada’s Business
Register or from another frame of
establishments provided by the industry-specific
program areas.
Respondents are identified based on the project
team’s specifications, designed to ensure a
selection of business establishments for each
industry sector to be tested.
o
o
Appointments are made with the person
within the business establishment who
usually completes questionnaires for
Statistics Canada.
The ideal respondent is the person in the
business who is most knowledgeable
about the data requested, who has access
to this data and who also has the authority
to release it.
o
o
Respondents are provided with an
explanation of the purpose of the cognitive
interview with emphasis on the fact that
Statistics Canada is consulting with them
to obtain their feedback and opinions.
The importance of their participation is
emphasized and they are given
assurances of privacy.
o
o
o
A consultant from the QDRC conducts a
one-on-one interview at the respondent’s
place of business and observers from the
client industry sector take notes during the
interview.
The interviewer walks through the
questionnaire with the respondent and
discusses issues as they arise.
The interview may be recorded with the
respondent’s acceptance.
Cognitive interviewing applied to
business surveys
o
o
The Unified Enterprise Survey (UES) began in
1997 with a pilot of a small number of industries
and has continued to the present time where it
now encompasses the majority of business
surveys conducted by Statistics Canada.
The UES is basically a single questionnaire with
a variant for each industry sector. There is a set
of core questions asked of every industry and
industry specific questions for each different
sector.
o
o
Core questions cover such topics as the
reference period covered by the survey
(financial year of the establishment),
revenues and expenses, personnel, sales
according to types of clients and their
location and, if appropriate, questions on
international transactions.
Industry specific questions refer to the
main commercial activity of the
establishment and industry characteristics,
namely sales activity specific to the sector.
o The purpose behind UES is to develop a
common approach with concepts
harmonized across all industries to:
o facilitate data collection and data capture
o reduce and better manage response burden
o to increase response and data quality.
Examples of issues tested in the
UES
Comprehension: Question structure
One of the difficulties encountered with the
2005 survey was in relation to a mismatch
between the question and one of the
answer categories.
F- Industry characteristics – automotive
repairs and maintenance
Sales of goods and services
1.
Repairs and maintenance – labour costs
________________________________________
2.
Repairs and maintenance- parts and accessories
used for repairs and services
_______________________________________
3.
Sales of merchandise, parts and accessories for
resale in their original state
________________________________________
o
This section referred to sales of goods and
services, but the first answer category asked to
record labour costs. Consequently, respondents
were confused.
o
The intention behind this question was to obtain
the sales value of labour costs billed to the client
but respondents failed to understand the
question in this way. The difficulty here was a
mismatch between question and answer, which
led to confusion.
Recommendation
o
Replace the expression
‘labour costs’
by
‘sales resulting from labour costs billed to
clients’.
Comprehension: Question wording
o
Another difficulty encountered was in
relation to the question wording in the
French version of the questionnaire,
resulting from the translation from English
to French.
E- expenses
1.
Cost of employment and personnel agencies
___________________________________
Coûts des bureaux de placements et des
services de location de personnel
______________________________
o
The expression ‘personnel agencies’
translated into ‘services de location de
personnel’ did not make sense to the
French respondents since the word
‘location’ in French means rental or
leasing.
Respondents did not understand the
meaning of the expression ‘services de
location de personnel’. As a result, many
respondents did not know what costs to
report in this column.
Recommendation
o Since ‘bureaux de placement’ was a more
common expression:
o leave out the reference to ‘services de
location de personnel’.
o include only this category in the question
Retrieval: multiple sources and
record retrieval
2006 survey on entertainment and leisure
Some respondents were involved in the health
fitness industry offering a variety of services
such as a spa, massage, tennis courts, pool and
exercise rooms. Some of the expense items
listed in the questionnaire did not correspond to
their books.
E- expenses
1.
Wages and salaries
________________________________
5. Contract workers
___________________________


Item 1 in the questionnaire asked for salaries in
Canadian dollars.
Item 5 asked for the dollar value attributed to
contract workers.
Some employees are hired on contract but the
costs are listed under salaries in item 1.
Respondents would not have reported any dollar
value in item 5 for contract workers and there
was no way for them to indicate that these
expenses were all under item 1 on salaries.
Consequently we would have ended up with an
unanswered question without knowing why.
Since expenses are part of the core
questions applicable to all industry
sectors, it is quite difficult to have
answer categories correspond to all
industry sectors.
Recommendation
Add a space for ‘comments’ under the
section on expenses to allow
respondents the possibility of
explaining the items they are unable
to provide as listed.
Judgment
o
Potentially sensitive information, strategic or
proprietary issues may prevent respondents
from revealing certain information.
2005 survey on service industries
o
Some respondents involved in “real estate,
leasing and property management” said they
would not provide information on revenues
because their company had a policy on
confidentiality which would not allow the release
of such information. They did, however, mention
that they would be willing to provide estimates.
Recommendation
Provide revenue categories to encourage
respondents to report information on
revenues they would not give if exact
figures were requested.
Communication
o
One of the difficulties encountered in the
communication phase of the process for
the 2006 Service industry survey was
related to response categories.
F- Sales according to product type
Architectural products
1.
Residential construction projects
______________________________
2. Office construction projects
______________________________
3. Construction projects for retail
businesses and restaurants
______________________________
4. Construction projects for hotels
_____________________________
5. Construction projects for hospitals
______________________________
6. Construction projects for educational
institutions
______________________________
7. Construction projects for industrial
buildings
_______________________________
Where would you report
construction projects for
airports?
Recommendation
Add significant categories identified by
respondents i.e. construction projects for
airports.
Conclusion
In a business environment, a respondent’s
main preoccupation is the success of his
business.
Few respondents will consider spending
time researching information if it is not
readily available or spending time trying to
understand questions that may not be
clear.
As indicated in this presentation, many
issues may contribute to response burden
and have a negative impact on the quality
of the results of a survey.
Cognitive interviewing:
o provides us with a greater
understanding of some of these issues
o allows us to improve a questionnaire
o allows us to increase the clarity of the
questions, reduce response burden and
result in increased data quality.