Collaborating for Quality:
A Cross-Discipline Approach to
Questionnaire Content Evaluation
in Business Surveys
Diane K. Willimack
Peter Gibson
U.S. Census Bureau
Outline
• Background
– Business surveys
– Response process
• Questionnaire development, pretesting &
evaluation
– Why collaborate?
– An example
– Benefits of the collaborative approach
3
Business Surveys (1)
• Nature of the collected data
– Financial
– Technical definitions
• Expect data to be in records
4
Business Surveys (2)
• Nature of response
– Person answers survey questions for the
business
– Cognitive response process
5
The Response Process in
Business Surveys
Cognitive response model
PLUS
Organizational processes
6
Tourangeau’s (1984)
Cognitive Response Model
1. Comprehension
2. Retrieval
3. Judgment
4. Communication
Survey
7
Response Process in Business
Surveys (1) (Sudman et al., 2000)
1.
2.
3.
Organizational
Aspects
4. Comprehension
5. Retrieval
6. Judgment
7. Communication
Business
Survey
8.
8
Response Process in Business
Surveys (2) (Sudman et al., 2000)
1. Encoding in Memory / Record Formation
2. Selection / Identification of Respondent(s)
3. Assessment of Priorities (Motivation)
4. Comprehension
5. Retrieval from Memory and / or Records
6. Judgment
7. Communication
Business
Survey
8. Release of the Data
9
Response Process in Business
Surveys (3) (Sudman et al., 2000)
1. Encoding in Memory /
Record Formation
2. Selection / Identification of Respondent(s)
3. Assessment of Priorities (Motivation)
4. Comprehension
5. Retrieval from Memory and / or Records
6. Judgment
7. Communication
Business
Survey
8. Release of the Data
10
Record Formation
• Management needs
• Regulatory requirements
• Accounting standards
• Knowledge domain of experts in
accounting practices
11
Putting the Pieces Together for Quality Response
Records
Respondent
Accountant
12
The Respondent in Business Surveys
• Answers survey questions for the business
– Cognitive response process
• Expect respondent to know –
– What is in records
– How to get them
• Knowledge domain of cognitive
survey methodologists
13
Response Process in Business
Surveys (4) (Sudman et al., 2000)
1. Encoding in Memory / Record Formation
2. Selection / Identification of Respondent(s)
3. Assessment of Priorities (Motivation)
4. Comprehension
5. Retrieval from Memory and / or Records
6. Judgment
7. Communication
Business
Survey
8. Release of the Data
14
Questionnaire Development,
Pretesting & Evaluation
• Challenges
– Technical
definitions
– Data availability
– Labor-intensive
response process
• Expertise
– Subject matter
– Accounting
practices
– Cognitive survey
methodology
15
Putting the Pieces Together for Quality Response:
Cross-Discipline Collaboration
Records
Respondent
Accountant
Cognitive
Survey
Methodologist
16
Questionnaire Development &
Pretesting
Methodologist role
• Cognitive testing
– Non-directive probes
get at cognitive steps
• Shortcoming:
– Limited by laborintensive response
process
– Unable to access
records
Accountant role
• Aid stakeholders in
specifying constructs
– Improve initial questions
– Align with records
• Review draft form &
protocol
• Shortcoming:
– Directive approach
17
Questionnaire Evaluation
Methodologist role
• Respondent
debriefings
– Post-collection
– Response strategy
• Shortcoming:
– Does not re-create
construction of the
response
– Does not validate
the response
Accountant role
• Review reported
data
– Comparison to
public data sources
– Aid identifying &
correcting reporting
errors
• Shortcoming:
– Lacks contact with
respondent
18
Cross-Discipline Collaboration:
Methodologist + Accountant
An Example
19
Example Draft Question
1. What is the amount of licensing fees or
royalty income received by your
company’s domestic operations?
2. Of the amount reported in item 1, how
much was from:
a)
b)
c)
d)
Patents?
Copyrights?
Computer Software?
Other (specify _______________)
20
Example Cognitive Protocol
1. Tell me in your own words what the terms “licensing
fees” and “royalty income” mean to you in the context
of this question.
2. Please explain how these are similar or different.
3. What sorts of “royalty income” or “licensing fees” does
your company receive?
4. How would you go about reporting “royalty income” or
“licensing fees” for your domestic operations? What
records would you use?
5. How would you go about reporting the amounts earned
from patents, copyrights, etc.? What records would
you use?
21
Example Technical Issues Identified
by the Accounting Expert
1. Forms of Intellectual Property (IP) tracked in the
corporate legal entity that directly owns the IP.
2. The legal office structures the IP licensing arrangements
(and not the Financial area).
3. Contracts consist of several forms of IP.
4. The company’s IP might be enhanced by the licensee
and resulting income combines the original with the
enhanced.
5. Attributing income to forms of IP (e.g., patents,
copyrights) is a theoretical exercise. Actual pricing
reflects opportunity costs, present value, profit potential,
value of future development.
22
Example Enhanced Protocol with
Technical Probes from the Accountant
1. Do you have “patents, trademarks, copyrights?”
2. Are these IP owned directly by the domestic legal
entity’s corporate or subsidiary business units?
3. Is any of this IP owned by a foreign subsidiary?
4. Do any of the “licensing fee or royalty income”
accounts contain a coding for income tax purposes?
5. Would there be a difference between the amount you
would report for “world-wide” or “domestic?”
6. Who handles licensing arrangements in sub-units?
23
Example Outcomes from the
Collaborative Interview
•
Burden:
–
–
–
•
Clarification of the construct being measured
–
•
Reporting total domestic “licensing fees and royalty
income” depends on data access.
Analysis of each licensing arrangement needed to
attribute income to individual forms of IP (e.g., patents,
copyrights).
Would not provide robust data.
Stakeholder reduced requirements, dropped questions
Protocol
–
Cognitive methodologist strengthened probes of
technical issues vis-à-vis data availability
24
Methodologist + Accountant =
Robust Pretest Interviews
Methodologist role
• More comfortable
with technical
nuances
• More effective
probing
Accountant role
• Gained insights into
variety of
respondents and
reporting issues
25
Methodologist + Accountant =
Robust Pretest Interviews
Methods
• Probing anticipated
problems
• Prepared to
investigate
hypotheses about
specific terms and
concepts
Results
• Respondent’s initial
response
augmented with
reaction to possible
alternatives
• Insights into burden
and data quality
26
Benefits of Cross-Discipline
Collaborative Approach
•
•
•
•
Improved draft questionnaire for pretesting
More effective pretest interviews
More efficient pretesting strategy
More robust results and useable
recommendations
• Additional expertise for review of reported
data
27
Thank you!
Diane K. Willimack
U.S. Census Bureau
[email protected]