Best Practices in Survey Development for the Evaluation of
Student Gains from Entrepreneurship Programs and Classes
Presented by: Jessica Menold
Source: http://blog.ivywise.com/blog-0/bid/156366/ACT-and-SAT-Test-Prep-Resources
2000 entrepreneurship courses are offered nationwide
and evaluating the impact of this education is critical
Source: http://epicenter.stanford.edu/page/about-programs
Today we will review a set of nine metrics, used to evaluate
eleven entrepreneurship assessment instruments
Content relevance and representativeness refer to the range
and limits of content coverage
Content to be assessed
Content relevance and representativeness set the boundaries of the
construct domain to be assessed by an instrument or measure
According to Messick’s theory, validity can be
differentiated into six aspects
Content
Structural
External
Source: http://galleryhip.com/post-it-notes-transparent.html
Downing and Haladyna advocate for the use of
qualitative evidence in the development test items
Content definition
Test Specifications
Item Verification
Item Tryout
Content definition links the construct that is to be measured
with the test specifications and items
Three metrics were derived for Content Definition:
1. Clearly defined boundary of the construct(s) of
the instrument or survey
2. Definition of the construct(s)based on literature,
job task analysis, and/or expert review
3. Documentation of methods used to select the
content presented5
Test specifications tie the content domain with the test
items and define the segments of the domain to be assessed
Three metrics were derived for Test Specifications:
4. Detailed rationale or process for breakdown of
proportions of content domain to be represented
by test items
5. Content areas and more specific content within
these areas clearly stated
6. Agreement on proportional breakdown by panel of
researchers or content experts
Item verification can help ensure that items reflect content
that is representative and relevant to the domain of interest
Two metrics were derived for Item Verification:
7. Initial items based on literature review, previous
instruments/survey, field experience
8. Items verified through panel of content experts on
both relevance and representativeness to the
content domain relative to the pre-defined
construct definition
Item “tryout”, or pretesting, is a method used to evaluate
the cognitive processes that items evoke from test takers
One was derived for Item Tryout:
9. Pretesting of items using think aloud or other
protocols to ensure the test takers engage in
the expected cognitive processes
We used these metrics as a lens to examine eleven
instruments used to evaluate entrepreneurial constructs
1.
2.
3.
4.
5.
6.
Entrepreneurial self-efficacy scale
NCIIA Entrepreneurship Inventory
Proactive Personality Scale
Entrepreneurial self-efficacy scale
General enterprising tendency
Individual entrepreneurial
orientation
7. Student entrepreneurial orientation
8. Entrepreneurial behavior inventory
9. Tolerance for Ambiguity Scale
10. Engineering Entrepreneurship
Survey
11. Entrepreneurial mindset index
We chose the NCIIA Entrepreneurship Inventory as
an illustrative example to review the use of each metric
Clearly defined boundary of the construct(s) of the instrument or survey
Definition of the construct(s)based on literature, job task analysis, and/or
expert review
Documentation of methods used to select the content presented5
Detailed rationale or process for breakdown of proportions of content
domain to be
Content areas and more specific content within these areas clearly stated
Agreement on proportional breakdown by panel of researchers or content
experts
Initial items based on literature review, previous instruments/survey, field
experience
Items verified through panel of content experts
Pretesting of items using think aloud or other protocols to ensure the test
takers engage in the expected cognitive processes
The selected instruments are strongest in content definition
and weakest in test specifications/blueprint creation
19%
25%
Met
Unmet
75%
Met
81%
Unmet
Content definition
Test Specifications
33%
46%
Unmet
Met
54%
Met
Item Verification
67%
Unmet
Item Tryout
In summary, two key improvements for entrepreneurship
instruments include:
The adoption and use of test
specifications and test blueprints
The use of content experts to verify
item representativeness and relevance
Thank you