Online Survey and
Assessment Techniques
Dr. Curtis J. Bonk
Associate Professor, Indiana University
President, CourseShare.com
http://php.indiana.edu/~cjbonk,
[email protected]
Pause…
What tools do you use or
know about?
Collecting
Evaluation Data
& Online
Evaluation Tools
Types of Data Collection
• Direct Observation in Work Setting
– By supervisor, co-workers, subordinates,
clients
• Collect Data By Surveys, Interviews,
Focus Groups
– Supervisors, Co-workers, Subordinates,
Clients
•
•
•
•
Self-Report by learners or teams
Email and Chat
Online Tests
E-Portfolios or Learning Logs
I. Online Testing
Tools
(see: http://www.indiana.edu/~best/)
Choice: Select companies that
specialize in online assessment.
Or: Use what the courseware
package gives ya…
Some things in life are free…!
Test Selection Criteria
(Hezel, 1999)
•
•
•
•
•
•
•
Easy to Configure Items and Test
Handle Symbols
Scheduling of Feedback (immediate?)
Provides Clear Input of Exam Dates
Easy to Pick Items for Randomizing
Randomize Answers Within a Question
Weighting of Answer Options
More Test Selection Criteria
• Recording of Multiple
Submissions
• Timed Tests
• Comprehensive Statistics
• Summarize in Portfolio and/or
Gradebook
• Confirmation of Test Submission
More Test Selection Criteria
(Perry & Colon, 2001)
• Supports multiple items types—multiple
choice, true-false, essay, keyword
• Can easily modify or delete items
• Incorporate graphic or audio elements?
• Control over number of times students
can submit an activity or test
• Provides feedback for each response
More Test Selection Criteria
(Perry & Colon, 2001)
• Flexible scoring—score first, last,
or average submission
• Flexible reporting—by individual
or by item and cross tabulations.
• Outputs data for further analysis
• Provides item analysis statistics
(e.g., Test Item Frequency
Distributions).
Web Resource: http://www.indiana.edu/~best/
II. Electronic Gradebooks
(Vockell & Fiore, 1993)
• Calculate scores, store info
• Weight scores
• Flag students with certain
characteristics
• Print reports by individual or
group
• Provide prompt feedback
• But inflexible, impersonal, &
can be incorrect
III. Online Survey
Tools for
Assessment
Perhaps you have heard of
my survey techniques?
Survey Tool Features
•
•
•
•
•
•
•
•
Maintain email lists and email invitations
Conduct polls
Adaptive branching and cross tabulations
Modifiable templates
Maintain library of past surveys
Publish reports
Technical support, chat advice
Different types of accounts—hosted,
corporate, professional, etc.
Still More Online Survey
Features
• Online help, FAQs, demos, tutorials,
white papers, glossaries, site maps,
tips, overviews
• Website evaluation tools or options
• Supports different types of items
(Likert, multiple choice, forced
ranking, paired comparisons, etc.)
• Customizable audience lists
• Launch and close surveys
More Online Survey Features
• Online viewing of assessment results
and question analysis (including
various graphical displays)
• Easy to author surveys
• Instant feedback to participants
• Custom reporting, editable images
and background, personalized
greetings
• Storage of results in Access or SQL
Still More Features
•
•
•
•
•
•
•
Ability to start and stop a survey
Reports on subgroups
Randomize q’s to reduce bias
Easy email reminders
Personalized banner or logo
Expanded question options
Ability to copy surveys
Web-Based Survey
Advantages
• Faster collection of data
• Standardized collection format
• Computer controlled branching and
skip sections
• Graphics can reduce fatigue
• Easy to answer clicking
• Wider distribution of respondents
Web-Based Survey Problems:
Why Lower Response Rates?
•
•
•
•
•
•
•
Low response rate
Lack of time
Unclear instructions
Too lengthy
Too many steps
Can’t find URL
Perceived as aggressive
Web-Based Survey
Solutions: Some Tips…
•
•
•
•
•
•
•
•
Send second request
Make URL link prominent
Offer incentives near top of request
Shorten survey, make attractive, easy
to read
Credible sponsorship—e.g., university
Disclose purpose, use, and privacy
E-mail cover letters
Prenotify of intent to survey
Tips on Authentification
•
•
•
•
Check e-mail access against list
Use password access
Provide keycode, PIN, or ID #
(Futuristic Other: Palm Print,
fingerprint, voice recognition, iris
scanning, facial scanning, handwriting
recognition, picture ID)
Sample Survey Tools
•
•
•
Zoomerang
(http://www.zoomerang.com)
IOTA Solutions
(http://www.iotasolutions.com)
QuestionMark
•
(http://www.questionmark.com/home.ht
ml)
Survey Solutions from Perseus
•
Infopoll (http://www.infopoll.com)
(http://www.perseusdevelopment.com/fromsurv.
htm)
Sample Survey Tools
•
Active Feedback
•
(http://www.activefeedback.com/af)
•
•
•
•
Majon International
(http://www.coolsurveys.com)
EZSurvey from Raosoft
(http://www.raosoft.com/)
•
SurveyShare
(http://SurveyShare.com; from
Courseshare.com)
Zoomerang Steps
1. Register: Sign Up for Basic
Membership (SKIP FOR NOW!)
2. Click on new survey.
3. Click on survey template (e.g.,
“Education” and Professor)
4. Delete, modify, or add 1-2 items.
5. Submit survey.
Zoomerang Steps
6.
7.
8.
9.
Modify Web greeting and submit.
Pick option #1 for submission.
Type Survey invitation & email message
Enter email addresses for potential
survey respondents
10. Decide on how share results
Zoomerang Steps
11. Launch survey
12. Click on “My Surveys” and browse
13. Click on “Email List” and browse
14. Click on “Account Info”
15. Other explorations…
SurveyShare Steps
1. Register: Sign Up (perhaps take Quick
Tour, Flash version)
2. Login
3. Click on Create Survey
4. Check out existing templates
5. Select one of the templates
6. Add one item, delete one item, modify
one item
SurveyShare Steps
7. Edit Survey Introduction and Title
8. Edit Closing and Submit
9. Select Survey Invitation Via Email
10. Type emails to complete the survey
11. Click add (to move emails to right)
12. Give email subject and message
SurveyShare Steps
13. Enter email subject line and text
14. Preview survey
15. Activate survey
16. Go to MySurveys
17. Click on Survey Info
18. Go to mailing lists
SurveyShare Steps
19. Add a person
20. Add list
21. Check out resources (books)
22. Check out help, quick tour, and sitemap
23. Perhaps create a survey from scratch
24. Perhaps create a survey from “My
Previous Surveys”
Any Reactions to
Online Surveys???
Fitting Surveys into
Evaluations Plans
Bonk’s Evaluation Plan…
Considerations in Evaluation Plan
8. University
or
Organization
7. Program
6. Course
5. Tech Tool
1. Student
2. Instructor
3. Training
4. Task
What to Evaluate?
1. Learner—attitudes, learning, use,
performance.
2. Instructor—popularity, course enrollments.
3. Training—internal and external components.
4. Task--relevance, interactivity, collaborative.
5. Tool--usable, learner-centered, friendly,
supportive.
6. Course—interactivity, participation,
completion.
7. Program—growth, long-range plans.
8. Organization—cost-benefit, policies, vision.
The 4 Levels
• Reaction
• Learning
• Behavior
• Results
Percent of Respondents
Figure 26. How Respondent Organizations Measure
Success of Web-Based Learning
90
80
70
60
50
40
30
20
10
0
Learner satisfaction
Change in
knowledge, skill,
atttitude
Job performance
Kirkpatrick's Evaluation Level
ROI
Ziegler, April 2002, eLearning
“…the key is not to measure
every possible angle, but
rather to focus on metrics
that are pragmatic and
relevant to both human and
business performance at
the same time.”
Some Final Advice…
Or Maybe Some Questions???