Online Student Peer Reviews
William J. Wolfe
Professor of Computer Science
California State University Channel Islands
[email protected]
Online Peer Reviews: Outline
•
•
•
•
•
•
Pros and cons
Implementation Considerations
Student Websites
Course Website
Peer Review Process
Results
Peer Reviews – Concerns
•
•
•
•
•
•
•
Students don’t know the subject.
Students are not skilled evaluators.
Students might send inappropriate messages.
Students will not do that much work.
Students will copy (cheat)!
Keeping track of the reviews is very difficult.
Student privacy.
Peer Reviews – Advantages
•
•
•
•
•
•
•
•
Students learn from each other.
Students get lots of feedback.
Students develop skills as evaluators.
Students learn to appreciate evaluation criteria.
Students see how they compare to their peers.
Students see the class from teacher’s perspective.
Students get to know one another.
Teacher plays role of supervisor
(A much better use of the teacher’s skills/knowledge).
Implementation Details
•
•
•
•
•
•
•
•
•
•
•
What type of assignment?
How many reviews does each student do?
How many reviews does each student get?
Who reviews whom?
Does the reviewer have to be “qualified”?
Will students grade fairly and accurately?
Anonymous reviews?
Grades based on peer reviews?
Grade the peer reviews?
Opportunity to revise based on reviews?
Peer review of the peer reviews?
Peer Review System
Course Website
•
•
•
•
•
Instructor sets up a course website
Web pages
Database
Scripts
Keeps track of all the activity/data
Peer Review Process
• Student:
– Logs onto the course website
• Unique password for each student.
– Sees list of URLs
• List of links to the student web sites
– Picks one from the list
• Accesses a student web site
• Finds the assignment
– Reviews the assignment.
– Submits an anonymous review:
• score (1 – 10)
• comment
– Goes back to the list of URLs and picks another.
Logon
System Flow
List of URLs
link 1
link 2
link 3
---
Score +
Comment
Student Web
Site
back
Assignment
back
List of URLs
Student Website
• Students use their own website.
• Students must have basic web skills.
• Students must have access to a web service.
• Students cannot (easily) hide their identity.
COMP 449
Human Computer Interaction
John Doe
Weekly Assignments
COMP 449 Assignment #1
COMP 449 Assignment #2
COMP 449 Assignment #3
COMP 449 Assignment #4
COMP 449 Assignment #5
COMP 449 Assignment #6
COMP 449 Assignment #7
COMP 449 Assignment #8
COMP 449 Assignment #9
COMP 449 Assignment #10
COMP 449 Assignment #11
COMP 449 Assignment #12
COMP 449 Assignment #13
COMP 449 Assignment #14
Score + Comment
Grading Criteria (Rubric)
Peer Reviews Received
Sample Peer Review
”Looks pretty good”
Perfunctory Reviews
perfunctory \pur-FUNGK-tuh-ree\ -adjective : Done merely to carry out a
duty; performed mechanically; done in a
careless and superficial manner;
characterized by indifference
Detailed Peer Review
You should have requirements that detail the concepts in section 4.2.
Although you had some very good points (i.e. the database should
look up student's degree requirements; view should list courses, etc...)
almost all your requirements can be more detailed. Go through section
4.2 (each of the sections) and think of what the program would need
to do to effective run. Some good examples of what requirements are
necessary are on others' websites, however I'll give some to you now:
1.Is there a timeline requirement?
2.Is there a requirement on how much(or how little) this will cost?
3.Is there security requirements?
4.Is there user view requirements?
These(and many other questions) are what you should answer in your
requirements definition document. Good luck on Assignment #3.
Let’s try it out:
http://faculty.csuci.edu/william.wolfe/ucd/online/
Score
Avg Review Score (Comp 449 Spring 05)
10
9
8
7
6
5
4
3
2
1
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Student Ranking
# Missing Homworks Comp 449 Spring 2005
50
45
40
Score
35
30
25
20
15
10
5
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22
Student Ranking
Average Score Given Comp 449 Spring 2005
10
9
8
Score
7
6
5
4
3
2
1
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22
Student Ranking
# Reviews Received Comp 449 Spring 2005
80
70
60
Score
50
40
30
20
10
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22
Student Ranking
# Reviews Given Comp 449 Spring 2005
160
140
120
Score
100
80
60
40
20
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22
Student Ranking
Length of Comments Given Comp 449 Spring 2005
45000
40000
35000
Score
30000
25000
20000
15000
10000
5000
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22
Student Ranking
Length of Comments Received Comp 449 Spring 2005
12000
10000
Score
8000
6000
4000
2000
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22
Student Ranking
Logon ID Comp 449 Spring 2005
35
30
ID #
25
20
15
10
5
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22
Student Ranking
Comp 449 Spring 2005 Assignment 1
Student ID:6
10
9
8
score
7
6
You Gave
Class Average
5
4
3
2
1
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16
student
10
9
8
7
6
You Gave
Class Avg
5
4
3
2
1
1
2
3
4
5
6
7
8
9
10
10
9
8
7
6
You Gave
Class Avg
5
4
3
2
1
1
2
3
4
5
6
7
8
9
10
Summary
Peer Review Process:
• Stimulated class activity.
• Some passionate participation.
• The “audience effect”: brought up all performance levels.
• Very accurate evaluations (as a whole).
• Immediate access to examples of good and poor work.
• Addressed late and incomplete work.
• Requires web skills.
References
1. Online Student Peer Reviews, Proceedings of ACM SIGITE
Annual Conference, Salt Lake City Utah, Oct. 28-30, 2004.
2. Student Peer Reviews in an Upper-Division Mathematics
Class, exchanges THE ONLINE JOURNAL OF TEACHING AND
LEARNING IN THE CSU, (From the Classroom), September, 2003.
3. Course Web Site:
http://compsci.csuci.edu/wwolfe/ucd/online
Password: GUEST
4. [email protected]
Acknowledgements
Carol Holder (Director of Faculty Development CSUCI)
Paul Rivera (Economics, CSUCI)
Harley Baker (Psychology, CSUCI)
Bob Bleicher (Education, CSUCI)
Ivona Grzegorzcyk (Mathematics, CSUCI)
Todd Gibson (Colorado Institute of Technology)
Michael Cook (Forstmann Leff).
Peer Reviews – How?
• Student Web Pages:
– Students post homework solutions on their own
web page.
• Course Web Site:
– Set up course web site to manage all the peer
review activity. Keep track of:
• Links to student web pages,
• Peer Reviews:
– Scores,
– Comments.
• Anonymous reviews.
The Course Web Site
Student Web Pages
Average Peer Review Score
Scoring Comparison
Number of Reviews
Software Engineering (CSC 4508):
34 students
Theory:
1 Assignment:
1,122 reviews.
15 Assignments:
16,830 reviews.
1 Assignment:
300 – 400 reviews.
15 Assignments:
5,212 reviews.
Fact:
Software Engineering (CSC4508) Fall 2002
Average Review Score
10
9
8
Score
7
6
5
4
3
2
1
0
1
3
5
7
9
11
13 15 17 19 21 23
Students
25 27 29 31 33
Software Engineering (CSC 4508) Fall 2002
Average Review Score Given
9
8
7
score
6
5
4
3
2
1
0
1
3
5
7
9
11
13
15
17
19
Student
21
23
25
27
29
31
33
Number of Reviews Received
(CSC 4508 Fall 2002)
400
375
350
325
300
275
250
225
200
175
150
125
100
75
50
25
0
1
3
5
7
9
11
13
15
17
19
21
Student Ranking
23
25
27
29
31
33
Number of Reviews Given
(CSC 4508 Fall 2002)
400
375
350
325
300
275
250
225
200
175
150
125
100
75
50
25
0
1
3
5
7
9
11
13
15
17
19
21
Student Ranking
23
25
27
29
31
33
Avg Deviation
(CSC 4508 Fall 2002)
2
1.9
1.8
1.7
1.6
1.5
1.4
1.3
1.2
1.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
3
5
7
9
11
13
15
17
19
21
Student Ranking
23
25
27
29
31
33
Avg Delta
(CSC 4508 Fall 2002)
0.2
0.15
0.1
0.05
0
1
3
5
7
9
11
13
15
17
19
21
-0.05
-0.1
Student Ranking
23
25
27
29
31
33
Software Engineering (CSC 4508) Fall 2002
Distribution of Scores
1600
1400
1200
Count
1000
800
600
400
200
0
1
2
3
4
5
6
Score
7
8
9
10
Logon Sequence
(CSC 4508 Fall 2002)
35
30
25
20
15
10
5
0
1
3
5
7
9
11
13
15
17
19
Students
21
23
25
27
29
31
33
Real Analysis (Math 351) Spring 2003
Average Review Score Received
10
9
8
score
7
6
5
4
3
2
1
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20
Student Ranking