Affective Assessment
of Team Skills
in Agile CS1 Labs:
The Good, the Bad, and the Ugly
Dawn McKinney and Leo F. Denton
http://www.cis.usouthal.edu/~mckinney/SIGCSE2005.ppt
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Affective Assessment
of Team Skills
in Agile CS1 Labs:
The Good, the Bad, and the Ugly
Dawn McKinney and Leo F. Denton
http://www.cis.usouthal.edu/~mckinney/SIGCSE2005.ppt
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
University of South Alabama



State university
13,000 students
Computer and Information
Sciences
 500 CIS majors
 Three specializations
 Undergraduate and
Masters Program
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
OUR CS1 ENVIRONMENT







2-semester course for majors
75 minutes per week closed lab
Java programming language
Graduate assistants for support
Required personal laptops
Supplemental Instruction
Small class sizes
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Bloom’s cognitive taxonomy
Evaluate solutions to complex problems
that involve class hierarchies and the
use of polymorphism.
Develop software that involves
the use of new classes based on
refining existing classes.
Perform arithmetic operations
on various data types.
Explain source code and
how it is produced.
Define three kinds of
program errors.
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Krathwohl’s affective taxonomy
Discuss specific personal
initiatives which demonstrate a
commitment to life-long learning.
Show strong work ethic and
initiative while working in
groups.
Use coding standards as
taught in the lectures.
Become aware that testing
is a part of the software
development life cycle.
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
NACE Top-10 characteristics most
wanted in college graduates










Communication skills
Honesty/integrity
Teamwork skills
Interpersonal skills
Motivation/initiative
Strong work ethic
Analytical skills
Flexibility/adaptability
Computer skills
Organizational skills
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Our study
Spring 2004






Team skills promoted in a closed lab
Integrated cognitive-affective objectives
Agile practices of Extreme Programming
Semester-long project
Quantitative and Qualitative Assessment
Improvements in Fall 2004
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Integrated cognitive-affective
objectives for team skills
Communication Communicate with students and
faculty about course concepts and
practices.
Cooperation
Cooperate with a team in an effort to
solve problems and develop software.
Commitment
Demonstrate a commitment to quality
software development with good
design and testing practices.
Demonstrate a strong work ethic by
attending class and participating fully.
Demonstrate adaptability in software
development practices.
Work Ethic
Adaptability
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Forming of teams




Students formed their own teams
Between 5 and 9 students
Team names
Problems with unbalanced skill levels
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Semester-long project





Teams were assigned the same project
Only during closed lab
“Real world” problem to solve
Three project iterations
Peer evaluations
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Agile Practices






Planning game
Small releases
Metaphor
Simple design
Test-driven
development
Refactoring
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO






Collective ownership
Continuous
integration
Sustainable pace
On-site customer
Coding standards
Stand-up meetings
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
High impact practices
Pair-programming
 Collective code ownership
 Test-driven development
 On-site customer
 Stand-up meetings

Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Pair-programming







Code in pairs only
Record pair programming experiences
Take turns at “driver” and “co-pilot”
“Pair pressure”
Cooperation
Communication of ideas
Develop and practice adaptability skills
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Collective code ownership





Code kept on a server
Any team member could change the code
Adaptability
Communication
Cooperation
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
On-site customer





Customer made various project changes
Instructor was the customer
Adaptability
Cooperation
Normalcy of change
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Test-driven development







Automated tool
New to students
Inexperienced programmers
Needed more instruction on TDD
Needed a more appropriate IDE
Adaptability
Cooperation
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Stand-up meetings







2 - 5 minute meetings
Each team member reports
Communication
Accountability
Cooperation
Commitment
Work ethic
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Assessment instruments
Peer Evaluation
 Final Peer Summary Evaluation
 Intrinsic Motivation Inventory (IMI)
 Anderson-Butcher’s Belonging Scale
 Observation

Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
First and second peer evaluations
1.
2.
3.
4.
5.
Specific positive contributions of this team
member
Best qualities this member added to the team
effort
Weaknesses of this member
Would you want this member on your team
again? (yes/no)
Overall level of contribution of this team
member (Low to High on a 5 point Likert scale)
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Frequency-of-use score as an unobtrusive
measure of the internalization of team skills




Review of student evaluations
Identification of team-based skills
Assignment of frequency-of-use score
Correlation of the frequency of use score with:
 Course grade
 First and second overall peer evaluation scores
 Final summary scores for each of the five skills
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Measuring affective factors



Affective factors correlate with achievement
In past semesters we found a decrease in
factors
Agile labs show increases in sense of
belonging and lack of pressure
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Significant positive correlations
Pearson Correlations with Frequency of Use
N
p-value
Course Grade
0.447*
24
0.029
1st Peer Evaluation
0.537**
24
0.007
2nd Peer Evaluation
0.661**
23
<0.0005
Work Ethic
0.690**
23
<0.0005
Communication
0.683**
23
<0.0005
Cooperation
0.615**
23
0.002
Commitment
0.677**
23
<0.0005
** Significant at the 0.01 level (two-tailed)
* Significant at the 0.05 level (two-tailed)
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Increase in belonging, decrease in pressure
Early/Late Factors
t-value
N
p-value
Pressure
-3.637**
23
0.002
Belonging
2.282*
23
0.035
Student Belonging
2.773*
23
0.013
** Significant at the 0.01 level (two-tailed)
* Significant at the 0.05 level (two-tailed)
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Sample peer evaluations data
Communication
Cooperation
Work Ethic
The Good
The Bad
“He is good at explaining
things to those who
are not grasping a
particular concept.”
“He sometimes talks
over weaker
members.”
“Maybe you should
discuss appropriate
behavior toward
women with him.”
“A good quality is her
positive attitude and
willingness to
participate and help
where needed.”
“Does not contribute
to group
discussions/effort
noticeably or
effectively.”
“I think he talks out of
turn, his comments
are inappropriate,
and he is
disrespectful toward
the customer.”
“She is willing to learn
and take extra steps
to dig for material. I
feel she is determined
to do well.”
“He is normally
absent and when
present never
wants to work on
the project.”
“Sleeps during class, is
absent a lot, doesn’t
really help during
lab, just kind of sits
back and watches”
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
The Ugly
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
To round up the bad…



Provide timely feedback on peer
evaluations
Supply instruction on team skills
Choose balanced and effective
teams
“He is set in his way of thinking”
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
To corral the ugly…


Deal with disrespectful or
uncooperative students
Require problem students to
work alone or among
themselves
“Maybe you should discuss appropriate behavior
toward women with him.”
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
To carry on the good…




Promote team skills
Provide a semester-long lab
project
Use agile practices
Conduct peer evaluations
“She has an open mind and wants to hear and
learn about what others bring to the table ”
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
Benefits of early team experiences






Assessment of affective objectives
Preparation for upper-level courses
Development of skills needed by industry
Natural active and cooperative learning
Women and other minorities
Sense of belonging
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA
http://www.cis.usouthal.edu/~mckinney/SIGCSE2005.ppt
Leo F. Denton
Dawn McKinney
[email protected]
[email protected]
Affective Assessment of Team Skills
SIGCSE 2005
St. Louis, MO
School of Computer and Information Sciences
University of South Alabama, Mobile, Alabama USA