Can students design software:
CS specific pedagogical research
Lynda Thomas
Aberystwyth University
and
Sweden Group
Overview
• What is CS education research?
• My involvement in CS education research
• Graduating students’ software designs
• Several studies
• Design results
• Educational implications
• Careers
Education research
• How do students learn?
• What should students learn?
• What do students learn?
• How should things be taught?
• How do we measure what has been learned?
•…
CS Education research
• How do students learn CS topics?
• What CS topics should students learn?
• What CS topics do students learn?
• How should CS topics be taught?
• How do we measure what has been learned in CS?
• Although CS Education research has things in common with education
research in general,
doing CS education research requires domain expertise
How did I get involved?
• Always thought it was too ‘soft’ an area ….
• But also always had an interest in why my students behaved as they did
• Various career happenings described later ….
• Bootstrapping – an NSF summer workshop in USA 2001-3!
What do I do in this area now?
• Met people at Bootstrapping and later through ACM SIGCSE
• Mainly work in International groups – hence Sweden group
• Try to get colleagues from Aberystwyth involved (more later)
An example – investigating software design
• Pose significant questions
• Link research to relevant theory
• Research question: How do CS students
understand ‘produce a design’
• What do others say about design?
• Use methods that permit investigation
• Provide a coherent chain of reasoning
• Examine student designs and
characterise understandings
• Replicate and generalize across studies
• Repeat experiments in different contexts
• Disclose research to critique
• Write papers and give presentations
Shavelson, R.J. & Towne, L. (Eds.) (2002) Scientific research in education, National Academy Press
Graduating Students’ Designs –
Through a Phenomenographic Lens from ICER 2014
Lynda Thomas
Anna Eckerdal
Robert McCartney
Jan Erik Moström
Kate Sanders
Carol Zander
How do students understand design?
More specifically:
• What do students think “produce a design” means?
• How can this inform our teaching?
How do we examine these questions?
• Give students a design brief
• Analyze their answers
• Assumption: the artifacts produced reflect what students think
producing a design means.
Design brief
Your brief is to design a "super alarm clock" for University students to help them to manage their own sleep
patterns, and also to provide data to support a research project into the extent of the problem in this community.
You may assume that, for the prototype, each student will have a Pocket PC (or similar device) which is
permanently connected to a network.
Your system will need to:
 Allow a student to set an alarm to wake themselves up.
 Allow a student to set an alarm to remind themselves to go to sleep.
 Record when a student tells the system that they are about to go to sleep.
 Record when a student tells the system that they have woken up, and whether it is due to an alarm or not.
 Make recommendations as to when a student needs to go to sleep. This should include "yellow alerts" when
the student will need sleep soon, and "red alerts" when they need to sleep now.
 Store the collected data in a server or database for later analysis.
 Report students who are becoming dangerously sleep-deprived to someone who cares about them (their
mother?). This is indicated by a student being given three "red alerts" in a row.
The server/database system (which will also trigger the yellow/red alerts) will be designed and implemented by
another team. You should, however, indicate in your design the behaviour you expect from the back-end system.
You only have 50 minutes, so produce the best design that you can in this amount of time – you are producing an
initial solution that someone (not necessarily you) could work from. Include as many artefacts as needed so
someone could fill in the details for your design.
Why is
this
good?
Return of the “super alarm clock”
• This design brief has been used before:
• “Scaffolding” Studies
• Fincher et al., A multi-national, multi-institutional study of student-generated
software designs, 2004
• Chen et al., The (relative) importance of software design criteria, 2005
• Tenenberg et al., Students designing software: a multi-national, multiinstitutional study, 2005
• Eckerdal et al., Categorizing student software designs: Methods, results, and
implications, 2006
• Replicated by Loftus, Thomas and Zander, Can graduating students
design: Revisited, 2011
• Has now been re-replicated with different problem – to appear
Results from these studies
• “Scaffolding” studies:
Overall quality low – most students’ designs added little or nothing to the
description in the brief
Hard to distinguish designs based on extraneous factors (like grades,
number of previous courses)
• Loftus et al. study:
Performance better than with the Scaffolding data, but well below the
level expected by the instructor. Social factors at work?
This Methodology
• Use students with similar backgrounds and experience –
3rd year, in context of final projects course, week 1
• Integrate task into appropriate course;
offer feedback that could be valuable for the course
• Use phenomenography to analyze the resulting 35 designs;
i.e. the data analyzed are design artifacts rather than interview transcripts.
Basic result – much better!
But – wanted more detail …..
Skumtomte (swedish marshmallow treat)
So, we also did a ‘phenomenographic analysis’
which gives …
• An outcome space describing different understandings of what it means to
‘produce a design’
• A list of critical features that distinguish the elements of the outcome
space
Outcome space
0. informal design; no formal artifacts.
1. Uses some formal notation for analysis, but no system structure or behavior.
2. Uses some formal notation for analysis, describes system structure
3. Uses some formal notation for analysis, describes some dynamic behavior
4. Uses some formal notation for analysis - the static and the dynamic are linked
5. The notations are relaxed, only the essential artifacts are included
(see the expert)
Category 0: informal design; no formal notation
Category 1: Uses formal notation, no system structure or behavior
Category 2: Uses formal notation, describes system structure
Category 3: Uses formal notation, describes dynamic behavior
Category 4: Uses formal notation, links static structure and behavior
Category 5: notations relaxed, only the essential artifacts included
Conclusions (general)
• Student designs still not as good as might be hoped given the background
and experience of the students
• Some students in their final year performed like a layperson
• Most students know there is a ‘language of computing’ to do designs but
only use it in a superficial way (use cases)
• Most students did not link structure and behavior in their designs
Conclusions (phenomenography)
• To gain more detail, we used phenomenography with design artifacts to
identify different understandings of what producing a design means
• We were then able to identify the critical features that distinguished
these outcomes
• These critical features can be used with patterns of variation to teach
these understandings
Critical features
Language of Computing
Structure of components
Behavior of components
Link structure and behavior
Appropriate artifacts as needed (expert)
If you want to know more about variation theory see the paper …
So, what? …….
• A frequent criticism of phenomenographic papers is ‘that is obvious’
• But is it? I think it tends to be ‘obvious’ after you see it
• What has this meant for my teaching?
• ‘round trip design’
• use cases through to behaviour
• If you can’t do that then you haven’t designed!
My (not so brilliant) (but lots of fun) career
Born in UK. Moved to Canada with family
BSc and MSc in maths in Canada
Want to ‘go home’! Need job!
Worked in computer industry in UK
Fall in love! Need US visa!
MSc in Computing in USA
Need job! Have babies!
Worked in a college in the USA - teaching
Kids bigger! Want better job!
Got PhD…………..
….Realised needed PhD. Realised UK PhD is all research
……In UK
Daughter quite ill
Should have capitalized on it, but go back to US…..
Want to ‘go home’ & be in Aber!
Worked in Uni in UK as a Teaching Fellow
Want more fun!
Started doing CS Ed Research
Want better job! Bit surprised!
Got Promoted to Senior Lecturer
Time for a change
About to retire
Or, in graphical form
P
R
E
S
T
I
G
E
Chart Title
ME
70
YOU?
60
50
40
30
20
10
0
PG-20s
Early career 30s
Prof in US 40s
Teaching
Fellow - 50s
SL - 60s
What does that mean for you?
• You are here to demonstrate your (probably not CS Education) research
• Where will you work (What kind of places do Sweden group work)?
• What will that be like? (PGCTHE?)
• How do you see your career?
• Teaching / Research
• ‘real’ life
• The future? TEF??
Thank You
I hope you get to do some of this sort of
research!