Evaluation Research into the
effectiveness of university learning
environments
Rob Phillips
Academic Consultant
Educational Design Project, School of Education, Curtin University,
Perth
Adjunct Associate Professor, School of Education, Murdoch
University, Perth
My Background
• 1980s
– Research scientist – theoretical chemistry and computer
science – Germany and Australia
• 1990s
– Interactive multimedia designer and project manager
• 2000s
– Educational designer
• 2010s
– Learning and Teaching Policy work
– Academic consultant – mentoring and educational
design
Introductions
• Why are you here?
authors to describe and
ion. It pr ovides both a
accessible ‘how-to’ guide
e-based learning.”
r Information Technology
ong
boration and
r ole-based e-learning
earch needed to design
ding practitioners
le, non-technical, and
evaluation of this
evaluating e-learning
o creating the
ary for students to take a
le-play simulations.”
Deputy Vice-Chancellor
ersity of Technology
e[ connecting with e-learning series
evaluating
e-learning
guiding research and practice
sis of the strengths and
about possible difficulties.
d pr ofessionals who
valuable resource for
staff involved in e-learning.
& Teaching at the
.
ng games and simulations
si P/L (Australia), a
rms.
Book advert
Disclaimer
• The book is largely about e-learning
• The ideas apply well to any learning
environment
– almost all include technology these days
Overview
•
•
•
•
•
•
Phenomena
The e-learning lifecycle
e-learning evaluation research
Cyclical evaluation research approaches
Divide and Conquer techniques
Work on own project
Fundamentals of Research
•
•
•
•
Inquiry into the behaviour of phenomena
Exercise of human curiosity
Attempting to solve problems in the world
Attempting to understand the world
In Broad Terms
• What sort of research do you do?
• What do you investigate?
• What are the main characteristics of this
research?
• Work in pairs
Fundamentals of Research
•
•
•
•
Inquiry into the behaviour of phenomena
Exercise of human curiosity
Attempting to understand the world
Attempting to solve problems in the world
What is a phenomenon?
• Inquiry into the behaviour of phenomena
• What is a phenomenon?
– an instance of something – an artefact or event
that is known through the senses
Natural Phenomena
• They exist naturally in our world
– the seasons, gravity and waves on the ocean
– we can observe and measure them
• Other natural phenomena cannot be directly
observed
– the inner workings of atoms and molecules
– their behaviour can be measured
Artificial Phenomena
• Phenomena/ artefacts which we create
through a design activity
• Tangible – aircraft, cars, etc.
• Intangible – computer software
Event Phenomena
• An artefact or event that is known through the
senses
• In Education
– a coming-together of teacher, learners and various
resources in a particular setting, usually following
some design activity
– Learning Environment
Summary
• Natural phenomena
• 1 phenomenon – many instances
• Artificial phenomena
– Many phenomena – many instances
• Event Phenomena
– Many phenomena – single instances
Paradigms vs Ideology
Paradigms
• The set of practices that define a scientific
discipline at any particular period of time
(wikipedia)
• A world view underlying the theories and
methodology of a particular scientific subject
(http://oxforddictionaries.com)
Ideology
• The set of beliefs characteristic of a social
group or individual
(http://oxforddictionaries.com)
• A comprehensive and coherent set of basic
beliefs about political, economic, social and
cultural affairs that is held in common by a
sizable group of people within a society.
(http://www.socialsciencedictionary.com)
Paradigms vs Ideology
• Is your research approach based on a
paradigm or an ideology?
Overview
•
•
•
•
•
•
Phenomena
The e-learning lifecycle
e-learning evaluation research
Cyclical evaluation research approaches
Divide and Conquer techniques
Work on own project
E-learning Artefacts and Environments
• We design and use e-learning artefacts
(intangible artificial phenomena)
– learning objects, learning management systems,
e-portfolios
• e-learning environments (event phenomena)
add the human context
– an interaction between learners, teachers and
resources in a set context
Inquiring About the World (IATW)
• New science unit for a new primary and early
childhood degree
– First offered in 2013
• Teaching problem
– How to make science interesting and engaging
• Design
– Authentic learning approach
– Students investigate a self-chosen environmental issue
over the whole semester
Learning Outcomes
• Develop lifelong learning skills including: analytical thinking,
information management, technology skills and
communication skills
• Apply processes of inquiry to carry out a scientific
investigation
• Report on the outcomes of the processes of inquiry using a
range of Information and Communication Technologies
• Evaluate the outcomes of the inquiry processes in the
context of the science and humanities content from the
Australian Curriculum
Assessment
1
Project Plan
15%
2
Report Draft
20%
3
Report
45%
4
Presentation
20%
Video
• IATW Flash video
The Learning Lifecycle
Learning lifecycle
Phase
Baseline
Design
Develop
Evaluate
Design
Trial learning env’t
Learning environment
Mature Learning
environment
The IATW Lifecycle
• On- campus and Open Universities Australia (OUA)
Date
Offering
July 2013
Curtin Semester 2
May 2014
OUA Study Period 2
July 2014
Curtin Semester 2
Nov 2014
OUA Study Period 4
May 2015
OUA Study Period 2
Overview
•
•
•
•
•
•
Phenomena
The e-learning lifecycle
e-learning evaluation research
Cyclical evaluation research approaches
Divide and Conquer techniques
Work on own project
My Invention
• The Kugelschreiber
• A better way to write!
Researching the Kugelschreiber
• I know that this technology is better than old,
pencil technology
• Compare the two, in writing a test
• Treatment and control groups
Researching the Kugelschreiber
• I know that this technology is better than old,
pencil technology
• Compare the two, in writing a test
• Treatment and control groups
Kugelschreiber results were slightly worse than
pencil results, but the difference was not
significant at the 90% confidence level. The
Kugelschreiber is no worse than a pencil.
Inquiry into Phenomena
• Natural
– Goal: observing how natural phenomena work
• and ultimately understand them
• Artificial
– Goal: design artefacts, improve their design and use
them
• and ultimately understand them
• Event
– Goal: design events, improve their design
• and ultimately understand them
Goals of Research
• Be clear about the phenomenon you are
investigating
• Clearly define the goals and questions of your
research
• Choose the research approach accordingly
Summary so Far
• We have thought about the phenomenon of
e-learning
• We have different research goals at different
stages of the e-learning lifecycle
• How will we conduct this research?
General Research Approaches
•
•
•
•
Pure versus applied research
Hard versus soft research
Research and development
Research then development
Becher, T. (1989). Academic tribes and territories: intellectual enquiry and the cultures of
disciplines. Buckingham: Society for Research into Higher Education.
Jones, C., Zenios, M., & Griffiths, J. (2004). Academic use of digital resources: Disciplinary
differences and the issue of progression Proceedings of the 4th Networked Learning
Conference. Lancaster University, England, UK.
Boyer’s Four Scholarships
•
•
•
•
Scholarship of discovery
Scholarship of integration
Scholarship of application
Scholarship of teaching
Boyer, E. L. (1990). Scholarship reconsidered - Priorities of the professoriate. New Jersey:
Carnegie Foundation for the Advancement of Teaching.
Boyer, E. L. (1996). The scholarship of engagement. Journal of Public Outreach, 1(1), 11-20.
Pasteur’s Quadrant
1.
2.
High
Pure basic research
(Bohr)
3.
Low
Quest for fundamental
understanding
Consideration of use
Low
High
Use-inspired basic
research
(Pasteur)
4.
Bird watching
Pure applied research
(Edison)
Stokes, D. E. (1997). Pasteur's Quadrant: Basic science and technological
innovation. Washington, D.C.: The Brookings Institution.
What is evaluation?
• The term is used imprecisely
• Evaluation implies judgement and decisions
• Formative evaluation:
– judgements about improving something
• Summative evaluation:
– judgements about the merit or worth of
something
Investigating a New Learning
Environment
• Questions:
– Does the environment function as designed?
– How can the environment be improved?
• These are evaluation judgements
• Is it any good???
Investigating a Mature Learning
Environment
• A study of a mature learning environment may
shed light on
– how learners engaged with the designed learning
environment
– why some learners achieved at different levels
– how some learners used the learning environment
to achieve a deeper understanding
• These are educational research questions
• The goal is increased understanding
Evaluation vs Research
• Studies of learning environments involve a
mixture of
– Evaluation: making judgements about improvement
and value and worth
– Research: increasing understanding
• Continuum between judgements and
understanding
– Does it function as designed?
Judgments
– How can it be improved?
– How does this lead to improved student learning?
• Similar methods – different goals
Understanding
IATW - currently
• Pre- and post-test
– Confidence with ICT
– Knowledge of scientific inquiry processes
• Anonymous survey of students
– Did they like it?
IATW questions to ask
• Was the website usable?
• What things didn’t work as well as they should?
• Was there evidence that students engaged with the
authentic task?
• Will the learning design scale up for 2000 OUA
students?
Overview
•
•
•
•
•
•
Phenomena
The e-learning lifecycle
e-learning evaluation research
Cyclical evaluation research approaches
Divide and Conquer techniques
Work on own project
The Learning Lifecycle
Learning lifecycle
Phase
Baseline
Design
Develop
Evaluate
Design
Trial learning env’t
Learning environment
Mature Learning
environment
Evaluation Research across the
Learning Lifecycle
Learning lifecycle
Baseline analysis
Design
Develop
Evaluate
Design evaluation
Formative evaluation
Formative evaluation
Summative evaluation
Effectiveness research
e-learning Research Approaches
• Need to apply to an artificial phenomenon
• Reflect the e-learning life cycle
• Cyclical research approaches
– Design-based research
– Action Inquiry
Design-based Research
• Focus on a designed learning environment
• A cyclical approach that originated in engineering
and other design fields
• Consistent with
– a mixture of evaluation and research
– Stokes’ views on quest for understanding and
consideration of use
Action Inquiry
• Focus on personal practice
• Local focus
• Little concern with theory
Drawing the Threads Together
• e-learning environments are designed
phenomena
– they follow a lifecycle
• Cyclical research approaches
• Mix of evaluation and research
– make sure it works first
• Now – onto the practical stuff
Overview
•
•
•
•
•
•
Phenomena
The e-learning lifecycle
e-learning evaluation research
Cyclical evaluation research approaches
Divide and Conquer techniques
Work on own project
Breaking Down the Complexity
• How to use all this information:
– It’s a complex situation, with
– many elements, which
– evolves over time
• It requires multiple research methods
• Use divide and conquer techniques and a
template to simply the issue
Divide and Conquer Techniques
• Evaluation research process diagram
• Five evaluation-research ‘forms’
• Break down the e-learning lifecycle
– To develop appropriate research questions
• Use an evaluation-research matrix
• Use template to combine into an evaluationresearch plan
Evaluationresearch
Process
Evaluationresearch
Process 1
Design Principles
• relatively humble ‘proto-theories’
• not ‘grand’ theories of learning
• Two roles for theory:
– Drawing on theory to support evaluation-research
goals and questions across the e-learning lifecycle
– Building theory from research outcomes
Evaluation Research across the
Learning Lifecycle
Draws on
theory
Learning lifecycle
Baseline analysis
Design
Develop
Evaluate
Design evaluation
Formative evaluation
Formative evaluation
Summative evaluation
Effectiveness research
Builds theory
Evaluationresearch
Process 1
The LEPO Framework
• Learning Environment, Process, Outcomes
Learning
environments
engage
in
design
work within
interact
with
Students
demonstrate
determine
Educational
Context
Learning
processes
facilitate
Teacher
s
assess
lead to
Learning
outcomes
facilitate
IATW Conceptual Framework
• Model for inquiry-based learning
• Model for information literacy
• Herrington’s 9-stage authentic learning model
Broad Evaluation-Research Questions
• The broadest questions that can be asked in
the study
• Open-ended, exploratory questions
• IATW examples:
– How can the unit design be improved for a large,
online OUA cohort?
– How effective is the unit at developing inquiry
skills in students?
Evaluationresearch
Process 2
IATW
• Stakeholders
– Centre for Teaching and Learning
– Head of School
– Standards bodies
• Participants
– Students
– Teaching staff
– LMS support staff
Activity
• Fill in page 1 of the template
• Discuss it with your partner
Divide and Conquer Techniques
• Evaluation research process diagram
• Five evaluation-research ‘forms’
• Break down the e-learning lifecycle
– To develop appropriate research questions
• Use an evaluation-research matrix
• Use template to combine into an evaluationresearch plan
Evaluation-research Forms
• Baseline analysis: Document the teaching and
learning context; needs analysis; assumptions,
literature
• Design evaluation: Evaluate the documented
design of the learning environment
• Formative evaluation: Make judgments about,
and suggest improvements to, the e-learning
environment
• Effectiveness research: Develop understanding of
how learners engage with learning tasks to
demonstrate learning outcomes
• Project management evaluation: judgements
about the conduct of a project
Evaluationresearch
Process 3
Specific Evaluation-research Questions
• Broad evaluation-research questions are hard
to answer
• Specific evaluation-research questions should
be answerable
– identify the evidence which is needed to answer
them
• Synthesise the evidence from the specific
questions to answer the broad questions
Examples of Specific Research Questions
Formative evaluation of the e-learning
environment
• How do learners use the learning environment?
• To what extent do learners engage with learning
tasks as intended?
• Are the learning tasks appropriate for the target
learners?
• How useful do learners find the e-learning
environment?
Divide and Conquer Techniques
• Evaluation research process diagram
• Five evaluation-research ‘forms’
• Break down the e-learning lifecycle
– To develop appropriate research questions
• Use an evaluation-research matrix
• Use template to combine into an evaluationresearch plan
Evaluationresearch
Process 3
Evaluation-research Matrices
• To map data sources and methods against the
research questions
• Examples from IATW
Baseline Analysis
Design Evaluation
Formative Evaluation
Effectiveness Research
Divide and Conquer Techniques
• Evaluation research process diagram
• Five evaluation-research ‘forms’
• Break down the e-learning lifecycle
– To develop appropriate research questions
• Use an evaluation-research matrix
• Use template to combine into an evaluationresearch plan
Evaluation-research Plan
• Use the template
• Work through the evaluation matrices and
develop questions which are relevant to your
project
• Specify your data sources
– Interview schedules
– Survey questions
Do the
evaluation
research
Overlap between Forms
Overview
•
•
•
•
•
•
Phenomena
The e-learning lifecycle
e-learning evaluation research
Cyclical evaluation research approaches
Divide and Conquer techniques
Work on own project
Evaluation-research Plan
• Use the template to start to develop your own
evaluation research plan
• Work through the evaluation matrices and
develop questions which are relevant to your
project
Application of the Approach
• PhD research
• Teaching improvement
• Preparation for grant applications
Where to publish your work?
Journal of Perspectives in Applied Academic Practice
http://jpaap.napier.ac.uk/index.php/JPAAP
International Journal for SoTL
http://academics.georgiasouthern.edu/ijsotl/v5n1.html
Invited essays, Research papers, essays
MERLOT Journal of Online Learning and Teaching
http://jolt.merlot.org
Journal of University Teaching and Learning Practice
http://ro.uow.edu.au/jutlp/
International Journal of Innovation in Science and Mathematics Education
IJISME is dedicated to encouraging and supporting early career researchers as
well as academics in science and mathematics who are new to education
research.
http://ojs-prod.library.usyd.edu.au/index.php/CAL/index