Evaluating Learning in Classroom Activities using Digital
Technologies.
Louise Starkey
Victoria University of Wellington
Access to digital technologies and the World Wide Web is changing some
learning activities that occur in classrooms. In a doctoral study of six
beginning teachers in secondary schools it was found that the purpose and
the learning that was occurring while using the digital technologies varied.
A matrix was designed and applied to evaluate learning activities that use
digital technologies. It was developed through consideration of current
thinking about the nature of knowledge, creativity, learning theories for the
digital age, and existing taxonomies. It includes five categories of digital
technology use; accessing information, presenting information, processing
information, gaming and communicating. This paper describes and explains
the digital age learning matrix.
Ideas about knowledge and learning are changing and this is reflected in changes that are occurring
in secondary schooling. It is moving from being a place where students learn what has already been
discovered and prescribed as ‘knowledge’ that they can then demonstrate in exams to a place
where, critical thinking, connections and knowledge creation are valued. This is evident by
comparing national curriculum documents from pre 1980s and the recently released 2007 New
Zealand Curriculum document (Ministry of Education, 2007).
Part of this paradigm shift in education is a result of digital technologies becoming ubiquitous,
allowing 24/7 access to information and people. The introduction of a networked world where
information can be found in seconds (with the right key words), mashed, further developed and
shared using web2.0 applications, is contributing to less importance being placed on remembering
facts and figures and increasing importance on the critique of information and sources, creativity
and the connections learners make (Tapscott & Williams, 2006).
Teachers reflect on their practice and are being studied by researchers as they strive to use digital
technologies in educationally meaningful ways. Therefore a way of evaluating such learning
activities is needed. Evaluating learning using digital technologies has been attempted in a number
of ways. Cassady (2002) summarises ways of evaluating learning in the cognitive domain that is
measured by improved student achievement over a project. This type of evaluation differs from one
which seeks to evaluate the potential and actual depth of learning in the design of an activity. An
effective evaluation tool would recognise the range of ways technologies can be used for learning
and the importance of critical thinking, creativity (knowledge creation) and making connections in
an information rich web2.0 world.
The use of critical thinking has been identified as important in the digital age as relatively quick
access to a wide range of information means that the user needs the ability to critically evaluate the
validity and value of information accessed. The difference to the past is the individual’s access to a
range of information and the way that information is accessed. In the past the library, a book or an
expert (eg. a teacher) were consulted, and the value or validity was less likely to be questioned.
(Rowlands, Nicholas et al., 2008). Critical thinking is an important aspect of learning when using
digital technologies.
Pulling together work by educational researchers, it could be surmised that ideas about
‘knowledge’ appear to be changing from something that is found in the heads of individuals or in
books to something that is not fixed, is debatable, accessible through a range of mediums and
created through networks, connections and collaboration (Bereiter, 2002; Gilbert, 2005; Siemens,
2004).
George Siemens (2004) developed ‘connectivism’ as a learning theory for the digital era. The
theory considers how people, organisations and technology work collaboratively to construct
knowledge. Siemens describes connectivism as:
‘the integration of principles explored by chaos, network, and complexity and selforganization theories. Learning is a process that occurs within nebulous environments of
shifting core elements – not entirely under the control of the individual. Learning (defined
as actionable knowledge) can reside outside of ourselves (within an organization or a
database), is focused on connecting specialized information sets, and the connections that
enable us to learn more are more important than our current state of knowing.’ (Siemens,
2004)
The connections that a student makes are an important aspect of learning in the digital age.
Creativity has been described in various ways. In Wikipedia it is notes that ‘more than 60 different
definitions of creativity can be found in the psychological literature’ (Wikipedia, 2007). Plato
described creativity when he compared two types of artist; true artists as those who bring into birth
some new reality as opposed to artists who deal only with appearances and not with reality
itself.(Plato, 360 BCE, as cited in Anderson, 1959, p. 57). He was making a distinction between
artists who were skilled at manipulating tools and artists who were able to show something new as
a result of manipulating the tools. This distinction is important. Taking this definition, being able to
reproduce an existing reality or knowledge should not be considered as demonstrating creativity in
schools.
A second definition of creativity comes from an educational setting. The National Advisory
Committee on Creative and Cultural Education in England gives a schooling based definition of
creativity as: ‘Imaginative activity fashioned so as to produce outcomes that are both original and
of value’ (1999, p. 29). This aligns with Plato’s definition.
Stephen Downes, a leading voice in elearning research and regular blogger on the use of digital
technologies in schools, recently defined creativity as ‘the manipulation of one’s experiences using
the tools at one’s disposal’(2007). In this definition, Downes implies that original ideas are the
result of experiences. This definition like the previous one requires a product, but differs in that it
specifies that tools are an important aspect of creativity.
Not all definitions of creativity include having a tangible product. In a Ted Talk presentation titled
do schools kill creativity?, Sir Ken Robinson defines creativity as ‘having original ideas that have
value’(2006). Combining the ideas of Plato, NACCCE, Downes and Robinson a model of
creativity can be developed (figure 1).
Figure 1: Creativity model based on definitions.
In the digital age the notion of connections and collaboration become explicit, with the model
underpinned by critical thinking.
Figure 2. Creativity in the digital age or Knowledge creation in the digital age.
Knowledge is rarely developed in isolation, and between the students’ ideas and the creative
product, there would usually be input from other people. The phenomenon of the long tail
(Anderson, 2006) means that through the World Wide Web, learners and knowledge creators are
able to connect with others in the world with similar interests who are able to critique and give
feedback, something that was not so easy prior to web2.0. The creativity model for the digital age
(figure 2) could be renamed ‘Knowledge creation in the digital age’.
These emerging ideas of knowledge creation, Connectivism and critical thinking in the digital age
form the basis for higher level learning in the ‘digital age learning matrix’.
The digital age learning matrix was developed to be used as a research tool when examining the
types of learning activities that incorporate the use of digital technologies in the classroom. It has
been used as the basis for discussion during interviews as teachers explain what they are planning
or how they have used digital technologies with their students, and a simplified version used with
secondary students to ascertain the level of learning that did occur as a result of their use of digital
technologies. The matrix has been trialled successfully with preservice teachers as they reflect on
the use and potential use of digital technologies in their teaching.
Existing taxonomies, while useful and relevant did not suit the needs of the tool (table 1). The
SOLO taxonomy (Biggs & Collis, 1982) includes a useful progression, but it fundamentally
focuses on the knowledge being within the learner, not including the learning or knowledge
creation that can occur between learners in a connected world as described by Siemens (2006). It
implies that students are learning what is already known and fails to acknowledge the importance
of connections with others in the process of learning. Bloom’s taxonomy puts knowledge at the
lowest level of cognitive processing which is now thought to be inappropriate. However, the
modified Bloom’s taxonomy (Anderson & Krathwohl, 2000) could be useful alongside the matrix
even though it is based on cognitive processes and is too linear and restrictive for the holistic
approach needed.
Bloom’s modified taxonomy has ‘create’ at the highest level in the cognitive domain, SOLO has
extended abstract- noting the link between the learning and other contexts. In a digital world,
“creativity” especially in terms of “creating knowledge” could be considered as the penultimate
learning experience for developing conceptual or procedural understanding. The ultimate level is
sharing and ascertaining that the created knowledge has value. This can be done through
connections in a web2.0 learning environment.
Six different levels of learning using digital technologies are identified in the research tool. The
basic level is when students do something within one context. This includes looking on the internet
for some information, posting pictures to a wiki, take a quiz to rote learn facts or post a comment
on a blog. The second level of learning is when students are being asked to make connections
within the context of intended learning. It may include a compare and share activity, how to do
something in different contexts or connecting to a person to share ideas. The third level of learning
requires students to demonstrate conceptual understanding of ‘big ideas’ (conceptual knowledge).
The fourth level involves critique and evaluation, exploring the limitations and potential of
information, sources or a process (procedural knowledge). The fifth level is where students create
new knowledge. At this level students develop an original product that provides a new reality,
using ideas or processes that they have critiqued and evaluated. The final level of learning brings
together the idea of connectedness, critique and creativity. It occurs when students share the new
knowledge through authentic contexts and gain feedback to measure value.
Digital age learning matrix.
Modified Blooms
taxonomy- cognitive
process domain.
Doing
Isolated information. Focus on
completing a measurable task.
1. Remember
recognizing
recalling
Thinking about connections
Connecting thinking.
Simple connections made within the
context of intended learning. Compare
and share.
2. Understand
interpreting,
exemplifying,
classifying,
summarizing,
inferring,
comparing,
explaining
3. Apply
executing,
implementing
4. Analyse,
differentiating,
organizing,
attributing (?)
5. Evaluate
checking,
critiquing
Thinking about concepts
Develop conceptual understanding of
‘big ideas’
Critiquing and evaluating.
Evaluating and critiquing to explore the
limitations and potential of information,
sources or a process.
Creating knowledge.
Creativity-Applying ideas, processes
and/or experiences to develop a new
reality.
6. Create
generating,
planning,
producing
Structure of
Observed
Learning
Outcomes taxonomy
Prestructural: students are simply
acquiring bits of unconnected
information, which have no
organisation and make no sense.
Unistructural: simple and obvious
connections are made, but their
significance is not grasped.
Multistructural: a number of
connections may be made, but the
meta-connections between them are
missed, as is their significance for
the whole.
Relational level: the student is now
able to appreciate the significance
of the parts in relation to the whole.
Extended abstract: the student is
making connections not only within
the given subject area, but also
beyond it, able to generalise and
transfer the principles and ideas
underlying the specific instance.
Sharing knowledge.
Sharing the new knowledge through
authentic contexts and gaining feedback
to measure value.
Table 1. Comparison of levels of learning with modified Bloom’s and SOLO taxonomies
The levels are not necessarily sequential, and not all learning activities in secondary schooling
should be aiming for the highest levels. The purpose of the matrix was as a research tool, one that a
teacher and the researcher can sit down together and discuss the aims and outcomes of a learning
activity that uses digital technologies. Ideally teachers will design activities that often include the
second, third and fourth level. The fifth and sixth level will occur less often and only after a
thorough understanding of concept(s) or process(es) is gained, depending on the aims of the
teaching and learning needs of the students.
.
In the following matrix, the five categories of digital technology use are matched with levels of learning.
Level of learning:
Doing
Thinking about
connections
Thinking about
concepts
Critiquing and
evaluating.
Creating
knowledge.
Sharing
knowledge.
Explanation of level
of learning:
Isolated information.
Focus on completing
a measurable task.
Connecting thinking.
Simple connections
made
within
a
context.
Compare
and share.
Develop conceptual
understanding of ‘big
ideas’
Evaluating
and
critiquing to explore
the limitations and
potential
of
information, sources
or process.
Creativity-Applying
ideas,
processes
and/or experiences to
develop a new reality.
Sharing the new
knowledge through
authentic
contexts
and gaining feedback
to measure value.
Digital technology
use:
Accessing
information
Presenting
Accessing:
Pictures
Graphs
Movies
Data
Information
Present information
using:
Sound
Pictures
Words
Video
Information from more
than one source is
connected or compared in
analysis.
Presented information has
clear connections across
formats or ideas.
Information explicitly
develops conceptual
understanding.
Presentation (or
explanation of
presentation) has explicit
conceptual underpinning.
Information and sources
are critiqued and
evaluated.
The presentation,
methods and results are
critiqued and evaluated.
New conceptual
understanding is
developed. Building on or
linking accessed
information.
Critiqued and developed
ideas or new knowledge
is presented.
Processing
information
Information is processed
or data/images are
manipulated in isolation.
Connections are made
between or within
processed
information/data or
images and relevant
concepts.
Gaming
Play a game
Take a quiz
Enter a virtual world
Links made between the
game/ quiz/ virtual world
and other knowledge.
The relevant concepts
within the game, quiz or
virtual world are
identified and explained.
The game, quiz or virtual
world is critiqued and
evaluated within a
conceptual context.
Original ideas are used to
create a knowledge
product in any medium.
Communicating
Send a communication
Receive a communication
Read a communication
Ideas compared and
shared with other learners
through a two way
conversation (written or
verbal)
Communication explicitly
develops conceptual
understanding.
Critique other peoples’
work or ideas.
Through interaction and
communication new
knowledge is constructed.
Table 2; Digital Age Learning Matrix.
Processed data or
information has clear
conceptual underpinning.
Process and product are
critiqued and evaluated.
Ideas and new knowledge
are developed.
The value of the product
is determined by the
quality and quantity of
feedback from beyond the
classroom environment.
Learning occurs when the
feedback is considered
and analysed.
Digital technologies are used in learning for a range of purposes. Teachers in case studies and
preservice teachers were observed designing and using activities that required students to
access, process or present information, interact in a gaming situation or communicate through
digital technologies. These five categories cover the range of observed learning activities that
use digital technologies.
Teachers in the research study design learning activities in which students access informationthis included downloading powerpoints, looking at websites, webquests, watching movies
from you tube or developed by peers and analysing graphs or data. Learning activities may
require students to present information. This could be in the form of contributing to a wiki
with words, pictures and/or sound, it could be to make a movie, a podcast, take photos of a
process. It could include students developing a multichoice quiz. Students may be asked to
process information. This includes manipulating data or information in a spreadsheet, altering
images through a programme like paint shop or developing mashups. Gaming includes
interactive programmes which have an element of competition. This includes learning
mastery activities where students take quizzes and gain instant feedback on their progress.
Communicating using digital technologies could be through comments on a blog or wiki,
through making podcasts and giving feedback on these, it could be through email or instant
messaging. Communications could be with individuals or groups and could be within the
learning community or beyond the classroom to a wider open or closed community.
The digital age learning matrix (table 2) combines the levels of learning with the categories of
digital technology use.
Teachers using the matrix highlight the boxes that reflect where they believe the level of
learning and digital technology use intersect for the learning activities they design. This forms
the basis of a discussion with the preservice teachers and is part of the research report for
research participants.
SUMMARY
As a result of observations of teachers and students a digital learning matrix was developed
that combines categories of observed use of digital technologies with levels of learning.
Building on the work of researchers in the field of knowledge and the digital age, creativity,
connections and critical thinking were identified as being crucial in underpinning the levels of
learning within the digital learning matrix. Existing taxonomies were not suited to the needs
of a researcher examining the learning as secondary school teachers and students use digital
technologies in their programmes. The resulting digital age learning matrix is designed as a
reflection tool for teachers and students as they consider the depth of learning that is
occurring in classroom activities which encourage participation in the information rich
web2.0 world.
REFERENCES
Anderson, C. (2006). The Long Tail. London: Random House.
Anderson, H. H. (Ed.). (1959). Creativity and its cultivation. New York: Harper and
Brothers.
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2000). A Taxonomy for Learning,
Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational
Objectives: New York:Allyn & Bacon.
Bereiter, C. (2002). Education and mind in the knowledge age Mahwah,
N.J:Lawrence Erlbaum.
Biggs, J., & Collis, K. (1982). Evaluating the Quality of Learning: the SOLO
taxonomy. New York: Academic Press.
Downes, S. (2007). Half an Hour: Creativity Retrieved 10 August 2007, from
http://halfanhour.blogspot.com/2007/08/creativity.html
Gilbert, J. (2005). Catching the Knowledge Wave? The Knowledge Society and the future of
education. Wellington: NZCER Press.
Cassady, J.C. (2002). Learner outcomes in the Cognitive Domain. in Johnston, J and Toms
Baker, L. (Ed). Assessing the Impact of Technology in Teaching and Learning. MI:
University of Michigan. Retrieved 23 February 2008 from
http://rcgd.isr.umich.edu/tlt/TechSbk.pdf
Ministry of Education. (2007). The New Zealand Curriculum. Wellington:Learning media.
NACCCE. (1999). All our futures: Creativity, culture and education. . Retrieved 17
August 2007, from http://www.dfes.gov.uk/naccce/index1.shtml
Robinson, K. (2006). Do schools kill creativity? . Ted Talks Retrieved 13 May 2007,
2007, from http://www.ted.com/index.php/talks/view/id/66
Rowlands, I., Nicholas, D., Huntington, P., Gunter, B., Withey, R., Dobrowolski, T.,
Tenopir, C., Williams, P., Fieldhouse, M., & Jamali, H. (2008). Information
behaviour of the researcher of the future. Retrieved 23 January 2008, from
http://www.jisc.ac.uk/whatwedo/programmes/resourcediscovery/googlegen.as
px
Siemens, G. (2004, April 5, 2005). Connectivism: A Learning Theory for the Digital
Age. Retrieved 15 March, 2006, from
http://www.elearnspace.org/Articles/connectivism.htm
Siemens, G. (2006, November 18). Knowing Knowledge. Retrieved November 18,
2006, from http://ltc.umanitoba.ca/KnowingKnowledge/index.php/Main_Page
Tapscott, D., & Williams, A. D. (2006). Wikinomics: How Mass Collaboration
Changes Everything. New York: Portfolio.
Wikipedia. (2007). Definitions of creativity. Retrieved 17 August 2007, from
http://en.wikipedia.org/wiki/Creativity#Definitions_of_creativity