Developing QuickSmart Online to Engage Learners
Doyle, Belson, Taber & Reading
ACEC2014 - DEVELOPING QUICKSMART ONLINE TO ENGAGE
LEARNERS
Helen Doyle, Stephanie Belson, Lorraine Taber & Chris Reading
University of New England, Australia
Keywords: learner engagement; online learning; numeracy
Abstract
Literacy and numeracy are identified as necessary skills for employment. QuickSmart
Online (QSO) was developed with the aim of closing the gap in numeracy skills to enable
the unemployed to break the cycle of long-term unemployment. QSO focuses on the learner
developing fast and accurate basic skills, which in turn develops their neural pathways,
allowing the learner’s working memory to be freed up to enable further learning. This
paper focuses on researching facilitator stories based on feedback from learners and
teachers, and on observations of QSO usage. These stories reported on the learner
experience during the initial development of QSO. The program was informally trialed for
a period of twelve months with learners, ranging in age from eight to the late fifties, from a
variety of learning institutions. There was some evidence of engagement with the program.
The five main aspects of the program that impacted on this engagement: learner
confidence, learner support, learner e-literacy, online environment style, and context of
learning are described. Key recommendations to increase learner engagement for the next
iteration of QSO are outlined.
More than seventy-five percent of employers in 2009 reported that their businesses were affected by
low levels of numeracy and literacy skills amongst their workforce (Ai Group, 2010), whilst the
Industry Skills Council (ISC) reported more than half of working age Australians have Language,
Literacy and Numeracy (LLN) problems (Industry Skills Councils, 2011). Successive Australian
Governments have reported low LLN skills of Australians. The Australian Government has cooperated with The Organisation for Economic Co-operation and Development (OECD, 2013)
resulting in an International Report on LLN skills of adults, which described such skills every
individual needed to participate in society. Capraro, Capraro, and Jones (2014) also stressed that
numeracy is an important skill for full participation in the workforce. The Science, Information and
Communication Technology, and Mathematics Education for Rural and Regional Australia National
Research Centre (SiMERR) received an Australian Federal Government (AFG) Grant in 2012 to
develop and produce an online version of QuickSmart, called QuickSmart Online (QSO) targeted at
Adult Job Seekers with identified low levels of LLN skills. This grant was part of the NBN-enabled
Tele-education Trial to support the Australian Government’s Digital Economy Goal for expanded
online education (Hand, 2013). This paper focuses on the numeracy component of QSO. First, some
background is provided about QSO, engagement with learning and the QSO software development
cycle, then, some results are discussed from the initial trialling of QSO. The results include the main
aspects of QSO that impacted learner engagement and recommendations for improving QSO to
increase engagement.
QuickSmart
QuickSmart (QS) was first developed in 2001 as a face-to-face (f-2-f) early intervention numeracy
program for middle-school students, followed by an additional literacy component. QS focused on
enhancing the students’ fluency in either numeracy or literacy (automaticity) through improving their
working memory. Students work in pairs with an instructor for thirty minutes, three times per week,
for an average of thirty weeks. Pegg, Graham, and Bellert (2005) defined automaticity as learners’
fluency and facility with basic number facts. They researched the links between working memory and
the ability to recall basic number facts and found that improvements made to a person’s processing
Page 1 of 8
Developing QuickSmart Online to Engage Learners
Doyle, Belson, Taber & Reading
speed of basic skills frees up his/her working memory capacity, which then becomes available to
address more difficult mathematical tasks. This research also showed that the improvements made to a
person’s working memory continued for at least twelve months following the completion of the QS
intervention program.
QuickSmart Online
The QS f-2-f program was the framework used to develop QSO numeracy. A team from SiMERR was
responsible for creating the content while an emerging software development company was contracted
for the technical development of the online environment. Like QS the development of QSO was
aimed at improving a person’s automaticity in numeracy, thereby freeing up his/her working memory
to allow him/her to perform more complex tasks. It is important to emphasise that QSO is not
intended to be a computer game. However, QSO does align with Whitton’s (2014) game definition of
providing a challenging activity and containing structure, rules, goal progression and rewards. The
numeracy component of QSO commenced trialling in April 2013. The remainder of this paper focuses
on this trial of QSO numeracy.
The QSO numeracy program consists of seven components: Warm-Up, Focus Facts, Flash Cards,
SpeedSheets, Fast and Accurate Basic Skills (FABS), Problem Solving (PS) and a game. After
enrolment into the program, the learner completes an eighty-question pre-test covering each of the
four basic mathematical operations. This was designed to establish a learner’s entry point into QSO.
Many learners have low e-literacy skills and/or low LLN. A calibration activity to assess a learner’s
keyboarding skills was included for the timed activities, i.e., Flash Cards, FABS and SpeedSheets. All
seven components of QSO were designed to help the learner engage with his/her learning.
Engagement with Learning
To better report on aspects of the engagement of learners when using QSO, it is first necessary to
clarify what is meant by engagement. Engagement, energy in action (Russell, Ainley, & Frydenberg,
2005), focuses on the connection between the learner and the activity. Care must be taken not to
confuse engagement with motivation, which is about energy and action and focuses on the reasons for
behaviour (Russell et al., 2005). Engagement is more likely than motivation to be affected by learning
experiences and rapport with people involved with those experiences. Students who are motivated are
not necessarily engaged. Teachers need to be able to design learning environments (f-2-f or online)
that will engage students.
Three distinct types of engagement: behavioural, cognitive and emotional, as described by Fredricks,
Blumenfeld, and Paris (2004), provide a useful framework for elaborating the concept of engagement.
Behavioural engagement involves: positive conduct, e.g., absence of non-disruptive behaviours; and
involvement in learning tasks, e.g., persistence. Emotional engagement involves: affective reactions in
learning situations, e.g., interest; and affective reactions to those delivering the learning, e.g.,
respecting teacher. Cognitive engagement involves psychological investment in learning, e.g., desire
to go beyond the requirements; inner psychological investment, e.g., desire to learn; and selfregulation, e.g., evaluating cognition when accomplishing tasks. Although categorising these three
types of engagement can assist in expanding perceptions of engagement, care needs to be taken as
confusion can result from these three types of engagement being “dynamically interrelated within the
individual” (Fredricks et al., 2004, p. 61). Such an expanded view of engagement, with three types,
provided a suitable framework for considering how learners were engaging with QSO.
QSO Software Development Cycle
QSO needed a software development cycle to monitor and evaluate each step of the development
process. The Most Significant Change (MSC) technique, developed by Davies and Dart (2005), was
favoured as a framework to collect stories from researching facilitators, hereafter called facilitators,
Page 2 of 8
Developing QuickSmart Online to Engage Learners
Doyle, Belson, Taber & Reading
working closely with the QSO trial. MSC is primarily a monitoring technique (Willetts & Crawford,
2007) involving collecting significant change stories from the people who are most closely involved
with a program and then the most significant of these is selected by the stakeholders. Adapting the
MSC technique for implementation in a specific evaluation situation, Willetts and Crawford (2007)
developed a monitoring and evaluation model, called the Monitoring and Evaluation Data Cycle
(M&E). Learning rather than accountability is the focus of the M&E Data Cycle.
The QSO Software Development Cycle (Figure 1), was created as an adaption of the M&E Data
Cycle. The six stages in the process were: 1) Identification - involves selecting the data to be captured
with indicators tracked throughout the life of the project; 2) Capture - involves collecting data,
through informal and formal processes, relevant to the chosen indicators; 3) Analysis - involves
analysing the raw data and developing recommendations for further software development; 4)
Development - (Dissemination in M&E Data Cycle) involves acting on the recommendations to
develop the next iteration; 5) Implementation - (Utilisation in M&E Data Cycle) involves the
implementation of the new iteration; 6) Assessment - involves assessing and reflecting on whether or
not the indicators in the Identification stage were the most appropriate and whether they need to be
refined in subsequent iterations. This paper focuses on the first three stages of the development cycle,
coloured green in Figure 1, as undertaken in the QSO trial.
Figure 1: QSO Software Development Cycle
Method
The QSO trial occurred in 2013, with three facilitators who worked with 40 early-school-leavers and
adult learners and 44 school-aged learners. The early-school-leavers participated in a youth-off-thestreets program and the adult learners in a government education program. Both programs required
the study of basic skills because these learners had been identified as having skills too low for
satisfactory employment. The school-aged learners struggled with mathematical skills, but were not
necessarily the lowest achievers in their respective cohorts.
Page 3 of 8
Developing QuickSmart Online to Engage Learners
Doyle, Belson, Taber & Reading
Unlike QS, where instructors work with pairs of students, QSO was designed for the learner to use
independently. For the trial both teachers/teacher aides and facilitators were present during each
session. The teachers/teacher aides were there to learn how to support the use of QSO in their class
rooms. The three facilitators were there to assist with overcoming any technical issues, and to observe
the learners, which informed the monitoring, and evaluation of the program. Each of the facilitators
attended at least one session weekly.
The research presented in this paper aimed to monitor and evaluate the trial of QSO to determine
improvements needed to increase student engagement. The first three stages of the QSO Software
Development Cycle (see Figure 1), as followed in the QSO trial, are now described. The first stage,
Identification, involved choosing the data to be collected. In accordance with the MSC approach, the
most significant change stories provided by the three facilitators were chosen. These facilitators had
the opportunity to observe the learner use of, and reaction to QSO and also to have informal
conversations with the learners and the teachers/teacher aides. The key indicator of interest was
student engagement. The second stage, Capture, involved the three facilitators writing their individual
stories recording their observations of student engagement with QSO, including identifying
significant changes that occurred. The third stage, Analysis, involved the facilitators collaborating
with an independent researcher to combine the three stories to synthesise the most significant
outcomes and impacts about learner engagement. The indicators of engagement, as evidenced in the
combined story, are reported across all three types, behavioural, emotional and cognitive, to
demonstrate the breadth of engagement.
The Story
During the Analysis phase information shared by the three facilitators about their observations during
the trial of QSO varied considerably, justifying the need to consider the stories from all three
facilitators, rather than just using feedback from one. Two important common themes were that QSO
gave the learner the opportunity: to improve basic number skills thus developing automaticity; and to
practice those skills with contextually appropriate problems. As a consequence of the improved skills,
learner confidence increased both within and beyond the learning environment.
However, consideration of the diversity within the three stories showed that the learner experience
varied accordingly to three key factors: perceived employment opportunities, learner age, and teacher
engagement. First, employment opportunities, as perceived by the learners, differed between
geographic locations with some learners believing that there was no point in engaging with the
program when there were no job opportunities relevant to their skill levels. Second, the learners varied
in age from eight to late fifties. Typically, school-aged learners could overcome technical issues and
engage from the outset and the early-school-leavers did not engage because of recent failure with the
school system, while adult learners could see the value in trying a new approach to learning LLN.
Third, teacher engagement decreased during the trial, when it became apparent that QSO was not
aligned to their specific curriculum requirements and that they were unable to access learner results to
map performance outcomes. Learners were more engaged when teachers were engaged.
Analysis of the stories showed that some learners were more engaged than others. Evidence of
engagement spread across all three types of engagement: behavioural - seeking assistance, persisting
with difficult tasks, completing work above minimum requirements, and assisting peers; emotional liking the facilitators, reacting positively to progress, and reacting positively to constructive
comments; and cognitive - using feedback, interpreting progress graphs, recognising when a fact is
“learnt”, linking progress to non-QSO life events, and acknowledging the value in learning.
Of most importance to the QSO team was identifying what had the greatest impact on learner
engagement and what recommendations could be made to inform the next iteration of QSO in the
Development stage of the QSO Software Development Cycle. The five main aspects of QSO found to
impact on learner engagement were: learner confidence; learner support; learner e-literacy; online
Page 4 of 8
Developing QuickSmart Online to Engage Learners
Doyle, Belson, Taber & Reading
environment style; and context of learning. These are elaborated below.
Learner Confidence
The trial commenced with learners displaying varying levels of confidence. There were many issues
that affected their confidence, most importantly, socio-economic status, family attitude to education,
fear of mathematics, and prior learning experiences. Despite lacking confidence, some learners were
excited to be part of the trial. Generally, the school-aged learners lacked confidence in their
mathematical abilities, however were confident using the online environment. Early-school-leaver
confidence was affected by previous failures at school. Being labelled as an early school leaver
defines a person as a failure, not achieving success in current societal norms (Schwab, 2012). The
confidence of many adult learners was affected by the fact that they previously achieved recognition
for attending and completing courses, yet were self-aware that they still lacked the basic skills.
Generally, adult learners were confident that QSO provided an alternate learning environment beyond
what they had previously experienced. They willingly persevered and continued to engage with the
program even when they experienced little academic progression due to technical issues.
Both starting level and automaticity were important influences on learner confidence. QS was
designed to start a lesson with Focus Facts, where the learner starts with facts already known and then
moves onto the unknown (Pegg et al., 2005). For the trial, QSO was designed so that every learner
started from the easiest Focus Fact, plus 2. For some this meant working on facts already known.
However, being able to answer the questions correctly helped the learner to develop confidence before
going on to the questions at a higher level for a new Focus Fact. Similarly, the rate at which
automaticity was achieved was linked to developing confidence. On completion of the trial, many
learners demonstrated greater confidence with most able to articulate that they had noticed
improvement in their confidence, both within and beyond the online learning environment.
Recommendation 1: QSO calibration be adjusted to start a learner practicing new skills at
one level lower than the level at which he/she tested successfully and to stop the learner from
spending anymore than six sessions on each Focus Fact.
Learner Support
There were four types of learner support involved: from facilitators; from progress feedback within
the online environment; from engaged teachers; and from peers. Initially, increased engagement with
QSO occurred when the learner had a facilitator encouraging him/her to get started and/or continue.
The facilitator continually encouraged the learner to attempt the questions. The facilitators realised
that QSO failed to replicate, within the online environment, what the instructor does in the f-2-f QS.
There are two types of progress feedback within the online environment: results, which are graphed in
a learner portfolio and incorrect answers which are displayed at the end of each activity. When the
learner engaged with his/her portfolio, there was a greater understanding of results and what was
necessary to achieve automaticity. Support from the teacher is important to a learner’s success in
QSO. Teachers were more likely to support learners if QSO helped the learners to achieve curriculumbased outcomes. Many teachers admitted to having poor e-literacy skills themselves and therefore
were not confident using QSO without facilitators to support them. A few teachers showed an obvious
lack of engagement with the learners and with QSO and generally when this occurred learners were
not engaged. Support from peers appeared to have more benefits for the peer who provided the
support than for the actual learner. In fact, some of the peers providing the support increased their own
confidence to such an extent that they went on to further study.
Recommendation 2: QSO incorporates features to replicate the facilitator in the classroom by
providing assistance through intelligent feedback and presenting progress graphs on completion of
Page 5 of 8
Developing QuickSmart Online to Engage Learners
Doyle, Belson, Taber & Reading
each activity.
Learner e-Literacy
Many of the learners had low levels of e-literacy, with a few having never used a computer prior to
the trial. The exception being some of the school-aged learners who had computers and internet access
at home. This lack of e-literacy had not been anticipated. Enrolment in QSO required each learner to
have an email address, which the majority of the early-school-leavers and adults did not have or if
they did they did not know how to access. Therefore, the facilitators were required to enrol each
learner with a username and password. This included a master list given to the teacher to assist those
learners who could not remember their details from session to session. The facilitators spent valuable
time in the beginning teaching basic keyboarding skills to the learners, including skills as simple as
the use of enter key and the numerical keypad on desktop computers. The learners tended to switch
between data entry methods throughout an activity between the numerical keypad, the numbers on a
standard keyboard, and pointing to the onscreen keyboard using the mouse. This then had the
unintended effect of compromising the calibration data in the timed activities, and hence the ability to
achieve automaticity.
Recommendation 3: QSO restricts learners to one entry method for numeric characters and
incorporates the capability for bulk enrolment of learners.
Online Environment Style
Three important aspects of the QSO environment style related to learner engagement: the interactivity
in screen layout, the capability for learners to have individualised programs and the opportunity for
non-judgemental anonymity. For the QSO trial, the screen was divided visually into three sections, the
centre, left side and right side. Screen layout design was found to have less impact on engagement of
the school-aged learner than the older adult learners. School-aged learners were observed to be more
adaptable and confident with the screen layout design, enabling them to proceed with few difficulties.
The early-school-leavers generally wanted the QSO screen design to be more interactive or game like.
Due to their generally poor e-literacy levels the older adult learners found the screen layout design not
to be intuitive and they required more help. The adult learners found the lack of explanation as to why
they were doing the activities and how they were to do the activities confronting, likening QSO to just
one test after another and expressing fear of being unsure as to what was expected next. This affected
their academic progress, particularly in the timed activities, and hence their automaticity.
Many learners were more engaged because the individualised learning program nature of QSO gave
them the opportunity to learn at their own pace. This was the first time they had felt like they were
actually achieving on their own merit. Previously some adult learners had “completed” courses in
LLN, however they still could not read or complete basic mathematical skills. These particular
learners gained an enormous amount of self-efficacy when they were engaged willingly with QSO to
achieve on their own merit.
Some adult learners articulated that the online learning environment was non-judgemental which
made it more comfortable than a f-2-f learning situation. This gave the learners the security to engage
with QSO and not be embarrassed by having incorrect responses made public. It had been observed
that many of these learners sat very quietly in a traditional classroom situation, not engaging and/or
not comprehending the classroom instruction.
Recommendation 4: QSO be more interactive by following a logical flow process through the
activities and presenting components of QSO when they are needed, making the environment more
App-like.
Page 6 of 8
Developing QuickSmart Online to Engage Learners
Doyle, Belson, Taber & Reading
Context of Learning
To engage the early-school-leaver and the adult learner with QSO, the trial identified the importance
of linking Problem Solving (PS) learning material to the course(s) being studied and the application to
their everyday life, e.g., Foundation Skills Course (FSK). From the adult trial, it was also evident that
teacher engagement and support for QSO is dependent on the learning material being linked to
relevant curriculum. The classroom teacher requires access to the learners’ results of the PS learning
material in order for them to be able to validate the learners’ progression through their overall course
of study. The engagement of the school-aged-learners was not linked to the learning material of the PS
activity. However, conversations with the school principal said that PS was a focus of their school and
would recommend that the learning material be linked to the Australian Curriculum (Mathematics).
Recommendation 5: QSO includes a teacher-accessible mapping of the problem solving activity
questions to the foundation skills package for numeracy and the Australian Curriculum
(Mathematics).
Conclusion
Engagement with QSO was evident to varying degrees across the range of learners who participated
in the trial. The fact that a variety of engagement indicators were reported indicates that the learners
were able to engage with QSO. The breadth of aspects of the QSO experience that impacted the level
of engagement indicates that designing an engaging online environment to “house” a learning
experience previously designed for a f-2-f situation is not such a straight-forward task. This is
especially true when the learners have low levels of literacy and/or numeracy and have previously
experienced failure in formal learning situations.
As a consequence of the trial, five key recommendations were reported to enhance QSO and thus
better engage the learner. These will inform improvements to be made to QSO in the Development
stage. Although these reported findings are specific to QSO, there are lessons to be learnt for other
educators designing online environments for learning basic skills. First, the designers must be aware
of learner background. Previous education experiences and level of e-literacy impact on learner
engagement. What is suitable for school-aged learners may not be suitable for early-school-leavers
and adults. Second, the designers need to build in suitable support structures. These should replicate,
as closely as possible, learner needs as previously identified in successful f-2-f learning situations.
Finally, the designers must set the learning activities in a suitable context. This would ideally include
a specific curriculum focus and linking to practical applications. Further research into building such
online environments is encouraged to maintain learner engagement.
References
Ai Group. (2010). National workforce literacy project: Report on employers views on workplace
literacy and numeracy skills. North Sydney, NSW.
Capraro, M. C., Capraro, R. M., & Jones, M. (2014). Numeracy and Algebra: A Path to Full
Participation in Community and Society? Reading Psycholgy, 1-15. doi:
10.1080/02702711.2012.739263
Davies, R., & Dart, J. (2005). The ‘Most Significant Change’ (MSC) Technique. A guide to its use
Retrieved 30/09/11, from www.kepa.fi/tiedostot/most-significant-change-guide.pdf
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the
concept, state of the evidence. Review of educational research, 74(1), 59-109.
Hand, T. (2013). eduONE Trial Evaluation Report (pp. 1-92).
Page 7 of 8
Developing QuickSmart Online to Engage Learners
Doyle, Belson, Taber & Reading
Industry Skills Councils. (2011). No More Excuses: An Industry Response to the Language, Literacy
and Numeracy Challenge, from http://www.isc.org.au/pdf/NoMoreExcuses_FINAL%20single
%20page.pdf
Pegg, J., Graham, L., & Bellert, A. (2005). The effect of improved automaticity and retrieval of basic
number skills on persistently low-achieving students. Paper presented at the International
Group for the Psychology of Mathematics Education, Melbourne, Australia.
Russell, V. J., Ainley, M., & Frydenberg, E. (2005). Schooling Issues Digest: Student Motivation and
Engagement, from www.dest.gov.au/school_education/publications_resources/schooling_
issues_digest_motivation_engagement.htm
Schwab, R. (2012). Indigenous early school leavers: Failure, risk and high-stakes testing. Australian
Aboriginal Studies, 1, 3-18.
Whitton, N. (2014). Digital Games and Learning: Research and Theory. New York, NY: Routledge.
Willetts, J., & Crawford, P. (2007). The most significant lessons about the Most Significant Change
technique. Development in Practice, 17(3), 367-379. doi: 10.1080/09614520701336907
Page 8 of 8