Evaluation of the
Early Learning
Languages Australia
trial
Final report
Department of Education
and Training
29 February 2016
ELLA Trial Evaluation Final Report
Contents
Acronym list.....................................................................................................................................i
Executive Summary ........................................................................................................................ii
1
2
3
4
5
6
7
Introduction ........................................................................................................................ 1
1.1
Evaluation approach .............................................................................................................. 2
1.2
Evidence sources .................................................................................................................... 5
1.3
Report structure..................................................................................................................... 7
Trial design .......................................................................................................................... 9
2.1
Tablet technology ................................................................................................................ 10
2.2
Software (‘app’) design ........................................................................................................ 13
2.3
Alignment of the trial to best practice ................................................................................. 15
Trial implementation ........................................................................................................ 18
3.1
The implementation process ............................................................................................... 19
3.2
Level of support received ..................................................................................................... 20
3.3
Implementation issues ......................................................................................................... 23
Trial delivery...................................................................................................................... 29
4.1
Introducing the trial ............................................................................................................. 30
4.2
Methods of trial delivery...................................................................................................... 33
4.3
The use of complementary activities ................................................................................... 35
4.4
Parent and guardian involvement ....................................................................................... 37
Trial engagement .............................................................................................................. 43
5.1
Children’s overall usage and participation ........................................................................... 44
5.2
Children’s usage of individual apps ...................................................................................... 53
5.3
Factors contributing to children using the apps .................................................................. 62
Trial impact and efficiency ................................................................................................ 66
6.1
Child outcomes .................................................................................................................... 67
6.2
Educator outcomes .............................................................................................................. 87
6.3
Funding and cost efficiency ................................................................................................. 92
Future considerations ....................................................................................................... 98
7.1
Trial limitations and refinements ......................................................................................... 98
7.2
Options for expansion ........................................................................................................ 100
7.3
Implementing the recommended option .......................................................................... 108
Appendix A : Overview of evidence sources ............................................................................. 110
Appendix B : Evaluation questions ............................................................................................ 115
Appendix C Literature review.................................................................................................... 117
Limitation of our work................................................................................................................... 139
ISBN 978-1-76028-705-4 [PDF]
ISBN 978-1-76028-706-1 [DOCX]
Liability limited by a scheme approved under Professional Standards Legislation.
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network
of member firms, each of which is a legally separate and independent entity.
Please see www.deloitte.com/au/about for a detailed description of the legal structure of Deloitte Touche Tohmatsu Limited and
its member firms.
© 2016 Deloitte Access Economics Pty Ltd
ELLA Trial Evaluation Final Report
Charts
Chart 2.1 : Prevalence of digital devices within the home of ELLA participants ......................... 11
Chart 2.2 : How easily can educators navigate the apps?........................................................... 13
Chart 2.3 : Has educator ability to navigate the apps improved since the trial began? ............. 13
Chart 2.4 : How easily can children navigate the ELLA apps? ..................................................... 14
Chart 3.1 : Educator agreement to the statement ‘Our service understands the support
available through the ELLA trial’ ................................................................................................. 20
Chart 3.2 : Educator agreement to the statement ‘Our service is able to obtain the support we
need’ .......................................................................................................................................... 21
Chart 3.3 : Extent to which educators agreed with the statement ‘I have received adequate
ongoing information from ESA throughout the ELLA trial’ ......................................................... 21
Chart 3.4 : Useful support tools (% of educators who nominated a mechanism as ‘useful’) ..... 22
Chart 3.5 : Issue type by resolution length ................................................................................. 25
Chart 3.6 : Issue resolution times by priority .............................................................................. 25
Chart 3.7 : Level of educator agreement to statement ‘the release of new apps has been
timely’.......................................................................................................................................... 28
Chart 4.1 : Do educators interact with the children when they are using the apps? ................. 34
Chart 4.2 : How do educators interact with children when they are using the apps? ............... 34
Chart 4.3 : Educator use of ELLA trial usage data ....................................................................... 35
Chart 4.4 : Types of complementary activities which are being facilitated by educators .......... 36
Chart 4.5 : Educator perceptions parent/guardian understanding and view of the ELLA trial... 38
Chart 4.6 : Educator views on changes in parent understanding of ELLA over time .................. 39
Chart 4.7 : Educator views on changes in parent attitude towards ELLA over time ................... 39
Chart 5.1 : App usage – average minutes per session, proportion of total sessions .................. 47
Chart 5.2 : App usage – average minutes per session, by child .................................................. 47
Chart 5.3 : App usage – proportion of sessions lasting less than two minutes .......................... 48
Chart 5.4 : App usage over time .................................................................................................. 49
Chart 5.5 : Educator perspectives on child engagement with the ELLA apps ............................. 49
Chart 5.6 : Distribution of child app usage by decile .................................................................. 50
Chart 5.7 : Frequency of app usage............................................................................................. 51
Chart 5.8 : Average number of children using each iPad ............................................................ 52
Chart 5.9 : Language exposure over time ................................................................................... 53
Chart 5.10 : Average sessions per user over time, by app .......................................................... 54
Chart 5.11 : Average minutes per session over time, by app ..................................................... 54
ELLA Trial Evaluation Final Report
Chart 5.12 : Summary of app popularity ..................................................................................... 55
Chart 5.13 : Cumulative app use, by app .................................................................................... 56
Chart 5.14 : Activity use by app, during first four weeks ............................................................ 57
Chart 5.15 : Activity popularity (by app), over time .................................................................... 58
Chart 5.16 : Proportion of language exposure, by app ............................................................... 59
Chart 5.17 : Proportion of language exposure over time, by app............................................... 60
Chart 5.18 : Total use (hours) and language exposure (as a proportion of total use), by activity61
Chart 5.19 : Most effective activities for language exposure ..................................................... 62
Chart 6.1 : Educator perspective on the number of children using the app language, as
observed by themselves or parents ............................................................................................ 69
Chart 6.2 : Educator views on how ELLA trial impacts changed over time ................................. 70
Chart 6.3 : Educator observations on children’s level of interest in the culture associated with
the language of the app being trialled ........................................................................................ 73
Chart 6.4 : Proportion of time spent on each activity, within Apps 6 and 7 ............................... 76
Chart 6.5 : Proportion of children reaching each level, by activity ............................................. 76
Chart 6.6 : Distribution of levelling scores .................................................................................. 77
Chart 6.7 : Parent/guardian observations of child use of words from the ELLA language at home78
Chart 6.8 : Parent/guardian observations of child showing interest in culture associated with
the ELLA app at home ................................................................................................................. 80
Chart 6.9 : Child use of tablet technology within the home ....................................................... 81
Chart 6.10 : Child use of tablet technology this year as opposed to previous use ..................... 81
Chart 6.11 Parent/guardian strength of agreement with ‘it is good for preschool children to
learn another language’ .............................................................................................................. 87
Chart 6.12 : Parent/guardian strength of agreement with ‘it is good for preschool children to
learn about another culture’ ....................................................................................................... 87
Chart 6.13 : Did educators feel increased confidence in their ability to incorporate language
learning as a result of the ELLA trial? .......................................................................................... 88
Chart 6.14 : Did educators feel increased confidence in their ability to incorporate cultural
awareness as a result of the ELLA trial? ...................................................................................... 88
Chart 6.15 : Educator confidence levels about incorporating digital technology at trial outset 89
Chart 6.16 : Did educator confidence levels about incorporating digital technology increase as a
result of the trial? ........................................................................................................................ 90
Tables
Table 2.1 : ELLA design relative to best practice principles in integrating digital technology in
preschools ................................................................................................................................... 15
ELLA Trial Evaluation Final Report
Table 2.2 : ELLA design relative to best practice principles in app design for early learners ..... 16
Table 2.3 : ELLA design relative to best practice principles for teaching additional languages in
preschools ................................................................................................................................... 17
Table 3.1 : Implementation length by provider characteristics (in weeks)................................. 27
Table 5.1 : Release cycle for ELLA apps ....................................................................................... 45
Table 5.2 : Overall usage statistics, by child ................................................................................ 46
Table 5.3 : Sessions per user per week, by various explanatory factors..................................... 63
Table 6.1 : ELLA trial cost per child illustration1 .......................................................................... 93
Table 7.1 : Recommended investments for ELLA trial 2016-18 ................................................ 108
Table A.1 : Data dictionary ........................................................................................................ 110
Table A.2 : Descriptive statistics for site visit sample ............................................................... 112
Table A.3 : Overall parent survey responses rate ..................................................................... 113
Table A.4 : Descriptive statistics for August Parent/Guardian Survey response rate ............... 113
Table A.5 : Descriptive statistics for December Parent/Guardian Survey response rate ......... 114
Table B.1 : Key evaluation questions ........................................................................................ 115
Table C.1 : Programmes related to the ELLA trial ..................................................................... 120
Table C.2 : Impacts of digital technology on learning effectiveness ......................................... 126
Figures
Figure 1.1 : Summary of ELLA apps ............................................................................................... 1
Figure 1.2 : Structure of the ELLA trial .......................................................................................... 2
Figure 1.3 : ELLA trial programme logic ........................................................................................ 4
Figure 1.4 : Evidence sources for ELLA trial evaluation................................................................. 5
Figure 6.1 : ELLA trial cost curve (illustrative replication only) ................................................... 94
Figure 6.2 : ELLA trial benefit-cost curve (illustrative replication only) ...................................... 95
Figure 7.1 : Summary of ELLA recommended expansion timeframe........................................ 109
ELLA Trial Evaluation Final Report
Acronym list
ACARA
Australian Curriculum, Assessment and Reporting Authority
ACEQA
Australian Children’s Education and Care Quality Authority
ACT
Australian Capital Territory
ASD
Autism Spectrum Disorder
ARIA
Accessibility/remoteness Index of Australia
BYOD
Bring your own device
DAE
Deloitte Access Economics
ECA
Early Childhood Australia
ELLA
Early Learning Languages Australia
ESA
Education Services Australia
EYLF
Early Years Learning Framework
ICT
Information communications technology
ID
Identity
HCC
Health care card
LDC
Long day care
LOTE
Language other than English
SES
Socioeconomic status
i
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Executive Summary
The Early Learning Languages Australia (ELLA) trial, which ran throughout 2015, was an Australian
Government initiative that involved the development and use of language-based mobile applications
(apps) for preschool children. The primary aim of the trial was to provide language exposure for
preschool children, in order to encourage further language learning in later years of schooling, and to
help address barriers to language education in the early years of education.
Deloitte Access Economics was engaged to conduct an evaluation of the ELLA trial, to consider
whether it was appropriately designed and effective in achieving its objectives.
ELLA trial overview
At the broadest level, the ELLA trial was developed by the Australian Government in the context of a
commitment to expand the teaching of languages in Australia. The Australian Government has a
specific goal of at least 40 per cent of Year 12 students studying a language other than English within
a decade. The ELLA trial was established as one part of the Australian Government’s endeavours
toward achieving this.
During the evaluation, the objectives of the ELLA trial were confirmed as being to:

expose preschool children to languages other than English through an online platform, to
improve cognitive development, encourage further language education and develop in
children an understanding of other cultures; and

contribute to transforming language education in Australia in the early years of schooling, by
addressing barriers to language education and initiating end-to-end language learning
opportunities for children.
In 2014, the Australian Government Department of Education and Training appointed Education
Services Australia (ESA) to manage (1) the design and development of language apps for preschool
children and (2) implementation of an ELLA trial at 41 preschool sites across the country.
Five languages were selected for the trial – Chinese (Mandarin), Japanese, Indonesian, French and
Arabic. Seven apps were developed for each language, with the apps being progressively released to
preschool trial sites throughout 2015. Trial sites were located in each state and territory, with a
mixture of metropolitan, regional and remote locations, and a range of settings such as school-based
preschools, stand-alone preschools and long day care (LDC) services. For the duration of the trial,
each site used apps in one language. Each site was supplied with iPads, which the children used to
access the apps. The first app commenced operating in the week of 22 February 2015. Apps were
subsequently released approximately every four to six weeks, with the final app being released in
early November 2015.
Each app featured the same characters ‘the polyglots’ in a different setting, including the playroom,
beach, birthday party, zoo, circus, park and town. Within each app, children could explore various
activities, designed to encourage learning in a play-based manner.1
1
Further information about the ELLA programme is available at the ELLA website or from the Commonwealth Department
of Education website.
ii
Deloitte Access Economics
ELLA Trial Evaluation Final Report
The overall usage profile of the apps comprised 1,868 children who used an app at least once. Key
usage statistics include 234,956 total user sessions and 27,661 total hours of use. On average, each
child used the ELLA apps for 29 minutes per week during the trial, comprising an average of four
sessions of around seven minutes each.
Evaluation overview
The evaluation of the ELLA trial was conducted over the course of the 2015 school year, and drew
upon on a range of information sources, enabling a mixed-method qualitative and quantitative
evaluation approach. These sources comprised: comprehensive app usage data; three rounds of
online surveys of educators at trial sites; two rounds of online surveys of parents and guardians of
children involved in the trial; two rounds of face-to-face visits with a sample comprising just over half
of the trial sites; a literature review on the use of digital technology for language learning in
preschools; information provided by ESA, including records from the ELLA help desk; and,
consultations with state and territory education authorities.
To establish the evaluation approach, a comprehensive evaluation plan was developed. This
document outlined the evaluation objectives and purpose, set out a programme logic model, put
forward a series of evaluation questions, and discussed the evidence sources and analytical
approach. The evaluation plan was approved by a Human Research Ethics Committee (Bellberry
Limited) and research approval was received from relevant state and territory education authorities.
An overview of the key evidence sources for the evaluation is shown in Figure i.
Figure i: Overview of evidence sources
Understanding of
usage and trends
Data
analysis
Breadth of
experiences
In-depth case
studies
Consultation
Understanding of
jurisdiction
contextual factors
Understanding of
contemporary
best practice
Literature
review
Surveys
Children
Trial sites
Families
Educators
Educational authorities
Expert opinion
External sources
Evaluation scope
Relative to the period over which benefits from participating in a programme like ELLA will
materialise, the trial occurred over the relatively short time period of time (one year). Consequently,
findings on the trial’s impact are based on early and emerging evidence. A more robust impact
assessment can only be undertaken after a longer time period has elapsed and outcomes have more
fully manifested.
Consistent with the objectives of the trial, this evaluation has focussed on children’s participation
and usage of the apps. It was not within the scope of the evaluation to assess the extent to which
iii
Deloitte Access Economics
ELLA Trial Evaluation Final Report
children became proficient in the language, although educators and parents were asked to provide
observations of language usage where possible.
As described above, the ELLA trial and this evaluation occurred in the context of the Australian
Government’s broader goal to significantly increase the proportion of year 12 children studying a
language. There are two primary ways the ELLA trial, and any subsequent expansion of the initiative,
can contribute to this objective:
1.
provide a foundation for expanding interest in language education among pre-school children,
in the expectation that some of these children will consequently be more likely to continue
with language education throughout their schooling; and
2.
providing a platform which could potentially be expanded to enable broader access to
language education, as a complement to language education teachers.
However, each of these points would require more detailed assessment and consideration over a
longer time frame. Therefore, while this evaluation cannot determine the extent to which these
points have been achieved, it has assessed the extent to which the trial has contributed to creating
the preconditions that would enable them to be realised.
Evaluation findings
Design of the ELLA trial
Evidence from surveys and site visits suggests that the ELLA trial has demonstrated promising results
in providing a model of enabling language exposure in preschool environments where the educator is
not a trained language educator and is usually not proficient in a language other than English.
Children and educators have found the iPads and apps easy to use, with educators consistently
stating that the ELLA apps are engaging and age-appropriate. These observations are supported by
the app usage data which shows children quickly adapted to using the apps and had were able to use
them in a relatively self-directed manner.
A review of literature pertaining to digital technology for language learning in preschools identified
features of best practice. The design of the ELLA trial features most of the identified best practice
elements. However, program design could be further strengthened through increased guidance for
educators on what to expect from children’s interactions with the apps and how the language
learning process typically transpires for preschool-aged children.
Implementation of the ELLA trial
The timeframe for implementation of the ELLA trial was relatively limited. Trial development
substantively occurred in the months leading up to the beginning of 2015. The first app was released
in February 2015.
Despite the compressed timeframe, implementation of the ELLA trial was relatively smooth. An
exception to this however was experienced in a few sites which encountered network issues (due to
state or territory government protected internet connections). This problem was more likely to arise
at sites operated by state education authorities, although they were limited to only two jurisdictions.
Engagement with sites through site visits and surveys indicated a relatively consistent view that the
support mechanisms available to sites were accessible and useful. In particular, the introductory face
iv
Deloitte Access Economics
ELLA Trial Evaluation Final Report
to face training workshop for two educators from every site and the consistent point of support
provided by ESA were considered valuable.
Delivery of the ELLA trial
Trial sites were provided with flexibility in many aspects of programme delivery within sites. This
approach was adopted so that each service could utilise the ELLA resources in a manner which suited
their local context and circumstances. Reflecting the flexibility afforded to sites, there was a
relatively high degree of variation between services regarding:

The manner in which the ELLA trial was introduced and delivered, such as whether sites had
designated times of the day for using the apps and whether they placed time limits on usage.
•
While many sites welcomed the ability to introduce and use the apps as they saw fit,
some sites would have liked more guidance on the optimal ways to support the ELLA
trial in their site, including on screen time, and in interaction with children while using
the apps.

Use of complementary activities to support the ELLA apps, which refers to the degree to which
sites ensured the apps were used with accompanying activities in order to reinforce app
content.
•
In the first round of site visits, the evaluation observed widespread variation in the use
of complementary activities, with some sites ensuring this was part of the ELLA
experience, while others had not explored this to any significant degree. However, sites
tended to become more systematic in their approach to integrating the ELLA apps with
their broader learning programme following a second face to face workshop for all sites
held by ESA in September 2015, where two educator representatives from each site
were given opportunity to share ideas, including the use of complementary activities.
This is explored further in Section 4.3.
Engagement with the ELLA trial
Almost three-quarters of children participating in the ELLA trial used the apps at least once a week,
as revealed by the detailed app usage data. App usage (as measured by minutes per session)
remained relatively consistent throughout the trial, supporting educator observations made during
site visits that children maintained a steady interest in the apps over the course of the year.
Some children spent a relatively short period of time with the apps, while others used them more
regularly and for longer periods. Overall, the most active 10% of children accounted for 33% of total
app use.
Each app was most popular when it was first released, with usage typically sharply increasing, before
plateauing gradually over time. Overall, App 3 (Polyglots at the Birthday Party) was the most popular
among the children throughout the trial, as measured by usage (see Figure ii). This seems to be a
function of the timing of this app’s release and inclusion within this app of activities that resonated
with children, including ‘cake’ – an activity which involved making and decorating a cake – which was
the most popular activity in the ELLA trial.
v
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Figure ii: Cumulative app use by app
Impact and efficiency of the ELLA trial
The ELLA trial has shown positive results in increasing language exposure for preschool children.
This is demonstrated by educator and parent/guardian observations on child engagement with, and
interest in, other languages and cultures, along with analysis of app usage data:

Engagement with the ELLA programme was high and sustained over the course of the year for
a considerable proportion of children, resulting in continued language exposure.

Most trial sites reported that at least some children were producing words and phrases from
the ELLA apps.
•
Although the evaluation cannot make definitive findings on language production,
observed use of the app language by some children indicates that language exposure
has been effective and children’s interest has been piqued.

In addition to the language exposure benefits, observations by educators and
parents/guardians also suggested that the ELLA trial was associated with the development of a
range of skills for participating children, including: care for property, sharing, digital literacy,
English literacy, collaborative learning and social skills. However, these observations were not
consistent across all educators, and some children experienced some or all of these benefits to
a greater extent than others.
With regard to the objective of transforming language education in the early years of schooling by
addressing barriers to language education, the ELLA trial also appears to have been successful to the
extent that an educator need not be language proficient in order for children to participate in, and
benefit from, language exposure.

As reported during site visits and through the educator surveys, over 60% of educators stated
they had increased confidence in incorporating language learning and a focus on
multiculturalism into their learning programme.
•
It is important to note however that a number of educators suggested that additional
training or guidance to enable them to familiarise themselves with the language at a
basic level, including correct pronunciation, would be beneficial and help increase their
confidence in delivering the trial.
vi
Deloitte Access Economics
ELLA Trial Evaluation Final Report

Some concerns were expressed about a lack of continuity of language learning for the children
once they leave preschool and commence school. It was noted in consultation with both
educators and jurisdictional education authorities that the potential impact of any ongoing
ELLA trial on language learning would be heightened if stronger language pathways existed for
children from preschool through primary and secondary education.
•
Given the trial has only been delivered to a single cohort of preschool children thus far,
it is too early to definitely assess the likelihood of the ELLA trial influencing children’s
language learning experience in subsequent years of education.
With regard to the cost of the ELLA trial, the cost during the trial phase was $9.8 million. It reached
1,868 students, representing a cost per student of approximately $5,246. The relatively novel nature
of the ELLA initiative means it is difficult to make a definitive finding regarding the cost effectiveness
of the ELLA trial at this time. However, there is potential for the ELLA trial to become progressively
efficient if it undergoes managed growth through extension of the trial phase – or if it were to
eventually becomes an ongoing programme under an appropriate business model.2 This is because:

a significant proportion of the cost associated with the ELLA trial has already been invested in
the development stages;

the marginal cost of distributing the trial to additional participants is low;

the evaluation has found positive indications that children are gaining language exposure
through participation in the trial; and

a range of benefits associated with participation may materialise over the longer term,
including for example, improved cultural awareness.
Future considerations
On balance, there appears to be a case for extending the ELLA trial period for a further two to three
years, should language exposure in the early years remain a policy priority for government. This
longer period of trialling is required to more definitively assess the outcomes resulting from
increased language exposure in preschool.
Several trial extension options were assessed against three key criteria: accessibility; cost efficiency
and effectiveness. The results of this analysis suggest a recommended expansion approach which
includes:

A staged expansion over the next three years, providing opportunity to establish a more
comprehensive understanding of demand for the trial and the level of resources required to
support the rollout (e.g. the extent of the marketing campaign required).

The development of high-quality online resources which form the basis of the support
provided to services, complemented by a technical help desk and face-to-face workshops in
the early stages of expansion (if funds are available).

Targeted financial assistance provided to support low socioeconomic status services in the
purchase of devices.
2
Note that it is not yet possible to undertake a definitive analysis of the ELLA programme’s cost effectiveness. While the
ELLA programme has clearly been well received over the trial period, and shown positive signs regarding language
exposure, the actual outcomes achieved will not become known for a longer period of time, and cost effectiveness analysis
would require a comparison group, so the value of these outcomes could be compared.
vii
Deloitte Access Economics
ELLA Trial Evaluation Final Report
This extended trial period also provides opportunities to build support for the ELLA programme from
potential interested stakeholders, such as state and territory governments, and to further develop
communities of practice in order to increase the sustainability of the ELLA programme over time.
The evaluation also found that there would potentially be merit in expanding the scope of the ELLA
trial to include other age groups, and potentially additional languages. However, given the
immediate priority is to extend access to preschool children and further evaluate the potential
impacts of the ELLA trial, this option should be revisited at a later point in time.
The recommended approach would result in an approximate spend of $7.9 million over the next
three years. However, the recommended option holds a degree of flexibility which allows the format
of the expansion (and associated costs) to be refined over time as further information regarding
demand for the ELLA trial comes to light.
Deloitte Access Economics
viii
Deloitte Access Economics
ELLA Trial Evaluation Final Report
1 Introduction
The ELLA trial is an Australian Government initiative that involves the development and use of mobile
applications (apps) which aim to (1) expose preschool children (i.e. children in the year before
primary school, typically aged four years old) to languages other than English and (2) raise interest in
further language learning in later years of schooling.
Five languages were selected for the trial – Chinese (Mandarin), Japanese, Indonesian, French and
Arabic. Seven apps were developed for each language, with the apps being progressively released to
41 preschool trial sites throughout 2015. Each app was made up of four to six activities (see Figure
1.3). Trial sites were located in each state and territory, with a mixture of metropolitan, regional and
remote locations, and a range of settings such as school-based preschools, stand-alone preschools
and LDC services. For the duration of the trial, each site used apps in one language. Each site was
supplied with iPads, which the children used to access the apps. A snapshot of the ELLA trial’s scope
is provided in Box 1.
Figure 1.1: Summary of ELLA apps
1
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Box 1: Snapshot of the ELLA trial’s scope

As at December 2015, 1,868 children had used the apps at least once throughout the year, and 162
educators involved in the trial.3

The largest site had 135 eligible children, while the smallest site had seven.

There were 41 sites involved in the trial, with the children spread across 74 preschool groups. Some
sites had only one group, while bigger sites had as many as six groups.

There were 12 trial sites in NSW, nine in Victoria, eight in Queensland, six in Western Australia, three
in South Australia, and one each in Tasmania, the Northern Territory and the ACT.

•
By geographic distribution, 22 were located in metropolitan areas, 17 in inner and outer
regional areas, and two in remote and very remote areas.
•
By service type, 13 were stand-alone preschools, 11 are preschools attached to a school, and
19 are LDC services with a preschool trial.
The tablet used in the trial was a 16GB iPad Air 2.
•
A set of six iPads (one educator iPad and five children iPads) was allocated for approximately
every 25 children at a preschool.
•
A total of 336 iPads were issued for the trial (56 educator iPads and 280 children iPads).
1.1 Evaluation approach
In late 2014, Deloitte Access Economics was appointed by the Australian Government Department of
Education and Training (the Department) to undertake an evaluation of the ELLA trial. The purpose of
the evaluation is to consider whether the ELLA trial was appropriately designed and effective in
achieving desired outcomes. A summary of the key parties involved in the ELLA trial is shown in
Figure 1.2
Figure 1.2: Structure of the ELLA trial
Commonwealth Department of
Education and Training
Education Services
Australia (ESA)
Education and
language
learning
authorities
Programme Evaluator
DT Millipede
Trial manager
Deloitte Access Economics
Software
developer
Reference
group
Trial sites
41 preschool services
Australia-wide
3
Throughout 2015, 97 of these children left their preschool and therefore did not participate in a full year of the ELLA trial.
2
Deloitte Access Economics
ELLA Trial Evaluation Final Report
The ELLA trial programme logic, which provides a detailed summary of the trial; including the trial
objectives, inputs, activities, outputs and expected outcomes is provided in Figure 1.3 below.
3
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Figure 1.3: ELLA trial programme logic
4
Deloitte Access Economics
ELLA Trial Evaluation Final Report
To establish the evaluation approach, Deloitte Access Economics developed a comprehensive
evaluation plan. This document outlined the evaluation objectives and purpose, set out a programme
logic model, put forward a series of evaluation questions, and discussed the evidence sources and
analytical approach. Following the development of the evaluation plan, Deloitte Access Economics
obtained ethics approval from a Human Research Ethics Committee (Bellberry Limited) and research
approval from state and territory education authorities.
The evaluation questions underpinning the approach are provided in Appendix B.
1.2 Evidence sources
This evaluation drew upon on a range of information sources, which each contributed a distinct
evidence base. These sources, described below, enabled a multi-faced qualitative and quantitative
evaluation approach and are summarised in Figure 1.4.
Figure 1.4: Evidence sources for ELLA trial evaluation
Source: Deloitte Access Economics
1.2.1
Usage data
Over the course of the evaluation, Deloitte Access Economics has been provided access to nonidentifiable app usage data. This report includes an analysis of app usage data associated with the
ELLA trial (i.e. from 8 March 2015 until 27 December 2015).

At the beginning of the trial, each child was required to select a unique avatar to represent
them (typically a photo of themselves or a picture of a toy they like to play with) Every time an
avatar was selected, all usage data was then attributed to that child’s unique identity (ID). This
allowed for each child’s usage habits to be captured and analysed over time.

The data captured included what time the interaction took place, which app and activity were
being used, and what kind of interaction took place (i.e. – it captured if the child was listening
to a recording of the target language, or popping bubbles).

Several key measures of child engagement and usage were generated from this data
(henceforth referred to as the ELLA app data). These measures included number of sessions
per child (indicating how often children would return to the apps), minutes per session
(indicating how engaged a child was with the apps once they had logged on), and the
5
Deloitte Access Economics
ELLA Trial Evaluation Final Report
proportion of language exposure (which indicated the share of time spent with the apps
children spent listening to the target language).

By comparing these measures across apps, and across time, the ELLA app data reveals patterns
of use and engagement levels among children involved in the trial.

Usage measures were also compared against several explanatory variables sourced from the
educator surveys, to attempt to identify factors that explained differences in iPad usage
among children.

Usage data from specific activities (which rewarded correct responses from users) was used to
represent child learning outcomes. This data was also compared against several explanatory
variables, to attempt to identify factors that may influence child outcomes.
A profile of the app usage data which informed this report is provided in Appendix A.
1.2.2
Online surveys
Three rounds of surveys were undertaken over the course of the evaluation. Specifically:

A baseline survey of trial sites was undertaken by ESA in April 2015 (the Baseline Educator
Survey). This survey sought to establish a baseline understanding of the implementation
process at each site and any relevant contextual factors (such as educator language and
technology knowledge).

In August/September 2015, Deloitte Access Economics conducted two mid-point surveys. One
of educators and directors (the August Educator Survey) and another of parents and guardians
of children participating in the ELLA trial (the August Parent/Guardian Survey).
•
The purpose of the mid-point surveys was to develop a comprehensive analysis of the
effectiveness of the implementation process, to flag any current issues and to provide a
preliminary indication of any impacts of the ELLA trial.

A final survey of educators and directors (the December Educator Survey), as well as parents
and guardians (the December Parent/Guardian Survey), was conducted by Deloitte Access
Economics in November/December 2015.
•
The final surveys sought to gauge educator and parent reflections on the trial in its
completeness. This included assessments of the trial’s impacts, learnings as to how
children can be most effectively engaged with the trial and any suggested
improvements.
A profile of the Baseline, August and December survey responses (i.e. the number of responses, by
variables of interest) is included in Appendix A.
1.2.3
Site visits
Consultations were held with a sample comprising over half of the 41 trial sites, in which each
selected site was visited at two points during the evaluation. The first round of site visits occurred
during June and July 2015. The second round took place in November 2015.
The sample comprised 21 trial sites, which were selected to provide a relatively representative
distribution of characteristics such as state or territory, ELLA app language, geographic location and
service type.
6
Deloitte Access Economics
ELLA Trial Evaluation Final Report
During these consultations, interviews were held with directors of the service, and any educators
who had been involved in the delivery of the ELLA trial. The purpose of these site visits was to
explore operation and performance of the ELLA trial from a site perspective in depth. The visits also
allowed for the evaluation team to develop a deeper understanding of the contextual factors
underpinning trial delivery through on-site engagement.
A profile of the trial sites which were engaged through site visits is included in Appendix A.
1.2.4
Literature
A review of literature on the use of digital technology for language learning in preschools was
undertaken. The purpose of this literature review was to develop an understanding of whether the
ELLA trial aligns with the available evidence on best practice approaches to:

language exposure in preschools; and

digital technology use in educational settings, focussed on preschools.
The findings of this literature review are provided at Appendix B, and referenced in the analysis of
trial design in Chapter 2.
1.2.5
Resources from Education Services Australia (ESA)
ESA was contracted by the Department to manage the design and development of suitable apps and
implement the ELLA trial. In its role as trial manager, ESA undertook several functions including
oversight of app development and release, and support for educators at trial sites (such as employing
a dedicated Early Childhood Liaison Officer, holding workshops and webinars, and providing a
helpdesk for technical support).
Information collected by ESA, such as trial service profiles and ELLA helpdesk reports, has been
shared with Deloitte Access Economics and forms part of the evaluation’s evidence base. A
consultation with ESA was also undertaken to ensure that any relevant knowledge acquired by ESA in
its role as trial manager was incorporated in the evaluation.
1.2.6
Education authorities
Deloitte Access Economics consulted with representatives from each state and territory education
authority, to understand how the delivery of the ELLA trial may be influenced by factors unique to
each jurisdiction. This included a discussion of the alignment between the ELLA trial and any existing
language education or technology policies within each state or territory, as well as relevant
considerations for any future development of the programme.
1.3 Report structure
The evaluation findings have been organised under several key headings, as outlined below.

Chapter 2: Trial design. This chapter analyses the available evidence supporting the
appropriateness of the design of the ELLA trial, with a focus on the use of tablet technology
and the software design, as well as the alignment of the ELLA trial design with best practice
principles.
7
Deloitte Access Economics
ELLA Trial Evaluation Final Report

Chapter 3: Trial implementation. This chapter considers the available evidence supporting the
effectiveness of the ELLA trial’s implementation, including the implementation process, the
level of support received and any implementation issues.

Chapter 4: Trial delivery. This chapter examines the available evidence supporting the
appropriateness and effectiveness of the delivery of the ELLA trial across sites, including how
the trial was introduced within sites, methods of trial delivery, the use of complementary
activities and how parents and guardians interacted with the trial.

Chapter 5: Trial engagement. This chapter analyses the available evidence on the
appropriateness and effectiveness of children’s engagement with the ELLA trial, including an
overview of children’s usage of the apps and an analysis of the factors potentially driving
differences in use.

Chapter 6: Trial impact and efficiency. This chapter provides an overview of the evidence
supporting the impact of the ELLA trial on child, educator and service outcomes across trial
sites. The chapter also includes an analysis of trial cost-efficiency, in light of the outcomes
achieved.

Chapter 7: Future considerations. This chapter analyses the options for the future of the ELLA
programme and draws on the evaluation findings to identify a possible way forward.
8
Deloitte Access Economics
ELLA Trial Evaluation Final Report
2 Trial design
This chapter analyses the available evidence on the appropriateness of the design of the ELLA trial,
with a focus on the use of tablet technology and the software design.
The key evaluation questions this chapter seeks to answer are:

Was the use of tablet technology to deliver the trial appropriate?
•
How user friendly were the iPads for the children and educators?
•
Were the iPads appropriate as an educational tool?

Was the design of the software appropriate? Is it fit for purpose?
•
Did the apps incorporate a level of functionality that was appropriate for their
purpose?
•
How user friendly were the apps for preschool aged children?
•
How user friendly were the apps for educators?

Does the ELLA trial design align with best practice principles for language learning and the
use of digital technology in preschools?
The evidence supporting the evaluation of this element of the ELLA trial has been drawn from ESA
documentation, the literature review, survey responses and site visit findings.
General findings on the design of the ELLA trial

Evidence collected from the ELLA trial suggests that the trial does provide a model which can
enable language exposure in preschool environments where the educator is not language
proficient.

Tablet technology appears to be an appropriate digital tool in the context of the ELLA trial,
with both the children and educators finding the technology highly functional and easy to
use.

There were initially some concerns among parents and educators about the impact of
children using iPads in preschool, particularly in relation to the risk of excessive use.
However, it was generally found that when supported by basic use guidelines and
supervision, the iPads and ELLA apps were an appropriate and effective play-based learning
tool.

The ELLA apps were found to be easy for both children and educators to navigate, with
children often helping each other to explore the apps, supported with minimal educator
guidance.

Educators found the ELLA apps to be appropriately targeted to the preschool age group,
incorporating a play-based design and themes relevant to the children’s interests and
broader development level.

Overall, the ELLA trial design is aligned to the majority of best practice elements in language
learning through digital devices in preschools – however, there is scope to increase guidance
to educators to further increase the utility of any such programme.
9
Deloitte Access Economics
ELLA Trial Evaluation Final Report
2.1 Tablet technology
There are two key aspects to the appropriateness of tablet technology within the ELLA trial: (1) the
extent of their functionality (including user friendliness); and (2) their use as an educational tool,
particularly for this age group of children.
2.1.1
Functionality of iPads
Prior to the introduction of the ELLA trial into the sites, there was already a moderately high level of
comfort in the use of touchscreen devices at both the educator and child level. Supporting this, the
Baseline Educator Survey found that 70% of educators were ‘very comfortable’ with using
touchscreen devices, while 30% were ‘somewhat comfortable’.
A minority of preschools mentioned in site visits that not all educators in the room were initially
confident using iPads. In some cases these educators were less engaged in the trial as a result, but
this was not seen as a significant issue as other educators could support the trial delivery. As a rule,
the vast majority of educators (as evidenced in educator survey results) were either already familiar
with iPads or were quickly able to familiarise themselves.
Regarding the use of the iPads and apps by children, the findings were even more positive than that
of educators. It was found that children were typically easily able to use the iPads and apps, with
many children already familiar with the functionality of the devices prior to the beginning of the trial.
The Baseline Educator Survey indicated that more than half the trial sites estimated that at least
three-quarters of their participating children had access to a touchscreen device at home. Responses
from parents and guardians themselves supported this, showing that the majority of respondents to
the surveys used digital devices within the home (Chart 2.1 below). Additionally, most survey
respondents (68%) stated that they use a tablet with their child at home frequently or most days. As
the survey was undertaken online, it is possible that the response is biased towards more technology
savvy users, however, all evaluation sources indicated a high and growing prevalence of technology
within the home.
10
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 2.1: Prevalence of digital devices within the home of ELLA participants
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Smartphone
Laptop or
desktop
computer
Tablet (iPad or
similar)
None
Source: Combined and weighted results from August Parent/Guardian Survey (276 responses) and December
Parent/Guardian Survey (241 responses)
Educators consistently indicated during interviews that almost all children were immediately able to
use the iPad without any significant difficulty. While children who had not used tablets before
sometimes required additional guidance in the early stages of the trial, it was stated that these
children became proficient very quickly, reflecting that children found use of both the iPads and apps
to be intuitive.
A number of educators also mentioned that they had initially been concerned that the iPads would
be easily damaged through everyday use. However, the cases supplied with the iPads were found to
be durable.
In recognition of the ease with which children were found to be able to use the iPads, there was a
consensus across survey responses and site visits that iPads were a suitable tool for use by preschool
aged children in the ELLA trial, noting that this also relied upon the suitability of the apps themselves.
2.1.2
iPads as an educational tool
A number of educators stated, in both survey responses and site visits, that they had initially been
concerned about the appropriateness of iPads as an educational tool within the classroom. These
concerns were primarily centred on:

children using the iPads at the expense of other important learning activities, such as physical
activities and playing outside; and

children reducing their level of social interaction due to the individual nature of iPad use.
In most cases, however, these fears were allayed during the trial. Based on site visits, educators were
generally found to be able to ensure that the amount of use was appropriately balanced with other
activities. It was noted that the iPads were commonly used in a social manner, with children often
seeking guidance from each other in pairs or small groups, encouraging social interaction and joint
learning experiences. However, the social use was limited somewhat in those sites that encouraged
11
Deloitte Access Economics
ELLA Trial Evaluation Final Report
use of the apps with headphones. Overall, the ease with which children interacted with the iPads,
and the swiftness with which learning outcomes could be observed, raised educator confidence in
the use of digital technology in the classroom.
On conclusion of the trial, educators were asked during site visits to reflect on the value of digital
technology and apps as a way of exposing children to other languages and cultures. The response to
this question was positive; with almost all educators noting that the high quality of the ELLA apps had
ensured the trial was effective in their site. It was stated that the apps kept the children continually
engaged and the incorporation of digital technology into the classroom had not proved difficult, in
fact resulting in some additional benefits as children and teachers began incorporating technology
into other learning areas such as research or storytelling.
Despite the alleviation of concerns over iPad use, there was a widespread view among sites that
additional guidance would have been beneficial regarding: how best to incorporate the iPad into the
learning environment; and, what was considered appropriate screen time for preschool aged
children. However, preschools typically emphasised that the apps and iPads were a supporting tool
only, which could not replace educator engagement with children. Indeed, the apps were generally
seen as complementing educator engagement. Isolated incidents of overuse of the iPads continued
to highlight the need for constant supervision by educators.
Parent and guardian perceptions on the use of iPads are outlined in Box 2.
Box 2: Parent/guardian perceptions regarding iPads
Site visits and educator survey responses indicated that there was some level of concern about how parents
and guardians would perceive the introduction of iPads into the preschool, particularly in sites which had not
used such technology prior to the ELLA trial.
Specific concerns were that parents would think the educators were performing a less active role than would
otherwise be the case as a consequence of the iPads being available, or that this would lead to an
undesirable amount of screen time in general for their children, particularly given the widespread availability
of digital devices at home. Most educators who voiced these concerns stated that after the trial had been
implemented, it became easier to communicate the benefits of the trial to parents as they could see the
high quality of the apps, and how much the children were engaged with them, first hand.
The August and December Parent/Guardian Surveys confirmed that the majority of respondents did not
have significant concerns about the use of iPads as a learning tool for preschool children. At the conclusion
of the trial, 76% of respondents to the parent and guardian survey stated they agreed that iPads were a
good way for preschool children to learn, with reasons for this stated to include:

Technology is the way of the future and developing digital literacy is important for all children.
•
It was also noted that the use of iPads in preschools helps to teach children the educational
value of digital devices, rather than simply being a passive instrument to be watched.

The functionality of iPads is engaging for children, encouraging play-based learning and self-discovery.

If appropriately supported in the classroom, iPads can complement other ways of learning (such as
educator led discussion, games or storybooks).
•
It was noted by a sizeable portion of parents and guardians that a balanced approach to iPad
use is necessary, with monitoring to ensure no overuse or misuse.
Opposition to the concept of iPads being used in preschools was voiced by only a minority of parents and
guardians (with 8% of survey respondents – averaged across the August and December Parent/Guardian
Survey disagreeing that iPads were a good way for preschool children to learn). In these cases, the most
12
Deloitte Access Economics
ELLA Trial Evaluation Final Report
common reason for this objection was a view that children would have an opportunity to engage with
technology in later years and that these formative years of childhood should be free from screen time and
focused on developing skills through social, play-based activities. However, there was a relatively consistent
message from parents and guardians, through the surveys, indicating that they felt the use of iPads in
preschools, if carefully monitored, was a positive addition.
2.2 Software (‘app’) design
There are two key elements to the appropriateness of software design within the ELLA trial: (1) the
ease with which educators can use and engage with the apps; and (2) the ease with which children
can use and engage with the apps, particularly in a setting in which educators are not required to
have an understanding of the languages being conveyed through the apps.
2.2.1
Educator navigation of the apps
As a general rule, educators found the ELLA apps quite easy to use. As illustrated in the charts below,
a clear majority of educators who responded to the surveys stated that they could navigate the ELLA
apps easily, with this proportion increasing over the course of the year until 85% of educators could
navigate the apps ‘very easily’ and there were no reported difficulties. As such, it is understood that
the reason a number of educators stated that their ability to navigate apps had not improved since
the trial began was due to an already proficient level of ability at the trial inception.
Chart 2.2: How easily can educators navigate
the apps?
90%
August Survey
80%
December Survey
Chart 2.3: Has educator ability to navigate
the apps improved since the trial began?
90%
70%
70%
60%
60%
50%
50%
40%
40%
30%
30%
20%
20%
10%
10%
0%
August Survey
80%
December Survey
0%
Very easily
Somewhat easily
Not easily
Source: Educator August Survey (105 responses) and
December Survey (78 responses)
Yes
No
Unsure
Source: Educator August Survey (105 responses) and
December Survey (78 responses)
It was also noted by several educators that children would often learn the functions within each app
more quickly than the educators. This often created an environment in which children demonstrated
use of new elements they had discovered in the apps among educators and other children. This
instilled confidence within the children and fostered collaborative learning.
Reflecting on this concept, one educator stated during site visits that ‘Learning the language together
is great for teacher/child relationships – it’s wonderful for the children to see the adults learning and
fosters true collaboration. It empowers the children. We both respect each other’s knowledge.’
13
Deloitte Access Economics
ELLA Trial Evaluation Final Report
2.2.2
Child navigation of the apps
Children used and engaged with the ELLA apps easily, with this finding supported through the site
visits, surveys and usage data. Drawing on the Baseline Educator Survey, a clear majority of children
were reported as successfully navigating the apps, even in the early stages of the trial (Chart 2.4).
Chart 2.4: How easily can children navigate the ELLA apps?
90%
Ease with which children can log in
using their avatar
80%
70%
Ease with which children can play with
the activities in the app
60%
50%
40%
30%
20%
10%
0%
Very easily
Somewhat easily
Not easily
Source: Baseline Educator Survey (37 responses for avatar login, 38 responses for ease of play)
Site visits confirmed this message, with a consensus that children were usually able to select their
own avatar, enter into the apps and navigate through various activities without any problems. It was
noted that in the beginning of the trial, it took some time for all children to remember to switch
avatars to their own each time they used the iPads. A minority of educators also stated that some
children sporadically needed help to work through an activity, but that these problems were easily
overcome. Educators found that children generally worked through any functionality issues by
watching and helping each other, rather than seeking guidance from the educator. This meant the
children often explored the apps together and used shared learning experiences to navigate their
way through the apps.
The apps themselves were almost universally considered to be appropriately targeted to preschool
children, and mostly easy for the children to navigate. Educators consulted during site visits generally
reflected that the app design and content were of a high standard, with the apps often described as
being ‘great’ or ‘fantastic’. It was also commented on during site visits that the apps seem to become
progressively more engaging for children as the year progressed. A number of educators particularly
noted that the play-based design of the apps was suitable for the age-group, encouraging social
interaction and providing useful links to the Early Years Learning Framework (EYLF). In this vein, the
content of the apps frequently aligned with the learning priorities of the respective site – reflecting
that each site had applied to utilise that particular language – and therefore complemented the
existing learning programme at each service.
A small number of educators stated during the site visits, predominantly in the early stages of the
trial, that they felt children were clicking buttons and flicking through the different app components
too quickly, ‘like a game’, rather than engaging with each of the activities in the manner intended.
While it is acknowledged that the ELLA apps were designed to be play-based and promote immersive
learning, these educators were concerned that children were not properly listening to the language,
which in turn, limited the effectiveness of the apps as a learning tool. However, some educators also
14
Deloitte Access Economics
ELLA Trial Evaluation Final Report
stated that the tendency to ‘flick through’ the apps reduced both (1) when an educator sat with the
child and asked them to explain the activities and what they were doing within each app; and (2) as
time progressed and the children became more familiar with the workings of the apps and their
favourite activities.
There was also a concern in a small number of sites that children were not being sufficiently
prompted to speak the target language, and occasionally reverted to English while engaging with the
app. It is noted, however, that responding to the apps in English does not necessarily mean that the
child using the app does not understand the language prompts. As such, this concern suggests that
increased guidance for educators on what to expect from children’s interactions with the apps, and
the language learning process, may be required, as opposed to being a design issue with the ELLA
apps.
2.3 Alignment of the trial to best practice
A literature review, included at Appendix B, identified a range of best practice principles in relation
to: (1) the integration of digital technology in preschools; (2) optimal app design features for early
learners and language education; and (3) teaching additional languages in preschools.
This section compares these best practice principles with the design features of the ELLA trial and
provides commentary on the alignment between the two. Additionally, feedback from parents and
guardians on the ELLA trial design is provided.
Findings for each area are outlined in Table 2.1, Table 2.2 and Table 2.3, and compared with the
design of the ELLA trial.
Table 2.1: ELLA design relative to best practice principles in integrating digital technology in
preschools
Best practice principles
ELLA trial
Tablets should not replace
good teaching
The ELLA trial was designed so educators did not have to
be familiar with the target language. However, several
educators have acknowledged the importance of
educator involvement while children engage with the
apps. As such, the alignment of the ELLA trial with this
principle depends upon the extent to which supporting
resources encourage and support educator engagement.
Provide a range of
pedagogical support for
teachers
ESA offered a range of materials to support educators’
use of the apps. While most educators found the support
helpful, there were some educators who felt there could
have been more support materials made available. The
additional support material could have included tips for
integration with the learning programme, research
summaries and a list of resources to support child
learning.
Provide technical support
to schools
ESA was contracted to operate the ELLA Helpdesk as part
of its contract management role, which was designed to
support sites encountering technical difficulties. This was
widely considered to offer timely and appropriate
support.
Alignment
Partially aligned
Partially aligned
Aligned
15
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Best practice principles
ELLA trial
Tablets are most effective
when used in a supportive
school and home
environment
While the preschools were supportive of the use of the
apps, they were not made available for use at home. A
number of parents stated that they feel the trial could be
more effective if available for use outside the preschool
setting.
The ratio of tablets to
students matters
ESA distributed approximately five iPads per group of 25
children. A ratio of one device for several children is
generally considered to be optimal for preschool children,
so this ratio is consistent with best practice.
The choice of app is
important
Alignment
Not aligned
Aligned
ESA, in conjunction with the Department and Millipede,
designed apps specific for their purpose, rather than
relying on publicly available apps.
Aligned
Source: Deloitte Access Economics literature review, ELLA trial app data, August educator survey, consultations
Table 2.2: ELLA design relative to best practice principles in app design for early learners
Best practise principles
Allow content creation
ELLA trial
Alignment
There are some activities within the ELLA trial which allow
the children to create their own content (such as Sandpit
or Colour & Create).
Aligned
Mixed learning methods
The ELLA apps are consistent in providing children
opportunities to learn through a variety of means. The
apps allows children to learn via visual means (through
the imagery on the screen), via aural means (by listening
to the target language), and via oral means (by speaking
into the microphone).
Personalised pathways
The ELLA trial offers four to six activities within each app.
The variety of tasks offered to children across these
activities allows them considerable scope to personalise
their learning experience.
Aligned
Given the frequency and length with which the apps have
been in use, it appears the content is engaging for
children. This is supported by positive reviews from
educators and parents/guardians.
Aligned
Engaging content
Appropriate hardware
Aligned
Educators have commented the screen size is appropriate
for children of this age and that iPads are intuitive and
easy to use.
Aligned
Encourage skills
development
The apps offer children opportunities to develop a variety
of foundation skills, such as colours, counting and
greetings. They are encouraged to develop their skills by
speaking the words when prompted by the apps.
Aligned
Source: Deloitte Access Economics literature review, ELLA trial app data, August educator survey, consultations
16
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Table 2.3: ELLA design relative to best practice principles for teaching additional languages in
preschools
Best practice principles
Play based learning is
important in early
language learning
Repetition
ELLA trial
The ELLA apps allow for children to engage in play-based
activities while being exposed to additional languages.
Aligned
Language learning through apps allows for more
repetition than through learning from an educator, as the
apps allow children to re-listen to the words and phrases
as they choose.
Develop listening skills
Language is learnt best thought through oral and aural
means, rather than through reading and writing,
especially considering most preschool children are
preliterate. Most language exposure through the ELLA
trial is done orally and aurally, allowing children to
develop their listening skills.
Use contextualised
language
Contextualised learning plays a large role in the ELLA
apps. The characters often use hands and facial
expressions, in addition to the target language, to convey
meaning.
Social and cultural context
Alignment
Aligned
Aligned
Aligned
The apps are designed to display the culture associated
with each language.
Aligned
Use of decontextualised
language (language
removed from context)
The app does not offer children the opportunity to use
decontextualised language. Such learning (such as
children developing stories or recalling past events in the
target language) must be educator led, and personalised
for each student. App-based learning cannot provide this
opportunity for children. As such, this is dependent on
the level of engagement of the educator and the
resources provided to support decontextualised language
use outside the apps.
Language production
Several activities employ the use of a microphone, which
rewards children for attempting to speak the target
language. Such activities encourage language production,
consistent with best practice.
Not aligned
Aligned
Source: Deloitte Access Economics literature review, ELLA trial app data, August educator survey, consultations
In summary, across all three facets of best practice, the ELLA trial appears to have been designed and
implemented largely in accordance with best practice principles.

While there is scope to offer more guidance to educators, the introduction of iPads and apps
largely appears to have been consistent which much of the evidence in the literature as to
effective use of digital devices in preschools.

Best practice in language learning is achieved in all but one facet (encouraging the use of
decontextualised language), for which there is scope to achieve alignment if educators are
sufficiently resourced to deliver supporting activities.

The apps incorporate all the elements of app design identified in the literature which are
consistent with best practice.
17
Deloitte Access Economics
ELLA Trial Evaluation Final Report
3 Trial implementation
This chapter provides an overview of the available evidence on the effectiveness of the ELLA trial’s
implementation.
The key evaluation questions this chapter seeks to answer are:

Was the trial able to be implemented effectively?
•
What was the implementation process at the trial sites?
•
Were there any impediments to a smooth trial implementation process?
•
Was the level and nature of support provided to the trial sites sufficient to facilitate a
smooth implementation process?
•
Was the initial availability of apps and subsequent app updates timely and accessible
for the trial sites?
•
How could the trial implementation process be improved in the future?
The evidence supporting the evaluation of this element of the ELLA trial has been drawn from the
survey responses, site visit findings and help-desk data.
General findings on the implementation of the ELLA trial

Implementation was generally smooth despite the limited timeframes, although there
were significant network issues at a few state and territory government operated centres
which caused delays in trial implementation.
•
The underlying drivers of these delays, particularly navigation of government school
firewalls, will require attention prior to any broader roll-out of the ELLA trial.

While the majority of services, as evidenced by survey responses, experienced some form
of implementation issue for which they required assistance to resolve (most commonly a
technical issue), these issues were usually minor and able to be addressed within a
relatively short timeframe.

ESA introduced a range of support materials for the implementation of the ELLA trial.
Educators felt the support mechanisms available to sites throughout the trial were
accessible and useful. In particular, the introductory training workshop, website and
communication with ESA were stated to be helpful tools.

Educators found that while restrictions on iPad use by children and waiting lists were
necessary at the beginning of the trial, as the year progressed and children learnt to selfmoderate their use, rules surrounding iPad use became less rigid and the devices became a
normal fixture in the room.

There are several ways in which the ELLA trial implementation process could have been
improved, including reducing the administration time for enrolling children and supporting
educators in delivering the ELLA trial around non-participating children.
18
Deloitte Access Economics
ELLA Trial Evaluation Final Report
3.1 The implementation process
The objective of the ELLA trial is to expose children to a language other than English using a series of
seven apps, delivered on digital learning devices. In introducing the trial, the Department supplied all
41 trial sites with their own set of iPads. Overall, 336 iPads were delivered to sites – 280 iPads for
children, and 56 iPads for educators. The sites were selected by the Department following an
application process in which sites nominated their preferred language and provided supporting
rationale.
Prior to the commencement of the trial, one to two educators or directors from each site took part in
an introductory training workshop. The main purpose of the workshop was to introduce each trial
site to the ELLA programme, including familiarising educators with the trial objectives and providing
an opportunity for educators to share ideas on how the trial could be introduced in each service. The
workshop also detailed the technical information educators required in order to be able to download
each app, create different groups, and how to set up a new user.
A suite of introductory materials were given to the trial sites at the workshop. These included:

literature on the benefits of tablets and language learning in early childhood education;

an ELLA ‘how to’ guide, with detailed instructions on how to set up and use the iPads,
download the apps, create a new group, add a new user, and other general iPad maintenance
issues;

contact details for the ESA helpdesk, in case any technical issues were encountered;

an outline of the activities in App 1;

a pronunciation and translation guide for App 1; and

a cultural guide for each language.
The workshop was followed by a visit to each site from an ESA liaison officer, whose role was to
support the sites in implementing the trial and methods which had been discussed during the
workshop. The ESA visits were made throughout the year, with not all visits occurring within what
would have been considered the implementation period.
Before children could participate in the trial, a parent/guardian of each child was required to
complete a consent form. Consent forms were distributed by each site to parents and guardians of
eligible children.4 Forms were returned to the service and then sent to ESA so the child could be
included as a trial participant.
Once ESA received and processed a child’s consent form, the educator could set up their profile. To
set up the profile each child selected an avatar, allowing them to easily log in as the correct user. At
most sites, a child’s avatar was a photo of themselves that had been uploaded to the app – parents
and guardians were also asked to consent to a child’s photo to act as the child’s avatar. A minority of
parents/guardians did not consent, so in these instances, children chose an image unique to them
(such as a favourite toy), which would act as their avatar.5
4
Children eligible for the ELLA trial are those eligible to attend their first year of primary school in 2016.
In accordance with the research approval granted by the South Australian Government, photos of children were not
permitted to be used for avatars in Government sites in South Australia.
5
19
Deloitte Access Economics
ELLA Trial Evaluation Final Report
The apps were released progressively throughout the year with supporting materials. The use of the
apps within services was therefore dependent on sites being able to access wifi which allows them to
download each new app as it is released. Once downloaded, wifi is not necessary to use the apps,
although wifi is needed to collect the usage data.
3.2 Level of support received
Given the uniqueness of the ELLA trial, it was anticipated there would be a number of issues in
implementing the programme at the trial sites. ESA was contracted by the Department to support
the sites through mechanisms including a help desk (details on help desk usage are provided in Box
3), an introductory training workshop, webinars, a website and a liaison officer.
3.2.1
Support during trial implementation
It was found that there was a very high level of understanding among educators about the forms of
support available throughout the ELLA trial. In the Baseline and August educator surveys, 100% and
87% of educators respectively, either agreed or strongly agreed that their service understood the
support available through the ELLA trial. This suggests that the means employed to educate the
participants on how the trial would operate, including the initial workshop, liaison site visits and the
provision of supporting resources6, were effective.
The higher proportion of educators who ‘strongly agreed’ with the proposition that they understood
the support available in the Baseline Educator Survey is potentially influenced by the fact that this
survey was only completed by one educator per service, which would most likely have been the ELLA
lead within that service. The August and November educator surveys, in contrast, reached a wider
breadth of educators and as such the average levels of understanding were lower. However, the
overall level of understanding increased over the course of the year (Chart 3.1 below).
Chart 3.1: Educator agreement to the statement ‘Our service understands the support available
through the ELLA trial’
70%
Baseline Survey
August Survey
December Survey
60%
50%
40%
30%
20%
10%
0%
Strongly agree
Agree
Neither agree nor disagree
6
General programme support resources included the Programme Implementation Guide, the Parent information brochure
and the Quick Start Guide. For each app an overview of the app, the song lyrics, the song soundtrack and artwork of each
character was provided.
20
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Source: Educator Surveys – Baseline (39 responses), August (98 responses), December (68 responses).
In regards to using the support mechanisms available, services overwhelmingly stated that help was
available when required. This was reiterated throughout the site visits, in which educators often
commented on the high level of support that had been available to help navigate any difficulties
encountered.
Chart 3.2: Educator agreement to the statement ‘Our service is able to obtain the support we need’
60%
Baseline Survey
August Survey
December Survey
50%
40%
30%
20%
10%
0%
Strongly agree
Agree
Neither agree nor disagree
Disagree
Source: Educator Surveys – Baseline (40 responses), August (98 responses), December (68 responses).
Following trial implementation, the positive commentary around the level of support provided to
trial sites did not diminish. The vast majority of educators (81% in August and 87% in December) who
responded to the educator surveys agreed or strongly agreed that they had received adequate
ongoing information from ESA throughout the ELLA trial (Chart 3.3 below).
Chart 3.3: Extent to which educators agreed with the statement ‘I have received adequate ongoing
information from ESA throughout the ELLA trial’
50%
August Survey
45%
December Survey
40%
35%
30%
25%
20%
15%
10%
5%
0%
Strongly agree
Agree
Neither
Disagree
Strongly disagree
Source: Educator Surveys - August (98 responses) and December (68 responses)
21
Deloitte Access Economics
ELLA Trial Evaluation Final Report
However, 66% of educators who responded to the August Educator Survey stated that they would
like to participate in further training about the apps, should it be made available.7The desired forms
of additional training, as stated by educators in free text responses, were centred on how to:

Encourage all children who have parental/guardian consent to use the apps (86% of educators)

Engage further with parents/guardians about the trial (76% of educators)

Introduce other language-based learning activities, to support what children are learning
about through the ELLA apps (74% of educators)

Introduce other cultural-based learning activities, to support what children are learning about
through the ELLA apps (73% of educators)

Introduce other learning activities not related to learning or culture, to support what children
are learning about through the ELLA apps (67% of educators)
Most helpful forms of support
3.2.2
The resources provided to sites over the course of the ELLA trial were generally considered
appropriate. The majority of respondents to the educator surveys stated that they found the support
materials ‘very helpful’ (53% in August and 68% in December), with the remainder typically finding
the materials ‘somewhat helpful’. Approximately 16% of respondents in August, and 7% in
December, found the materials ‘not very helpful’ or indicated that they had not used them.
Chart 3.4displays the support mechanisms which were found to be the most useful for educators
throughout the ELLA trial. Other useful means of support were mentioned as being the overview
provided with each app and the reminders from ESA (phone calls/emails) at key stages throughout
the trial.
Chart 3.4: Useful support tools (% of educators who nominated a mechanism as ‘useful’)
60%
August Survey
December Survey
50%
40%
30%
20%
10%
0%
Training ELLA weekly ELLA website ELLA Liaison
workshops
email
Officer
ELLA
Helpdesk
ELLA
Facebook
page
ELLA
webinars
Early Years
Learning
Framework
Outcomes
Source: Educator Surveys - August (98 responses) and December (81 responses). Note, the Early Years Learning Framework
Outcomes were not included as an option in the August Survey.
22
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Educators who attended the introductory training workshop8 felt this was an effective introduction
to the trial, and contained important information on the objectives of the trial and how to introduce
the apps within sites. It was noted in site visits that the trial was implemented more successfully
when educators had a strong understanding of the intended benefits of the ELLA trial and the
workshop helped to engage educators, generating enthusiasm and ideas.
Educators also reiterated through the August and December survey responses, and site visits that:

sharing ideas between sites was highly useful, and it would have been beneficial if the
Facebook page (or a similar mechanism) was used more actively and by a greater number of
educators;

it was useful having a point of contact (ESA) through which to seek help with initial
implementation issues; and

the ESA Helpdesk was highly responsive and helpful.
3.3 Implementation issues
While implementation was generally smooth, a few technical and administrative issues were
encountered by services during the implementation of the ELLA trial. A discussion on the nature and
prevalence of these issues, and how this impacted the timeliness of implementation, is provided
below.
3.3.1
Technical issues
Feedback gathered from services during site visits suggested that the implementation of the ELLA
trial into services had been relatively smooth. Despite this, approximately half of the services
indicated in the Baseline Educator Survey that there had been technical difficulties encountered
during the implementation of the ELLA trial. This figure was even higher in educator responses to the
August and December surveys, at approximately 70% of services stating that they had experienced
technical difficulties. The contrast between these conflicting forms of feedback (a smooth
implementation process but a relatively high proportion of sites with some form of technical issue)
suggests that while technical issues were prevalent across trial sites, they were typically minor, and
managed in a manner which did not significantly impede implementation of the trial, leaving
educators at most sites positive about the implementation experience. In a minor number of sites,
however, there were significant technical difficulties early on which sometimes persisted and created
major delays to these sites effectively implementing the ELLA trial (see below).
The technical difficulties encountered by services, as described in the August and December educator
surveys, included:

Difficulties with the internet (33% of responses): including trouble connecting at the start of
the trial due to government protected internet connections (particularly in Western Australia,
South Australia and the ACT), download limits, server problems and wifi issues. Generally
these services reported that the internet issues were resolved at the beginning of the year, in
some cases requiring additional investment. A minority of services (five) reported persistent
connection issues due to wifi variability, resulting in apps being slow to download.
8
The introductory training workshop was followed by a second workshop later in the year, which was focused on ongoing
programme delivery rather than programme implementation. Survey responses presented in Chart 3.4 refer to both
training workshops collectively.
23
Deloitte Access Economics
ELLA Trial Evaluation Final Report

Difficulties with the iPads (30% of responses): including iPads freezing, children accessing the
password settings and locking the iPads, issues with the microphones and the iPads running
out of battery quickly. Three services reported iPads which stopped functioning completely.
There were also several reports of damage to the iPad, including screens cracking or the iPad
being left in the rain.

Difficulties with the apps (18% of responses): the most commonly reported issue with the
apps was difficulty downloading new apps onto the iPads. A number of services reported that
sometimes the apps would download onto only some of the iPads.
Other reported issues included syncing and accessing the app data and ensuring the children’s
avatars were in the correct user group.
The implementation process was more difficult for government services in South Australia and
Western Australia, which experienced problems with the firewalls. In these cases, the problem took
many weeks to resolve and required a significant time investment from staff members. South
Australian sites indicated that the final solution which enabled them to participate in the trial was
only a temporary fix and a more permanent solution will need to be developed if the trial is to be
rolled out more broadly.
ESA Helpdesk Data, as provided in Box 3 on the following page, provides an overview of the type and
number of implementation issues encountered by trial sites.
24
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Box 3: Helpdesk data
The ESA helpdesk was set up to support educators encountering difficulties of a technical nature. Requests for
support were logged by ESA, and classified by severity (either major or minor), and by problem type
(administration, software, deleted avatar, internet, hardware, data and other).
Over the trial period, 142 requests for assistance were made to the ESA helpdesk, of which 44 were classified
major and 98 minor. Unsurprisingly, the number of requests by most problem types decreased over time. (Only
requests for avatar deletion increased towards the end of the year, as children left the service.). Administrative
issues, software issues and data deletion were the main causes of a helpdesk call, making up a 75% of all issues
reported between them (Chart 3.5).
Chart 3.5: Issue type by resolution length
Number of issues reported
40
>7 days
30
≤7 days
20
10
0
Source: ESA Helpdesk
The length of time between when an issue was created and when it was resolved varied significantly. Minor
issues were predominately resolved on the same day, whilst major issues were typically resolved between
eight and 30 days (Chart 3.6). 48% of all issues were resolved within seven days. 8 issues remained unresolved
at the conclusion of the trial period.
Chart 3.6: Issue resolution times by priority
Number of issues reported
40
Same day
2-7 days
8-30 days
>30 days
Unresolved
30
20
10
0
Minor
Major
Source: ESA Helpdesk
25
Deloitte Access Economics
ELLA Trial Evaluation Final Report
3.3.2
Administrative issues
While technical issues were the most commonly reported problem faced by trial sites, it was also
noted in consultations with trial sites that the process of ensuring consent forms were appropriately
filled out, returning them to ESA and then the time lag before children were added to the system
caused issues for some sites. Several sites stated that parents were displeased at the length of time it
took to register the child after consent had been given, reflecting on the difficulties of explaining to
children and parents that the child couldn’t use the iPads until the process had been completed.
In this vein, a theme emerging from the site visits was the difficulties associated with providing the
ELLA trial to participating children in a manner which did not exclude non-participating children. The
current format of the ELLA trial requires that only children in the year before formal schooling, and
with parental consent, could use the apps. Educators stated that non-participating children (such as
three year olds who were too young for the trial or those who hadn’t received parental consent)
would express a desire to use the apps and would watch the children who were eligible to use them.
Sites would have preferred to not exclude non-participating children from use.
3.3.3
The timeliness of implementation
The implementation issues explored above had an impact on the timeliness in which the ELLA trial
was successfully integrated into each site. As such, one way to measure the magnitude of
implementation issues across different service types is by observing the number of weeks it took
each preschool to begin using the iPads.
Table 3.1 indicates the length (in weeks) for different types of preschools to commence using their
iPads.9 Consistent with the findings discussed above, government services took, on average,
significantly longer to commence using the iPads than independent services and LDC centres.
Government services in South Australia, Western Australia and New South Wales all experienced
delays in commencing the trial. This was driven primarily by the difficulties in accessing the ELLA apps
through government protected internet connections.
9
For the purpose of this exercise, it was assumed the ELLA trial commenced at Week 10 of the school year, as this was two
weeks after the initial educator workshop. This allows sufficient time for services to receive their iPads and other
programme resources.
26
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Table 3.1: Implementation length by provider characteristics (in weeks)
Provider type
Attached-to-school (Government)
Attached-to-school (Independent)
Long day care
Stand-alone (Government)
Stand-alone (Independent)
3.5
1.0
0.6
3.8
0.1
Socioeconomic status of area in
which service is located
High
Medium
Low
0.2
1.3
1.8
Service location
Metropolitan
Regional
Remote
1.7
1.2
0.5
Source: ELLA trial iPad data
Prior to the commencement of the ELLA trial, it was hypothesised that remote sites may experience
greater wifi connectivity issues, and would have trouble implementing the trial. However, it appears
remote sites had little trouble connecting to the internet and accessing the apps, with the average
delay in commencing use only 0.5 weeks. This finding is supported by consultation with jurisdictional
educational authorities, in which it was often stated that the infrastructure established to support
distance education was generally of a high quality and would in most cases enable provision of the
ELLA trial in remote areas. It should be acknowledged however, that there are only two remote sites
in the sample.
Aside from implementation issues, feedback from the August and December educator surveys and
consultations found that the timeliness of the app releases was generally considered to be
appropriate, although some educators felt that App Four was released too quickly after App Three –
there were only four weeks between the two apps. Chart 3.7 shows that at least 85% of respondents
agreed or strongly agreed that app release was timely across the Baseline, August and December
educator surveys. It is noted that there is a general shift from ‘strongly agree’ to ‘agree’ between the
Baseline and later surveys, potentially indicating that while release of the first app was considered
timely, this was less emphatically so the case for the remainder of the apps.
27
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 3.7: Level of educator agreement to statement ‘the release of new apps has been timely’
60%
Baseline Survey
August Survey
December Survey
50%
40%
30%
20%
10%
0%
Strongly agree
Agree
Neither
Disagree
Source: Educator Surveys – Baseline (39 responses), August (98 responses) and December (68 responses).
Note: Responses to the August and December surveys were for the statement ‘release of new apps has been timely’ while
responses to the Baseline Educator Survey were to the statement ‘release of the first app was timely’.
In summary, the evidence indicates that while there were several technical and administrative issues
identified throughout the implementation process, the level of support underpinning the delivery of
the ELLA trial in the trial sites mitigated the impact of these difficulties. However, the more
significant issues, such as navigating protected internet connections within government school
services and how the delivery of the ELLA trial effects non-participating children, will need to be
considered in any future expansion of the trial.
28
Deloitte Access Economics
ELLA Trial Evaluation Final Report
4 Trial delivery
This chapter provides an overview of the available evidence on the appropriateness and effectiveness
of the delivery of the ELLA trial across sites.
The key evaluation questions this chapter seeks to answer are:

How do services deliver the trial?
•
How was the trial introduced in the participating sites?
•
What methods do educators use to engage children with the apps?
•
What complementary activities are delivered to support the ELLA trial within services?

How are parents/guardians involved in the trial?
•
Was the involvement of parents/guardians throughout the trial appropriate?
The evidence supporting the evaluation of this element of the ELLA trial has been drawn from the
survey responses, ELLA trial data and the site visit findings.
29
Deloitte Access Economics
ELLA Trial Evaluation Final Report
General findings on the delivery of the ELLA trial

There was variation in how educators introduced the ELLA trial into their sites. This reflected
the flexibility afforded to sites by the trial, which was intended to allow each service to find
an approach which best suited their circumstances. However, some educators stated they
would have appreciated more guidance as alluded to previously.

Following this, there was also a high degree of variation in (1) the way the ELLA trial was
delivered within sites; (2) the use of complementary activities to support the ELLA apps; and
(3) the level of parent and guardian interaction. There were other less pronounced
inconsistencies, such as the use of headphones in some sites.

The ELLA app data has not been widely used by educators to inform trial delivery and as
such, the format of the data provision to educators may need revisiting.

While a range of complementary activities have been implemented across the trial sites, in
general there was an inconsistent and ad hoc approach to supporting the ELLA apps through
broader integration with the learning programme. However, this did improve as the trial
went on.

Specifically, the level and consistent use of complementary activities increased following the
second workshop, where educators were given opportunity to share ideas and reiterate the
importance of engaging children in language use outside the app. As such, additional
resources for educators or further avenues to share ideas across sites may increase the
number of activities undertaken.

Parents and guardians have generally had relatively high levels of understanding and
positive perceptions of the ELLA trial. Further, 99% of all parents/guardians whose children
were invited to participate in the ELLA trial gave consent for their child to participate in the
trial, indicating a strong willingness to allow their children to be involved.

However, a number of parents/guardians expressed a desire for more information, including
further resources to support their child’s learning at home, child level progress updates, and
more frequent and detailed information updates about the ELLA trial.
4.1 Introducing the trial
There were several components to be considered in the introduction of the ELLA trial into the site,
including: (1) introducing children to the trial; and (2) integrating iPads into the site.
4.1.1
Introducing children to the trial
Generally, educators stated that they held an initial group discussion with their class to introduce the
ELLA trial. In this session they showed how the iPads and apps worked, explained about different
languages and cultures and described how the children would be learning about a particular
language and culture over the course of the year.
The demo function of the app was used to demonstrate in more detail how the apps could be
explored by the children. Sites with interactive screens or projectors stated this was an advantage for
using the ELLA apps in group settings. The introductory song was also used as a group activity to help
familiarise the children with the ELLA trial.
30
Deloitte Access Economics
ELLA Trial Evaluation Final Report
This initial group session was also used to establish any ground rules about how the iPads and apps
would be used within the site. This included explanations of any roster or waiting list instructions,
how to find each child’s individual avatar (and that this should be done each time the iPads were
used), care of use guidelines and how the iPads would be set up.
While the ‘group introductory session’ approach was the most common, several sites stated they
placed the iPads into the room without a formal introduction. This allowed children to ‘find’ the
iPads, introducing the ELLA trial organically. Prior to the introduction of the iPads, a few services
populated their play space with words and cultural symbols from the target language.
Several sites stated that they introduced the ELLA trial within a broader unit on cultural awareness,
diversity and different ‘ways of being’. Introducing the concept of different languages and cultures
through examples of foreign language songs or videos was popular, as were examples of different
languages from bilingual children or parents. A number of sites also stated that they used a globe or
map to show the children the country or countries where their ELLA trial language was spoken and
discussed how far this was from Australia.
4.1.2
Integrating the iPads in the site
The way in which services physically integrated the iPads into the site varied. Educators typically
introduced the iPads in a manner which suited the circumstances of each service, often trying a
variety of options until settling on an approach that suited best. The Baseline Educator Survey found
that there were three predominant ways of positioning the iPads within the site:

Creating an ELLA corner or space in which the iPads could be accessed at any time.
•
It was noted that such areas were designated with consideration to the noise the iPads
could make and how this may impact the rest of the site.
•
Some services established ‘cultural corners’ which also included cultural play items (such
as costumes, pictures, books or food).

iPads were placed on tables for use at designated times.
Set up modifications over time
The educator surveys found that the majority of educators (approximately 60% of responses to both
the August and December surveys) had changed the way that the iPads were set up in the site since
the beginning of the trial. Reasoning for this change (noting different and sometimes opposite
approaches were undertaken in each site) included:

Offering children a variety of different ways and contexts through which to engage with the
iPads – maximising opportunities to play in a less structured format and ‘keep things fresh’.
•
Providing cushions, bean bags and mats to allow children to lie down while using the
apps. This was also suggested to encourage use of iPads in small groups rather than
individual use.
•
Broadening the area in which the iPads could be used to more evenly distribute noise,
providing space for children to use the iPads on their own or in a quiet place.
•
Broadening the times in which children could use the iPads in efforts to either (1)
increase use or (2) spread the use of the iPads more evenly over the course of the day.

Placing additional restrictions on iPad use in order to increase supervision and limit disruption
to the classroom.
31
Deloitte Access Economics
ELLA Trial Evaluation Final Report
•
•
•

Restricting the area in which the iPads could be used to increase supervision,
concentrate the noise and/or limit opportunity for inappropriate use or damage of the
iPads.
Restricting the number of iPads available at any one time to increase supervision
capabilities and limit opportunities for too much ‘chaos’.
Modifying the set-up of the room to make it easy to separate the engagement of
children eligible for the ELLA trial and those who are not.
Addition/removal of timers, rosters or waiting lists to ensure equitable use.
The site visits conducted in the concluding weeks of the trial found that as the year progressed, many
sites loosened their usage restrictions on the iPads. Educators stated that as the novelty associated
with the iPads lessened, the digital tools became a normal fixture in the room. Educators were able
to maintain adequate supervision of the apps through looser structures, resulting in less set ‘iPad
times’, less formal monitoring of session times (i.e. with timers) and less set ‘iPad spaces’. Children
drifted to the iPads when they wanted to use them, and engaged in the iPads in whatever format
they liked (sometimes solo, sometimes in groups, sometimes with headphones). One educator noted
that this approach was more consistent with the EYLF and allowing children the freedom to follow
their interests.
Controlling noise
A logistical issue that emerged when iPads were introduced to some preschool was noise. It was
found that when in use, the iPads had the potential to create considerable noise and disrupt other
activities, although the extent that this was a significant issue varied based on room layout. This
problem was stated to be compounded by the fact that the iPad cases supplied made it difficult to
adjust the volume of the iPads.
There were several methods employed by sites in order to minimise the impact of this issue,
including:

restricting the number of iPads in the room at any one time (i.e. a number of services allowed
a maximum of two iPads to be in use);

allowing iPads to be taken into different areas – which allowed children to find a quiet space
(even outside) in which they could hear the iPad more clearly;

creating a separate area for iPad use; and/or

purchasing and encouraging the use of headphones.
Of the services which made the decision to use headphones with the apps, there were mixed reviews
on this approach. A number of services stated that the use of headphones had influenced the ELLA
trial positively, resulting in the children being able to achieve a higher level of engagement with the
apps as they could hear more clearly while having minimal impact on social interactions. These sites
stated that the children still worked interactively with the apps, pointing out areas of interest and
helping each other. Conversely, a number of sites stated that they purchased headphones. Some
sites continued to use the apps only with headphones, while others did not persist with headphone
use. At least one site expressed a concern among educators that headphones reduced peer
interaction and also reduced interaction with educators.
32
Deloitte Access Economics
ELLA Trial Evaluation Final Report
4.2 Methods of trial delivery
The methods of trial delivery employed across sites were also diverse. This section discusses the
variety of ways in which educators engaged children with the ELLA apps.
4.2.1
Methods of engaging children
In order to allow educators flexibility in determination of the most appropriate way to implement the
trial in their site, minimal prescriptive guidance was given as to how the ELLA trial should be
delivered. As a result, educators engaged the children with the apps in a variety of ways. Aspects
which varied across sites included:

Exposure to the iPads: Some services made the iPads available only at prescribed time periods
in the day (such as before lunch, or after rest time) for an hour or two at a time. Other services
kept the iPads out at all times. Some services did not ever make the iPads available for general
use but instead accommodated children’s requests for use when asked. Similarly, some
services made the iPads available each day, while others would only introduce the iPads once
or twice a week.
As stated in Chapter 4.1, there was a general trend of increasing children’s exposure to the
iPads over time as the devices became a normal fixture of the room.

Time limits on app use: While time limits were often used, there were varying approaches to
this. In some sites, time limits appeared largely notional, while in others usage time was closely
monitored and limits enforced. The time limits referred to were usually between 5-20 minutes
per child per session. Educators reported that in most instances children were happy to share
the iPad with another child when their time was up. In services where no time limit was
imposed, educators reported encouraging other activities if they noticed a child had been
using the iPad for an extended period of time.
Educators were generally of a consensus that in the absence of a time limit or supervision,
there was a small sample of children that would continuously play with the iPad if allowed,
reducing their exposure to the rest of the preschool programme.
Again, strict time limits and supervision appeared to lessen over the course of the year once
the iPads became normalised in the preschool environment. Educators stated that over time
children matured in their use of the iPads and required less formal use arrangements.

Nature of integration with learning programme: A number of educators noted through
consultation that the flexibility afforded to educators in the nature and extent of activities they
were expected to deliver suited the preschool environment. In busy times, educators could
limit the ELLA trial to children using the iPads, and when they found time to facilitate a
supporting activity it was considered a bonus. However, this flexibility also resulted in a high
level of variation across sites as to the extent of engagement with the ELLA apps.
4.2.2
Educator guidance of app use
The majority of educators played an active role in supporting the trial’s use. 77% of responses to the
August Educator Survey stated that the educator interacted with the children while they were using
the ELLA apps (Chart 4.1 below). This proportion increased to 94% by the time of the December
Educator Survey, which may be attributed to the influence of messaging from the second educator
workshop and liaison site visits that educator guidance was important for supporting trial outcomes.
Consultation evidence supports this observation, with a number of educators stating during site visits
33
Deloitte Access Economics
ELLA Trial Evaluation Final Report
that they increased their level of interaction with children while they were using the apps following
advice received during the second workshop or a liaison visit and found this had a positive influence
on child engagement.
As shown in Chart 4.2, methods of interaction were typically stated to be (in order of popularity at
the time of the August Educator Survey) (1) using the app alongside the child; (2) guiding children’s
use and (3) answering the child’s questions – or a combination of these actions. Interestingly, in the
December Educator Survey educators responded that answering questions was a more regular
occurrence than guiding use. This suggests that the nature of interaction between educators and
children matured over the course of the year from a more instructional model at the beginning of the
trial, to responsive interactions in the later months. Additionally, a number of educators stated that
they would ask the children to repeat things they’d heard or to explain what they are doing or what
they understand, reinforcing the learning through questioning.
Chart 4.1: Do educators interact with the
children when they are using the apps?
Chart 4.2: How do educators interact with
children when they are using the apps?
50%
100%
August Survey
90%
December Survey
August Survey
December Survey
40%
80%
70%
30%
60%
50%
20%
40%
30%
10%
20%
10%
0%
0%
Yes
No
Unsure
Source: Educator Surveys – August (105 responses) and
December (78 responses)
Use the app alongside Guide children’s use
the child(ren)
Answer children’s
questions
Source: Educator Surveys – August (105 responses) and
December (78 responses)
Educator use of app data
While it was initially envisioned that the ELLA trial usage data may be drawn upon by educators to
help inform their method of trial delivery, this was not found to be occurring on a widespread basis.
In both the August and December Educator Surveys it was found that over 50% of educators used the
ELLA trial data less than once a month, or not at all. Indeed, the usage of the data appeared to
decrease slightly over the course of the year (Chart 4.3 below). These results were reinforced in site
visits.
34
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 4.3: Educator use of ELLA trial usage data
45%
August Survey
December Survey
40%
35%
30%
25%
20%
15%
10%
5%
0%
Regularly (at least once a
week)
Occasionally (at least
once a month)
Rarely (less than once a
month)
Never
Source: Educator Surveys – August (98 responses) and December (68 responses)
Of those educators that had accessed the app usage data, just over half of the respondents stated
that they had found the data helpful – primarily using it to inform child reports for parents and to
monitor usage. A number of educators stated that the data helped them to further understand
learning patterns of certain children (i.e. noting if children like spending times on mazes) and
allowing educators to adjust the learning programme to cater for this.
However, 47% of respondents stated that they had not found the data helpful. It was commonly
noted that the time commitment required by other preschool activities made it difficult for the data
to be accessed and analysed on a regular basis. It was suggested that as the data only showed use
and not ‘how well’ the child was progressing with the apps, it was not overly useful.
4.3 The use of complementary activities
There was a high degree of variability in the nature and extent of complementary activities delivered
by educators to support the ELLA trial. At the most intensive end, one site developed a book detailing
each child’s journey with the ELLA trial throughout the year, including photos of the child dressing in
traditional attire, cooking traditional cuisine, playing with the apps and recording words that were
learnt. This book was then distributed to parents so they could more fully understand how the ELLA
trial was influencing their child’s learning. In contrast, a number of sites were found to not have
facilitated any complementary activities over the year.
While site visits found that complementary activities were most often facilitated in an ad-hoc
manner, and sometimes not at all, the majority of educators (79% of respondents to the August
Educator Survey and 82% to the December) indicated that they had introduced at least one related
activity to complement children’s experiences of the apps. It was found that in most sites the
number of complementary activities fluctuated in accordance with other activities occurring in the
site, as the activities were generally conceptualised as ‘extras’ rather than a regular component of
the learning programme. For instance, in the final consultation with sites, many services noted that
they had not engaged in complementary activities in the later stages of Term Four as they had been
too busy.
35
Deloitte Access Economics
ELLA Trial Evaluation Final Report
As seen in Chart 4.4 below, the most popular complementary activities were centred on cultural
education, followed by singing songs in the target language and building play structures which
replicated activities in the ELLA apps. Site visits found that complementary activities which were
natural extensions of the existing learning programme (such as counting first in English, then the
target language, or repeating the colours in the target language during painting time) were more
likely to be implemented, while activities which were new to the site (such as building a rocket ship,
or cooking) were less frequently introduced.
Chart 4.4: Types of complementary activities which are being facilitated by educators
70%
August Survey
December Survey
60%
50%
40%
30%
20%
10%
0%
Cultural
education
Singing songs Building a play Telling stories Oral language Performances
structure
games
Source: Educator Surveys - August (98 responses) and December (71 responses) Other includes: Creation of games to
reinforce language learning; Art projects which reinforce cultural and language learning; Decorating the room with language
or cultural symbols; Cooking; Incorporating greetings (and less commonly other words) into everyday use; Using target
language when teaching related activities (such as numbers and colours); Integrating app games into play time, i.e. making
a real cake for an educator’s birthday or playing in the sandpit using the target language words; Creating simulated play
spaces such as a Japanese shop or restaurant; Reading books in the target language or about the target culture; Exposing
children to native speakers of the target language (parents/school language teachers/other educators); Building a cultural
garden; Playing sports associated with the target language; Skyping children in a similar centre overseas; Watching videos in
the target language or set in the relevant cultural setting.
It is also noted that a few of the trial sites complemented the ELLA trial with language instruction
from a qualified language teacher. In these sites, the sessions delivered by the language teacher
were typically distinct from the ELLA trial (i.e. did not incorporate use of the apps or the activities
within the apps into the lesson) but did provide opportunity for increased language exposure in a
different format.
Generally, it was found that the networking opportunities provided through the ELLA trial were
helpful for the participating educators in terms of stimulating ideas for complementary activities.
Some educators stated that they looked at the Facebook page for new ideas, though it was noted
that a higher level of activity on the page would have been appreciated. It was also found during
consultation that a number of sites had increased their undertaking of complementary activities
following the second workshop, after feeling inspired by stories emerging from other trial sites. This
is supported by the educator survey data, which shows an increase in the provision of
complementary activities across all types of activities from August to December (Chart 4.4).
36
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Educators who responded to the August and December educator surveys consistently stated that
they predominately orchestrated complementary activities on an ad hoc basis, in response to
children’s requests and interests or in response to a natural juncture into the existing learning
programme, as opposed to imposing a fixed activity schedule. However, a number of services did
state that they imposed a baseline target for a minimum amount of complementary activities – such
as at least three activities a term, or once a week.
For educators who stated that they did not engage in complementary activities (less than 15% of
responses), the cited reasons were:

the opportunity hasn’t arisen naturally in the curriculum;

there is not enough time for additional activities;

the educators do not feel confident enough in the language to help the children, and have no
access to native language speakers or ways in which to teach themselves; and

the educator doesn’t have as much knowledge of the ELLA trial than other educator(s) in the
service (i.e. those who attended the workshop) and as such doesn’t feel adequately qualified
to deliver complementary activities.
During site visits, some services stated they did not want to place too much emphasis on one
language or culture, as this could be insensitive, or not consistent with an ethos about appreciating
all cultures.
It was also noted in site visits that if more support was provided, in the form of additional resources
or more effective ways of communicating across sites to share ideas, educators would be more likely
to engage in supporting activities. Some examples of additional resources that would be helpful were
given as:

an educator app, to teach educators the language while they familiarise themselves with the
apps;

transcripts for the educators, so they can translate and pronounce the words;

learning materials (such as flash cards with the words on them);

suggested resources and websites educators could use to improve the learning experience for
children; and

better infrastructure to improve networking with other preschools (educators noted the
Facebook page existed, but commented it was not widely used).
4.4 Parent and guardian involvement
The nature of parent and guardian involvement with the ELLA trial varied across sites, but was
generally found to be of a relatively low level – with minimal interaction other than providing
consent for participation, varied levels of interest in the trial’s operation, requests for supporting
activities to use at home, and isolated incidents of parents helping to deliver complementary
activities.
4.4.1
Initial parent and guardian interactions with the ELLA trial
Parents and guardians were informed about the ELLA trial at the beginning of the year, and were
required to provide written consent if their child was to participate. The vast majority of respondents
to the August and December parent/guardian surveys reported that they were informed about the
37
Deloitte Access Economics
ELLA Trial Evaluation Final Report
ELLA trial through their preschool (90%), with the remainder stating they heard about the trial
through media (5%) or through their child (2%).
Educators reported that they were generally able to convey the expected benefits of the ELLA trial to
parents relatively easily. As noted above in Box 2, while a small portion of parents initially had
concerns with the ELLA trial – in most cases these concerns were dispelled over time.
However, some educators did state they would have appreciated additional ELLA material to help
them illustrate the benefits of digital technology and language learning in early childhood education.
It was also mentioned in site visits that the initial phases of the ELLA trial were implemented quite
quickly and the rushed nature of this made it difficult to engage with parents more meaningfully on
participation, trial design or the choice of language at each site.
Several services noted that they felt parent/guardian engagement would be an important
consideration for the future roll out of the ELLA programme – and could be supported with a higher
level of resources than what was provided in this trial. These services noted that they felt parents
were still not entirely clear on the primary objective of the trial and were confused about the
distinction between language exposure and language learning, which caused difficulties for managing
expectations.
4.4.2
Parent and guardian understanding and perceptions of the ELLA trial
There was a general consensus among ELLA trial sites that parents both understood and were
broadly positive about the ELLA trial. In the Baseline Educator Survey, educators were asked about
the initial understandings and perceptions of the ELLA trial with parents and guardians. As seen in
Chart 4.5 below, the feedback was quite positive, with over three-quarters of educators stating that
parents both understood and had positive views of the ELLA trial.
Chart 4.5: Educator perceptions parent/guardian understanding and view of the ELLA trial
Parent/guardians understand the
ELLA trial
80%
70%
Parents/guardians have a positive
view of the ELLA trial
60%
50%
40%
30%
20%
10%
0%
Strongly agree
Agree
Disagree
Neither agree nor
disagree
Source: Educator Baseline Survey, 40 responses
Following this, the August and December educator surveys asked educators to reflect on whether
parent understanding or positive views of the ELLA trial had increased over time. Chart 4.6 and Chart
4.7 show that most educators felt at least some parents had increased their understanding of the
38
Deloitte Access Economics
ELLA Trial Evaluation Final Report
ELLA trial since the trial had begun, and that views of the trial had become increasingly positive over
time.
Chart 4.6: Educator views on changes in
parent understanding of ELLA over time
Chart 4.7: Educator views on changes in
parent attitude towards ELLA over time
60%
60%
August Survey
50%
August Survey
December Survey
December Survey
50%
40%
40%
30%
30%
20%
20%
10%
10%
0%
Parent/guardians
Some
Parent/guardians
understand it parents/guardians do not understand
better
understand it
it better
better than others
Unsure
Source: Educator Surveys - August (98 responses) and
December (69 responses)
0%
Parents/guardians There is a mix of Parents/guardians
have a more
views
do not have a
positive view
more positive view
Unsure
Source: Educator Surveys - August (98 responses) and
December (69 responses)
Parent/guardian understanding of the ELLA trial was also tested in the August and December
parent/guardian surveys which asked respondents to nominate what they thought the ELLA trial
involved (% respondents) and which were the most important aspects. The list below includes the
proportion of respondents who ranked each item as important, and which elements of the ELLA trial
were ranked as most important, according to parents.
3.
Learning about another language (99% respondents) – Most important
4.
Learning about another culture (95% respondents)
5.
Using an iPad (84% respondents)
6.
Using apps (80% respondents)
7.
Playing games (80% respondents) – Least important.
It is noted that the responses to these questions were almost identical between the August and
December parent/guardian surveys, indicating that understandings of the nature of ELLA trial did not
change over the course of the year.
Service engagement with parents and guardians
Reflecting the diversity in approaches to the ELLA trial across services, survey responses and site
visits indicated that there was also a high level of variation in the approach to parent/guardian
interaction adopted across the trial sites. Some services held information nights and encouraged
parents to sit with their children and use the apps. Other services used the ELLA trial as a mechanism
through which to increase engagement with multiculturalism within their area and invited bilingual
parents to share their skills or held cultural days. Conversely, some services did not report any
engagement with parents/guardians.
Two services consulted with during site visits reported that parents and guardians who spoke the
language being trialled were interested in being involved in the learning process. They occasionally
came into class and helped the children with their pronunciation, and attempted to teach them some
39
Deloitte Access Economics
ELLA Trial Evaluation Final Report
new words. The educators at these services felt this support was unquestionably beneficial for the
children and their language outcomes. It was noted by a number of services during the final round of
consultation that parent interest in the ELLA trial grew over the course of the year as they began to
observe their children using words from the target language at home.
Educators also commonly reported that a number of parents requested extra material to help the
child continue their language learning in the home. In response to this, a number of educators
developed word lists or flash cards that parents could use at home. Some parents even reported
finding language classes for their child so they could continue their learning following the completion
of the ELLA trial, demonstrating that the ELLA trial has the potential to raise community engagement
in language learning.
Approximately three-quarters of educator respondents to the August Educator Survey stated that
they were anticipating further engagement with parents/guardians about the ELLA trial over the
remainder of the year. However, less than 50% of respondents to the December Educator Survey
stated that they had indeed engaged further with parents/guardians. This was supported by
consultation findings, which found limited engagement with parents/guardians across most sites.
General comments from educators who responded to the educator surveys relating to
parent/guardian engagement included:

It was often noted that allowing the parents to trial the app in demo mode or sit down with
their child and watch them using the app was an effective means of engaging
parents/guardians and increasing their understanding as to what the ELLA trial involved.

A number of centres reported a lack of interest from parents and guardians about the ELLA
trial, stating that they offered interaction opportunities but these were not taken up.
•
Preschool programs operating within a LDC service noted particular difficulty with
engaging parents/guardians as their contact with parents was often limited due to early
drop off and later pick up times.
•
It was noted that parent interest in the ELLA trial generally correlated with parent
interest in their child’s preschool experience overall – with engaged parents asking more
questions about the ELLA trial.

Some parents expressed high expectations regarding the ELLA trial and actively sought
opportunities through which they could support their child’s language development. There
was some reporting of confusion regarding the nature of the ELLA trial – with these parents
expecting the language learning to be a more structured learning programme.
•
Many parents/guardians asked to access the app at home.

It was noted by several services that parent/guardian engagement and interest with the ELLA
trial heightened as the children started to speak the app language at home. As the language
skills increased and the potential benefits of the trial become clearer, parents become
increasingly interested in the trial.

Several services also stated that parent management committees were interested in the trial,
and the implications of delivering the ELLA trial in the service.

Two educators responding to the December Educator Survey stated that their service was
culturally diverse and they were wary of appearing they were promoting one culture over
another, so did not actively strive to engage parents in the ELLA trial – focussing instead on
engaging parents/guardians in other areas, such as literacy and numeracy.
40
Deloitte Access Economics
ELLA Trial Evaluation Final Report
It was also noted in consultations that the discussions on multiculturalism and inclusion stimulated
by the ELLA trial helped to promote a more welcoming environment for multicultural families
attending the service.
4.4.3
Ongoing parent and guardian involvement with the trial
Approximately half the services consulted with during the site visits stated that they kept their
parents/guardians regularly updated with the trial, either through newsletters or various forms of
digital media, such as Kinderloo. The other half of services did not attempt to keep families informed
(often stating this was because families did not demonstrate a high level of interest when the trial
was introduced).
Parents and guardians were also kept informed about the ELLA trial through their children, with 74%
of respondents to the August and December parent/guardian surveys stating that their children had
spoken about the ELLA apps at home. It was reported that children talked about the games they
were playing, the words they were using and expressing interest in other languages, cultures and
visiting other places. It was also reported that children informed their parents when new apps were
available and expressed wishes to download the apps on home devices. Only several parents
commented on the level of discussion of the apps over time and these noted that their child had
mentioned the ELLA apps more frequently in the latter half of the year.
Additionally, 79% of respondents to the August and December parent/guardian surveys stated that
their children had spoken to them about using iPads at their preschool. These discussions were
usually based on the excitement and enjoyment associated with being able to use the iPads, limits on
iPad use, the content of new apps and learning with others. In the December Parent/Guardian
Survey, a number of parents noted that their children’s confidence using iPads, and willingness to
talk about the iPad use, increased over the course of the year.
Reflecting on a combination of both these factors, 68% of respondents to the August
Parent/Guardian Survey, and 79% to the December Parent/Guardian Survey, stated that they had
received enough information about the ELLA trial to date. 19% in the August Parent/Guardian
Survey, and 13% in the December Parent/Guardian Survey, stated that they had not received
sufficient information. Those who would like further information expressed a desire for:

Notifications when new apps are released and what theme they are based on.

Further resources (or access to the apps if possible) so they could support the learning at
home;
•
Requests included further information on the content of the apps and what the children
are learning, such as word lists and cultural topics covered.
•
Methods and strategies for the optimal parent involvement at home to support
language learning.

More progress updates on their child’s learning, including outcome targets and milestones.

More about the ELLA trial generally, including:
•
More information on the aims, structure and evaluation methods underpinning the trial;
•
How much time children are spending on the ELLA trial each day; and
•
What the ELLA trial will mean for their child’s language learning after the preschool year
is complete, including whether the trial will be available in later years.

An ability to try the apps before the child.
41
Deloitte Access Economics
ELLA Trial Evaluation Final Report
4.4.4
Development of an app for families
One the most commonly reported requests from parents, as evidenced by consultations with
educators and responses from the parent/guardian surveys, was an ability to use the apps at home.
There were various contended benefits associated with this, including:

An ability for parents to increase their understanding of the ELLA trial and the quality of the
apps, reducing the likelihood for any parent opposition to the trial stemming from
misunderstanding;

An ability for parents to engage with their children’s language learning journey;
•
It is noted that while preschool educators will only be engaged with an individual child
over a one year period, cultivating parent engagement with their child’s language
learning will have more significant impacts over time. As such, greater parent
involvement with the ELLA trial may further enhance the benefits of the trial through
encouraging continued parent appreciation of, and investment in, language learning.
•
Additionally, several services noted that the ability for children to use the ELLA trial with
their parents in the home may be an effective way of promoting parent interest in their
children’s learning more generally. It was noted that the play-based nature of the apps
may be in some cases more accessible for families than traditional learning modes, such
as reading story books, and provide an avenue for shared learning experiences that may
not otherwise take place.

Increased exposure to the ELLA trial for children, particularly children who attend the service
less frequently than other children. It would also allow the apps to be used after the
completion of preschool, to some extent lessening the issues with continuity of language
exposure inherent in the current ELLA trial format.
However, despite these noted benefits, there was still a level of reservation in a number of educators
(approximately one quarter of sites consulted with) about allowing the ELLA apps to be downloaded
at home. These services noted that educators are skilled in encouraging children to engage with the
apps in a manner which supports positive learning, and that misuse or overuse of the apps in the
home may reduce the educational value of the apps.
42
Deloitte Access Economics
ELLA Trial Evaluation Final Report
5 Trial engagement
This chapter provides an overview of the available evidence on the appropriateness and effectiveness
of children’s engagement with the ELLA trial.
The key evaluation questions this chapter seeks to answer are:

Do children use the programme?
•
To what extent are children engaged with the apps?
•
What are the patterns of usage?
•
What factors contribute to children using the apps more or less frequently?
The evidence supporting the evaluation of this element of the ELLA trial has been drawn from the
ELLA trial data, the survey responses and site visit findings.
Although children have not been directly consulted during the evaluation, the detailed app data
reveals significant evidence on children’s usage of the apps. Information from surveys and site visits
also provides qualitative evidence from educators, directors and parents/guardians on the level of
child engagement with the trial.
43
Deloitte Access Economics
ELLA Trial Evaluation Final Report
General findings on ELLA trial engagement

72% of children use the apps at least once a week. Nearly all children spend between five and
ten minutes with the apps in once session.

App usage (as measured by minutes per session) remained consistent throughout the trial.

The number of sessions per child declined in the final term of the year. However, this is most
likely due to children having less access to iPads (as term four is typically busy with other
activities), rather than a drop-off in interest.

Not all children engaged equally with the apps. There are some children that spent a relatively
short period of time with the apps, while others use it more regularly, and for longer periods.
10% of users account for 33% of total app use.

75% educators report children use the iPads in groups of two or more.

Each app was most popular when it was first released. Usage would increase sharply after app
release, before plateauing gradually over time.

While all apps received significant attention when they were first released, App 3 appears to
have been the most popular among the children throughout the trial.

Cake (from App 3) and Lift operator and Pizza café (from app 7) appear to have been the most
popular activities.

App 2 and App 7 were the most successful at exposing children to the target language, with
over 30% of time spent with these apps spent listening to the target language.

The main factor driving differences in use among children was service type – generally,
children attending long day care centres used the apps more regularly. This was to be
expected, as children attending long day care centres are typically there for more days a week
than children attending other types of preschool services.
5.1 Children’s overall usage and participation
The extent to which children use the ELLA apps and participate in the trial more broadly are key
indicators of their engagement with the trial.
5.1.1
Children’s usage of the ELLA apps
Information on children’s use of the ELLA apps was primarily sourced from the app data. The analysis
presented in this report relates to all seven apps, with data included up to the week starting 28
December 2015. The release schedule for these apps is included in Table 5.1. In total, 41 weeks of
app data has been analysed.
For the purpose of the evaluation, the trial was assumed to commence on 8 March 2015. While
usage actually commenced two weeks earlier, very few children had begun using the apps at that
time. As almost 50% of children were active by 8 March 2015, this was used as the starting point for
analysis and is referred to as week 1 throughout the report.
44
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Table 5.1: Release cycle for ELLA apps
App
Release cycle
App 1 – Polyglots in the Playroom
Released 22 February 2015
App 2 – Polyglots at the Beach
Released 13 April 2015 - week 6 for purpose of the evaluation (week
15 of the calendar year)
App 3 – Polyglots at the Birthday Party
Released 9 June 2015 - week 14 for purpose of the evaluation (week
23 of the calendar year)
App 4 – Polyglots at the Zoo
Released 6 July - week 18 for purpose of the evaluation (week 27 of
the calendar year)
App 5 – Polyglots at the Circus
Released 10 August – week 23 for the purpose of the evaluation
(week 32 of the calendar year)
App 6 – Polyglots in the Park
Released 28 September – week 30 for the purpose of the evaluation
(week 39 of the calendar year)
App 7 – Polyglots in the Town
Released 2 November – week 35 for the purpose of the evaluation
(week 44 of the calendar year)
There were 1,771 children that completed the ELLA trial – that is, of the 1,868 children involved in
the trial, 1,771 children were still eligible to use the apps as at week 41. Only these children that
completed the trial have been included in the ELLA app data analysis presented throughout this
chapter.
Over the trial, there have been around 235,000 sessions, an average of five sessions per child per
week (after controlling for school holidays) – see Table 5.2 below. A session is defined as
commencing when a user (child) logs into an app, and continues until another user logs in, or the
iPad is idle for at least one minute.10 It is important to note that each time a user switches between
apps, this was recorded in the app data as separate sessions. This means a child may have used the
iPad for a longer amount of time than indicated by the session time (i.e. they may have used several
apps consecutively in the one ‘iPad session’).
During these sessions, there was an accumulated 27,661 hours of app usage, which equates to an
average of nearly 28.5 minutes per child every week. On average, each child spent 7.1 minutes using
an app every time they logged on.
10
Sessions lasting less than 30 seconds were excluded from reporting, as it was assumed these short sessions were
occurring only when a child selected the wrong user.
45
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Table 5.2: Overall usage statistics, by child
As at week 41
Total users
Number of sessions
Sessions per child per week
Total hours
1,771
234,956
4.0
27,661
Average minutes per child per week
28.5
Minutes per session (average session length)
7.1
Number of users above average minutes
418 (24%)
Sessions lasting at least 5 minutes
116,491 (50%)
Sessions lasting at least 10 minutes
55,945 (24%)
Source: Deloitte Access Economics analysis of ELLA app data
Note: Total users was not used as the denominator in the per child analysis. A weighted average of eligible users per week
was used, as this better captured when children were late in joining the trial, or if a service had no usage due to school
holidays. This is explained further in Box 4.
Several other assumptions underlie this data analysis, as outlined in Box 4.
Box 4: Major assumptions underpinning the app data analysis

Active user – An active user was defined as a user who had completed their consent form (and was
therefore eligible to participate in the trial), and their service was not on school holidays or
encountering technical difficulties preventing use of the iPad.

Sessions or minutes per user – where practical, this analysis presents figures on a per active user (or
child) basis. This per active user measurement controls for preschools that may have experienced
delays in implementing the trial, and controls for school holidays. For example, total usage appears to
decrease by almost 40% between weeks 16 and 17 of the trial. However, this can be explained by
nearly half the children involved the in the trial going on school holidays. If only the children who
attended preschool in week 17 are considered to be active users in this week, the total minutes per
user were unchanged, and is more reflective of child use of the apps in that week.

Children not selecting their avatar – while educators encouraged children to log in and out using
their own avatar, it was reported that children did not always remember to do so. If children use the
iPad without logging the previous user out, this will overstate the length of each session, but
understate the number of sessions. This issue cannot be controlled for in the analysis.

Children joining the preschool midtrial – it is assumed that a child only became eligible to join the
trial the week they first used the iPad. For example, if a child first used the apps in week 10, it was
assumed they only gained permission to participate in the trial in week 10, and they were assumed to
be ‘inactive’ in weeks one to nine. There is a risk this will understate the number of children eligible
to use the iPad, as a child may have been eligible but never logged in; however, reports from
consultations suggest educators encouraged all eligible children to use the iPads at least once.

Children leaving the preschool midtrial – throughout the trial, 97 children were removed from the
trial, and these children’s usage statistics have been removed from the sample. The analysis therefore
only pertains to children who completed the trial.
46
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Most sessions (54%) were between two and ten minutes long, as shown in Chart 5.1.11 There were a
relatively small proportion of sessions lasting over 15 minutes (11%). During the second round of
consultations, educators often suggested they had relaxed enforcement of ‘time limits’ yet the app
data revealed slightly fewer sessions that last longer than 15 minutes, compared to the mid-way
point of the trial.12 Therefore, educator observations that children’s ability to self-regulate their use
of the iPads increased over the course of the year is supported by trial data.
Chart 5.1: App usage – average minutes per session, proportion of total sessions
30%
As at week 22
25%
As at week 41
Sessions
20%
15%
10%
5%
0%
0 to 1
1 to 2
2 to 5 5 to 10 10 to 15 15 to 20
Minutes per session
20+
Source: Deloitte Access Economics analysis of ELLA app data
Analysis of session length conducted on a per child basis shows that the majority of children (1,502,
or 85%) spent, on average, between five and ten minutes per session with an app (Chart 5.2).
Chart 5.2: App usage – average minutes per session, by child
Number of children
1,600
As at week 22
As at week 41
1,200
800
400
0
0 to 2
2 to 5
5 to 10
Minutes per session
10 to 15
15+
Source: Deloitte Access Economics analysis of ELLA app data
11
The national physical activity guidelines for screen time suggest children under the age of five should spend no more than
one hour per day with electronic media. For a link to the guidelines, see here.
12
It should also be noted that the time limit imposed by educators may affect the interpretation of app data results, in
terms of usage providing an accurate reflection of the extent to which children want to use the apps and therefore their
level of engagement.
47
Deloitte Access Economics
ELLA Trial Evaluation Final Report
5.1.2
Child engagement over time
Educators noted that engagement had generally been consistent over the duration of the trial, with
interest levels peaking with the release of each new app. Educators also commented that children’s
use of the iPads became calmer as the trial progressed. They reported that the novelty factor
associated with iPad use had lessened, so most children were no longer rushing to play with the
iPads each morning. Educators felt this was overall a positive for the children, as interest levels were
still high and users were able to calmly interact with the apps without pressure from other children.
The ELLA app data supports this viewpoint, as the proportion of ‘short’ sessions (sessions lasting less
than two minutes), have declined as the trial has progressed (Chart 5.3). This suggests children are
less likely to rush through the activities, and more likely to spend time with the apps, potentially
increasing their exposure to the target language.
Chart 5.3: App usage – proportion of sessions lasting less than two minutes
40%
30%
20%
10%
0%
1
3
5
7
9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
Week
Source: Deloitte Access Economics analysis of ELLA app data
Educator views on child engagement are largely supported though the app data, which reveals child
usage patterns were largely consistent throughout the trial – usage would increase with the release
of a new app, and then slowly decline. Chart 5.4 demonstrates that session length increased with the
release of a new app, and slowly diminished over time. While this pattern is also true of the number
of sessions, there was a decline in the number of sessions per user towards the end of the trial.
The reduced number of sessions towards the end of the trial may be due to Term Four being
particularly busy with other activities (as reported during the second consultation), rather than an
indication of reduced interest.
48
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 5.4: App usage over time
12
Session per user
Minutes per session
9
6
3
0
1
6
App 2
released
11
16
21
App 3
released
26
Week
App 4 released
31
App 6
released
36
41
App 7
released
App 5 released
Source: Deloitte Access Economics analysis of ELLA app data
The dotted lines represent averages across the trial.
The patterns of usage following release suggest that interest in the apps was maintained quite
consistently throughout the year, and the process of incremental release contributed to this.
Further evidence of the continued high levels of child engagement can be elicited from the educator
survey responses, with 82% of educators reported children were ‘very engaged’ with the apps by the
conclusion of the trial, a marginal increase from the previous survey (Chart 5.5). 86% of educators
reported that child engagement had either increased or remained unchanged.
Chart 5.5: Educator perspectives on child engagement with the ELLA apps
70%
August survey
December survey
60%
50%
40%
30%
20%
10%
0%
Very engaged
(consistently)
Very engaged
(inconsistently)
Somewhat engaged
(consistently)
Somewhat engaged
(inconsistently)
Source: Educator Surveys – August (101 responses) and December (81 responses)
49
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Variation in usage between children
5.1.3
During site visits, it was commonly stated that the children enjoyed using the ELLA apps. There was
however, a range of usage among children. Most children were reported to have used the iPads
regularly throughout the trial, although a minority were reported to rarely use the iPads.
Additionally, educators noted a spectrum of interest among those who used the apps regularly, with
some children preferring to play one game before moving on and others who prefer to use the iPad
for extended periods of time.
This usage distribution is captured through the app data, where it is clear a disproportionate share of
total usage can be attributed to the most regular users of the apps. Chart 5.6 shows that if all
children are ranked by their total usage (in minutes), the top 10% of users were significantly
overrepresented in the total usage statistics, accounting for 33% of total time with the apps. By way
of contrast, the lowest 50% of users account for only 21% of total time with the apps highlighting a
possible risk that this inequality of use will result in different language outcomes across children.
Chapter 6 includes a more detailed discussion on the link between iPad use and possible language
outcomes.
Percentage of use each decile accounts for
Chart 5.6: Distribution of child app usage by decile
40%
As at week 22
As at week 41
30%
20%
10%
0%
1
2
3
4
5
6
7
8
9
10
Increasing use of the apps with users grouped in deciles, by total minutes of use
Source: Deloitte Access Economics analysis of ELLA app data
Despite the uneveness of use, it is still the case that the majority of children (72%) logged on at least
once a week (Chart 5.7). The proportion of children that logged on less than once a week increased
to 28% at the end of the year (up from 20% in mid-year). Educators reported managing the trial in a
less structured manner as the trial progressed, allowing children to self-regulate their use, which may
explain this slight increase in children logging on less frequently. Over the trial, very few children
50
Deloitte Access Economics
ELLA Trial Evaluation Final Report
logged in more than four days a week, but as the number of days each child involved in the trial
attends preschool is not known, it is difficult to gauge the significance of this result.13
Chart 5.7: Frequency of app usage
80%
Proportion of children (as at week 22)
Proportion of children (as at week 41)
Proportion of children
60%
40%
20%
0%
Four or more days Two to four days
a week
per week
Once or twice a
week
Less than once a
week
Source: Deloitte Access Economics analysis of ELLA app data
5.1.4
Group usage of the apps
During site visits, educators noted that some children enjoyed using the apps by themselves,
whereas other children enjoy using the apps while interacting with other users around them. During
these interactions, children were observed to compare activities and exchange ideas on how to
perform the task on their screen. Some children were reported to prefer sharing the same iPad
among two or more children, but in these instances, it was reported that children mostly wanted to
be the ‘driver’ controlling the iPad.
This variation in the number of children using one iPad was confirmed by the educator survey
responses, as shown in Chart 5.8 with children reported by educators as using the iPads individually,
in pairs and in small groups. Educator survey responses suggest there was a slight move away from
using the iPads individually, as children became more likely to use the iPads in pairs.
13
For example, it is likely children attending long day care centres attend for more days a week, compared to children
attending stand-alone preschools, increasing the frequency with which they would access the iPads and use the apps.
Further investigation is undertaken later in this chapter.
51
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 5.8: Average number of children using each iPad
45%
August Survey
40%
December Survey
35%
30%
25%
20%
15%
10%
5%
0%
1
2
3
4
more than 4
Source: Educator Surveys – August (101 responses) and December (71 responses)
It is important to note that due to group usage of the iPads, the app data understates the level of
child engagement with and exposure to the apps. This is because only one child can be logged into
an iPad at a time, but there may be one or more other children alongside the ‘driver’ who were
gaining exposure to the language.
Group usage of the apps, compared to individual usage, also has implications for the number of iPads
that may be required by a service, outlined in Box 5 below.
Box 5: Number of iPads
In both the Baseline Educator Survey of trial sites and the August Educator Survey, just over half of the
respondents (68% and 57% respectively) indicated that the service had enough iPads to sufficiently cater for
the ELLA trial. This proportion had increased to 70% by the December Educator Survey. Some reasons as to
why services wanted more iPads are cited below:

Children typically want to be in control of the iPad, and are not satisfied when they can only
participate as an observer

Children generally want to play on the iPads at the same time, rather than spread out equally over
the course of the day.
Most educators noted that in limiting the number of iPads available, children were required to gain a range
of valuable skills, including collaborative learning, patience and turn taking.
5.1.5
Language exposure
Each app comprises between four and six activities, and within each activity, children are exposed to
different words in the target language. The ratio of time spent listening to language, relative to total
time spent with an app, is an important measure, as it outlines the degree to which children were
exposed to the target language.
Over the duration of the trial, approximately 27% of all time spent with the apps had been spent
listening to the target language. The proportion of language exposure was inconsistent over time,
52
Deloitte Access Economics
ELLA Trial Evaluation Final Report
making it difficult to validate or repudiate educators’ observations that children’s engagement with
the listening activities in the apps increased towards the end of the year (Chart 5.9). This issue is
further explored in Chapter 5.2, as the language exposure for each app may vary.
Chart 5.9: Language exposure over time
35%
30%
25%
20%
15%
Language exposure
10%
5%
0%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
Week
Source: Deloitte Access Economics analysis of ELLA app data
Note: the black dotted line represents the average throughout the entire trial
5.2 Children’s usage of individual apps
Further insight on children’s engagement can be gathered from analysis of patterns of usage on an
individual app basis.
5.2.1
App use over time
It was mentioned in site visits that app popularity varies over time – with apps being most popular in
the period immediately following their release, and interest diminishing over time. Educators
consulted during the site visits also stated that while children are likely to use the most recent app,
they will occasionally return to older apps. Both these observations are affirmed by Chart 5.10, which
demonstrates that the app children log into the most, as measured by the average number of
sessions per user, was always the most recently released app.
53
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 5.10: Average sessions per user over time, by app
Sessions per week per user
6
App 1
App 2
App 3
App 4
App 5
App 6
App 7
Average
4
2
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
Week
Source: Deloitte Access Economics analysis of ELLA app data
Each app kept children engaged for longest when it was first released, suggesting there is an initial
burst of enthusiasm associated with the release of a new app (Chart 5.11). Following this initial spike
in usage, the length of each session across each app was fairly consistent throughout the remainder
of the trial.
Chart 5.11: Average minutes per session over time, by app
15
Minutes per session
12
App 1
App 3
App 5
App 7
App 2
App 4
App 6
Average
9
6
3
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
Week
Source: Deloitte Access Economics analysis of ELLA app data
5.2.2
App popularity
Throughout the site visits, no single app emerged as the most universally popular. Each app was
nominated at least once as the children’s favourite app at different trial sites.14 However, the app
data can be analysed to understand whether some apps are in fact more popular than others.
When comparing apps, it is important to recognise that each app has been available for a different
length of time and, as shown above, the longer an app has been released, the less it is used by
children. Comparing each app by usage over the duration of the trial will either:
14
Note that, during the period when some of the second site visits occurred, app 7 was yet to be released.
54
Deloitte Access Economics
ELLA Trial Evaluation Final Report

overstate the popularity of older apps if total time is used, as these apps have been available
to use over a longer time span; or

understate the popularity of the older apps if average time is used, as popularity naturally
diminishes with time.
It is therefore important to compare between apps across a comparable timeframe. Analysis has
been undertaken using two complementary time periods:

the first four weeks after each app was released, indicating initial popularity levels; and

the final four weeks of the trial to date, to determine the popularity after each app has settled
into a consistent pattern of use.
Finally, app popularity is measured using two methods:

Number of sessions – this reflects the attraction of an app, as a large volume of sessions
suggests children enjoyed the app sufficiently to return to it next time they logged in.

Minutes per session – this reflects the level of attention a child gives an app after logging in. A
long session suggests the app holds children’s attention for a longer period of time, implying
children are engaged while using the app.
Examining the first four weeks of each app, App 1 appears to have been the most popular app based
on the average number of sessions per user per week (Chart 5.12). However, taking into account the
average length of each session, it becomes apparent App 1 was used for the least amount of time per
session. This result is most likely attributable to the initial unfamiliarity of the ELLA trial to children
(discussed above), potentially resulting in children frequently logging in and out as they familiarised
themselves with using the app. This may explain the elevated average number of sessions and the
relatively lower average session length over the first four weeks after the release of App 1.
Chart 5.12: Summary of app popularity
25
App 1
App 2
App 3
App 5
App 6
App 7
App 4
20
15
10
5
0
Number of sessions per
user
Minutes per session
Number of sessions per
user
First four weeks (post app release)
Minutes per session
Final four weeks
Source: Source: Deloitte Access Economics analysis of ELLA app data
NOTE: App7 only became widely in use during week 36 of the trial (out of 41 weeks). For the analysis of the most recent
four weeks, rather than include data from weeks 38-41 (as this would include data from the initial spike in use), only data
from weeks 40 and 41 is included.
Discounting App 1, App 3 appears to be (marginally) the app most children logged into when it was
first released (12 sessions per user); it also kept children engaged, with just less than 10 minutes per
session over the initial four week post-release. App 7 could potentially also be considered the most
popular app, with users averaging 10.2 minutes for every session. The sessions per user measure
55
Deloitte Access Economics
ELLA Trial Evaluation Final Report
suggests it has been unpopular, but this could be attributable to general reduced activity with the
iPads in Term Four (as mentioned above), rather than a reflection of App 7’s popularity.
Turning attention to the final four weeks, only App 7 was used with any regularity, presumably as it
was the most recently released app, while App 3 still receives slightly more attention than other
apps. The length of each session across most of the other apps seems fairly consistent as well,
indicating there is not a clearly preferred app. Two notable exceptions are App 7, (which is likely to
be experiencing additional attention as it is still the most recently released app), and App 6, which
appears to be marginally less popular and engaging than the other apps.
To reinforce the relative popularity of each app, Chart 5.13 shows the cumulative hours of use, by
app. This confirms the popularity of App 3, as it is the most used app despite being available for 13
weeks less than Apps 1 and 2.
Chart 5.13: Cumulative app use, by app
6,000
Hours of use
5,000
4,000
App 1
App 3
App 5
App 7
App 2
App 4
App 6
3,000
2,000
1,000
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
Week
Source: Deloitte Access Economics analysis of ELLA app data
5.2.3
Most popular activities within the apps
During the second round of site visits, educators reiterated that the interactive activities (those
encouraging movement or voice responses) were generally stated to be among the children’s
favourite activities in the apps. During previous consultations, several educators noted children
preferred the apps with a specific task to complete (such as Cake), rather than the apps which have
no conclusion (such as colouring-in activities, or Sandpit).
Examining app usage data for the first four weeks post release of each app reveals the relative
popularity of different activities – see Chart 5.14. Note that for the purpose of this analysis, a
sub-session refers to the number of times a user selects to enter each activity. Within a single
session, a user often changes between activities, so there are multiple sub-sessions within each
session.
56
Deloitte Access Economics
Deloitte Access Economics
App 1
App 2
App 3
App 4
App 5
App 6
Supermarket
Pizzacafe
Lift operator
Library
Building site
Warmup
Simon says
Racetreasure
Obstacle maze
Movie builder
Cultural video
Memory cards
Flippart
First aid
Face painter
Sub-sessions per user (left hand axis)
20
2
10
1
0
0
Minutes per session
30
Clown trainer
Picnic
Juicebar
Fruit sorter
Food puzzle
Feeding time
Songtree
Hide & seek
Egg & spoon
Cake
Balloons
Submarine
Shell
Feedfish
Colour & create
Sandpit
Rocket
Playmat
Phone
Maze
Number of sub-sessions
ELLA Trial Evaluation Final Report
Chart 5.14: Activity use by app, during first four weeks
3
Minutes per sub-session (right hand axis)
App 7
Source: Deloitte Access Economics analysis of ELLA app data
57
Deloitte Access Economics
App 1
App 2
Source: Deloitte Access Economics analysis of ELLA app data
App 3
App 4
App 5
App 6
Supermarket
Pizzacafe
Lift operator
Library
Building site
Warmup
Simon says
Racetreasure
Obstacle maze
Movie builder
Cultural video
Memory cards
Flippart
First aid
Face painter
Clown trainer
Picnic
Juicebar
Fruit sorter
50%
Food puzzle
60%
Feeding time
Songtree
Hide & seek
Egg & spoon
Cake
Balloons
Submarine
Shell
Feedfish
Colour & create
Sandpit
Rocket
Playmat
Phone
Maze
ELLA Trial Evaluation Final Report
Chart 5.15: Activity popularity (by app), over time
Share of total app use (first four weeks)
Share of total app use (whole trial)
40%
30%
20%
10%
0%
App 7
58
ELLA Trial Evaluation Final Report
As can be seen in Chart 5.14, the most popular activity by a significant margin, (as measured by the
number of sub-sessions per user) was Cake (from App 3), while Lift Operator and Pizza Cafe (from
App 7) have the longest lasting sessions (2.7 and 2.6 minutes per session respectively).
The relative popularity of each activity, within each app, was broadly unchanged over time (Chart
5.15). The song, which accompanies each app, was the least popular activity across all the apps. At
the initial release of App 1, it appeared as if there was some interest in listening to the song, with
usage across the first week representing almost 5% of total use of App 1. However, over the duration
of the trial, and across all the apps, usage of the song had decreased to negligible levels, suggesting
children were skipping the song when they logged into the apps.
An interesting lesson to emerge from Chart 5.14 is that, on average, there were noticeably fewer
sub-sessions per user in the later apps, but the length of each session increased. This finding lends
additional weight to the hypothesis that children became more familiar with the apps as the trial
progressed, and therefore became more sophisticated in their usage. For example, rather than
jumping haphazardly between activities, they appeared to choose their preferred activity, and use it
for an extended period of time.
5.2.4
Language exposure by app
Language exposure varied more significantly by app, rather than over time. Apps 2 and 7 were the
most successful at exposing children to the target language, with children spending 32% of time
listening to the target language (Chart 5.16). By comparison, only 16% of time on App 1 was spent
listening to the target language.
Chart 5.16: Proportion of language exposure, by app
90%
Language
80%
Non-language
70%
60%
50%
40%
30%
20%
10%
0%
App 1
App 2
App 3
App 4
App 5
App 6
App 7
Overall
Source: Deloitte Access Economics analysis of ELLA app data
Examining usage over time by app (Chart 5.17), there is little evidence to support some educators’
observations that children are ‘playing games’ without listening to the target language, or that
children spent more time listening to the target language as they became more familiar with the
apps. Language exposure by app appeared to be largely consistent throughout the trial. It should be
noted this does not preclude educators’ observations that children increased concentration levels
59
Deloitte Access Economics
ELLA Trial Evaluation Final Report
while using the app – it may be that children used the apps in a similar manner throughout the trial
(therefore meaning usage data remained unchanged), but began paying more attention to the target
language when it was spoken.
Chart 5.17 : Proportion of language exposure over time, by app
50%
45%
40%
App 1
App 2
App 3
App 5
App 6
App 7
App 4
35%
30%
25%
20%
15%
10%
5%
0%
1
3
5
7
9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
Week
Source: Deloitte Access Economics analysis of ELLA app data
5.2.5
Language exposure by activities
Examining app data by activity, Colour & Create, Shell, (from App 2), Feeding Time (from App 4) and
Building Site (from App 7), were the most effective in exposing children to the target language, in
that the largest proportion of time with these activities was spent listening to the target language
(Chart 5.18). In the case of Colour & Create and Building Site these apps are also highly popular,
suggesting these activities were more likely to expose children to the target language. On the other
hand, Feeding Time, was a relatively less popular activity, and therefore less likely than other, more
popular activities, at successfully exposing children to the target language.
60
Deloitte Access Economics
100
Total use (hours per week) (left hand axis)
100%
80
Language exposure (right hand axis)
Average language exposure
App 1
Deloitte Access Economics
App 2
App 3
App 4
App 5
App 6
80%
60
60%
40
40%
20
20%
0
0%
Proportion of time spent listening to target language
Maze
Phone
Playmat
Playspace
Rocket
Sandpit
Colour & create
Feedfish
Playspace
Shell
Submarine
Balloons
Cake
Egg & spoon
Hide & seek
Playspace
Songtree
Feeding time
Food puzzle
Fruit sorter
Juicebar
Picnic
Playspace
Clown trainer
Face painter
Firstaid
Flippart
Memory cards
Playspace
Cultural video
Movie builder
Obstacle maze
Playspace
Race treasure
Simon says
Warmup
Building site
Library
Lift operator
Pizza cafe
Playspace
Supermarket
Hours per week
ELLA Trial Evaluation Final Report
Chart 5.18: Total use (hours) and language exposure (as a proportion of total use), by activity
App 7
Source: Deloitte Access Economics analysis of ELLA app data
61
ELLA Trial Evaluation Final Report
Presenting this information in a different format, Chart 5.19 demonstrates the effectiveness of each
activity at exposing children to the target language by considering their:

popularity (measured in average minutes per user); and

degree of language exposure (measured as the proportion of time spent listening to the target
language, as opposed to playing games).
Activities located in the top right quadrant can be considered both popular and effective at delivering
the target language. Notably, relatively popular activities (such as Cake and Colour & Create) also
demonstrated above average effectiveness at language exposure (these are located in the top-right
quadrant of Chart 5.19). Other activities, such as Memory Cards and Feeding Time (in the top left
quadrant), were relatively unpopular, despite their effectiveness at exposing children to the target
language. There were also a large amount of activities in the bottom-left quadrant, which are both
unpopular, and did not expose children to language as effectively.
Degree of language exposure (proportion of time
spent listening to target language)
Chart 5.19: Most effective activities for language exposure
60%
Memory cards
Colour & create
Feeding time
Building site
50%
Juicebar
40%
Cake
30%
Maze
20%
Submarine
10%
Warm up
Sandpit
0%
0
2
4
6
8
10
12
14
Increasing popularity (minutes per user)
Source: Deloitte Access Economics analysis of ELLA app data
Note: An activity in App 6 – Cultural Video was designed to focus on cultural, rather than language, exposure and has
therefore been excluded from this analysis.
5.3 Factors contributing to children using the apps
To shed further light on factors driving the patterns of usage outlined in this chapter, it is useful to
consider other evaluation evidence sources.
There are a range of factors that may be influencing how regularly children use the apps. During site
visits, educators were generally reluctant to make any definitive statements concerning factors
62
Deloitte Access Economics
ELLA Trial Evaluation Final Report
driving differentiation in app usage, with most putting difference in use down to ‘personal
preference’.
There are several other factors which could potentially be driving app usage:

language – different languages may resonate differently with children;

socioeconomic status15 – children of different socioeconomic backgrounds may have different
home environments, including level of parental engagement in their education, and therefore
may respond differently to the trial;

location – location may affect services’ ability to access the internet, especially for regional
and remote services;

provider type – different providers may employ different methods in engaging children,
including the opportunity to use the apps more frequently (e.g. children generally attend LDC
services more days a week than other provider types);

did children have previous language experience at the centre? – this examines whether or
not child familiarity with languages other than English changes their behaviour with the apps;
and

educator comfort with iPads as an educational tool – an educator’s comfort with the tool is
expected to influence how it is integrated into the classroom, and therefore will influence
usage patterns.
A cross-tabulation method was used to attempt to identify the key factors driving use (for this
exercise, use was estimated by the average number of sessions per user per week). It was identified
that children who attended LDC centres were higher weekly users than other children. This was to be
expected however, as children enrolled in LDC centres typically attend for longer hours and more
days in a week, and therefore have more opportunities to access the iPads. In light of this finding, all
other potential factors were considered after separating out long day care providers, and all other
providers. Results are shown in Table 5.3 below:
Table 5.3: Sessions per user per week, by various explanatory factors
Long day care
Non-long day care
Overall
Language
Arabic
French
Indonesian
Japanese
Mandarin
3.15
3.65
6.17
6.50
6.04
2.72
2.85
2.69
2.69
3.13
2.88
3.08
3.91
3.88
3.97
Site SES status
High
Medium
Low
5.64
5.09
4.75
2.22
3.29
2.89
3.07
4.00
3.44
15
The socioeconomic status of each child is unknown. Rather, the socioeconomic status of the suburb the preschool is
located in has been applied to all children attending the same preschool.
63
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Long day care
Non-long day care
Overall
Location
Metropolitan
Other2
5.13
5.16
2.81
2.82
3.41
3.57
Educator comfort with iPads
Very comfortable
Somewhat comfortable
5.32
4.32
2.94
2.68
3.95
2.95
Previous language experience
Yes
No
5.76
4.45
2.68
2.88
4.20
3.26
Source: DAE analysis of ELLA app usage data
2
Given the small number of remote services, these were combined with regional services.
As seen in Table 5.3 above, users in the left hand column (children from LDC centres) are always
more regular users than their counterparts in non-LDC centres, aligning with the cross-tabulation
outlined above.
Preschools which reported to have educators who are ‘very comfortable’ with iPads as an
educational tool appears to be the only factor which possibly explains differences in usage among
children. Both within LDC and non LDC centres, children attending preschools where educators were
‘very comfortable’ with iPads as an educational tool demonstrated more frequent usage. This is
possibly a reflection that educators who were more comfortable allowed children increased access to
the iPads. It may also be the case these children enjoyed using the apps to a greater level as
educators were able to provide a highly engaging experience.
Focussing on different languages, there is little evidence to suggest one language resonates more
with children than any other. However, the most popular languages within LDC centres (Indonesian
and Japanese) are the least popular within non-LDC centres. This suggests there are other factors
driving high language usage, rather than the popularity of any language. It should also be noted that
as each preschool was only given one language, there may be some other, unobservable differences
between preschools driving this result. To thoroughly test different levels of popularity between
languages, each child would need access to each language.
There appears to be little variation in usage by socioeconomic status – this is likely due to the fact
that nearly all children quickly learnt how to use the iPads and apps at their preschool. This means
that the possible effect of a lack of exposure to digital technology among lower SES children was
likely made redundant by the speed with which all children could use the iPads effectively.
Services in different locations exhibited, on average, similar usage habits, although the limited
number of remote services made it difficult to test whether or not rurality influences usage.
Finally, it was not universally found that children who had previous language experience used the
iPads more frequently. While this was the case within LDC centres, it was not true within other
preschools.
It should be noted the findings articulated above are not definitive, as it could be the case that there
are unobserved, underlying factors which influenced the results. For example, different usage
patterns between preschools may be driven by different rules relating to iPad use. Further, much of
64
Deloitte Access Economics
ELLA Trial Evaluation Final Report
the data is analysed at the preschool level, however it may be the case the children within the same
preschool have different experiences during the trial. There are often different groups within
preschools, and several different educators within each group (each of which may employ different
trial delivery methods). The analysis should be interpreted in light of these caveats.
65
Deloitte Access Economics
ELLA Trial Evaluation Final Report
6 Trial impact and efficiency
This chapter provides an overview of the available evidence on the impact of the ELLA trial on child,
educator and service outcomes across trial sites. The efficiency of the ELLA trial, that is the
relationship between the impact of the trial and the benefits invested in it, is also evaluated.
The key evaluation questions this chapter seeks to answer are:

Does the ELLA trial make a difference to children’s outcomes?
•
What outcomes for children have been observed to date?

What impact has the ELLA trial had on educators and services involved?

Has the ELLA trial been cost-efficient?
•
Are there any ways in which the ELLA programme could increase its efficiency?
The evidence supporting the evaluation of this element of the ELLA trial has been drawn from the
survey responses, ELLA trial data and the site visit findings.
66
Deloitte Access Economics
ELLA Trial Evaluation Final Report
General findings on the impact of the ELLA trial

Educators and parents/guardians commonly reported that at least some children were
producing words and phrases from the ELLA language and using them in context – though this
varied significantly by child.

It was found that over time, children used a higher number of words and became more
comfortable speaking in the target language. It was generally understood by educators that
comprehension levels were higher than speaking – that is, a larger number of children
understood words from the language even if they were not observed to be using the words.

There were some reports of the ELLA trial inspiring increased interest in the culture associated
with the ELLA apps, or multiculturalism more broadly, but this was not consistent across sites.

The ELLA trial was associated with the development of a range of skills for participating
children, including: caring for property, sharing, digital literacy, collaborative learning and
social skills.

There were also several casual links observed by educators between the ELLA trial and
improved English skills for children, and general development for autistic children. It is
expected that targeted research would need to be undertaken to further explore this link.

Educators, parents and guardians were largely positive about the trial, and most expressed a
desire for the trial to continue in the future.

Educators, on average, felt more confident about using technology or integrating language
learning in their learning programme as a result of the ELLA trial.

Despite the positivity, there are some lingering concerns about the impact of the ELLA trial,
including a lack of continuity of language learning for the children once they leave preschool
and variations in the level of educator support for the ELLA trial.

Because some children use the apps considerably more than others, not all children are
receiving the same level of exposure to the apps or the language(s).

Given that a significant proportion of the cost associated with the ELLA trial has been invested
in the development stages, and the marginal cost of distributing the programme to additional
participants is low, the ELLA programme will become increasingly cost-efficient as participation
grows.
6.1 Child outcomes
The outcomes arising from children participating in the ELLA trial can be grouped into three broad
categories:

language outcomes (the amount of language exposure and production being observed in
children participating in the trial);

cultural awareness (children’s understanding of the culture associated with the target
language of their app); and

any other skills associated with the trial.
This section discusses these outcomes as observed by analysis of app data, observations from
educators and observations from parents and guardians.
67
Deloitte Access Economics
ELLA Trial Evaluation Final Report
The focus of the ELLA trial was on benefits associated with language exposure, rather than language
learning. However, in the broader context of the ELLA programme objectives, including increased
participation in language learning among secondary school students, and the realisation of benefits
associated with language learning, it is pertinent to look at the impact of the ELLA programme on
language production to the extent that this was observed. The relationship between the ELLA trial
and the realisation of benefits, and the ability of the evaluation to assess this, is described in more
detail in Box 6.
Box 6: The link between language exposure, language learning and benefits
Within the spectrum of language learning, there is a hierarchy of outcomes – beginning with exposure and
leading towards fluent communications. Even if language production does not occur, there are benefits
associated with language exposure exclusively.
The benefits to language exposure at an early age include an increased propensity to successfully produce
language later in life, increased awareness of other languages and cultures and more advanced cognitive
capabilities. These benefits are described in greater detail in the literature review presented in Appendix C.
To the extent that a number of these outcomes are immediately observable in children, the evaluation has
been able to make an assessment as to the ELLA trial’s effectiveness. However, many of the benefits associated
with language exposure will only become apparent over a longer time frame.
While there is a theoretic link between language exposure in early years and increased language learning later
in life, there are a range of factors that may impact on the achievement of this ultimate outcome, including a
scarcity of language programmes for children immediately after preschool and the inconsistency of connected
language pathways between preschools, primary schools and secondary schools.
Deloitte Access Economics, based on Krashen and Terrell (1983)
In light of the above, the assessment of the ELLA trial’s effectiveness within this evaluation has been primarily
focused on the programme’s ability to expose children to languages and to look at early signs of effective
exposure, including educator observations of language production. This approach is appropriate given the
overarching objective of the ELLA trial is to increase the number of Australian children studying languages later
in life.
68
Deloitte Access Economics
ELLA Trial Evaluation Final Report
6.1.1
Educator observations
6.1.1.1
Educator observations on language outcomes
Educator observations, as collected through the three educator surveys, display a wide range of
variety in estimates of the proportion of children who are using the app language. Less than 10% of
educators observed ‘all’ children speaking the language (though this increased over time), with
responses fairly evenly split between ‘Most’, ‘Half’ and ‘Few’ (averaging 31%, 28% and 30% across
the three educator surveys respectively). By the time of the December survey, no educators stated
that there had been no children observed to be using the language.
Chart 6.1: Educator perspective on the number of children using the app language, as observed by
themselves or parents
35%
Baseline Survey
August Survey
December Survey
30%
25%
20%
15%
10%
5%
0%
All
Most
Half
Few
None
Source: Educator Surveys - Baseline (39 responses), August (109 responses), December (69 responses)
Similarly, the survey findings relating to the proportion of educators that felt the impacts of the ELLA
trial increased over the course of the trial were also mixed (Chart 6.2). Aside from an
acknowledgement that children were learning more words from the apps as the year progressed,
there was less evidence that the use of language, and the level of language production, among
children was also increasing over time. Similarly, just over half of educators responding to the
surveys noted an increase in interest in children’s understanding of culture or demonstration of a
continued interest in other languages and cultures over the course of the year.
69
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 6.2: Educator views on how ELLA trial impacts changed over time
100%
August Survey
90%
December Survey
80%
70%
60%
50%
40%
30%
20%
10%
0%
Children learning new Use of the language Children showing
Children learning
Children
words from the apps
among children
more interest in and more when the app is
demonstrating
increasing
understanding of combined with other continued interest in
other culture
learning activities other languages and
cultures
Source: Educator Surveys - August (98 responses), December (69 responses). Note, the category ‘children demonstrating
continued interest in other languages and culture’ was only included in the December survey.
Site visits supported the finding that there was a range of reported outcomes associated with
language exposure and production. A number of educators were highly positive about the language
production they had observed over the course of the year, stating that some children were using
words and phrases from the apps in real life contexts, unprompted. The most commonly stated uses
of the languages, as observed by educators or parents, included singing the app songs, counting,
using greetings or naming colours and foods. At the other end of the spectrum, there were a small
number of educators who believed participating children were not speaking the target language at
all, indicating that some children, while receiving language exposure, were not yet at the language
production stage.
The most common observation was that children were using and understanding the language while
engaging with the apps, but did not tend to use language outside of the apps unless prompted by
educators. In the final round of consultations, approximately one quarter of sites stated that they
doubted the participating children would use the words from the apps in context unless prompted.
This further emphasises that language outcomes related to emergent speech and fluency are not
direct outcomes of the ELLA trial and would be reliant on future language learning.
There was, however, evidence to suggest that over time children increased both their knowledge of
language and the ability to use the language they had learned in context. The mid-year consultation
and surveying period found that educators and parents were observing children using words from
the ELLA apps, particularly singing the songs and using the greetings. For instance, the following
statements were typical of those made by educators in the August Educator Survey:

‘They use the words incidentally during the day. When we were focusing on colour they would
use the Indonesian words when selecting crayons. They sometimes count in Indonesian….
Recently they have been telling us that they are hungry in Indonesian.’

‘Out of the two groups of 20 children in our room I have not heard any of them using words
from the app outside of the context of actually playing the app. I've heard my colleagues
reporting that a few of their children use some words occasionally. I hear children repeating
'sounds' from the app (in the context of playing the app), but I don't believe they have a
70
Deloitte Access Economics
ELLA Trial Evaluation Final Report
thorough understanding of the fact that those sounds are actually words that translate to a
corresponding word in English, therefore they are not able to meaningfully use them in any
other context.’

‘They love saying hello and goodbye in the language. We have heard most children using it’
In contrast, the consultation and survey period conducted at the conclusion of the trial found more
constant observations of children using the language unprompted, and/or in the correct context.
Educators noted in consultations that children’s use of language increased as the year progressed
and they became more comfortable using the language. Accordingly, survey commentary
surrounding language observation in the December Educator Survey included:

‘We regularly hear children singing songs form the apps such as the heads-shoulders song,
colours, the greetings – they say the fruit names and when using the apps you can hear them
repeating the names. Parents say they hear them at home using the language.’

‘There is a wide variation on the language used, with a few children using a lot and most of
them using a little. Most of their usage revolved around colours and numbers, although there
are some children who also use other words that they have particularly become familiar with.

‘Almost daily we will hear someone using some Japanese, away from iPad use. Single words
such as colours and numbers are mixed with English sentences. Short phrases are used in play –
answering the telephone, and at meal times, “I like”.

‘I have not noticed the children using the language in their own play, but when questioned or
encouraged they can repeat the phrases/words they have learned.’

‘We use the language daily but only at times that are meaningful, e.g. greetings and farewells;
colours; numbers and counting; foods (mostly ones that the children are familiar with eating
themselves).’
It was also stated by most educators that comprehension levels were felt to be quite high, even for
children who were not observed speaking words in the target language. For instance, some
educators would use activities to gauge understanding levels, such as asking everyone wearing a red
top (with ‘red’ spoken in the target language) to stand up and noted that no members of the class
needed further instruction.
6.1.1.2
Factors influencing language learning
The second round of site visits were used to explore in greater depth, factors which educators felt
might influence the rate or extent to which participating children were learning the target language –
noting that it was universally reported that there were differing levels of language ability across the
classroom. This question garnered mixed responses from services, with no definitive factors
emerging as influential. Common responses included:

Usage: while usage was typically noted as having some impact on the rate of language
learning, educators most often felt that usage was not the dominant factor.
•
Consistency of use (rather than quantum of use) was forwarded by one centre as being
influential. As an LDC service, it had noted that children who came every day of the
week, and were therefore more highly exposed to the apps, picked up more words in
the target language than children who attended the service only one or two days a
week.
•
In consultation with a formal language teacher at one of the trial sites, the teacher
noted that the ability for children to engage with the language on a daily basis (as
71
Deloitte Access Economics
ELLA Trial Evaluation Final Report
•
opposed to their interaction with them once a week) had reinforced basic words to a
greater extent than what they had observed in years prior to the ELLA trial.
However, it was also stated that children who were not high users (or even users of the
apps at all) had also been observed singing the songs from the apps and using words.
This suggests that constant exposure to the ELLA trial over time, including interactions
with children who were engaged with the apps, had allowed them to learn elements of
the language without being directly involved.

Confidence: it was contended that more confident children (in terms of communication most
particularly) were observed using the language more often as they were comfortable
experimenting and using new words.

Technology lovers: educators also noted that children learn in a variety of different ways and
for some children, iPads and use of digital technology were found to be a highly effective
medium. These children, who were confident and naturally curious about the apps, tended to
learn language at a faster rate than those who were less digitally literate.

Bilingual: interestingly, while services were explicitly asked if bilingual children engaged with
the apps any differently than monolingual children in the first round of site visits and it was
determined there was no material difference – in the second round of site visits, several
services noted that children who already spoke another language were picking up the ELLA
language more quickly than other children.

Educator engagement: It was also noted that the extent of educator engagement was an
influential factor in the rate of language learning. Educators noted that when they (1) sat with
children and asked questions about the app while they were using it; and/or (2) incorporated
words from the apps in everyday use in the room; this had the effect of increasing language
learning. An example of this, as found in a response to the August Educator Survey, is provided
below:
‘The children in my room were not using the language at all by approximately half way through
the year. It wasn't until explicit teaching took place and the educators began really teaching
the language to the children that they actually began responding. We used the fruit names for
morning tea. And incorporated other French words and sayings into our transitional routines.’
6.1.1.3
Educator observations on cultural awareness
As with language knowledge, the level of observed cultural awareness and the degree to which the
ELLA trial facilitated an increased focus on cultural discussions, varied across sites. The majority of
educators in both the Baseline Educator Survey and the August Educator Survey (74% and 79%
respectively) stated that children were at least ‘somewhat interested’ in the culture associated with
the language of the app being trialled at their preschool, as shown in Chart 6.3. This proportion
increased to 88% in the December Educator Survey, potentially reflecting an increased focus on
cultural awareness that may have been encouraged during the second training workshop or through
service liaison visits.
72
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 6.3: Educator observations on children’s level of interest in the culture associated with the
language of the app being trialled
70%
Baseline Survey
August Survey
December Survey
60%
50%
40%
30%
20%
10%
0%
Very interested
Somewhat interested Not very interested
Unsure
Source: Educator Surveys – Baseline (38 responses), August (96 responses), December (38 responses)
Site visits determined that it was common for services to incorporate the cultural aspects of the ELLA
trial into a broader discussion on multiculturalism which often existed prior to ELLA. This made it
difficult to attribute cultural learnings directly to the influence of the ELLA trial. It was also found that
there was sensitivity around placing a singular focus on the culture portrayed in the app, at the
expense of other cultures prevalent in the preschool.
However, a number of observations recorded in the educator surveys pointed to the fact that
children’s cultural awareness had increased due to activities associated with the ELLA trial. These
outcomes relate to both an increased knowledge of the places from which the target language
originates, and an acceptance of cultural diversity more broadly. For instance, this included:

‘The increase in general knowledge about Indonesia. I love it when they show me a picture of
something they have seen in an app and then in a book. Of course we have spent time talking
about it, e.g. the Garuda Pancasila and completing art and craft activities, but I think it is so
wonderful that they remember.’

‘Cultural confidence. Our students are aware of their cultures and are proud to share stories or
words from their language or ask about another child's culture. We feel this year that there are
more mixed social groupings.’

‘They seem to be more aware of other children's differing cultural backgrounds and the
children who speak another language were regarded as 'special' as they could already do what
the others were just learning to.’
Of the sites that had incorporated a focus on the specific culture attached to the apps, it was
reported in site visits that the children had generally responded positively. Children displayed an
interest in the country of origin, and some cultural elements that differ from what they are familiar
with. Parent observations relating to children’s increased interest in different cultural aspects are
discussed in detail below.
73
Deloitte Access Economics
ELLA Trial Evaluation Final Report
6.1.1.4
Other learning outcomes associated with the ELLA trial
In addition to language and cultural outcomes, a host of other outcomes attributable to the ELLA trial
have been identified. Educators contended (through both site visits and educator survey responses)
that the ELLA trial encouraged development of a range of cognitive and social skills to varying
degrees in the participating children, such as:

caring for property;

understanding of other cultures in general (rather than just the culture exposed to through
app use);

understanding of language (including English) as a form of communication;

understanding of different languages and where they are on maps;

digital literacy;

developing areas targeted in the apps (such as counting or colours);

persistence and focus to complete activities;

sharing and collaborative learning;

turn-taking;

increasing confidence and pride for multicultural and/or bilingual children;

understandings of time and rostering (through use of a time minute limit and/or waitlist);

social skills – particularly for developmentally delayed or autistic children;

self-regulation and self-autonomy;

improving understanding of English;

technology skills (particularly for those who don’t have access at home); and

concentration and maturity.
It was noted by educators in educator survey responses that these skills were continually developed
over the course of the trial, improving with time. Educators did contend, however, that while the
listed benefits are attributable to the trial, it is likely that many of these skills would have been
developed through other means in the absence of the ELLA trial.
There were also some trial outcomes that educators were more confident attributing directly to the
ELLA trial than others. For instance, the positive impacts on self-regulation, sharing and collaborative
learning and care for property were consistently reported confidently across services. The impacts on
understandings of English, and the positive social development for autistic children16, however, were
reported with a higher level of uncertainty as to whether the ELLA trial had been the driving factor in
this changed behaviour. For instance, numerous educators noted that ‘the level of literacy has been
higher in the room this year’ or ‘the children using the apps more often have displayed large
increases in their English skills’, but then caveated this with an observation that this outcome may
have been a coincidence and not explicitly linked to participation in the ELLA trial.
16
A brief overview of the current evidence supporting increased learning outcomes for autistic children through the use of
digital technology is provided in the literature review in Appendix C.
74
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Data analysis
6.1.2
This section draws on the ELLA trial data, specifically the progress through apps, to make an
assessment of child outcomes.
6.1.2.1
Using levelling data as a proxy for language outcomes
There are several activities within the ELLA apps containing logic which change the difficulty of the
activity in response to the user’s performance. Such activities are said to contain ‘levelling’
information – and a user’s progression through these levels indicates their ability to accurately
complete the tasks within each activity.
Within Apps 6 and 7, three activities have been designed so progression through the levels can
possibly be interpreted as a demonstration of language learning. In theory, the successful completion
of tasks is contingent on a user understanding the meaning of particular words in the target language
(although in practise players may be able to advance between levels due to their persistence, rather
than knowledge). These three activities are:

Race treasure

Pizza café

Supermarket.
It therefore may be possible to infer language outcomes, or language ability, based on performance
throughout these ‘levelling’ activities. While this is not an exact measure for ability as other factors
likely feed into a child’s ability to advance to further levels, for the remainder of this analysis, it is
assumed that, broadly, users who progress to higher levels have demonstrated a stronger
comprehension of the target language.
6.1.2.2
Performance of children based on levelling activities
It is firstly worth noting these activities had varying levels of popularity within their respective apps.
Race treasure is close to the most popular activity in App 6, Pizza café is the most popular in App 717,
while Supermarket is one of the least popular activities. Notwithstanding the small sample size, it
does not appear children’s behaviour changed when faced with levelling activities – that is, the
levelling aspect of an activity did not appear to influence whether or not children played an activity,
as shown in Chart 6.4.
17
It should be acknowledged that Pizza café also has a ‘free play’ section, so it is possible its popularity is driven by this,
rather than the fact children enjoy the ‘levelling’ element of the activity.
75
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 6.4: Proportion of time spent on each activity, within Apps 6 and 7
40%
30%
20%
10%
App 6
Supermarket
Pizzacafe
Lift operator
Library
Building site
Warmup
Simon says
Race treasure
Obstacle maze
Cultural video
Movie builder
0%
App 7
Source: ELLA app data
The progress children were able to make within each levelling activity is demonstrated in Chart 6.5
below. Only a small fraction of children were able to progress to the final level of each activity. Less
than 40% of children successfully completed level 2 of Race treasure, suggesting this was the most
difficult of the levelling activities. The number of users advancing through each level on Pizza café
slowly declined as the levels got harder, although, once past level 4, 35% of users managed to reach
the final level. This pattern is largely true for Supermarket as well – there are fewer and fewer users
able to advance to each level, although 44% who reach level 7 are able to reach level 10.
Chart 6.5: Proportion of children reaching each level, by activity
120%
Race treasure
100%
Supermarket
80%
Pizza cafe
60%
40%
20%
0%
1
2
3
4
5
6
7
8
9
10
Level
Source: Deloitte Access Economics analysis of ELLA app data
Note: the highest level for Race treasure was level 6, the highest level for Supermarket was level 10, and the highest level
for Pizza café was level 9.
76
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Overall, a total of 229 children reached the highest level in at least one of the levelling activities, and
22 managed to reach the highest level across all three activities.
In order to summarise their performance using the levelling activities, each child was assigned a
‘levelling score’. A child’s levelling score was the sum of the highest level reached across the three
activities (with scores from Pizza café and Race treasure scaled up to a maximum of 10). The
maximum score was therefore 30. The distribution of scores is shown in Chart 6.6.
However, it was considered that simply summing up the highest level each user reached is unlikely to
best represent demonstrated knowledge, as it fails to consider how many attempts each child took
to progress. For example, it is more likely that a child who progresses between levels on their first
attempt has a better understanding of the language, as opposed to a child who takes 10 attempts.
Therefore, an additional weighting was added to the ‘levelling score,’ reducing the score of users
who required many attempts to advance between each level. The greater the number of attempts,
the larger the reduction that was applied. This is henceforth referred to as the ‘weighted levelling
score’.
Chart 6.6: Distribution of levelling scores
600
Levelling score
Number of children
500
Weighted levelling score
400
300
200
100
0
0 to 5
5 to 10
10 to 15
15 to 20
20 to 25
25 to 30
Levelling score
Source: ELLA app data
Using either measure, a majority of children had scores between 0 and 10 – in other words, they
were able to reach (on average) level 3 in each activity. Given the number of potential levels, this
indicates that children were generally unable to progress far through the levelling activities. Despite
what most educators reported, the data appears to suggest that only a minority of children were
able to fully absorb the language and demonstrate this learning by progressing through the various
levels of those activities.
Several attempts were made to determine what factors influenced a child’s levelling score. However,
it was found that the most dominant factor (by a significant margin) was the number of attempts at
each activity – in other words, the higher the amount of times a child attempted each activity, the
more likely they were to progress further. This could be interpreted that children who were most
engaged with the trial learnt the most, and were therefore most keen to participate in the levelling
77
Deloitte Access Economics
ELLA Trial Evaluation Final Report
activities; alternatively it could be the case that these activities reward persistence rather than
knowledge, and children were only able to progress after making multiple attempts, not because
they were able to understand the language. The analysis was unable to reveal which of these
alternatives was more likely.
6.1.3
Parent/guardian observations on ELLA trial outcomes
6.1.3.1
Parent/guardian observations on the impacts of the ELLA trial
There were many reports, received through both the site visits and parent/guardian surveys, that
parents and guardians had observed their children speaking the app language at home. Most
typically these observations were related to the use of greetings, counting, colours or singing the
introductory song. There were even several reports of some children knowing words in the app
language (such as colours or numbers) but not in English.
This was reaffirmed in the parent/guardian surveys in which over 70% of respondents stated that
they had observed their child using words from the language outside of preschool (Chart 6.7 below).
The proportion of parents who observed their child using words from the language increased from
73% to 78% over the second half of the ELLA trial.
Chart 6.7: Parent/guardian observations of child use of words from the ELLA language at home
100%
90%
August Survey
80%
December Survey
70%
60%
50%
40%
30%
20%
10%
0%
Yes, have observed language
being used
No, have not observed
language being used
Unsure if lanugage has been
used
Source: August Parent Guardian Survey (252 responses), December Parent Guardian Survey (199 responses)
Specific observations relating to the language outcomes of children participating in the ELLA trial, as
observed by parents and guardians, included:

Children using the words learnt through the app in context. Many parents noted the pride that
children felt when showcasing their language skills. The most commonly reported words used
outside the home were numbers, colours and greetings.
•
‘We travelled overseas and my child was doing cannon-balls into the swimming pool
counting in Japanese to 10 before jumping in the pool. The Japanese tourists were highly
excited that he could do that at 4 years of age.’
78
Deloitte Access Economics
ELLA Trial Evaluation Final Report
•
•
•
‘I took him to a Lebanese restaurant, and he ordered his drink in Arabic (like the
character does in the app, I think), and he used Thank you and Bye. And the waiter said
his pronunciation was very good. Amazing.’
‘My daughter tells me the Arabic names for food I’m cooking with (eggs, milk, butter)’.
‘Sometimes [she] will deliberately slot in a Japanese word when she is discussing a topic.
She is always very proud to make the connection.’

Children teaching parents/guardians and siblings the words.
•
‘My child is speaking Japanese at home and teaching me words. My child is drawing
pictures of things he knows in Japanese (eg. Numbers, colours, fruit) and writing the
Japanese word next to them.’
•
‘My child counts to 10 in Indonesian and so does his younger sister, and she doesn’t even
attend preschool’.
•
‘He sings the songs frequently. Says hello and goodbye to people in Japanese. Uses words
for like and dislike daily at the dinner table. Plays schools with his older sister and he
plays the teacher instead of the student which has encouraged her to learn a few basic
words also.’
•
‘[My child] comes home keen to apply new words that he has learnt at school in the
home context. He is even teaching his 2 year old sister to count in French. He often
describes things in French’.

A small proportion of parent/guardians noted that their child’s use of language had increased
progressively over the course of the year.
•
‘She quite often comes out with a French word or song from the app, more so as the year
has gone on’.
•
‘In the beginning of the year we rarely heard him use any Indonesian words. He
increasingly uses Indonesian. He will tell us the colours and body parts. He tried to teach
me how to say hello and good morning but became frustrated with me because I wasn’t
pronouncing them properly.’

One parent/guardian who had prior knowledge of the language being learnt stated that their
child’s pronunciation had improved.

Several parents/guardians noted that while they could observe their child speaking the
language, they could not tell if the child was using the words in context or with the right
pronunciation as the adult didn’t speak the language themselves.
Those who stated that the children had not learnt any language generally stated they had not talked
about the ELLA trial at all with their children or that the child enjoyed playing on the iPads but
seemed to like the games more than the language learning.
6.1.3.2
Parent/guardian observations relating to cultural awareness outcomes
Reflecting the findings from educators, an interest in the culture associated with the language of the
ELLA apps was observed less frequently by parents and guardians than the use of words from the
language. However, a larger proportion of parents observed their children displaying interest in the
culture associated with the ELLA apps at home in December (49%) than in August (37%). This
suggests that children’s interest levels in the culture associated with the apps may have increased
over the course of the year as children became more familiar with the ELLA apps.
79
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 6.8: Parent/guardian observations of child showing interest in culture associated with the
ELLA app at home
60%
August Survey
December Survey
50%
40%
30%
20%
10%
0%
Yes, have observed interest
No, have not observed
interest
Unsure of interest levels
Source: August Parent Guardian Survey (252 responses), December Parent Guardian Survey (199 responses)
Specific observations in relation to cultural awareness outcomes of children participating in the ELLA
trial, as observed by parents and guardians, included:

An interest in places, food and other cultural objects associated with the language being
explored in the ELLA app.
•
‘He gets all excited when he hears the word Japanese, anywhere!! If any of his brothers
come across anything Japanese they go straight to [my child] and tell him, and always
greatly appreciated by [my child], lots of smiling and nodding. We like to get books from
the town library about Japan. We practice using chop sticks.’
•
‘She notices when Indonesia is mentioned on TV or out in the world, and points things
out to us that remind her of things relating to Indonesia which she has learned about at
school’.
•
‘She is keen to go to Japan and loves sushi and Japanese food, which we make at home
sometimes, as well as chicken teriyaki. She has also asked me quite a lot about the war
with Japan. As Darwin was bombed by the Japanese, we have had discussions about why
our countries were enemies and how they are now friends and allies, for instance.’
•
‘He wanted have long shirt (Arabic traditional costume) on Eid. He bought one and he
was so happy.’

An awareness of different cultures and ways of being.
•
‘My son is always asking questions about people who he notices are different to him’.
•
‘Talks about how far other countries are away from us and has his own globe in his
bedroom now that he is always looking at’.
•
‘He is now more observant of accents and the implication of accents, i.e. that the
individual must speak another language’.
80
Deloitte Access Economics
ELLA Trial Evaluation Final Report
6.1.3.3
Observations relating to other child outcomes
In addition to observed language use and cultural outcomes, digital literacy was the next most
commonly discussed outcome of the ELLA trial by parents and guardians. The results of the
parent/guardian surveys suggested that child use of tablets outside of preschool has increased over
the course of the year, as shown in Chart 6.9 and Chart 6.10. Many parents/guardians stated that
their children had become confident using iPads and touch screens, able to navigate through
different apps with ease.
Chart 6.9: Child use of tablet technology
within the home
Chart 6.10: Child use of tablet technology this
year as opposed to previous use
50%
40%
August Survey
December August
Current tablet use
35%
Tablet use last year
30%
25%
40%
30%
20%
20%
15%
10%
10%
5%
0%
0%
Frequently
(weekly)
Most Days
Occasionally
(monthly)
Rarely
Source: Weighted average of the Parent/Guardian
Surveys - August (231 responses) & December (176
responses)
Never
No change
More often
Less often
Unsure
Source: Weighted average of the Parent/Guardian
Surveys - August (231 responses) & December (176
responses)
And finally, while social outcomes such as turn taking and care for property were not reflected in
parent and guardian observations, a number of parents/guardians discussed the impact of the ELLA
trial on their child’s confidence. It was remarked in several survey responses that the child’s ability to
showcase their language skills or cultural knowledge, had increased their confidence in conversing
with others. It was suggested that this was driven by holding a skill which others did not, or through
the ability to evoke positive reactions in native speakers of the language.
It was also observed in several parent/guardian observations that children expressed heightened
interest in other languages and cultures. This supports educator observations that bilingual children
gained pride for their own language abilities over the course of the trial. For instance, one parent
observed that ‘She likes to name things in Japanese when she knows them. She’s also more interested
in other languages spoken fluently in our family (French and German) which previously we’ve
struggled to pique her interest in’.
6.1.3.4
Overall parent/guardian perception of ELLA trial
Generally, parents and guardians endorsed the ELLA trial, with some referring to the trial as
‘wonderful’ or ‘fabulous’. Typically, parent/guardian survey responses indicated that they felt the
trial was beneficial for their child. As a whole, they appeared to be appreciative of the diverse
potential outcomes associated with the ELLA apps, with more than 90% of parents/guardians
agreeing or strongly agreeing with the propositions that it is beneficial for preschool children to learn
about another language and another culture (Chart 1.1 and Chart 6.12).
81
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 6.11Parent/guardian strength of
agreement with ‘it is good for preschool
children to learn another language’
70%
Chart 6.12: Parent/guardian strength of
agreement with ‘it is good for preschool
children to learn about another culture’
80%
August Survey
December Survey
60%
August Survey
70%
December Survey
60%
50%
50%
40%
40%
30%
30%
20%
20%
10%
10%
0%
0%
Strongly agree
Agree
Neither
Disagree
Strongly
disagree
Source: Parent /Guardian Surveys - August (257
responses) and December (207)
Strongly agree
Agree
Neither
Disagree
Strongly
disagree
Source: Parent /Guardian Surveys - August (257
responses) and December (206)
Some of the perceived benefits associated with language learning, as stated by parents and
guardians, included:

it enables children to conceptualise living in a multicultural society;

it increases opportunities for travel and work in the future;

it encourages brain development and other cognitive skills;

children learn languages more easily at a younger age;

learning another language reinforces understanding of the primary language; and

it improves children’s confidence.
Additionally, some of the perceived benefits associated with cultural learning, as stated by parents
and guardians, included:

encouraging children to think and understand concepts of difference and cultural awareness;

important that children begin to be exposed to diversity at an early age, particularly given the
multicultural nature of Australia and the increasingly globalised world; and

encourage tolerance and respect for other people and other perspectives – potentially leading
to a more harmonious and accepting society in the future.
6.2 Educator outcomes
While child outcomes were the primary focus of the ELLA trial, educators also noted a number of
positive outcomes which could be attributed to the trial. Most commonly this included: (1) an
increased confidence in incorporating language and culture; and (2) increased confidence in using
digital technology to support early childhood learning.
6.2.1
Increased educator confidence in incorporating language and culture
The educator surveys asked educators to indicate whether they felt more confident in their ability to
incorporate language learning and culture as a result of the trial. The results, confirming a positive
impact of the ELLA trial on educator confidence, are provided in Chart 6.13 and Chart 6.14 below,
which show that over 70% of educators felt increase confidence in incorporating language learning. It
87
Deloitte Access Economics
ELLA Trial Evaluation Final Report
can be observed that the proportion of educators stating their confidence increased as a result of the
ELLA trial grew as the year progressed.
Chart 6.13: Did educators feel increased
confidence in their ability to incorporate
language learning as a result of the ELLA
trial?
Chart 6.14: Did educators feel increased
confidence in their ability to incorporate
cultural awareness as a result of the ELLA
trial?
80%
August Survey
December Survey
70%
80%
August Survey
December Survey
70%
60%
60%
50%
50%
40%
40%
30%
30%
20%
20%
10%
10%
0%
Yes
No
Unsure
0%
Yes
Source: Educator Surveys - August (91 responses),
December (71)
No
Unsure
Source: Educator Surveys - August (91 responses),
December (72)
Affirmative comments as to why educator confidence in language and culture integration had
increased included:

The ELLA trial provided exposure to a wide number of methods through which to incorporate
language learning into the educational programme – including through play and immersion
rather than just structured language lessons.
•
‘The apps gave me so much [sic] ideas and strategies to teach language to young
children, the interactive and fun based language learning will stimulate children’s
interest and desire, that’s for sure!’

The format of the trial allowed children to engage with and learn the language at their own
pace in a positive environment.
•
‘You learn as the children go. Because it is a new skill to both educators and children it
was good to learn together. Learning a new language is daunting but breaking it down
into categories made it easier and following the children’s interest with their progression
of the phrases was good.’

The trial provided exposure to the impact that teaching children to communicate in another
language and understand another culture can have. This was driven by the observed high level
of interest children had in the ELLA apps and how quickly they picked up words and phrases.
•
‘I have seen how easily the children learn a new language, and how they take to it. They
have no fear about making mistakes and just enjoy themselves. They are happy to learn
new things and absorb them quickly. So there is no need to think it would be hard to
incorporate foreign languages.’
•
‘I have more knowledge of Arabic than I previously did and the ELLA apps help with
pronunciation and meaning. I am also more confident because I have seen how the
children have embraced the learning and the confidence it has given them to know
languages other than that of their families.’
88
Deloitte Access Economics
ELLA Trial Evaluation Final Report

The realisation that the educator did not need to speak the target language in order to
facilitate the children engaging with and learning the language.
•
‘Most definitely are more confident teaching language, but with using the iPads. The
work is done for you. If I had to teach the language without the iPad I would probably
not do it…pronouncing those words for me is very difficult’
•
‘When reinforcing the words I kept referring back to the iPads to get the words correct.’
•
‘Realisation that it is OK for me to be learning this alongside the children – so I don’t
have to already know a second language for it to be valuable.

The trial highlighted how the cultural experiences of children and families within the service
could be used to raise awareness of multiculturalism and to increase connections across the
service. While many services stated that the ELLA trial facilitated conversations around cultural
awareness and openness, other services already employed an extensive cultural program
within their service and the ELLA trial complemented existing approaches.
•
‘Creates entry points for family and community members to share their culture across
the centre’.
•
‘Having had this opportunity and being able to discuss it with other professionals has
provided lots of information and ideas about how cultures can be integrated into early
childhood programs’.
•
‘The ELLA Trial has encouraged lots of parents to “open up” about their own culture and
heritage’.
•
‘The ELLA trial has made me think about the cultures that are currently represented
within our centre and how little we know about them. Making resources, planning
activities and singing songs based around the apps and the French language has made
me think I could adapt them to other cultures’.
•
‘We are more comfortable and have seen the effect it has on the confidence of the
children from those cultures when they see their culture is acknowledged and valued’.
It was noted in some responses that this increased confidence was linked to the ability to use the
ELLA apps and resources, and without these educators would revert back to the informal and ad hoc
approaches to language education they employed prior to the trial. Others stated they would use the
learned techniques regardless of whether the trial was continued or not.
Comments as to why confidence had not increased included:

Uncertainty that the trial had resulted in any actual language learning as it appears more
focussed on language awareness.

Educator unfamiliarity with the language and the correct pronunciation has limited potential
confidence growth. It was noted that this may be improved with further training or guidance.
•
‘As an educator/teacher I like to really know and understand what I am teaching. I felt
that I didn’t have enough understanding of the language and to gain a strong
understanding of this one area of my program would take a lot of time’.

Lack of time for the educator to properly familiarise themselves with the ELLA apps and
resources, resulting in a lower level of engagement with the trial than would have been
preferred.

In some instances the service already had a strong emphasis on cultural awareness so the ELLA
trial did not alter current practice.
88
Deloitte Access Economics
ELLA Trial Evaluation Final Report
6.2.2
Increased educator confidence in incorporating digital technology
Responses to the August Educator Survey found that at the beginning of the trial there was a fairly
even spread of confidence levels relating to digital technology among participating educators, with a
slight majority (56%) of respondents feeling confident or very confident about incorporating
technology (Chart 6.15).
Following this, as shown in Chart 6.16, in August most educators (69% of responses) stated that their
confidence in incorporating digital technology had increased as a result of the ELLA trial. This
proportion grew to 83% of educator responses by the December Educator Survey, indicating that
confidence continued to develop over the course of the year. Of those educators who stated that
their confidence had not improved (6% of responses in December), it was commonly stated that the
reason for this was because they were already confident using technology prior to the ELLA trial. Site
visits suggested that those educators who felt more confident using technology were looking for
other ways to integrate technology, outside of the ELLA trial.
Chart 6.15: Educator confidence levels about incorporating digital technology at trial outset
40%
35%
30%
25%
20%
15%
10%
5%
0%
Very confident
Confident
Somewhat
confident
Not confident
Source: August Educator Survey (95 responses)
89
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Chart 6.16: Did educator confidence levels about incorporating digital technology increase as a
result of the trial?
90%
August Survey
80%
December Survey
70%
60%
50%
40%
30%
20%
10%
0%
Yes
No
Unsure
Source: Educator Surveys - August (95 responses) and December (72 responses) Educators cited numerous characteristics
of the format of the ELLA trial which facilitated their growth in technology confidence:

The ELLA trial highlighted the effectiveness of iPads and apps as a learning tool for preschool
children. A key fixture in this effectiveness was noted as being the flexibility iPads afforded
children to work at their own pace, in a non-competitive format. This characteristic was in
some cases stated to be particularly beneficial for certain cohorts, such as quiet children or
children with developmental delays.
•
‘I did not really use the iPads…however after seeing how well the children enjoy using the
iPads and how they have learnt so much I would definitely use them in my class. I am
quite amazed at how much children have learnt using the iPads. The apps have
reinforced some basic Kindy concepts which has been really good, especially for the
lower achievers’.
•
‘It has been encouraging to see quiet/shy students interact with technology in a way they
don’t with educators. That face to face interaction which is intimidating for some is
removed when they are using an iPad.’
•
‘I have experienced how easy it is for children to use and the learning opportunities to
suit all developmental levels.’
•
‘The ELLA apps are well designed, non competitive and with a positive purpose, this fits
in well with our philosophy. Not all apps are designed this way so we would have to be
very careful which ones we put in the classroom.’

Being exposed to the iPads over the course of the year has made the devices a familiar tool
and they are being increasingly used outside the ELLA trial. iPads were also found to easier to
integrate into a preschool environment than desk-top computers.
•
‘More knowledgeable about what quality programs look like, can see how iPads can be
integrated in the classroom if available all the time, knowledge of how children will
interact with the iPads.’
•
‘The iPads have opened so many possibilities with their portability.’
•
‘More regular use and consistency of use of technology in the learning programme. Not
just for “one-offs”.’
•
‘The iPads are so intuitive. The children picked up things much quicker than the staff. I
have loved moving away from the notion of a computer station.’
90
Deloitte Access Economics
ELLA Trial Evaluation Final Report

Confidence in using the iPads has extended into other technological devices, such as Apple TV
and interactive white boards.

Concerns that use would be difficult to manage were generally found to be without merit, with
rosters and timers helping children to self-regulate their time.
•
‘The children self-regulating their use was a surprise to the educators.’
•
‘The children were quite capable of regulating their use of the iPads themselves and they
were not “over-used”.’
•
‘Honestly, I am not a digital technology person and I had reservations about having them
in the classroom. I thought the iPads would take over and the children would not want to
do anything else. This year has changed my mind. Some of the children need to be
prompted to move on from the iPads but generally they monitor themselves. Some days
the children choose to just leave them in the basket and not even use them and other
days there is a line up (for this we use a 10 minute sand-glass timer).’

Concerns that the iPads would be easily damaged were also discovered to be unfounded, with
the cases providing good protection and children generally following established use
guidelines.
•
‘We had never used iPads before and were nervous about how the children would
manage e.g. breakages etc. but they have listened well and follow ‘rules’ really well.
However, there was still a level of reservation from a minority of educators about the use of digital
education. Reservations included:

A need to balance technology use against other experiences matched with a belief that
exposure to iPads occurs enough in the child’s home.

A philosophy that young children learn best with concrete materials they can feel and
manipulate.
6.2.3
Other outcomes for educators as a result of the ELLA trial
Site visits and surveys found numerous other impacts on educators which resulted from the
introduction of the ELLA trial within their service. When educators were asked to reflect on the
broader impacts of the ELLA trial, some common reflections were that the trial:

Provided increased opportunities to network with other services and learn from others.

Inspired educators to think more creatively about pedagogy and the way they deliver the
educational programme – including more focus on play-based and technology based learning.

The supporting material provided for the ELLA trial, including the impacts of language learning
on brain development, was noted by several educators as being useful context for reinforcing
the educational aspects of preschool for families.

An opportunity to redesign their learning trial to incorporate a more immersive approach to
multiculturalism, including increased interaction with families.

An environment in which children can ‘teach the teacher’, encouraging collaborative learning
and instilling confidence in participating children.
The following statements were made by educators responding to the December Educator Survey at
the conclusion of the trial, and reflect the positive impressions of the trial.

‘Amazing experience, so glad to have been a part of it’.
91
Deloitte Access Economics
ELLA Trial Evaluation Final Report

‘The apps are incredible! App 7 has been developed so accurately to suit their interests and to
make language and culture as rich as possible. The apps make learning so, so fun.’

‘I feel very strongly that the intention of bringing interest back into learning other languages to
children by introducing and engaging at a young age has been a resounding success and I
would strongly encourage learning further languages in the new year…even filtering this
learning to other rooms (we are a 0-5 year centre)’.

‘It has been one of the best learning experiences of my 25 year teaching career. I am in awe
daily of how quickly the children learn and have loved this learning.’
Although uncommon, there were a small number of less positive reflections:

The ELLA trial was time consuming and resulted in other components of the educational trial
receiving less attention.

The behaviour of children was more difficult when iPads were introduced – including children
becoming impatient, constantly asking for the iPads and being disruptive until it was their turn.

It was more difficult to engage children in active social play at times when iPads were being
used in the classroom.
6.3 Funding and cost efficiency
The evaluation has considered the cost efficiency of the ELLA trial, which is a function of: the
outcomes of the trial; the number of children participating in the trial; and the level of resources
required for establishing and delivering the trial.
While the evaluation can make general observations regarding cost efficiency, it is not yet possible to
undertake a definitive cost effectiveness analysis. This reflects that while the ELLA trial has clearly
been well received and shown positive signs regarding the pre-production and early language
production stages of language learning, the actual benefits will not become known for some time.
The cost effectiveness analysis would also require a comparison group, so the value of these
outcomes could be compared.
6.3.1
The cost of the ELLA trial
The cost of the ELLA trial during the development and the 2015 trial period was $9.8 million. It
reached 1,868 students, representing a cost per student of approximately $5,246. Given the
innovative nature of the ELLA trial, no direct comparisons through which to determine the cost
efficiency of the trial on a per child basis have been identified.
If costs were considered at a ‘per app’ rate, rather than a ‘per child’ rate, the total cost of the ELLA
programme came to $280,000 per app ($9.8 million divided by 35 apps18). However, given the $9.8
million budget for the ELLA trial included training materials, support materials, project management,
the introductory and mid-way workshops the cost of supplying the iPads, and a host of other cost
items, the per app development cost is likely to be significantly less than the $280,000 per app.
Nevertheless, this figure can be compared with the development of similar apps – in particular, the
development of other enterprise apps (apps designed to satisfy the needs of a group of people,
rather than an individual user) to provide a high level point of comparison. Surveys conducted by IT
18
This assumes the trial involved the development of 35 separate apps – a series of 7 apps across each of the 5 languages.
92
Deloitte Access Economics
ELLA Trial Evaluation Final Report
services firm Kinvey suggest that the costs associated with developing an enterprise app average to
approximately US$270,000, with a fifth of respondents spending between US$500,000 and US$1
million on their app development (Kinvey, 2014). Reporting slightly lower costs, similar research
conducted by IT research firm Clutch places the median cost of an app between US$38,000 and US
$171,000, with a maximum cost as high as $727,000 (Clutch 2015).
6.3.2
Increasing cost efficiency over time
Reflecting that the evaluation has examined the ELLA programme during a trial phase, the most
significant cost investments associated with it (i.e. the development of the ELLA apps and supporting
resources) have been undertaken. As such, the per-unit cost associated with delivering the ELLA trial
is expected to decrease as the number of participants increases, provided an appropriate model for
managed growth and sustainable app maintenance and ongoing development is put in place.
Table 6.1 shows the per child unit costs of the ELLA trial in the initial year (2015). Assuming no
additional apps were developed and the trial was made available to the same or an increased
number of sites, the estimated per child costs for a further three years of the trial have been
estimated under three illustrative scenarios (base case, medium growth and fast growth). In each
scenario, the average cost per child decreases over the three year period after the initial year. The
trial costs are assumed to be $9.8 million for the pilot trial, with ongoing annual costs of $8 million
dollars over the next three years.
Table 6.1: ELLA trial cost per child illustration1
Base case scenario
No. participating children
Ave. cost per child
Medium growth scenario
No. participating children
Ave. cost per child
High growth scenario
Year 1
Year 2
Year 3
Year 4
1868
1868
1868
1868
1868
$5,246
$3,337
$2,700
$2,382
$2,191
1868
3,736
7,472
14,944
29,888
$5,246
$2,225
$1,157
$635
$353
1868
7,472
29,888
119,552
335,357
$5,246
$1,335
$386
$112
$41
3
4
No. participating children
Ave. cost per child
Trial year
2
1
Future cost of ELLA trial assumed to be $8 million dollars over three years, spread equally over each year.
2
Assumes no new sites receive the apps, and a new 1,868 children participate in the trial each year (i.e. the same number of
trial participants per site as current levels).
3
Assumes the number of sites and the number of children that receive the apps doubles each year.
4
Assumes the number of sites and the number of children that receive the apps quadruples each year (until all children
[1]
enrolled in preschools are reached in year 4 ).
Assuming that the trial quality remains constant (that is, that an increase in participants does not
adversely impact trial quality and, therefore, trial outcomes), the cost efficiency of the ELLA trial is
expected to increase as it expands.
[1]
All children enrolled in preschools is sourced from ABS (2015), Preschool education, Australia, 2014, Table 1: Preschool
education, Australia, 2014, catalogue number: 4240.0. There were 324,624 children enrolled in preschools nationwide in
2014.
93
Deloitte Access Economics
ELLA Trial Evaluation Final Report
As seen in Figure 6.1 below, there will always remain a base per child cost associated with the
delivery of the ELLA trial. This is driven by variable costs such as administration and support for
services which will increase as participation numbers increase. However, the fixed costs of the ELLA
trial, those which do not vary with participation numbers (such as app development, data storage,
advertising and provision of set resources) will decrease on a per unit basis as the trial grows.
Therefore, as the number of participants grows, the ELLA trial cost per child decreases towards the
level of the base per child cost.
Figure 6.1: ELLA trial cost curve (illustrative replication only)
Per child
cost/benefit of
ELLA trial
Fixed
development
costs
ELLA programme
benefit curve
Total benefit to society
ELLA programme
cost curve
Base per child cost
Break even point
Number of
participating
children
Source: Deloitte Access Economics
It can also be assumed that there is a minimum benefit per child as a result of participating in the
trial. At this stage of the trial, it is difficult to determine the magnitude of this benefit. While there
are positive signs that the benefits associated with early language exposure would be gained by
participating children, many of the benefits associated with learning a language, and language
exposure, will not become apparent until a later time period. Also, the nature of the benefits of the
ELLA trial, including increased cultural awareness and acceptance, community cohesion and
increased interest in language learning are in many cases intangible and difficult to quantify. It is also
noted that engagement with the ELLA trial was highly variable on a per child basis, and as such, the
benefits accruing to each child would vary significantly.
As the magnitude of the costs and benefits are at this stage unknown, Figure 6.2 below is purely
illustrative. However, it demonstrates – at a theoretical level – how the aggregate benefit of the trial
increases as the number of participants increases, leading to an increase in cost effectiveness. As
long as the per child benefit of participating in the ELLA trial is larger than the per child base cost, the
ELLA trial will have a higher probability of producing a net benefit for society as the number of
children participating in the trial increases.
94
Deloitte Access Economics
18
ELLA Trial Evaluation Final Report
Figure 6.2: ELLA trial benefit-cost curve (illustrative replication only)
Per child cost of
ELLA trial
Break even point
Base case
Medium growth scenario
High growth scenario
Fixed
development
costs
As the number of participating
children increases, the ELLA trial
per child cost moves towards the
base per child cost
ELLA trial average
cost curve
Base per child cost
Number of
participating
children
Source: Deloitte Access Economics
Ultimately, given (1) the significant investment associated with trial development, (2) the potential
benefits of the ELLA trial and (3) the relatively low marginal cost of expansion, there is a case for
facilitating the expansion of the trial to a greater number of services, provided this is done in a
managed way.
The potential for partial cost recovery
6.3.3
While it is anticipated that the Australian Government will remain the primary funder of the ELLA
trial going forward, there are two avenues through which a portion of the ELLA costs may be funded
by an alternative source:
1.
Charging a price for the ELLA trial and recouping a portion of the cost of delivery from services.
2.
Co-funding the ongoing costs of the ELLA trial with state and territory governments.
6.3.3.1
A user-pays system
Given the positive reception of the ELLA trial by sites, and an overwhelming indication that services
wish to continue delivering the trial, a potential opportunity exists to charge services for the apps.
This is supported by the fact that several sites noted the apps were a powerful marketing tool for
their services, as parents were eager to enrol their children at a site which offered language exposure
and learning.
The potential to charge for ELLA apps was tested with sites and it was found that only 6% of educator
responses to the December Educator Survey stated that they would be unwilling to pay for the ELLA
apps. 34% indicated they would be willing to pay and 60% stated they were unsure. Educators noted
that the following considerations were important:
95
Deloitte Access Economics
20
ELLA Trial Evaluation Final Report

The high quality of the apps, and the positive experience of the trial would support a move to
purchase the apps in the future.
•
‘This is something I cannot commit to without going to my finance committee. However,
if this became available for us to buy I would encourage my school to do so as I think it
has been a great experience for the children and has helped their growth and
development.’
•
‘We think it has been an excellent program and we are just coming to grips with it now,
as are the children. To be able to continue on would be fantastic. We would even look at
getting additional languages…’
•
‘Depending on cost obviously, but we know it is a high quality program with high benefit
in the classroom’.

A need to get sign off from councils or the broader community
•
‘We have a strong family committee who runs great fundraising events, we always look
for good program or projects for the children…learning language will be a good program
for the children, I’m sure the families would like to support this.’

The price of the programme
•
‘Hard to say what a reasonable amount would be. The price of the app would need to be
reasonable to be competitive as schools may wish to install them on more than just the 5
iPads provided – i.e. 50 iPads across the school with 7 ELLA apps on each at $1 per app is
$350 for one language…I guess we would be prepared to pay around $300-$500.’
•
‘The major expense for us would be furnishing the iPads. It’s a significant outlay to
purchase 5 iPads…and then the additional expense of an ELLA subscription. I think the
program is worth it, but I particularly value language learning…I don’t know that I could
convince my colleagues. I think a reasonable amount for a year’s classroom subscription
would be $100-$150.’
Those who were reluctant to pay for the apps generally cited a lack of available finances, equity
implications and/or competing priorities:

‘As our service has a number of families on HCC and concession cards which reduce their
Kindergarten Fees and already struggle to pay those reduced fees… children that have limited
access and exposure to using iPads etc have really enjoyed having the exposure to ICT and
exploring a new language…so being free has enabled those families to actively participate in the
program.’

‘Preschool funding being minimal, this program would be an expense the preschool couldn’t
afford. A lot of other things more important and would be more beneficial to the children than
paying for this program. Like paying for support workers’.

‘As we are a community kindergarten it would need to be a cheap option as funds are limited for
us. I would think that it would need to be government funded so that all children have equal
access’.
It is noted that the equity implications of imposing any charge for the ELLA programme would need
to be considered, to the extent that the imposition of an additional cost-barrier may preclude
participation from any families or services. This is considered in greater detail in Chapter 7.
96
Deloitte Access Economics
ELLA Trial Evaluation Final Report
6.3.3.2
State and territory contributions
There is also a potential that states and territory governments increase their involvement in the ELLA
programme, and may provide support through in-kind or financial assistance. The consultations held
with education authorities found that generally, the states and territories were supportive of the
concept of an expanded ELLA trial, and that the trial generally aligned well with local policies related
to increased use of digital technology in early years education and heightened focus on language
learning and cultural exposure.
As such, to the extent that there is scope for the Australian governments to increase collaboration in
the delivery of the ELLA programme, and for the state and territory governments to progress
towards language learning and digital education policy objectives through the ELLA programme,
further cost efficiencies may be achieved. Additionally, if the ELLA programme was to be expanded
into primary education at any point in time, it would be necessary for state and territory
governments to increase their role in the delivery of the ELLA programme given their jurisdiction
over the schooling system.
97
Deloitte Access Economics
ELLA Trial Evaluation Final Report
7 Future considerations
This chapter, informed by the Evaluation findings, provides an overview of considerations for the
future of the ELLA trial. This includes both suggested refinements to the existing trial and an analysis
of the options for future expansion.
7.1 Trial limitations and refinements
While educators and the wider community, including jurisdictional state and territory authorities,
were broadly positive about the potential benefits of the trial, there were several concerns that were
raised about the trial’s efficacy.
At a high level, these concerns included:

Continuity in language learning: the lack of a clear path for language learning for children on
the completion of the ELLA trial raises questions as to the long-term effectiveness of the ELLA
programme. It is noted that this issue relates specifically to the outcome of ‘language learning’
as opposed to ‘language exposure’.

Ensuring adequate language exposure: there was variation in the use of apps by children
within and across sites. While this is appropriate in a pre-school environment, it does suggest
that children could not be expected to have a minimum level of language understanding at the
conclusion of the trial.

Level of educator engagement: it was found that the level of trial integration (and potentially
effectiveness) across sites was highly variable and dependent on the individual approach of
each educator. As such, educator engagement is crucial for successful implementation of the
ELLA trial and methods to ensure consistent and high levels of engagement should be
considered in relation to any future broadening of the trial.

Educator confidence: stemming from concerns with educator engagement, there were
concerns among several educators (noting that this was a minority group, as discussed in
Chapter 6.2) that they did not hold the required skills to adequately deliver the ELLA trial in
their service. These concerns stemmed from a lack of confidence in teaching a foreign
language, as well as how to integrate digital technology in the classroom (i.e. unsure of
appropriate screen time limits for children).

Exclusivity of the ELLA trial: it was difficult to exclude non-participating children (three year
olds or those not provided with consent) from using the apps when they were so engaging for
other children. Many educators noted that they could not see a logical reason why three year
olds could not participate in the trial as they would easily be able to navigate the apps, and by
starting a year earlier, would have the opportunity to increase their language exposure.
7.1.1
Suggested refinements
Building on the concerns articulated above, over the course of the trial – many educators and
parent/guardians made suggestions as to how the ELLA trial could be improved, and the impact of
the trial further enhanced. These are discussed below.
98
Deloitte Access Economics
ELLA Trial Evaluation Final Report
7.1.1.1
Additional guidance
There was a common theme that services would have benefited from increased guidance at both the
implementation stage and throughout the trial. This included guidance on trial delivery, supporting
activities and parent/guardian engagement. However, it is important to note the trial was purposely
designed to allow educators the freedom to implement the trial in a way that suited them best. This
was done to observe the effectiveness of different methods used to introduce iPads and language
learning into a preschool environment.
The types of additional material, which sites indicated they would appreciate was predominately in
the form of:

A translation sheet or educator app which allowed educators to more confidently support
children in their language acquisition – it is noted that an educator app is currently under
development for inclusion in the ELLA trial from next year.

A more clear delineation of the ELLA trial’s aims and objectives (i.e. is it language exposure or
language acquisition) in order to manage expectations of educators and families accordingly. It
was also stated that it would be helpful if such a document also clearly mapped linkages to the
EYLF and provided guidance on how the ELLA trial could most effectively be implemented in
the room, including advice on screen time.

A resource bank which provided educators online access to a stockpile of relevant, high
quality, appropriate materials to be used for complementary activities. However, it was noted
that there may be difficulties associated with the Australian Government or ESA promoting
external material.

More explicit guidance as to appropriate amounts of screen time for children.
7.1.1.2
Broadening of the ELLA trial
It was suggested by numerous parents and educators that the trial could heighten its impact if
broadened to allow for participation by children of broader age groups (potentially developing more
later age apps and integrating with primary school). A desire for greater language choice within the
trial was also expressed. Potentially, multiple languages could be explored within a single preschool,
especially as a focus on one language and culture can be seen to contradict an ethos of
multiculturalism.
7.1.1.3
Networking between educators
Educators regularly stated they felt they would have benefited from increased interaction with other
sites to share ideas on how to best deliver the ELLA trial. While there was acknowledgement of the
Facebook page, educators felt this was not being sufficiently utilised.
7.1.1.4
App design
While feedback on the software design was typically positive, several suggestions for further
enhancements and improvements to the apps were made by both educators and parents/guardians
throughout the trial. Suggestions included:

a ‘back’ arrow to improve navigation between activities;

parental controls on the iPads - allowing the settings to be locked and preventing children
from changing them;

increased opportunities for children to speak the target language;
99
Deloitte Access Economics
ELLA Trial Evaluation Final Report

provision of a screen cover to prevent damage;

a timer on the screen which begins when a child logs into the trial;

using real life images, rather than cartoons, to further support children’s cultural awareness
outcomes; and

use of the child’s avatar within the app activities as a means to increase child engagement.
7.2 Options for expansion
This section works through the key decision points for the potential expansion options for the ELLA
trial. A series of design principles and parameters were established to ensure that the options
analysis was undertaken with consideration of the trial objectives and the anticipated direction of
the next phase of the ELLA trial, based on Australian Government consideration of evaluation
findings.
Design criteria
To evaluate the merits of the potential options for the expansion of the ELLA trial, three design
criteria were established.

Accessibility: the extent to which each option provides opportunity for increased participation in
the ELLA trial. This criterion also includes consideration of equity.

Cost-efficiency: the extent to which each option minimises the cost per child of delivering the
ELLA trial (as discussed in Chapter 6.3).

Effectiveness: the extent to which the trial achieves its desired objectives of increasing language
exposure and learning in Australia. This includes consideration of the quality of the programme
and how this may change over time.
Throughout this options assessment, the above design criteria are applied to each option. This
illustrates the interactions and trade-offs between the considerations of accessibility, cost-efficiency
and effectiveness.
Parameters for consideration
While the below analysis of potential options is based on the independent findings of the evaluation,
to ensure the analysis is able to provide practical guidance for decision making, several parameters
have been set. These parameters relate to the Australian Government’s intended direction for the
ELLA trial:

An ELLA trial budget of approximately $8 million over the next three years.

The major priority of the ELLA trial in the immediate future (the next three years) is to boost
participation among preschool aged children in use of the ELLA language apps.

In the immediate future, the apps developed as a result of the ELLA trial will remain a
government asset. As such, no analysis of divestment of the ELLA programme has been
undertaken.
Four elements are discussed below – support for services, recruitment of additional sites financial
assistance for services, and scope of the trial.
100
Deloitte Access Economics
ELLA Trial Evaluation Final Report
7.2.1
Support for services
Ongoing support for ELLA trial participants can be divided into two key components (1) responding to
ad hoc questions and technical issues regarding trial delivery; and (2) providing opportunities for
information sharing and educator development.
The various support mechanisms available are discussed below, drawing on evaluation findings
outlined in Chapter 3.2, and are categorised as either delivery or development support.

Programme delivery support
•
Online and paper based resources: These were found by educators to be useful support
mechanisms. Additionally, as these resources have already been developed, the
marginal cost of distribution to participating services will be minimal.
•
Webinars: While the webinars conducted throughout the trial were noted as being
useful by some educators, this form of support was not as popular as either the website
or networking opportunities – potentially due to the reduced flexibility of needing to
logon to computers at a set time.
•
Help desk: As the help-desk has the potential to service a large number of participating
sites with centralised resources, it presents a cost-effective way of ensuring smooth
implementation and providing reasonably responsive technical support to services on an
ongoing basis.

Programme development support
•
Workshops/networking opportunities: The workshops were the most popular support
service offered throughout the ELLA trial, reported by educators as inspiring increased
engagement with the ELLA trial, and as such, potentially raising outcomes.
•
Site visits: The site visits were found to be a useful form of support by services.
However, given that workshops are both more cost effective than individual site visits,
and were more popular with educators, it is expected that workshops would be
prioritised over site visits in any ongoing support for the ELLA trial.
A trade off exists between the cost-efficiency of online resources and the effectiveness of face-toface support. During the second round of site visits, the importance of face-to-face demonstrations,
as opposed to viable alternatives, was tested with services. Educators stated that they appreciated
the value of the workshops, and the opportunity to network with other educators. Some educators
felt a combination of introductory materials and online webinars or videos would be sufficient to
enable sites to implement the ELLA trial effectively.
101
Deloitte Access Economics
ELLA Trial Evaluation Final Report
However, despite the increased costs, evaluation findings suggest that face-to-face demonstrations
would be beneficial, particularly in the early phases of the ELLA trial:

Educators participating in the ELLA trial stated that the workshops were highly valuable in
providing direction on how to most effectively deliver the ELLA trial at their site, and for
generating enthusiasm for the trial.

Educators also suggested that face-to-face inductions increase the likelihood that the ELLA trial is
delivered in a manner which maximises trial outcomes. Indeed the evaluation found that
interactive sessions were more likely to encourage changes in practice, encourage innovative
pedagogical approaches and ‘develop the educational model’ – all which would work to enhance
the effectiveness of the ELLA trial

Additionally, face-to-face demonstrations would act as a further marketing tool for the trial and
potentially increase participation from services that would be unlikely to apply unless witnessing
the apps first-hand, thereby also increasing the accessibility of the trial. Indeed, consultations
with educators found that a number of educators and families were initially sceptical of the ELLA
trial until they saw the high quality of the apps and the ease with which children engaged with
them.
It is noted that investment in more intensive forms of support, such as workshops and a phone-based
help desk, would be more effective if concentrated in the initial stages of trial expansion. As the ELLA
programme becomes more established, and networks of educators develop increased capacity to
provide support (through online forums, on a regional basis etc.), the need for more intensive
support may reduce.
Educators involved in the ELLA trial indicated that they would be willing to act as advocates and
provide guidance to new programme participants where practical, creating communities of practice
that will, over time reduce dependence on formal networking opportunities and increase the
sustainability of the ELLA programme. Specifically, it is expected that the need for organised
networking opportunities will decrease (and correspondingly become less costly) as informal
networks build in strength. The strength of these networks, and their ability to function sustainably
over time, will be more likely if leaders are identified early, and supported to develop in their role.
Evaluation findings therefore suggest that although face-to-face induction is not a mandatory
requirement to support effective implementation, the accessibility and effectiveness of the trial would
likely be enhanced if some level of face-to-face interaction was facilitated. It’s expected that informal
networks (online and regional) will reduce the need for organised face-to-face interactions as they
become more established over time.
Reflecting on the above, and seeking to balance the need for a responsive and comprehensive
support service to facilitate effective programme delivery with a cost-efficient approach, it is
anticipated that the ongoing support for the programme will include:
1.
The provision of standard online and paper-based resources to support implementation and
delivery.
2.
A central help-desk to field technical and implementation issues. There is also the potential to
shift this help-desk to an online service in the future if demand significantly reduces.
3.
The facilitation of face-to-face workshops at a regional level, to encourage meaningful
engagement with the ELLA programme and innovative approaches to delivery – particularly in
the early stages of trial expansion.
102
Deloitte Access Economics
ELLA Trial Evaluation Final Report
4.
Identification and support of regional leaders to help develop informal networks that will
sustainably support the ELLA programme over time.
Summary of recommended option for service support

Evaluation findings suggest that the most appropriate format for introducing the ELLA trial to services is
through the development of high-quality online resources, which allow for flexibility in the number of
services participating in the programme.
•
If additional funds are available, face-to-face awareness demonstrations would be beneficial in
both encouraging participation, providing networking opportunities for educators and advising
on best-practice implementation of the ELLA trial.

Ongoing support for the ELLA programme should include online and paper based resources, a technical
help desk, to be administered by ESA and some form of workshop/networking opportunity in the early
stages of trial expansion.

More intensive support measures should be weighted towards the early stages of expansion, to aid in
building momentum and capacity at a regional level. Over time, networks of participants (online and
regional) are expected to replace the need for organised face-to-face workshops.

It is noted that cost efficiency is weighted more strongly than accessibility and effectiveness in the
recommended option, as it is deemed that the programme will still be accessible and effective under the
medium option.
7.2.2
Recruitment of additional sites
One option for the ELLA trial going forward is an expansion in the number of services participating in
the programme. The extent of resources required to facilitate the recruitment of additional sites to
the ELLA trial depends heavily on the desired rate of expansion.
There are several factors which need to be considered when determining what the optimal
expansion pathway would look like:

Currently, there is no clear picture of future demand for the ELLA programme. The response to
the trial has been positive, however it is not yet known how a ‘bring your own device’ (BYOD)
requirement, or reduced support levels, may impact on participation rates.

An understanding of the level of resources required to encourage services to participate in the
ELLA programme, and to support this participation, will be further developed over time as the
trial expands. For instance, the effectiveness of word of mouth advertising will be more
apparent after the opportunity to observe how many services apply for the ELLA trial in the
absence of a deliberate marketing campaign in 2016.

As the trial increases in size, the overall ongoing cost of administrating the trial also increases
(assuming the project management responsibilities remain unchanged). Given this, measures
103
Deloitte Access Economics
ELLA Trial Evaluation Final Report
should be put in place to ensure that demand for the trial does not exceed a level in which the
quality of support for participating services may be compromised. If the quality of support was
threatened.
In the light of the above, a staged expansion in which additional services are added to the trial at a
controlled rate, offers the opportunity to establish a more comprehensive understanding of demand
for the program and the level of resources required to support the expansion – increasing the ability
of the Australian Government to support the expansion of the ELLA program with adequate resources
in a cost effective manner.
As discussed above, there is currently uncertainty regarding the level of demand for the ELLA trial,
which will influence the nature and magnitude of the marketing and communications strategy
required. Given the evaluation finding that the ELLA trial has been generally well received by sites,
and confirmation that many educators involved in the trial would be comfortable advocating the
merits of the ELLA trial at a regional level, there is solid evidence to suggest that a word-of-mouth
campaign may be highly effective, potentially reducing the need to invest in paid advertising.
It is expected that the level of resources required to encourage participation from the desired
number of sites will become clearer over the course of 2016 as the impact of word of mouth
advertising will be able to be monitored through the number of services which apply to participate in
the trial next year in the absence of any additional marketing activity.19 The other advantage of this
approach is to allow a longer time period to examine app usage in an environment in which the
release of new apps is less frequent. In other words, to what extent will children maintain interest in
the apps over a longer time frame? The evaluation has considered this to the extent possible but a
longer timeframe will enable more detailed analysis.
Given the recommendation to take-up a staged approach to the ELLA trial expansion, and the
recognised uncertainty in future demand, it appears the most cost-efficient option for marketing and
communications would be to begin with a low-scale approach, and increase marketing and
communications over time if deemed necessary to facilitate the desired expansion rate.
Summary of recommended option for recruitment of additional services

Evaluation findings support a staged expansion of the ELLA trial over the next three years, providing
opportunity to establish a more comprehensive understanding of demand for the program and the level
of resources required to support the expansion.

As demand for the ELLA trial and the strength of word-of-mouth advertising will become more apparent
over the course of this year, it is advised that the marketing strategy be determined at the end of 2016.
Depending on expected demand, this will consist of either a minimalist marketing campaign centred on
social media and word of mouth advertising, or an extensive campaign including print and radio
advertising.
19
It is noted that this will only be an accurate reflection of the strength of informal advertising if current participating
services are made fully aware of this arrangement and their ability to recommend the programme to other interested
services.
104
Deloitte Access Economics
ELLA Trial Evaluation Final Report
7.2.3
Financial assistance for services
Distinct from the format of technical and programme delivery support provided to services is an
option regarding the nature of (if any) financial or in-kind support provided to services to either (1)
encourage participation; or (2) support equity objectives.
The figure above highlights the two extreme approaches to device provision, on the left side of the
spectrum a blanket BYOD model, which would maximise cost-efficiency but may lead to equity
implications as services with limited ability to access funds would be excluded from the trial. At the
other extreme, while provision of devices for all participating services would maximise the
accessibility of the ELLA trial, the high cost implications associated with this approach render it an
unlikely option from a cost-efficiency standpoint. Between these two options sit an array of midpoint possibilities, including providing financial support to targeted services.
While the proportion of services that currently have access to iPads is unknown, it is expected that
many services would either already have tablet devices or be able to access tablets without
significant barriers. Access to devices is also supported by a range of ad-hoc grant programs in
various jurisdictions to encourage growth in the use of digital tools in educational institutions. For
example, in the past 3 years, Queensland issues grants to kindergartens to allow them to purchase
iPads. However, given the sporadic nature of these grants programs, it is contended that to
appropriately balance the facilitation of equitable access to the ELLA trial and the cost-efficiency
parameters of the trial, there should be some level of funding or in-kind support available to support
services for which purchase of an iPad presents a prohibitive cost barrier.
One possibility for increasing the cost-efficiency associated with financing additional support is to
partner with a hardware provider that may facilitate the large scale distribution of discounted
hardware.
Summary recommended option for financial assistance

A portion is funding is made available to support low SES services in the purchase of devices through
which to support the delivery of the ELLA trial.
105
Deloitte Access Economics
ELLA Trial Evaluation Final Report
7.2.4
Scope of trial
Options
Scope of ELLA
programme
Accessibility
Cost efficiency
Effectiveness
Maintain current scope
Lowest
Highest
Lowest
Increase age to early
primary years
Highest
Medium
Highest
Increase to Australian
Curriculum languages
Highest
Lowest
Highest
Recommended
option
• In the immediate
future, maintaining
the current scope is
the most cost efficient
means of increasing
participation.
• In 2019, it would be
worth considering an
expansion of scope.
Given the high quality of the current ELLA apps, and the cost-efficiency gains to be made through
increased participation, the immediate priority is to extend trial participation to a greater number of
preschool services rather than investing in the development of additional apps. It is not envisioned
that there will be large software investments within the next three years. However, in the mediumterm, there are two distinct areas for further expansion of the trial; the extension of the programme
into other age groups and the development of additional languages (noting these are not mutually
exclusive extensions).
As such, the discussion below is for consideration in the future (at least not until 2019), and
dependent on the structure and success of the ELLA trial at that point in time.
Inclusion of additional age groups
The options considered in this section are:
1.
Maintaining current scope and limiting the ELLA trial to preschool
2.
Expanding provision of the ELLA trial into other age groups
The positive reception of the ELLA trial raises obvious questions as to the extension of the
programme into other age groups. The logic of this expansion is supported by the fact that one of the
major concerns raised about the format of the ELLA trial (by both educators and educational
authorities) was the lack of continuity for children and the risk that the language learnt over the
course of the year would be lost when the child entered primary school.
Consultations with the educational authorities found that LOTE is typically not required in primary
schools until Year Three, although this varies between jurisdictions. Given this, additional apps
created for the early years of primary school may provide an opportunity to bridge this gap in
language learning that currently exists between preschool and the later years of primary school. The
breadth of the ELLA trial objectives could then also be progressively expanded to incorporate a
higher focus on language learning, in addition to language exposure, as links with formal primary
school LOTE programmes become easier to facilitate. Such an evolution of the ELLA trial, to the
extent that this increases the likelihood of children learning an additional language later in life, would
increase trial effectiveness.
Another possibility for expansion is to allow younger children to access the apps (if deemed age
appropriate). One concern with the current format of the ELLA trial, as noted throughout the
evaluation, is that by restricting enrolment to four year olds, the iPads can potentially act as a divisive
106
Deloitte Access Economics
ELLA Trial Evaluation Final Report
feature in the room. In these cases, allowing three year olds in shared classrooms to access the ELLA
apps may reduce difficulties in classroom management for the educators as well as providing
children with a longer period of exposure to the trial for those services. It is noted, however, that a
formal extension of the ELLA trial to three year olds may have implications in the context of current
funding arrangements for early childhood services. Moreover, extending the trial to three year olds,
while seemingly a logical extension, is not expected to be an immediate priority and could be
reassessed at some point in the future.
As discussed in Section 6.3, there is also a cost-efficiency argument to be made for any increase in
participation of the ELLA trial as the greater the number of participants, the lower the unit cost of
programme delivery per child.
Development of additional languages
The options considered in this section are:
1.
Maintaining current scope and delivering the ELLA trial in the current five languages
2.
Expanding the number of languages offered by the ELLA trial to the 11 curriculum languages
It was however noted by a number educators, families and educational authorities throughout the
evaluation that a greater choice of languages within the ELLA trial would be desirable, allowing
services increased flexibility to cater to their community demographic or better link with local
primary school language trials.
However the ELLA programme in its current form, as a one year trial for pre-school aged children,
often with no formal links to language learning in the subsequent years of early primary school, is
focused on facilitating language exposure rather than language learning. The main objective of the
programme is to expose children to language. The language being exposed is a secondary
consideration.
As such, unless increased continuity of language learning can be achieved through either (1) the
extension of the ELLA trial into the early years of primary school; or (2) the extension of formal LOTE
programmes into the early years of primary schools in each jurisdiction; there are minimal gains to
be made from the development of additional languages – as any language will suffice to provide
language exposure.
If there is potential for a continued language learning experience to be established for children,
reaching from preschool through to secondary education, then the decision to invest in the
development of additional languages becomes more pertinent. It is suggested that the decision as to
whether additional languages should be developed is revisited at a later date and evaluated on the
basis of the potential for language learning pathways to be developed, and the success of the ELLA
trial in facilitating language learning (rather than exposure).
107
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Summary of recommended option for expansion of trial scope

In consideration of the analysis parameter that expansion into preschools is the Australian
Government’s immediate priority, and in recognition that increased participation in the current model is
the most cost-effective means of achieving this, there should be no extensions to trial scope in the short
term.

Evaluation findings support an expansion of the ELLA trial into the early years of primary school in the
medium term, to assist with language continuity for children and increase the potential for language
learning, as well as language exposure, to occur as a result of participation in the ELLA trial.
•

The Evaluation findings also suggest that it would be logical to include three-year olds in the ELLA
trial for the same language continuity objectives and to increase inclusivity in mixed-age
classrooms. However, it is not expected that this will be an immediate priority, and can be
reassessed at some point in the future.
The Evaluation findings support the expansion of the ELLA trial to incorporate the 11 languages
endorsed by the Australian Curriculum. However, this is conditional on language pathways first being
established for children between preschool and primary school. As such, the development of additional
languages should not take place until deemed feasible by a future evaluation.
7.3 Implementing the recommended option
The following table provides a summary of the potential investments to be made in the ELLA trial
over the next three years, as supported by the recommended options for a staged extension and
expansion of the trial detailed above. The cost estimates are based on a costing study undertaken for
ESA by Dandolo Partners. Any modifications to the cost estimates made in this work have been
documented in the assumptions column in Table 7.1 below.
The table below only considers investments that will be made in the next three years. It is noted that
any potential expansion to the ELLA scope would be revisited in 2019.
Table 7.1: Recommended investments for ELLA trial 2016-18
Investment
Assumptions
Cost (over 3
years)
Project management office
Derived from cost estimates
$1.3 million
Administration of the help
desk
The cost estimates for the labour associated with the
administration of the Help Desk have been revised down by
50% in recognition that the facilitators of the Help Desk would
typically hold other program management responsibilities and
as such, cost efficiencies would be achieved.
$1.0 million
App maintenance and
storage
Marketing and
communication
Derived from cost estimates
The cost of the marketing campaign will depend on the
strategy chosen – which in turn is dependent on information
gathered through the initial expansion stages (i.e. how
effective the word of mouth advertising emerges to be) this
has been presented as a range.
This range has been derived from the cost estimates but
assumed that the advertising campaign (for both the minimal
campaign and the more extensive campaign) are repeated on
an annual basis, rather than only in Year 1.
$0.065 million
Between $0.6
million and $2.5
million
108
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Investment
Development of support
materials
Financial assistance for low
SES services
Assumptions
Cost (over 3
years)
Derived from cost estimates
The quantum of this funding is flexible and dependent on trial
funding and the extent to which equitable access is a priority.
For the purpose of this exercise it has been assumed that the
quantum of funding allocated to financial support will be
determined on the amount of funding made available from
the marketing and advertising campaign, in the context of an
$8 million budget. I.e. if marketing costs $2.5 million, only $0.5
million will remain for financial assistance.
$1.5 million
Between $0.4
million and $2.3
million
Face to face workshops to
Derived from cost estimates for ‘ELLA Awareness
encourage communities of
Demonstrations’ and assumes approximately 600 workshops
practice
per annum.
Source: Deloitte Access Economics adaption from DandoloPartners (2015)
$1.1 million
As seen in the table above, the total quantum of funding the extension of the ELLA trial is anticipated
to be approximately $7.9 million over three years. This estimate assumes approximately $3 million in
spending on the combined sum of marketing and financial assistance, while the proportion allocated
to each is to be determined at a later date. It is expected that the investment associated with this
floating funding will be concentrated towards the beginning of the expansion period, in order to
encourage increased engagement early on and maximise potential impact.
A summary of these investments, and their expected impact on trial participation, are provided in
the figure below.
Figure 7.1: Summary of ELLA recommended expansion timeframe
Source: Deloitte Access Economics
109
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Appendix A: Overview of evidence
sources
This appendix provides a more detailed overview of the evidence sources informing this interim
report.
A.1 App data from DT Millipede
This appendix describes the app data provided by the software developer – DT Millipede – from a
technical perspective.
The ELLA apps have been designed to collect raw data every time a child or educator interacts with
the iPad (i.e. data is collected every time an iPad is touched). Identical raw data extracts are provided
to Deloitte Access Economics, ESA and the Department by DT Millipede once a week. Fields are
included for app, centre and (de-identified) user identification, enabling changes in app, centre and
individual use to be analysed over time. There is also an event field, with further descriptive columns
for the tasks undertaken – these fields are explained in the data dictionary in Table A.1.
Table A.1: Data dictionary
Data field
Description
Device_time
The device time is a unix EPOCH UTC (the number of second since 01/01/1970).
Language
Which language does the entry relate to?
App
Which app does the entry relate to? (1 through 7).
Centre
Which preschool is the data entry from?
Group
Each group has their own ID. This is useful for comparing different groups within the
same preschool.
Year
The year the observation took place.
Month
The month the observation took place.
Week
The week of the trial.
Day
The day of the month the observation took place.
Day of year
The day of the year the trial took place, with January 1 being 1, and December 31
being 365.
User_id
The unique user id attached to each child.
Type
The nature of the child’s interaction with the iPad. (access, app, activity or language).
The analysis is mainly interested in activity and language.
Event
Each interaction has a range of associated events. For example:
st
st
 Select user: A child has logged in on the log-in page
 Begin: a child has selected an activity to enter
 Interaction: a child has touched the screen to perform a task within an activity
id1
Each event has a range of associated values in id1. The analysis is mainly focussed on
which activity the children are choosing to engage with.
110
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Data field
id2
Description
Each id1 entry has a range of associated values in id2. For example, within the maze,
there are several actions a child can take, or several phrases they are exposed to:
 maze.tam.problem_thankyou – Tam says ‘thank you’ in the target language
 talo.found – the child find Talo (one of the characters) in the maze.
The functional specifications and language transcripts of each app are also provided to Deloitte
Access Economics, which allows code in the raw data to be matched with app usage. This has been
used to understand app usage, popular activities within the app, language exposure and progression
through activities.
Finally, DT Millipede also provides two summary reports in Microsoft Excel:

event_summaries – This report presents the total time, mean and median usage time, in
seconds by each site over the duration of the entire trial to date. It also divides usage into
activity type (for example, time spent playing with maze or the spaceship), and interaction
type (for example, the user opened a gate, or popped a bubble).

event_by_group_by_week – This report presents the time, mean and median usage time, in
seconds, by each site for each week of the trial. It also divides usage into activity type (for
example, time spent playing with maze or the spaceship).
These reports have been used to cross check results from Deloitte Access Economics’ analysis.
A.2 Profile of educator and director surveys
At least one August Educator Survey was received from every trial site.

In total, 112 responses were received, 14 of which were incomplete.

Of the 112 responses, 22 (20%) were from directors, while 85 (76%) were from educators. The
remaining 4% did not state their role.
At least one complete December Educator Survey was received from 37 out of the 41 trial sites. One
of the four trial sites that did not complete a survey submitted an incomplete survey.

In total, 81 responses were received, 13 of which were incomplete

Of the 81 responses, 15 (19%) were from directors, while 63 (78%) were from educators. The
remaining 4% did not state their role.
A.3 Profile of trial sites for face-to-face visits
The descriptive statistics for the sample of 21 sites participating in the face-to-face visits are
compared to those for the total trial population in Table A.2.
111
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Table A.2: Descriptive statistics for site visit sample
Variable
Site visit sample
No. of sites (% of total
sample)
All sites
No. of sites (% of total
sites)
State/territory
NSW
5 (24%)
12 (29%)
Victoria
5 (24%)
9 (22%)
Queensland
4 (19%)
8 (20%)
WA
2 (10%)
6 (15%)
SA
2 (10%)
3 (7%)
Tasmania
1 (5%)
1 (2%)
NT
1 (5%)
1 (2%)
ACT
1 (5%)
1 (2%)
French
3 (14%)
8 (19.5%)
Indonesian
5 (24%)
8 (19.5%)
Mandarin
4 (19%)
8 (19.5%)
Arabic
5 (24%)
8 (19.5%)
Japanese
4 (19%)
9 (22.0%)
15 (71%)
22 (54%)
13 (62%)
22 (54%)
Inner/outer regional
7 (33%)
17 (41%)
Remote/very remote
1 (5%)
2 (5%)
Low
6 (29%)
11 (27%)
High
7 (33%)
9 (22%)
Stand-alone-preschool – Government
3 (14%)
4 (10%)
Stand-alone-preschool – Independent
4 (19%)
7 (17%)
Pre-school-attached to school – Government
5 (24%)
8 (20%)
Pre-school-attached to school – Independent
2 (10%)
3 (7%)
Long-Day-Care-With-A-Preschool-Programme
7 (33%)
19 (46%)
Language
Capital city
Yes
ARIA categorisation
Metropolitan
Socio-economic status^
Service type
^ Socioeconomic status (SES) was determined on the basis of the ABS index of relative advantage and disadvantage.
Suburbs in the lowest 20% of scores were classified as low SES, while suburbs in the highest 20% of scores were classified as
high SES.
A.4 Profile of parent/guardian surveys
For the August Parent/Guardian Survey, at least one response was received from a parent or
guardian at 36 of the 41 sites. 309 responses were received from parents or guardians in total, of
which 59 were incomplete.
For the December Parent/Guardian Survey, at least one response was received from a parent or
guardian at 35 of the 41 sites. 241 responses were received from parents or guardians in total, of
which 41 were incomplete.
112
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Table A.3: Overall parent survey responses rate
August Parent/Guardian Survey
December Parent/Guardian Survey
Responses
309
241
Total children
1764
1771
Proportion
17.5%
13.6%
Note: There 41 responses from the August Parent/Guardian Survey, and 24 responses from the
December Parent/Guardian Survey, which did not state which ELLA trial site the child of the
respondent was attending and therefore could not be included in the more detailed analysis below.
Table A.4: Descriptive statistics for August Parent/Guardian Survey response rate
Parent/guardian
responses
Total
parent/guardians
Percent of total
parents/guardians
89
70
31
50
9
3
12
4
461
458
245
240
159
24
60
117
19.3%
15.3%
12.7%
20.8%
5.7%
12.5%
20.0%
3.4%
51
62
35
46
74
311
348
346
339
420
16.4%
17.8%
10.1%
13.6%
17.6%
Remote/very remote
158
107
3
990
750
24
16.0%
14.3%
12.5%
Socio-economic status^
Low
Medium
High
45
157
66
442
772
550
10.2%
20.3%
12.0%
Stand-alone-preschool – Independent
37
59
287
323
12.9%
18.3%
Pre-school-attached to school –
Government
55
416
13.2%
Pre-school-attached to school –
Independent
42
157
26.8%
Long-Day-Care-With-A-PreschoolProgramme
75
581
12.9%
Variable
State/territory
NSW
Victoria
Queensland
WA
SA
Tasmania
NT
ACT
Language
French
Indonesian
Mandarin
Arabic
Japanese
ARIA categorisation
Metropolitan
Inner/outer regional
Service type
Stand-alone-preschool – Government
113
Deloitte Access Economics
ELLA Trial Evaluation Final Report
^ Socioeconomic status (SES) was determined on the basis of the ABS index of relative advantage and disadvantage.
Suburbs in the lowest 20% of scores were classified as low SES, while suburbs in the highest 20% of scores were classified as
high SES.
Table A.5: Descriptive statistics for December Parent/Guardian Survey response rate
Parent/guardian
responses
Total
parent/guardians
Percent of total
parents/guardians
74
33
57
19
21
1
1
11
450
456
252
243
162
23
64
121
16.4%
7.2%
22.6%
7.8%
13.0%
4.3%
1.6%
9.1%
33
40
54
65
25
354
302
337
438
340
9.3%
13.2%
16.0%
14.8%
7.4%
Remote/very remote
145
70
2
999
749
23
14.5%
9.3%
8.7%
Socio-economic status^
Low
Medium
High
47
94
76
442
776
553
10.6%
12.1%
13.7%
Stand-alone-preschool – Independent
25
57
291
314
8.6%
18.2%
Pre-school-attached to school –
Government
49
422
11.6%
Pre-school-attached to school –
Independent
11
161
6.8%
Long-Day-Care-With-A-PreschoolProgramme
75
583
12.9%
Variable
State/territory
NSW
Victoria
Queensland
WA
SA
Tasmania
NT
ACT
Language
French
Indonesian
Mandarin
Arabic
Japanese
ARIA categorisation
Metropolitan
Inner/outer regional
Service type
Stand-alone-preschool – Government
^ Socioeconomic status (SES) was determined on the basis of the ABS index of relative advantage and disadvantage.
Suburbs in the lowest 20% of scores were classified as low SES, while suburbs in the highest 20% of scores were classified as
high SES.
114
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Appendix B: Evaluation questions
Table B.1: Key evaluation questions
Evaluation question
1
2
3
4
5
Sub-evaluation questions
Was the overall design of
the ELLA programme
appropriate and based
on evidence
1.1
What is the evidence regarding the link between early
exposure to other languages and educational and
developmental outcomes?
1.2
Is there evidence that the ELLA trial was designed in
accordance with available evidence from research literature
and demonstrated good practice approaches in comparable
programmes?
Was the design of the
software appropriate? Is
it fit for purpose?
2.1
Did the apps incorporate a level of functionality that was
appropriate for their purpose?
2.2
How user friendly were the apps for preschool aged children?
2.3
How user friendly were the apps for educators?
2.4
Could educators easily engage with children about and during
use of the apps?
3.1
Was the initial availability of apps and subsequent app
updates timely and accessible for the trial sites?
3.2
To what extent were children engaged with the apps?
3.3
Was the involvement of parents/guardians throughout the
trial appropriate?
3.4
Why did children use the apps more or less?
3.5
Were iPads an appropriate platform for children’s use of the
apps?
3.6
Were services adequately supported in use of the apps within
the sites?
3.7
Did services and their staff receive adequate and appropriate
training in use of the apps throughout the trial?
3.8
How often were the apps used?
3.9
What were the patterns of usage?
3.10
Did usage vary across languages?
3.11
Why did usage vary across certain apps or app series?
3.12
Why did usage vary across sites?
Did the ELLA programme
make a difference to
children’s outcomes?
4.1
What did children learn as a result of the ELLA trial?
4.2
In what ways did exposure to another language through the
ELLA trial contribute to the development of children?
What would be the cost
of expanding the
programme to more
children?
5.1
What was the cost of the ELLA trial and its components (fixed
and variable; operating and capital)?
5.2
What did it cost to implement?
5.3
What did it cost to operate?
5.4
Are there economies of scale which will reduce the per child
cost if the programme is expanded?
Was the programme able
to be implemented
effectively? Did children
use the programme?
115
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Evaluation question
6
Are there other options
or pathways for the ELLA
model?
Sub-evaluation questions
6.1
What are the key lessons from the ELLA trial that should be
applied to any future programme or service model?
6.2
What are the options for models of ongoing delivery of early
childhood language education utilising digital technology?
6.3
What model is recommended?
6.4
What role should the Australian Government play in this area?
6.5
What role should state and territory governments play in this
area?
116
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Appendix C Literature review
Introduction
The objective of the Early Learning Languages Australia (ELLA) trial is to expose preschool children to
languages other than English using a selection of language applications (apps) to be used on iPads in
the classroom. The languages selected in the trial are Chinese (Mandarin), French, Indonesian, Arabic
and Japanese. The trial is part of an Australian Government commitment to increase the number of
students studying additional languages in the later years of schooling. Deloitte Access Economics has
been engaged to undertake an evaluation of the ELLA trial, which is being conducted in 2015.
A literature review was undertaken to provide a qualitative understanding of the current rationale
for, and best practice approach to, using digital technology for language learning in preschools.
Accordingly, a number of research questions were identified, which are outlined in Appendix A.
Approximately 65 documents were canvassed in the course of the literature review, and
consequently sorted to determine relevance to the ELLA trial. The literature review methodology is
also detailed in Appendix A.
The findings of the literature review are structured as follows:

Chapter One illustrates the current policy context in relation to both language education in
preschools and the integration of digital technology in preschools.

Chapter Two details the literature supporting the underlying rationale for the learning of
languages and the use of digital technology in preschools.

Chapter Three presents the literature findings relating to the best practice approaches to the
use of digital technology in preschools, the teaching of languages in preschools and the design
of apps.
1. Policy context
This chapter outlines the policy context for the ELLA trial, including the growing emphasis on the
importance of language education in Australia, the national current early childhood education policy
framework and emerging policy design relating to the use of digital technology in educational
settings. The design of the ELLA trial, and how this aligns with the policy context, is also considered.
Language education
Over the past 40 years, policy on language education in Australia has advocated for a greater uptake
of language learning (Lo Bianco & Slaughter, 2009). Policy documents recognise the advantages of
learning a foreign language, specifically citing the benefits to cognitive development, the
improvements to intercultural understandings and the broadening of social, personal and
employment horizons (ACARA, 2011). The recently released White Paper Australia in the Asian
Century (2011) suggested that Australia should increase Asian languages education, to prepare
students for their inevitable exposure to Asia, given the rapid growth of several Asian economies.
The National Statement on Asia Literacy in Australian Schools 2011-2012 supports this
recommendation, stating that learning an Asian language will provide students with a competitive
edge in an increasingly globalised world.
117
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Based on the findings of these recent policy reports, the Australian Government has committed to
increase the number of students studying a second language in Year 12 to 40% within a decade, up
from 13% in 2013 (Australian Government, 2013). Specifically, the Government intends to:

develop a national curriculum for 13 languages;

work with the states to make language study compulsory between Year 5 and Year 10;

actively recruit language educators, including adding specialist language educators to the
Skilled Occupation List;

improve educator training courses;

actively recruit for specialist language educators through Teach for Australia;

consider options to increase language study in senior secondary years; and

provide preschool children the chance to study a language other than English, through the use
of digital technology.
The ELLA trial is intended to support this stated government objective by exposing preschool children
to languages other than English through the use of iPads. It is anticipated that early exposure will
encourage further language learning in later years of education.
Early childhood education
The current national early childhood education policy framework for learning outcomes is outlined in
Belonging, Being & Becoming: The Early Years Learning Framework (Department of Education,
Employment and Workplace Relations, 2009). The framework consists of five outcomes:
1.
Children have a strong sense of identity
8.
Children are connected with and contribute to their world
9.
Children have a strong sense of wellbeing
10.
Children are confident and involved learners
11.
Children are effective communicators
The policy framework encourages a programme focus on the promotion of creativity and interaction
in the learning processes, as well as high levels of engagement and communication (Yellend &
Gilbert, 2013). As such, the apps being used throughout the ELLA trial have been mapped to the Early
Years Learning Framework (EYLF), specifically to outcomes 4 and 5.
The use of tablets in early childhood education
The use of tablets in education is still a relatively novel concept, although results from initial trials
suggest tablets support positive educational outcomes for young children.
Most states and territories have implemented trials to ascertain the effectiveness of digital tablets in
education (for a list of past programmes, see Table C.1 below). As most of these programmes are
either ongoing or only recently completed, tablet use in education, particularly early childhood
education, is yet to feature explicitly in policy documents. While the use of digital technology is
encouraged, there is no specific policy objective to increase the use of tablet technology as a
teaching aid. Some states, however, do offer targeted support and guidance in this area. For
118
Deloitte Access Economics
ELLA Trial Evaluation Final Report
example, the Western Australia Department of Education outlines methods and techniques
educators can employ to support the use of tablet in an educational setting.20
Comparable programmes
A number of programme trials which are comparable to the ELLA trial have been identified (Table C.1
below). The learnings associated with these programmes have been incorporated in the discussions
of best practice in the following chapters.
20
This documentation can be found on the WA Department of Education website
119
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Table C.1: Programmes related to the ELLA trial
Study name
iPads for learning.
In their hands.
Use of tablet
technology in the
Classroom
Smart Classrooms
Early childhood
iPad initiative
Author(s)
I&J
Management
services
Goodwin, K
Jurisdiction
Victoria
NSW
Queensland
Government
Queensland
Western
Australian
Department of
Education
WA
Target
participants
Primary and
Secondary
students
Grades 3-6
Years 8-10
Grades 1 and 2
Study description
Key student outcomes
The purpose of the trial was to assess if iPads
enrich children’s learning experience across
all fields of education. Over 650 iPads were
placed in 10 primary, secondary and special
schools around the state. The trial ran for 13
months. Schools were provided with $100
spending money per iPad, to download apps.
Students found learning was more
The trial was rolled out to six classrooms
across three primary schools. In some classes
there was one iPad for each student; in
others, there was one iPad between two
students. Schools were provided with $50
spending money per iPad, to download apps.
iPads place additional demand on
The iPads were used in four classrooms across
two schools. One school gave one iPad to
every student; the other shared the iPads
between students. The trial ran for 6 months
Outcomes improved when there
The trial was implemented in 19 schools in
metropolitan, regional and rural areas, with a
focus on teaching literacy and numeracy.
Over 850 iPads were distributed to schools,
and were used by 1,671 students. iPads were
generally used on a shared basis.
Students usually disinterested with
fun with an iPad
Most teachers noted iPads
improved learning outcomes
iPads had a greater impact on
younger students
teachers
Both students and teachers found
that iPads supported and
enhanced learning
was one iPad for every student
learning were engaged when using
the iPad
Teachers reported the iPads
created a more relaxed learning
environment
120
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Study name
Schools going
mobile
Author(s)
Pegrum, M;
Oakley, G &
Faulkner, R
Exploring the
pedagogical
applications of
mobile
technologies for
teaching literacy
Pegrum, M;
Oakley, G &
Faulkner, R
Studying Asian
languages with
web technology
Salt Group
Jurisdiction
WA
WA
Victoria
Target
participants
Kinder – year 12
Kinder – year 12
Secondary
students
Study description
Key student outcomes
This was a two phase project, with
researchers examining the use of digital
technology in 10 schools across Western
Australia. The focus was on literacy
outcomes, but this evolved throughout the
programme. The initial phase was concerned
with schools existing practises with handheld
technologies. Only schools already using
digital technology in the classroom were
eligible.
At younger levels, one iPad for
This is the second phase of the above
programme. It involved six case study
schools, again with a focus on teaching
literacy. The focus of this study was on
teachers views on the benefits and challenges
of digital technology
Preliminary empirical studies show
The study was conducted over two years. 51
schools partook in the first year; 41 schools in
the second year. The study sought to uncover
whether digital technology improved learning
outcomes in language education.
The iPad generated a positive
many children was the preferred
model.
Children are more motivated and
engaged when using iPads
the iPad improves test results
among primary students
iPads work best for younger
students, as the iPad allows
freedom for play-based learning.
student attitude toward learning
Students took responsibility for
their learning
Teachers were confident the iPad
was leading to increased language
skills
121
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Study name
iPlay, iLearn, iGrow
Author(s)
Yellend, N &
Gilbert, C
Jurisdiction
Victoria
Target
participants
Study description
Children aged 0-6
The study was designed to explore the use of
tablet technology with young children. There
were 95 children (including 20 in
kindergarten) involved in the trial. The trial
ran for six weeks. iPads were introduced to
the classroom as necessary.
Key student outcomes
Tablets stimulated conversation
Students self-regulated their use of
the tablets
International examples
AlphaEU
NA
Lazar, A
McPake, J &
Stephen, C
Various
countries
Children aged 2-6
Scotland
Preschool
The study used digital media to promote
multilingualism in the EU, through digital
alphabet books, alphabet-related games and
interactive activities. It aimed to develop
perception and recognition of sounds and
concepts of at least one foreign language, and
to then understand and use the language in
an interactive manner.
A four-week trial teaching Gaelic in
preschools, rolled out across several
playrooms. The trial was aimed at reducing
teaching load and acting as a new strategy to
more effectively teach Gaelic to preschool
children. The app involved arranging a
sequence of photos that either the child or
teacher would take on the iPad, and then add
captions and related sound files. The children
could therefore repeatedly listen to the
retelling of familiar events, which stimulated
them to respond in Gaelic.
iPads found to effectively teach
additional languages
Children were found to be
enthusiastic about their learning.
Those who were relatively
unfamiliar with iPad use and
navigation quickly learnt from their
peers.
In doing so, the app encouraged
collaboration and scaffolding
among students
Ground rules needed to be set by
teachers in order to ensure that
the iPads were equitably shared.
122
Deloitte Access Economics
ELLA Trial Evaluation Final Report
2. Supporting rationale for ELLA
This chapter documents the literature on the benefits of learning a second language, the benefits
associated with early exposure to foreign languages and the benefits of integrating digital technology
in preschool classrooms. Together, these three separate elements form the evidence base which
underpins the rationale of the ELLA trial design.
General benefits of learning a second language
The review of the literature identified three common benefits to learning a second language:

improved cognitive function;

cultural awareness; and

improved health.
Improved cognitive function
Studies have demonstrated that learning a second language at an early age improves (among other
things) children’s cognitive development and problem solving skills. Furthermore, these effects have
been shown to continue beyond childhood, with bilingualism shown to offset age-related declines in
executive functions, such as memory, reasoning, and problem solving ability (Bialystock and Craik,
2010; Gold et al, 2013).
Cultural awareness
With the rise of globalisation, interactions between people of different languages and culture have
been increasing steadily. The ability to engage and communicate with people of different languages
and cultures is fundamental to participation in the global economy (Graddol, 2006). Learning a
second language has been shown to facilitate cultural awareness, and improve people’s
understanding of human behaviour (Baker, 2006; Crozet and Liddicoat, 1997). Not only does this
prepare people for the increasingly globalised economy, it can also lead to improvements in social
unity and harmony. UNESCO highlights the importance of language diversity, stating that language
diversity improves cohesion, and makes countries more effective members of the international
community. This is encapsulated in the quote below.
“Languages are indeed essential to the identity of groups and individuals and to their peaceful
coexistence. They constitute a strategic factor of progress towards sustainable development
and a harmonious relationship between the global and the local context”.
Mr Koichiro Mastuura, Director General of UNESCO, 2008
Improved health
Studies have shown learning a second language can delay the onset of dementia (Craik et al 2010).
Criak et al posit that bilingualism is a cognitively demanding skill, and therefore increases the brain’s
resistance to damage.
Early exposure to a second language
Early exposure to a second language has been found to have multiple benefits, including:
123
Deloitte Access Economics
ELLA Trial Evaluation Final Report

Increased probability of successfully acquiring the second language;

Increased learning and development outcomes in other areas;

Heightened language support for bilingual students.
Second language acquisition
It has been consistently found in literature that the early years are a crucial period for the successful
acquisition of a second language. A literature review focused specifically on the role of early language
intervention in kindergarten found that ‘there is a wide consensus that infants develop two separate
but connected linguistic systems during the first years of life’ and as such, the early years are ‘a
critical period for a child to learn a second language’ (Griva & Sivropoulou, 2009).
The review found that:

children’s enjoyment and openness towards other languages and cultures are ‘key factors’ for
the potential benefits of early language learning; and

this early dual language exposure does not delay development in either language.
Generally, the literature points to the fact that ‘younger children are more receptive to language
learning and develop a more native-like pronunciation when second language learning begins before
the onset of adolescence’ (Stewart, 2005). This is attributed to the developmental stage of younger
children, in that they are more receptive to language learning and the adaptability it requires.
Research finds that younger students question language structure to a less extent, which helps with
pronunciation and the learning of different grammar structures.
Supporting this, a study using 100 Chinese-Spanish bilinguals looked at the difference in language
ability between those who began learning the second language early in life (between 3 and 6 years of
age) and those who began learning at the age of 16 years or older. The study found that the learning
mechanisms of early learners were not fundamentally different to native speakers, although they still
may differ from native speakers in the ultimate success of learning the language. Conversely, those
who started to learn the second language at an older age found that their learning mechanisms were
fundamentally different from native speakers, having to rely on reference to their first language and
learnt problem-solving mechanisms rather than accessing the intrinsic learning mechanisms which
children use to learn a first language (Granena, 2013).
Increased outcomes in other learning areas
Aside from the increased ability to learn a second language, research has also indicated that the
introduction of a second language in early years increases outcomes in other learning areas. An
Australian study found that the introduction or maintenance of a second language in the early years
‘improves cognitive abilities, positively influences achievement in other disciplines, and results in
higher achievement test scores’ (Jones Diaz, 2014).
Supporting this, an American study conducted in 2004 found that ‘possessing a second language
allows children to participate in two social worlds and become more attuned to subtleties of
communicative interactions, expanding cognitive abilities, creative thinking, adaptability and
problem solving skills which are all transferable to other academic areas’(Stewart, 2005). This study
also found that exposure to a second language develops ‘an understanding of geographical and
124
Deloitte Access Economics
ELLA Trial Evaluation Final Report
cultural perspectives that enhances learning in other classes such as social studies, science, art and
music’.
Reduced language loss for bilingual students
As a side-note, it is found in literature that exposure to a second language in preschools is of great
importance to bilingual students. Jones Diaz (2014) argues that explicit policy initiatives for children
between birth and five years of age are needed in Australia as ‘language shift in early childhood and
primary education occurs when children are exposed to English-only educational settings at a young
age’. In light of this, the introduction of, or maintenance of, alternative languages in preschool is
expected to be of importance in reducing the language loss of bilingual children attending English
speaking preschools.
Digital technology as an educational tool
While evidence relating to the effectiveness of digital technology on the learning of a second
language in preschools was the primary objective of the literature review, it was found that there
have been limited studies conducted at that level of specificity. As such, this section first presents the
findings relating to digital technology and the teaching of other languages, and then discusses digital
technology as an early childhood educational tool more broadly.
Second language learning and digital technology
In relation to the increased opportunities presented by digital technologies, one of the primary
constraints in providing for language education in preschools is the difficulty of recruiting and
retaining well qualified language educators (Stephen et al, 2012). In this context, technology offers
an alternative teaching method to relying on the language capabilities of preschool teachers
(Nemeth & Young, 2013). A national position statement made in the US on the use of technology in
education drew an explicit link between the emergence of new technologies and the ability to better
support second language speakers. This is highlighted in an excerpt from the position statement
which is given below.
‘Digital technologies allow teachers to find culturally and linguistically appropriate stories, games,
music, and activities for every child when there may be no other way to obtain these resources…with
technology, adults and children can hear and practice accurate pronunciations so they can learn one
another’s languages. If teachers do not speak a child’s language, they may use technology to record
the child’s speech for alter translation and documentation of the child’s progress. As linguistic and
cultural diversity continues to increase, early childhood educators encounter a frequently changing
array of languages. Appropriate, sensitive use of technology can provide the flexibility and
responsiveness required to meet the needs of each new child and ensure equitable access for children
who are dual language learners’
– National Association for the Education of Young Children, 2012
Further, a study looking at the use of computers in aiding second language learning found that ‘as
computer games enable language to be put in the context of dialog, they enable language to be
situated. This provides an ideal environment for language learning as verbal information is given just
in time and is provided in an appropriate context’. The study also found that the use of personal
avatars creates a high degree of immersion which increases motivation and is a key factor for
successful language learning (Peterson, 2010).
125
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Supporting this, a recent Spanish study found multimedia to be a useful tool in early childhood
language learning. Multimedia uses interactive games to (1) increase vocabulary, (2) improve
pronunciation, (3) reinforce learning through repetition, and (4) contextualise learning, through
engagement with shapes, colours, sounds and letters (Agudo, Rico & Sanchez, 2015).
Digital technology in early years education
Due to the relatively recent emergence of digital technology as an educational tool, the integration
of technology into preschool classrooms was initially met with a level of concern by both educators
and parents. However, as the impacts of digital technology on learning outcomes become clearer,
these concerns have largely subsided. There is currently a weight of evidence to suggest that on
balance, digital technology enhances the learning experience for preschool children and results in
increased child outcomes in relation to learning and development.
Evidence supporting digital technology as an educational tool
It is generally considered that children born into a Westernised society after the 1980’s are
technologised, and engage with technological artefacts on a daily basis. As this engagement with
technology is vastly different from the childhood experience of parents, teachers and early years
practitioners, the integration of technology into the learning environment of children has been met
with some level of concern, specifically ‘negative assertions about technology being socially
detrimental for children…despite contemporary empirical research substantiating technologies as
tools around which pro-social, collaborative and helping interactions can emerge’(Arnott, 2013).
In contradiction to these reservations, a comprehensive literature review looking specifically at the
influence of technology on learning in the early years found that technology typically had a positive
impact on most areas of learning, and that the impact was conditional on the child’s age,
experience, time spent using technologies and gender (Hsin et al, 2014). The literature review found
the following frequencies among the 87 selected studies (from a pool of 273) relating to the learning
effectiveness impacts of digital technologies.
Table C.2: Impacts of digital technology on learning effectiveness
Domain
Developmental domain
Learning effectiveness
Positive
Negative
No difference
Depends
Cognitive aspect
53
2
20
47
Social aspect
13
1
4
3
Emotional aspect
10
0
1
1
Physical aspect
2
0
0
0
Positive
Negative
No difference
Depends
Language and literacy
26
1
16
32
Math
8
0
2
7
Science
4
0
1
3
Digital literacies
17
1
0
5
Cognitive abilities
7
0
2
3
Others
2
0
0
0
Cognitive domain of development
Source: (Hsin et al, 2014)
126
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Impact of digital technology on literacy outcomes
An American study used a software program designed to facilitate early learning literacy
development to engage 265 kindergarten children from a high-risk community. 151 students in eight
experimental classrooms used the program for approximately 15 minutes per day. Additionally, 114
students in seven non-intervention classrooms had varying amounts of access to older hardware and
software that was not integrated into the classroom in a systematic manner. Pre and post testing of
students found that students in the experimental classrooms performed significantly better than
non-intervention students on one literacy test (TERA-2) and better (though not to a significant
extent) on the other form of testing, the Lindamood Auditory Concepualization Test (Tracey & Young,
2006).
A second American study found that providing iPads to 266 kindergarten students improved their
literacy scores, specifically in the Hearing and Recording Sounds in Words subtest, in which the iPad
group scored 2.1 points higher than the control group (Bebell & Pedulla, 2015). Subsequent years of
education showed that the iPad group experienced continued greater benefits, with better reading
and writing skills.
Impact of digital technology on social interactions
In relation to the impact of digital technologies on social interactions between children, the following
studies found positive correlations:

A Scottish study analysing the impact of digital technologies in a kindergarten found that ‘one
of the fundamental aspects of children’s interactions around technologies is the need to
negotiate with, and mediate, the other children’. The study observed children managing the
social aspects of sharing the technology between themselves, and concluded that the
introduction of technology in the classroom ‘moves beyond technological affordances to
considering children as active agents who have considerable influence over their own social
experiences’(Arnott, 2013).

A recent Australian study found that after the distribution of several forms of digital
technology in a class room ‘the most common activity children engaged in were ‘social
interactions’…including cooperating together using a device’. It was found that even though an
iPad is typically a solitary device, children rarely used it by themselves. Instead, groups of
children would watch and encourage the user (Bird, 2013).

The literature review of 87 studies (outlined above) of relevance found that ‘only one reviewed
study revealed that the use of a touchscreen increased children’s behaviour of pursuing
individual goals instead of collaboratively achieving the same goal…most of the studies,
however, showed that various technologies support children’s social development’.
Impact of digital technology on learning style
Aside from social learnings, the experiments with the use of digital technologies in early years
education have generally found that the technologies encourage a high level of self-motivated
learning. It has been found that ‘kindergarten children benefit from technology when technological
tools are integrated into learning tasks that allow children to work at their own pace with adult
support’ (Gimbert & Cristol, 2004). Specifically, tablets have been found to be an appropriate tool for
engaging learning among preschool students. A study analysing the use of touch screen tablets to
increase literacy in young children found that ‘in contrast to traditional computers, touch screen
127
Deloitte Access Economics
ELLA Trial Evaluation Final Report
tablets provide an easier to use and more intuitive interface for a child’ (Neumann & Neumann,
2014).
More generally, an Australian study found that the distribution of iPads and iPod touches in a
kindergarten practice gave the participant children an increased independence and agency in their
choices and experiences of play. Overall, the early years practitioners involved in the study found
that ‘the introduction of this technology has strengthened many areas of early education practice
and encouraged our children to be more capable, enthusiastic and independent learners’ (Forbes,
2013). Similarly, the Western Australia study highlighted in Table C.1 found that ‘the standout
conclusion in the current literature is that m-learning (learning using a mobile device) is highly
engaging for students’ and cited ‘greater student ownership of learning processes’ as a key
contributor to the increase in student motivation (Oakley et al, 2012).
Digital technology in autistic education
Many countries have adopted greater use of digital technology, particularly tablets, to facilitate
better learning for students with special needs, including Autism Spectrum Disorders (ASD). Most
research in this area has been confined to small clinical trials, involving 1-4 children, due to the
heterogeneous nature of the ASD student population. As such, general claims are difficult to
support. Nevertheless, several studies have looked specifically at how digital technology can assist in
better academic outcomes for ASD children, as complementary to standard teaching practice.
A study involving 4 students with ASD, aged between 11 and 13 years, looked at the effect of iPad
use on academic outcomes (Tanner, Dixon & Verenikina, 2010). The research found that the
performance of the children improved across the board, although the extent varied between 15%
and 60%. While academic improvement was clearly noted, the level was very sensitive to the
individual. This is in line with past pedagogical research that has found education for ASD children
needing to be tailored to the specific student, in order to optimise learning outcomes.
In Western Australia, there was a similar trial assessing whether iPads were useful educational tools
for students aged 5-13 with special needs, over an 8 month period (Johnson, 2013). The study found
that autistic children in particular were very responsive to tablets being used as a learning aide.
An American study examined the comparative differences in instructional effectiveness between
iPads and traditional learning materials such as flashcards, specifically in the case of students with
ASD (Neely, Rispoli, Camargo, Davis & Boles, 2013). Once again, the trial was with a very small
sample of two students. It found that the use of iPads reduced the incidence of challenging
behaviour from ASD students, whilst simultaneously improving academic engagement, serving a
motivating purpose.
Other forms of technology have also found to be beneficial in improving learning outcomes for
children with ASD. Specifically, virtual reality technology was particularly effective as it was able to
offer individualised treatment – previously noted to be of special import for children with ASD – and
providing opportunities for the child to have active control (Parsons & Cobb, 2011). The children
were found to be able to meaningfully interact and respond to various stimuli during trials.
Furthermore, behaviours and responses could be practised and reinforced, and facial expressions
could be better integrated into the virtual reality experience, which helped build social and
emotional skills.
This area continues to be an emerging field of research, with a study currently underway in NSW
looking at the effectiveness of digital technology for children with ASD, between kindergarten to
Year 12 (Tanner, et al., 2010). It is expected that in the coming years there will be a growing interest
in conducting longitudinal surveys and research in this field, to isolate the real effect of digital
technology in improving learning outcomes for children with ASD.
128
Deloitte Access Economics
ELLA Trial Evaluation Final Report
A growing acceptance
As the evidence base supporting digital technology as an education tool grows, and as society
becomes more accustomed to the integration of technology in everyday life, it is anticipated that the
historical concerns of parents and educators will be further dispelled. Supporting this, a study
conducted across several European countries regarding parental attitudes towards technology found
that rather than feeling concern towards the use of technology in preschools, parents felt a level of
anxiety around the lack of use of technology.
Specifically, the study found that ‘parents felt there was a ‘generational digital gap’ between how
their children interact with digital technologies at home and in their early years settings’. Whilst the
parents still believed in and valued teachers and the classroom, a desire was expressed for
classrooms with no boundaries that ‘bound the home experience with that to be found in early years
settings’ and stated that pedagogical practices needed to incorporate the changing technologies
(Palaiologou, 2014).
3. Best practice approaches
Given that both language education and the use of tablets are relatively new additions to early
childhood teachers’ pedagogical repertoires there is a level of uncertainty surrounding:

how to best integrate tablets into a preschool environment;

which app design features are most useful for preschool students; and

the best techniques for teaching a second language in preschools.
This section explores the findings of previous programmes and studies in relation to best practice
digital technology approaches in the preschool setting, and early learning language techniques.
Digital technology integration in preschools
The emergence of touchscreen devices has expanded the role digital technology can play in early
childhood education. Several programmes have introduced tablets into both preschool, primary and
secondary school classrooms, and have found that tablets have a broadly positive impact on student
educational outcomes. While this literature review focusses on the integration of digital technologies
in preschools, findings from the younger year levels of primary schools have also been included to
ensure all relevant learnings were canvassed.
Lesson 1 – Tablets do not replace good teaching
Tablets have proven to be an effective and engaging learning tool. However, studies have found that
tablets works best as an instrument to support, rather than replace, teachers. To achieve the best
outcomes, teachers should organise activities which guide and support students, but also allow some
scope for students to explore their own work. Apps, when designed well, create an environment of
enquiry for students, and teachers reported better outcomes when some responsibility for learning
fell on the students (I & J Management Services, 2011).
As digital technology is still relatively new, teachers may require support in expanding their teaching
techniques to unlock the potential of new tablet technologies in early childhood education (Yellend &
Gilbert, 2013). However, given the myriad of ways a tablet can be used, there is no ‘typical’ lesson
129
Deloitte Access Economics
ELLA Trial Evaluation Final Report
structure which can be detailed, therefore no single ‘best method’ for teachers using tablets
(Goodwin, 2012).
Lesson 2 – Provide a range of pedagogical support for teachers
As mentioned in Lesson 1, the notion of best pedagogical practices in relation to digital technology in
education is still evolving. Consequently, most programme managers developed a range of tools to
support teachers and students (Western Australian Department of Education, 2014; I & J
Management Services, 2011; Oakley et al, 2013).
The most popular tool developed was a forum for educators to suggest new ideas and discuss
educational techniques, which allowed educators to improve the quality of the educational
experience they could provide. This was established in six of the programmes listed in 0. Other tools
used are listed below:

At least three programmes produced a suite of support resources such as handbooks, tips and
guides, resource lists, protocols and research summaries. These were provided directly to
schools, and also uploaded to a website dedicated to the programme.

The Victorian programme iPads for learning: In Their Hands established a twitter account, to
increase the channels of communication between the programme managers and participating
schools.

Five programmes provided professional learning days for teachers participating in the tablet
programme, for teachers to develop their skills, and learn how to introduce digital technology
to the classroom.

The Early childhood iPad initiative in Western Australia appointed a project coordinator, who
was the first point of call for trial schools if they encountered any issues.

The AlphaEU project in Europe made e-learning modules available in order to inform and train
teachers and classroom facilitators (Lazar, 2014).
These tools were used to support a smooth implementation of the programmes. Some schools noted
there was a significant cost associated with the time committed to educating teachers and to gaining
familiarity with the apps. Support tools were found to reduce these time costs, and improve teacher
quality (Western Australian Department of Education, 2014).
An alternative method of pedagogical support is through the provision of technology to complement
the tablet, to enhance children’s learning experience. For example, children’s learning outcomes
were found to improve when their tablets were integrated with other digital technologies, such as
interactive whiteboards and Apple TVs (Western Australian Department of Education, 2014).
However, such technology is costly to provide.
Lesson 3 – Provide technical support to schools
Technical issues can often arise with the use of digital technology, so programme managers often
provided technical support for schools. The literature acknowledged the importance of solving
technical issues promptly, as inoperative technology deters children from using digital devices
(Goodwin, 2012). Two types of technical problems have been identified: network issues and software
issues (Oakley et al, 2013).
130
Deloitte Access Economics
ELLA Trial Evaluation Final Report

Network issues, such as internet connectivity, are particularly a problem for remote schools.
The tablet’s usefulness as a teaching tool may be limited in cases where internet connectivity
is inadequate.

Software and hardware issues, such as an app freezing, or issues with iTunes accounts, can
affect schools regardless of location. Throughout the trials, technical issues were solved more
efficiently if the programme managers organised specialist resources dedicated to providing
support, rather than relying on general support (such as Apple staff) to solve technical issues.
Lesson 4 – Tablets are most effective when used in a supportive school and home environment
Student outcomes were generally better when the teacher used the app as an enabler to broader
lessons. Ideally, the teacher would encourage activities that transfer a child’s interaction with the
app into the real world, enabling children to consolidate what they have learned (Goodwin, 2012).
In the Victorian trial, iPads for Leaning: In their Hands, schools made deliberate efforts to promote
the trial to parents, finding that widespread parental support is an important part of using iPads in
education. As the iPad is a mobile device, and apps are typically readily available through the Apple
app store, students were encouraged to continue using the apps at home with their parents.
Educators found students learned better when parents were included in the educational process.
To keep parents engaged and informed, some schools provided regular emails and newsletters
encouraging parents to explore the apps with their children. However, schools also acknowledged
the risks in relying on parental support, as parents were sometimes found to be dismissive of digital
technology use in education. This was particularly the case if tablets were not used by parents and
were an unfamiliar concept. As such, to the extent that it is possible, parents should be informed of
the positive influence tablets can have on child development and educational outcomes.
Lesson 5 – The ratio of tablets to students matters
Early childhood students were found to work best when sharing one tablet between several students
(Oakley et al, 2013). However, there is a trade-off to consider when deciding the optimum number of
tablets per children. Fewer tablets will encourage greater collaboration and communication between
children (two important benefits of using tablets in an early childhood education setting), but will
also allow less scope for children to personalise their learning experience, as they experience less
face-time with the app. As learning often comes from interaction with the tablet, some teachers
report uncertainty as to whether the benefits accrue to students who are merely watching (Oakley et
al, 2013).
It has also been found that in order to ensure each child is able to enjoy the benefits of interacting
with the iPad, ground rules may need to be established by teachers, to encourage children to share
the iPads equitably (McPake and Stephen, 2016).
Lesson 6 – The choice of app is important
In previous programmes, programme managers provided spending money for teachers and students
to purchase publicly available apps. Teachers would therefore be required to research appropriate
apps for children, increasing the time spent with the tablet (Western Australian Department of
Education, 2014). There is also the risk students waste time by using ineffective and poorly designed
apps. However, the upside to this approach is that students can personalise their learning experience
131
Deloitte Access Economics
ELLA Trial Evaluation Final Report
due to the wide range of educational apps available (I & J Management Services, 2011). The greater
choice and flexibility afforded to students was associated with positive outcomes, as students will
choose content most relevant to them.
Teaching a second language in preschools
The research describes various approaches towards teaching languages to preschool aged children.
Some accepted techniques in supporting language learning amongst young children include:

Play based learning is important in early language learning - it has been recommended that
play be utilised as a method of teaching young children to develop language abilities, with one
study finding that ‘play is considered to be a powerful, flexible, amusing and pleasant learning
experience which promotes oral communication and interaction’ and encourages children to
use language in meaningful exchanges (Griva & Sivropoulou, 2009).

Repetition – this gives the child more than one opportunity to understand what is being said
(Clarke, 2009). The greater the frequency with which a child hears new words and sounds, the
more familiar they become and the easier it is for them to learn the language (Harris et al,
2011). One of the key steps in language learning is identifying and recognising words, sounds
and basic language structures, which is facilitated by greater engagement (Lazar, 2014).

Develop listening skills – preschool children are generally preliterate, and therefore need to
hear the language spoken in order to learn it. A common approach to encourage children to
listen is to rely on songs, rhymes and games, which ensure children maintain their attention
(Clarke, 2009). Studies have found that children quickly become familiar with the gist of stories
and teachers can adopt a question-and-answer style of storytelling to encourage language use
(Mhathuna, 2008).

Using contextualised language – learning is aided through the use of hands-on activities and
visual materials (Clarke, 2009). Visual cues, facial expressions and hand gestures can all provide
meaningful contextual clues to children, which can support children in understanding new
words and phrases. Learning by doing is also an important part of contextualised learning
(Harris et al, 2011).

Social and cultural context - It can also be beneficial to provide social and cultural context to
language learning. The use of cultural symbols, music and food encourages engagement
among children, as well as reinforces opportunities to develop intercultural understanding.
This is particularly relevant for educators who have little experience in language teaching
(Nemeth and Simon, 2013).

Use of decontextualised language – as language skills develop, students may be able to learn
without relying on visual materials (Clarke, 2009). An example of decontextualised learning is
through dialogic reading, where children learn through developing stories, or recalling past
events. Educators should question or prompt children, encouraging them to actively
participate in a dialogue with their teacher (Parish-Morris et al, 2013).

Language production – as their skills develop, children should be encouraged to use the
second language (Clarke, 2009).
Optimal app design features for early learners and language education
As mentioned above, studies have so far focussed on the use of tablets as an educational tool, with
limited research into app-specific features that encourage learning among preschool aged children.
The ELLA trial is unique in that an app has been designed specifically for the trial; it is therefore
132
Deloitte Access Economics
ELLA Trial Evaluation Final Report
important to determine the optimal design features for apps to support early childhood education.
Features listed below were found to improve child educational outcomes consistently across most, if
not all, programmes.

Allowing content creation - at an early childhood level, teachers and students generally
expressed a preference for content-creation apps, also referred to as ‘productivity’ or
‘constructive’ apps, instead of content receiving, or instructive apps (see for example Yellend
and Gilbert, 2013, and Oakley at al, 2013).21 Such app design has been found to pique curiosity
and engender commitment from children, which facilitates longer-term learning (Agudo, Rico
& Sanchez, 2015).
The app should allow for directed (or modelled) play, so teachers can direct and assist children,
and open ended play, allowing children to explore as they wish. The app should also allow for
framed play –a combination of directed and open-ended play. All three play types have proven
to be effective in delivering outcomes to preschool children.

Mixed learning methods - the apps should provide access to multimodal learning. In other
words, there should be opportunities for children to learn via visual, spatial, oral and aural
means (Yellend and Gibert, 2013). This is particularly pertinent for language exposure, as
relative to relying on traditional educational tools (such as story books), the technology can
increase oral and aural learning opportunities (Strickland & Grantham, 2013).

Personalised pathways - the app should provide multiple tasks and pathways for children to
complete, allowing them to personalise their learning experience. Early child outcomes
improve if a child has some control over their learning, as they can adopt practises best suited
to themselves, and develop at their own pace (I & J Management Services, 2011).

Engaging content - the apps should be engaging and encourage persistence. To fulfil the
tablet’s potential as a learning device, the tasks within the app should be sufficiently complex
that children maintain attention and do not get bored with the content. On the other hand,
the content should not be too challenging, as children will not use the device if they do not
understand the activities (Goodwin, 2012). The tasks within the app should be interesting, fun,
and something children can relate to. If possible, the children should be able to replicate
activities within the app in the real world.

Flexibility in play options - the apps should provide context and opportunities for solitary and
social play (See for example, Yellend and Gilbert, 2013 and Oakley et al, 2012). A key feature of
childhood learning with tablets is that the preferred apps encourage collaboration, interaction
and peer to peer learning among children. The apps should also stimulate discussion between
children, educators and parents. Alternatively, children may want to use the app by
themselves, which can also provide educational benefits for the child. The optimal app design
will therefore allow for opportunities for individual play as well.

Appropriate hardware - the display needs to be appropriately sized, to get maximum benefit
from tactile interactions.

Encourage skill development - playing with the apps should provide an opportunity to
encounter and use foundational skills for learning (Yellend and Gilbert, 2013). In the case of
the ELLA trial, the apps should allow children opportunities to practise their language and
literacy skills, not just to listen and watch the characters speak another language.
21
A constructive app is more open-ended, and allows users to create their own content, while an instructive app involves
the app delivering a predetermined task, which elicits a homogenous response from the user.
133
Deloitte Access Economics
ELLA Trial Evaluation Final Report
4. Implications for the ELLA trial
The findings of the literature review largely support the rationale for, and the design of, the ELLA
trial. The introduction of languages is consistent with bipartisan policies directed to increasing
language uptake in Australia, while the focus on Asian languages is consistent with the policy
objective of increasing interaction with Asia.
The ELLA trial design, while a novel concept, appears prima facie to be consistent with features of
optimal programme design located within the literature. There are three key reasons this is
considered to be the case:

Introducing a language in preschool, as opposed to in the later years of schooling, brings a
range of identified benefits including an increased ease in second language acquisition, more
native pronunciation and increased learning outcomes in other areas.

Digital technologies have been found to increase learning outcomes in preschools, particularly
given children’s increasing digital literacy and the ability for children to work at their own pace.

Digital technology has been identified as a method through which to increase second language
learning in early years, in the absence of a second-language speaking teacher.
The literature did identify several questions which will be worth considering throughout the ELLA
trial in order to maximise learning outcomes. These include:

Is the introduction of iPads to the classroom supported with the following predicators of
success:
•
a range of pedagogical options for the teacher to utilise;
•
high levels of technology support;
•
linkages to broader lessons and integrations with the real world;
•
high levels of parental support; and
•
a suitable iPad:child ratio?

Does the app and supporting language activities include repetition, stories, songs, rhymes,
games, contextualised language and encourage language production?

Does the app design allow for multi-modal learning, engage children, encourage persistence,
provide for both social and solitary play and give opportunities to practice speaking the
language?
134
Deloitte Access Economics
ELLA Trial Evaluation Final Report
References
ACARA. (2011). ‘The Shape of the Australian Curriculum: Languages’, Australian Curriculum
Assessment and Reporting Authority.
Agudo, J. E., Rico, M. & Sanchez, H. (2015). Multimedia games for fun and learning English in
preschool, Centro Universitario de Mérida, Universidad de Extremadura, Spain.
Arnott, L. (2013), ‘Are we allowed to blink? Young children’s leadership and ownership while
mediating interactions around technologies’, International Journal of Early Years Education,
Volume 21. Number 1.
Arthanat, S., Curtin, C. & Knotak, D. (2013). ‘Comparative observati0ns of learning engagement by
students with developmental disabilities using an iPad and computer: a pilot study’.
Asia Education Foundation. (2011). ‘National Statement on Asia Literacy in Australian Schools 2011–
2012’.
Australian Government. (2011). ‘White Paper on Australia in the Asian Century’.
Australian Government. (2013). ‘The Coalition’s Policy for Schools: Students First’.
Baker, C. (2006). ‘Foundations of Bilingual Education and Bilingualism’, Clevedon: Multilingual
Matters.
Bebell, D., & Pedulla, J. (2015). ‘A quantitative investigation into the impacts of 1:1 iPads on early
learner’s ELA and math achievement’. Journal of Information Technology Education:
Innovations in Practice, 14, 191-215.
Bialystok, E & Craik,F. (2010). ‘Cognitive and Linguistic Processing in the Bilingual Mind’, Current
Directions in Psychological Science 19: 19
Bialystok, E., Craik,F & Freedman, M. (2013). ‘Delaying the onset of Alzheimer disease: Bilingualism as
a form of cognitive reserve’, Neurology. 75(19): 1726-1729
Bird, J. (2013), ‘A world of digital possibilities’, Every Child, Volume 19, Number 4.
Clarke, P. (2009). ‘Supporting Children Learning English as a Second Language in the Early Years (birth
to six years)’, Victorian Curriculum and Assessment Authority.
Clutch (2015) ‘Cost to Build a Mobile App: A Survey’, accessible at https://clutch.co/appdevelopers/resources/cost-build-mobile-app-survey
Crozet, C. & Liddicoat, A. (1997). ‘The challenge of intercultural language teaching: engaging with
culture in the classroom,’ In: Lo Bianco, J., Liddicoat, A. and Crozet, C. (eds), Striving for Third
Place. Intercultural Competence Through Language Education. Canberra: Language Australia,
pp 113-125
135
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Department of Education, Employment and Workplace Relations. (2009). ‘Belonging, Being and
Becoming: The Early Years Framework for Australia’, Canberra: Australian Government.
Forbes, E. (2013), ‘How can digital technology work for you’, Every Child, Volume 19, Number 4.
Gimbert B. & Cristol, D. (2004), ‘Teaching Curriculum with Technology: Enhancing Children’s
Technological Competence During Early Childhood’, Early Childhood Education Journal, Volume
31. Number 3.
GO8. (2007). ‘Languages in Crisis: A Rescue Plan for Australia’, The Group of 8 Universities
Gold, T., Kim, C., Johnson, N., Kryscio, R. & Smith, C. (2013). ‘Lifelong Bilingualism Maintains Neural
Efficiency for Cognitive Control in Ageing’, The Journal of Neuroscience 33(2): 387-396
Goodwin, K. (2012). ‘Use of Tablet Technology in the Classroom’, State of New South Wales,
Department of Education and Communities
Graddol, D. (2006) English Next. UK: British Council – available at:
http://englishagenda.britishcouncil.org/sites/ec/files/books-english-next.pdf
Granena, G. (2013), ‘Individual Differences in Sequence Learning Ability and Second Language
Acquisition in Early Childhood and Adulthood’, Language Learning, Volume 63, Number 4.
Griva, E. & Sivropoulou, R. (2009), ‘Implementation and Evaluation of an Early Foreign Language
Learning Project in Kindergarten’, Early Childhood Education Journal, Vol 37.
Harris, J., Golinkoff, R. M. and Hirsh-Pasek, K (2010), ‘Lessons from the Crib for the Classroom: How
Children Really Learn Vocabulary’, In S. B. Neuman & D. K. Dickinson (Eds.) Handbook of Early
Literacy Research, New York: Guilford Press.
Hsin, C., Li, M. C., Chin-Chung, T. (2014), ‘The Influence of Young Children’s Use of Technologyon
their Learning: A Review’, Educational Technology & Society, Vol 17, Number 4.
I & J Management Services. (2011). ‘iPads for Learning - In Their Hands Trial: Evaluation Report’,
Report for the Victorian Department of Education and Early Childhood Development.
Jones Diaz, C. (2014), ‘Institutional, material and economic constraints in languages education :
unequal provision of linguistic resources in early childhood and primary settings in Australia’,
International Journal of Bilingual Education and Bilingualism, Volume 17, Number 3.
Johnson, G. M. (2013). ‘Using tablet computers with elementary school students with special needs:
the practices and perceptions of special education teachers and teaching assistants’. Canadian
Journal of Learning and Technology. 1-12.
Kinvey (2014) ‘State of Enterprise Mobility Survey’, accessible at
http://resources.kinvey.com/docs/State+of+Enterprise+Mobility+Survey+2014+-+Kinvey.pdf
Lazar, A. (2014). Pre-primary educators and the use of digital resources to raise early foreign
language awareness. Procedia –Social and Behavioural Sciences
136
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Lo Bianco, J., & Slaughter, Y. (2009). ‘Second Languages and Australian Schooling’, Australian Council
for Educational Research.
McPake, J. & Stephen, C. (2016). New technologies, old dilemmas: theoretical and practical
challenges in preschool immersion playrooms, Language and education, 30:2, 106-125
Mhathuna. M. M. (2008). ‘Supporting children’s participation in second-language stories in an Irish
language preschool’, Early Years, Volume 28, Number 3.
Neely, L., Rispoli, M., Camargo, S., Davis, H. & Boles, M. (2013). ‘The effect of instructional use of an
iPad on challenging behaviour and academic engagement for two students with autism’.
Research in Autism Spectrum Disorders, 509-516.
Nemeth K. & Simon F. (2013). Using technology as a teaching tool for dual language learners in
preschool through grade 3. Young children, pp 48-52.
Neumann, M. & Neumann, D. (2014), ‘Touch Screen Tablets and Emergent Literacy’, Early Childhood
Education Journal, Volume 42.
Oakley, G., Pegrum, M., Faulkner, R. & Striepe, M., (2012), ‘Exploring the Pedagogical Applications of
Mobile Technologies for Teaching Literacy’, Report for the Association of Independent Schools
of Western Australia.
Oakley, G., Pegrum, M., & Faulkner, R. (2013). ‘Schools going mobile: A study of the adoption of
mobile handheld technologies in Western Australian independent schools’, Australasian
Journal of Educational Technology, 66-81.
Palaiologou, I. (2014), ‘Children under five and digital technologies: Implications for early years
pedagogy’, European Early Childhood Education Research Journal.
Parish‐Morris, J., Mahajan, N., Hirsh‐Pasek, K., Golinkoff, R. M., & Collins, M. F. (2013), ‘Once upon a
time: parent–child dialogue and storybook reading in the electronic era’, Mind, Brain, and
Education, 7(3), pp. 200-211.
Parsons, S. & Cobb, S. (2011). ‘State-of the art of virtual reality technologies for children on the
autism spectrum’. Journal of Special Needs Education, 26:3, 355-366.
Peterson, M. (2010), ‘Massively multiplayer online role-playing games as arenas for second language
learning’, Computer Assisted Language Learning, Volume 23, Number 5.
Stephen, C., McPake, J. & McLeod, W., (2012), ‘Playing and learning in another language: Ensuring
good quality early years education in a language revitalisation trial’, European Early childhood
Education Research Journal, Volume 20, Number 1.
Stewart, J. H. (2005), ‘Foreign Language Study in Elementary Schools: Benefits and Implications for
Achievement in Reading and Maths’, Early Childhood Education Journal, Volume 33, Number 1.
Strickland, B. & Granthman, M. (2013), ‘A review of the literature on technology in second and
foreign language learning’, The Language Research Centre at the University of Calgary.
137
Deloitte Access Economics
ELLA Trial Evaluation Final Report
Tanner, K., Dixon, R., & Verenikina, I. (2010). ‘Digital technology in the learning of students with
autism spectrum disorders (ASD) in applied classroom settings’.
‘Technology and Interactive Media as Tools in Early Childhood Programs Service Children from Birth
through to Age 8’, (2012), A joint position statement of the National Association for the
Education of Young Children and the Fred Rogers Centre for Early Learning and Children's
Media at Saint Vincent College.
Tracey, D. H., Young, J. W. (2006), ‘The impact of an Integrated Learning System on High-Risk
Kindergartners’ Achievement’, Technology and Early Literacy.
Western Australian Department of Education. (2014). ‘Teach, Learn, Share’, Western Australian
Department of Education.
Yellend, N., & Gilbert, C. (2013). ‘iPlay, iLearn, iGrow’, Melbourne: Victoria University
138
Deloitte Access Economics
Limitation of our work
General use restriction
This report is prepared solely for the internal use of the Department of Education. This report is not intended to
and should not be used or relied upon by anyone else and we accept no duty of care to any other person or
entity. The report has been prepared for the purpose of the ELLA trial evaluation. You should not refer to or use
our name or the advice for any other purpose.
Contact us
Deloitte Access Economics
ACN: 149 633 116
Level 2
8 Brindabella Circuit
Brindabella Business Park
Canberra Airport ACT 2609
GPO Box 823
Canberra ACT 2601 Australia
Tel: +61 2 6263 7000
Fax: +61 2 6263 7004
Deloitte Access Economics is Australia’s preeminent economics advisory practice and a
member of Deloitte's global economics group. For
more information, please visit our website
www.deloitteaccesseconomics.com.au
About Deloitte
Deloitte refers to one or more of Deloitte Touche
Tohmatsu Limited, a UK private company limited
by guarantee, and its network of member firms,
each of which is a legally separate and
independent entity. Please see
www.deloitte.com/au/about for a detailed
description of the legal structure of Deloitte
Touche Tohmatsu Limited and its member firms.
Deloitte provides audit, tax, consulting, and
financial advisory services to public and private
clients spanning multiple industries. With a
globally connected network of member firms in
more than 150 countries, Deloitte brings worldclass capabilities and deep local expertise to help
clients succeed wherever they operate. Deloitte's
approximately 200,000 professionals are
committed to becoming the standard of
excellence.
About Deloitte Australia
In Australia, the member firm is the Australian
partnership of Deloitte Touche Tohmatsu. As one
of Australia’s leading professional services firms.
Deloitte Touche Tohmatsu and its affiliates provide
audit, tax, consulting, and financial advisory
services through approximately 6,000 people
across the country. Focused on the creation of
value and growth, and known as an employer of
choice for innovative human resources programs,
we are dedicated to helping our clients and our
people excel. For more information, please visit
our web site at www.deloitte.com.au.
Liability limited by a scheme approved under
Professional Standards Legislation.
Member of Deloitte Touche Tohmatsu Limited
© 2016 Deloitte Access Economics Pty Ltd