Chapter 4
Section
Ministry of Education
4.07 Literacy and Numeracy
Secretariat
Follow-up on VFM Section 3.07, 2009 Annual Report
The Ministry of Education (Ministry) is responsible
for the system of publicly funded elementary
and secondary school education in Ontario. Its
responsibilities include developing the primary and
secondary school curricula, setting requirements
for student diplomas, and providing funding
to school boards. The Ministry also set up the
Education Quality and Accountability Office
(EQAO)—a government agency—to provide
independent assessments of student achievement by testing students in reading, writing,
and mathematics. The Ministry’s Literacy and
Numeracy Secretariat (Secretariat), the subject
of one of our 2009 audits and this follow-up, was
established in November 2004 to help more than
4,000 elementary schools across 72 school boards
to meet student-achievement targets. From the
time it was established in 2004 to March 31, 2011,
the Secretariat spent $505 million ($340 million
by March 31, 2009), with almost $423 million
($288 million by March 31, 2009) transferred to
school boards.
The Ontario government made a significant
commitment to improving student achievement
when, in 2004, it set a goal that 75% of all 12-yearolds (grade 6 students) would score a level-three
Chapter 4 • Follow-up Section 4.07
Background
standard (approximately a B average) on provincewide testing for reading, writing, and mathematics
by 2008. Although the Ministry had not achieved
this goal by 2008, substantial progress had been
made over the five years previous to our 2009 audit,
and the percentage of 12-year-olds achieving the
provincial standard increased from an average of
56% in 2003/04 to 65% in 2007/08. We stated at
the time of our audit in 2009 that further increasing
this percentage would be a challenging undertaking
and noted a number of improvements that could be
made to help achieve this goal. Some of our more
significant observations at the time of the 2009
audit were:
Although the Secretariat and the school
boards we visited had done some limited
assessment of the effectiveness of the secretariat programs, further analysis was required
if the Secretariat was to ensure that its funding of almost $288 million had been directed
to the initiatives that provide the most benefit.
School board improvement plans had been
initiated to help teachers, principals, and
school board staff plan and implement
strategies to improve student achievement.
The Ministry had developed a framework for
an effective improvement-planning process.
However, neither the Secretariat nor the
boards we visited documented, monitored, or
•
•
355
356
2011 Annual Report of the Office of the Auditor General of Ontario
reported on the plans to the extent necessary
to assess whether the plans were contributing to improved student achievement. Also,
because it exercised only limited oversight,
the Secretariat did not have the information
needed to identify patterns and trends among
school boards, so it could not identify the
most successful initiatives and share them
with other boards.
Secretariat program funding was not always
allocated to school boards and schools with
the greatest need. Rather, funding allocation
was based on average daily enrolment or the
reason a given amount of funding went to a
school board could not be fully explained by
the Secretariat. We found that, for one major
program, the board with the greatest number
of schools designated as low-performing
received only $17 per student, while several
boards with no schools designated as lowperforming received more than twice this
amount per student.
The Secretariat routinely uses certain boards
as “bankers” to act as distributors of funds
to third parties or other school boards. We
questioned the need for such arrangements
and noted that there is no memorandum of
understanding or agreement between the
Secretariat and the banker boards outlining
respective roles and responsibilities, accountability relationships, reporting requirements,
and service levels to be provided. Also, the
Secretariat paid banker boards administrative
fees that in some cases appeared excessive.
We made a number of recommendations for
improvement and received commitments from the
Ministry that it would take action to address our
concerns.
•
Chapter 4 • Follow-up Section 4.07
•
STANDING COMMITTEE ON PUBLIC
ACCOUNTS
The Standing Committee on Public Accounts held
a hearing on this audit in May 2010. In Decem-
ber 2010 the Committee tabled a report in the
Legislature resulting from this hearing. The report
contained six recommendations and requested that
the Ministry of Education report back to the Committee with respect to the following:
what measures the Ministry is considering to
make better use of cohort tracking to assess
the progress of the same group of students
over time;
the criteria the Ministry uses to assess school
board improvement plans, including an
update on the template that the Ministry has
developed for the boards to use and how it
plans to communicate best practices to all
school boards;
the results of the Ministry’s review of the
effectiveness of the Secretariat’s various programs, specifying the review criteria used, the
results of previous reviews, and any program
changes that have resulted from the reviews;
the Ministry’s most recent data on the amount
of funding allocated to secretariat programs
on the basis of needs and the amount based
on enrolment, its criteria for assessing
whether program money has been spent
effectively, whether it has identified any
program funding that should be redirected,
and the percentage of total funding that was
actually spent on the intended services;
an interim report from the Ministry on
its review of lead or banker board use,
specifying:
• whether the Ministry is on track to complete its review by the end of the 2010/11
fiscal year;
• what initiatives formerly administered by
lead/banker boards have been returned to
the Ministry;
whether
the Ministry will continue to use
•
lead/banker boards and, if so, what criteria
it will use to select them and to monitor
their expenditures; and
what
criteria the Ministry will use to deter•
mine the appropriate levels of payment for
•
•
•
•
•
Literacy and Numeracy Secretariat
services provided by lead/banker boards
and for reviewing their expenses; and
an update of the Ministry’s assessment of data
from the study it commissioned to compare
the consistency of students’ report card marks
with their grade 3 and grade 6 EQAO achievement results and whether the Ministry will
consider ongoing tracking of the correlation.
The Ministry responded to the Committee in
March 2011. A number of the issues raised by the
Committee were similar to our observations. Where
the Committee’s recommendations are similar to
ours, this follow-up includes the recent actions
reported by the Ministry to address the concerns
raised by both the Committee and our 2009 audit.
•
357
MEASURING AND REPORTING ON
PROGRAM EFFECTIVENESS
Recommendation 1
The Ministry of Education should develop more comprehensive indicators for measuring and reporting on
its effectiveness in improving student achievement. In
addition to reporting the percentage of 12-year-olds
who are at or above the provincial standard, it should
also consider reporting changes in the gap between
the top-performing and lower-performing student
groups and schools, as well as how specific student
cohorts perform over time while participating in the
programs and initiatives intended to improve their
performance.
Status of Actions Taken on
Recommendations
According to information received from the
Ministry, substantial progress has been made on
implementing all of the recommendations in our
2009 Annual Report. For example, the Ministry
reported the completion of thorough reviews of
the board improvement-planning process, of all
the Secretariat’s program initiatives, and of the
correlation between report card marks and EQAO
results. These reviews resulted in, for example, the
development of new criteria for the board planning
process and the elimination, revision, or expansion
of several programs. The Ministry also completed
the implementation of the Ontario Statistical
Neighbours System, a system with demographic,
school, and student performance information used
for program planning, and the Ministry is training
all boards in how to use it. The status of the action
taken on each of our recommendations at the time
of our follow-up was as follows.
The Ministry informed us that it uses EQAO test
results over time as one method for measuring
improvements in student learning and achievement
in grades 3 and 6. Cohort tracking now provides the
Ministry with an additional way to assess student
progress and gives some insight into the effectiveness of the Ministry’s previous efforts to support
students. Such tracking has also become one of the
means to identify areas where performance could
be improved, so that information can be developed
about how to focus support on these identified
areas of need.
Now that the EQAO has begun to regularly track
cohorts of students, the Ministry has determined
that most cohorts have been improving in reading,
writing, and mathematics. For example, of the
120,000 students who attended school in Englishlanguage school boards and who wrote both the
grade 3 assessment in 2006/07 and the grade 6
assessment in 2009/10, 10% more met the standard
in reading in grade 6 than in grade 3, 6% more
met the standard in writing, but 8% fewer met the
standard in mathematics. For the 5,700 students
who attended school in French-language school
boards and who wrote both assessments, the results
for reading, writing, and mathematics improved by
24%, 6%, and 20%, respectively.
Chapter 4 • Follow-up Section 4.07
Status
358
2011 Annual Report of the Office of the Auditor General of Ontario
The Ministry informed us that cohort-tracking
data will continue to be used going forward from
2009/10 as a needs assessment and improvement
tool for school boards. The Ministry noted that it
uses EQAO test results over time for its planning
purposes and for selecting priorities. The Ministry
also tracks the proportion of students at each EQAO
performance level, one through four (similar to
letter grades). Level three, equivalent to a B grade,
is an indication that the student has demonstrated
most of the required skills and is currently at the
provincial standard. Cohort tracking also includes
analysis of specific groups of students, such as
males and females, English- and French-language
learners, and students with special needs. It compares the results for specific groups such as these
to overall student results to help inform its plans to
reduce the gaps in achievement.
The Ministry also indicated that it has introduced changes to the assessment, evaluation, and
reporting of elementary student achievement by
implementing a new fall progress report that allows
teachers to give parents personalized feedback on
each student’s strengths and on steps required for
improvement without grading students.
Chapter 4 • Follow-up Section 4.07
SCHOOL BOARD IMPROVEMENT PLANS
Recommendation 2
To ensure that the improvement-planning process is
sufficient to support boards, administrators, principals, and teachers in helping students to improve
results and progress toward the provincial standard
in achievement testing, the Ministry of Education
should:
implement a formal improvement-plan review
process to help ensure that all of the necessary
components of an effective plan are included;
require that school boards post improvement
plans online to enhance accountability and
transparency;
consider adopting the practice followed in some
other provinces of using a formal contract with
school boards that would require school boards
•
•
•
to periodically report their results in achieving
the goals in their improvement plans; and
properly document the result of its monitoring
efforts along with any required corrective action
to be taken and any subsequent follow-up where
plans are not complete.
•
Status
We were informed that a ministry working group
put in place a formal improvement-plan review
process that articulates the criteria for good board
improvement plans and provides an assessment
template for improvement planning. These criteria
include:
assessment of student needs and
achievements;
identification of SMART (specific, measurable, attainable, results-oriented, time-bound)
goals;
development of indicators to measure student
achievement;
identification of required human and financial
resources;
identification of professional learning needs
for educators;
a process for data collection and analysis;
clearly defined responsibilities for plan implementation and monitoring; and
comparison of planned results with actual
outcomes.
The Ministry’s student achievement officers will
use these criteria to assess whether board goals are
appropriate and are currently being met, and will
document their review of the school board plans
and provide feedback to boards that includes information on best practices.
The Ministry has also developed a refined version of its School Effectiveness Framework to be
used in the board improvement planning process to
provide guidance for assessing student and professional learning needs. In addition, the Ministry
has developed and provided boards with other
resources, such as examples of SMART goals to help
set targets and develop indicators for monitoring
student achievement.
•
•
•
•
•
•
•
•
Literacy and Numeracy Secretariat
MONITORING AND FUNDING OF
PROGRAM INITIATIVES
Recommendation 3
To ensure that student achievement initiatives are
effective and that limited resources are used appropriately, the Literacy and Numeracy Secretariat should:
formally evaluate how well all its program
initiatives contribute to improving student
achievement, and modify or eliminate the less
effective initiatives;
ensure that its program funds are allocated to
the areas of greatest need;
ensure that program funds are being spent for
the intended purpose;
ensure that expenditures made by the Council
of Ontario Directors of Education are appropriately approved and supported; and
•
•
•
•
• reconsider pre-flowing funds to “banker” school
boards.
Status
The Ministry noted that, in 2010, it formalized a
three-year plan to assess the Secretariat programs
and initiatives. This review was guided by a logic
model designed to systematically assess whether
a program contributed to student learning and
achievement, by aligning program goals, inputs,
outputs, activities, and performance measures.
Overall, the Ministry indicated that it had performed an initial assessment of all its programs. As
a result of this initial review, one initiative, Leaderto-Leader, has been eliminated. Several other
initiatives have been revised or expanded since we
completed our 2009 Annual Report. Both external
and internal research has been used as input for
the assessment of student achievement initiatives,
including the following reviews:
In December 2010, an internal review was
completed on the Ontario Focused Intervention Partnership (OFIP) program, which
focuses on schools where fewer than 50% of
the students met the provincial requirement
of level three on EQAO testing. The 2009/10
results for the 137 OFIP schools showed an
overall average improvement of five percentage points, and in 62 of these schools more
than 50% of the students met the provincial
standard. The study noted that the strongest
indicator of success reported by the 62 successful schools was their self-assessment using
the School Effectiveness Framework. The
results of this study were shared with school
boards.
An external study contracted by the Ministry
and completed in November 2010, Collaborative Inquiry and Learning in Mathematics
(CILM), identified that CILM is an appropriate approach for teaching and learning
math. Based on these findings, the Ministry
expanded CILM to all boards. In addition,
internal research done by the Ministry on the
•
•
Chapter 4 • Follow-up Section 4.07
Although the Ministry advised us that it has
not formally required boards to post improvement
plans online, we were informed that all boards
have done so. The Ministry also noted that it has
not entered into any formal contracts with school
boards to periodically report their results in achieving the goals in their improvement plans. However,
revisions to the Education Act pursuant to Bill 177,
the Student Achievement and School Board Governance Act, 2009, increased the responsibility of
boards to report publicly regarding their plans to
promote student achievement and well-being.
The Ministry informed us that it has put
additional monitoring processes in place, with
staff making three annual visits to school boards
regarding their improvement plans. The first visit in
the fall helps develop the board improvement plan
and ensure that the Ministry’s criteria are included
before the final plan is submitted to the Ministry.
The second visit, in the middle of the school year, is
used to monitor the plan against the SMART goals,
indicators, and targets to identify areas that require
corrective action. The final visit, at the end of the
school year, is to document, review, and evaluate
the impact on student achievement.
359
360
2011 Annual Report of the Office of the Auditor General of Ontario
Student Work Study Teacher Initiative found
a number of factors contributing to the success of students’ work moving from level two
to level three (the provincial standard). The
Ministry has shared these success factors with
school boards.
In January 2011, another external study was
completed on the Schools on the Move initiative to identify which factors contributed to
student achievement in the 32 participating
schools that faced challenging circumstances.
In particular, it identified four factors contributing to student success that could be transferred to other schools: support for a common
code of behaviour throughout the school; a
feeling of responsibility among teachers for
student learning; teachers’ commitment to
the notion that all students can learn successfully; and the explicit teaching of literacy
skills to students. The Ministry has indicated
that it shared these findings with schools in
the Ontario Focused Intervention Partnership
program.
The Ministry advised us that it analyzes final
report-backs from the school boards to assess
whether board program funding has been allocated
to the areas of greatest need. In our 2009 audit
we noted that for almost 70% of the funding over
the previous five years, either the funding had
been based on student enrolment or the Ministry
could not fully explain its allocation method. In the
2010/11 fiscal year, the Ministry indicated that the
majority of its program funding was now allocated
on the basis of need. In addition, the Ministry
indicated that it had instituted further accountability measures for school boards by specifying the
funding criteria, reporting requirements, payment
schedules, and allowable expenditure categories
in transfer payment contracts to help ensure that
funds were spent for the purposes intended.
The Ministry advised that it recently approved
and released Administration Fee and Lead Board
Guidelines with criteria for transfer-payment
expenditures to third-party organizations, such
Chapter 4 • Follow-up Section 4.07
•
as the Council of Ontario Directors of Education
(CODE) and boards that act as lead boards.
In the guidelines, clarification is provided with
respect to the overall rationale for the selection
of lead boards, and the criteria that need to be
considered and included in determining whether
funding to a lead board is required. These include:
whether it would be more efficient and effective in delivering the services;
whether it would provide a more economical
delivery to ensure value for money; and
whether specific expertise resides with the
lead board.
•
•
•
CONSISTENCY OF STUDENT
ASSESSMENTS
Recommendation 4
To help ensure that students are being assessed in
a consistent way, the Ministry of Education should
monitor the results from different types of assessment,
especially those from report cards and Education
Quality and Accountability Office (EQAO) tests, to
identify any major discrepancies for follow-up.
Status
In March 2010, the Ministry completed a study
that examined the correlation between report
card marks and EQAO results for grades 3 and 6.
The study found that report card marks and EQAO
results appear to be reasonably well aligned and
that classroom-based assessments are relatively
consistent over time. It also found that report card
marks in the two years prior to the EQAO assessments are consistent predictors of EQAO results.
The study found that the relationship between
report card marks and EQAO results was virtually
the same for grades 3 and 6, and that assessment
practices of EQAO and teaching to the curriculum
were relatively well aligned. The study also found
that report card marks were more closely aligned
with EQAO results for female than for male
students.
Literacy and Numeracy Secretariat
The study also looked into variations between
report card marks and EQAO results based on
grade level, curriculum, and student gender. The
Ministry noted that there will be some variation in
these assessments, because the nature and purpose
of the large-scale EQAO assessment are very different from those of classroom assessment. The
EQAO assessment reflects a point in time, whereas
report cards summarize a longer period of time and
include consideration of greater specific circumstances pertaining to the student.
The Ministry is reviewing these findings and
further reviewing its capacity to perform similar
statistical analysis in the future.
ONTARIO STATISTICAL NEIGHBOURS
INFORMATION SYSTEM
361
strategies for improving student achievement, the
Ministry of Education should consider granting them
direct access to the Ontario Statistical Neighbours
information system. This would be more cost-effective
than school boards having to develop and maintain
their own systems.
Status
The Ministry indicated that it has completed the
development of the online Ontario Statistical Neighbours system, which gives school boards direct
access to useful information necessary for improving
student achievement. At the time of our follow-up,
50 of the 60 English district school boards and 11
of the 12 French district school boards had received
training from the Ministry. The Ministry expected
the training to be complete in fall 2011.
Recommendation 5
Chapter 4 • Follow-up Section 4.07
To ensure that all school boards and schools can
obtain useful and relevant information to develop