Performance Management & Information Team
County Hall
Taunton
Somerset
TA1 4DY
KEY STAGE 4 2014 - PRELIMINARY ANALYSIS
Part 2: Transition Matrix Comparison
1.
1.1
1.2
2.
2.1
2.2
3.
3.1
INTRODUCTION
The accompanying tables provide the second part of the analysis of 2014
Preliminary GCSE data. In previous years this analysis has been produced using
the Jesson Framework – but this year it compares data against the 2013 national
transition matrices. This enables data to be presented showing pupils making 3
levels of progress and those who achieve the GCSE grade C threshold.
As with the first part of this analysis it is produced to enable an early comparison
of results to be made both within and between schools, by providing data to
compliment the internal analysis of results that schools produce themselves. By
making this analysis available early it is intended to help support the process of
school improvement and raising standards – it is not intended that this data is for
public release. This note provides an explanation of the data is the tables so
please ensure that a copy is circulated with the data.
SOURCES OF DATA
The data analysis is based on the same set of GCSE subject result data that was
used in Part 1 of this analysis – ie the data available to on 27 August which
means that changes to grades made since that date will not be reflected in this
analysis. GCSE data was provided by schools from SIMS Exams Organiser for
June 2014 results together with data for previous examination sessions. In all
cases of multiple entries in a subject the pupil's best-grade is used for analytical
purposes. The subjects covered are listed in Appendix A.
Pupils’ outcomes at GCSE are plotted against their average KS2 performance
(see Appendix B), with comparisons made to the 2013 national transition matrix
tables. GCSE performance is therefore set against national outcomes in terms of
progress and pupils obtaining GCSE grade C or better. Schools can assess and
compare the performance of pupils in each band, and in the school overall,
against national data.
SCHOOL LEVEL DATA ANALYSIS
The national transition matrix tables for each subject can be used to calculate the
proportion of pupils at each KS2 level making 3 levels of progress in a GCSE
subject, and also to determine the chances of a pupil reaching GCSE grade C
from. School results from the 2013/14 year 11 are then used to compare against
national percentages (from 2013), and thereby provides an indication of where
there has been performance above or below the national expectation. The
national transition matrix tables are included as part of the spreadsheet of data.
3.2
There are two files of data which are included in this analysis which provide
school and subject level data, as set out as follows:
Levels of Progress (file: Levels of Progress Summary Data 2014.xlsx)
Based on a pupil’s average KS2 level it can be calculated whether or not a pupil
has made 3 levels of progress for each GCSE subject. (There are a small
number of pupils cannot be included in this calculation because they do not have
a KS2 result).
The school tables show the number and percentage of pupils in each subject that
have made 3 levels progress (and those that have not therefore). In addition to
overall summary data for the subject there are 4 columns showing the percentage
of pupils making 3 levels of progress from their KS2 starting point.
In addition to a table for each school, the same data is presented in a subject
table. This enables comparison with other establishments and also with the 2013
national averages and the 2014 average for Somerset secondary schools for
pupils making 3 levels of progress in each subject which appear at the top of the
table. In addition, at the right of the table is a column which flags if a school has
an overall lower rate of pupils making 3 levels of progress than the 2013 national
average in that subject.
Attainment at GCSE Grade C and above (file Grade C+ summary Data
2014.xlsx)
The same raw data is also the source of the second file of tables. Again this
provides a series of tables for each school showing the performance in each
subject and a subject level summary table to set all schools alongside one
another.
The school table shows the percentage of pupils at each KS2 level who have
reached grade C or better at GCSE. Alongside the percentage outcomes are a
series of column shown for each band the difference from the national
expectation. A negative difference figure indicating that fewer pupils than would
be expected have achieved grade C+, whilst a positive figure indicates that more
than the expected number of pupils has reached GCSE grade C. (The number of
subject entries is shown since the performance of small groups of pupils either in
total, or from a particular KS2 level, is unlikely to give a representative number on
which to draw conclusions).
Again the subject level table brings together data from each school to enable a
comparison between schools to be made. There are 2013 national averages and
2014 Somerset averages are shown at the top of the page, allowing each school
to set it performance against other schools and performance.
4.
4.1
Q&A REGARDING USE OF THE TRANSITION MATRIX DATA
The analysis is intended to provide schools with an early comparison with other
schools. As always there are limitations to the way in which the data can be
used. The following questions and answers may help in the interpretation of the
data:4.1.1 How does this analysis differ from the first part that has already been
issued? The first part of this year’s analysis was based on FFT D
estimate data which includes an inbuilt level of challenge for most schools.
The Part 2 analysis is based on transition matrix data which simply looks
at the national rate of conversion from KS2 to GCSE in each subject.
However, an additional feature of the transition matrix analysis is that it
provides a breakdown of data by KS2 level so it may identify an ability
group where there are issues or where there has been particular success.
4.1.2 What about changes to this year’s GCSE results? Schools should be
cautious about making judgements where this year’s results are known to
be missing or under review. Any amendment to the subject grades at the
C/D borderline would change the school ‘Difference’ figures on the C+
analysis, whereas any change to grade has the potential to change the
figure for pupils making 3 levels progress. Even where a school does not
expect its own results to be amended please remember that the data for
other schools may be revised.
4.1.3 We had more year 11 students entering some GCSE subjects than are
included in calculations – why is this? It is only usually possible to
calculate levels of progress for students who have KS2 data and a GCSE
result. Similarly, it is only possible to include students in a KS2
performance band if they have KS2 results.
4.1.4 Why is this analysis not provided for all GCSE subjects? In order to
provide a meaningful comparison there needs to be a large enough
number of schools and students entering the subject. The subjects that
are included are based on the number of entries in Somerset in the
previous year – these subjects account for around 93% of full GCSEs
taken in Somerset. The analysis is for the Full GCSE subjects only and is
based strictly on the specified subjects. This means a subject such as
Fine Art appears separately from Art & Design, whereas minority subjects
such as Film Studies are not included.
4.1.5 Is the analysis solely about pupils attaining GCSE grade C or better?
Whilst the Grade C+ part of the analysis focuses on pupils reaching that
key attainment level, the levels of progress tables reflect attainment
across the ability range. As always schools are advised to use a range of
data and judgements to give a rounded view of performance.
4.1.6 What can I tell about the GCSE result for individual students? The
analysis is based on the expectation that a certain percentage of a group
of pupils will either achieve a GCSE grade C+ or make 3 levels of
progress. It is designed to show whether the group of pupils as a whole
have over/under-achieved in a subject. Whilst pupils’ results lie behind
the school data, the analysis is not intended to identify individual pupils
who have over or under performed. The under lying data used to create
the analysis tables can be made available on request.
4.1.7 Can I use the analysis on its own to determine how successful our school
has been at GCSE? No performance data should be considered in
isolation – it needs to be used to support other data and judgements.
Individual schools will be aware of local issues that may have influenced
performance which are outside the scope of this analysis. The data may
point to areas worthy of further internal investigation within a school, eg
the outcomes in one subject being particularly at odds with the figures for
other subjects, and other schools, may indicate a particular strength or
weakness in a subject.
4.1.8 Are the school results likely to be distorted by the data for a few individual
students? Young people are exposed to a range of influences in the 5
years between the end of KS2 and GCSE. By focusing on the bigger
picture of attainment across a large group of students it helps reduce the
influence of outside factors. Clearly data for a larger group of pupils is
less likely to be influenced by the performance of a few individual results.
4.1.9 What should I look for when comparing the KS2 performance bands?
Because data for a small number of pupils can be distorted by a few
individual results, it is important to be sure that there are sufficient pupils
in a group to enable a fair comparison to be made. Once that is
established, a comparison can be made with: the national average; the
local authority average; performance of other schools; or that same KS2
group in other subjects. It is important to remember that the grade C
threshold will be much easier to achieve for pupils in the level 5 KS2
group than it is for those in the level 3 KS2 group. It will be seen from the
national average data, that the more able the KS2 group the higher the
rate of conversion to Grade C+. If it is found that pupils in one of the KS2
groups is consistently performing significantly worse/better than would be
expected this may indicate that this is an ability group or a curriculum area
that merits extra support/has benefited from extra support and attention.
4.1.10 In the subject tables why is data suppressed when there are less than 10
results available? For the sake of completeness, on the school tables,
data is included whenever the establishment has entries in a subject, but
on the subject summary tables data is not shown for small groups since
the data may not be sufficiently robust to be compared with other schools.
Even where the data is provided, schools should avoid over-reliance on
data for smaller groups of pupils.
4.1.11 The tables show the percentage of the cohort with results. What value is
this data? Whilst entry policy should not affect the chances of achieving
grade A*-C or the level of progress, it is always better to make a like-forlike comparison with other schools that have a similar number/proportion
of pupils entering the subject. In addition to the overall percentage, the
spread of pupils between the KS2 levels may give an indication of the
level of ability of the pupils entering that subject. eg if a school offers the
traditional science subjects of Biology, Chemistry and Physics this will be
reflected not only in a lower number of pupils entering the single sciences
but may skew that group towards the lower ability range. Also, it may be
helpful to distinguish between examinations which are entered as a
default option in one school, but which is only entered by students with a
genuine enthusiasm for the subject in another school.
4.1.12 Does it matter that the transition matrix data refers to 2013 GCSE
outcomes. Whilst dramatic change from year to year is usually unlikely,
there may have been changes to the exam syllabus or marking regime
which need to be remembered when looking at the data. If a change from
modular to linear assessment has an impact on the grades awarded the
rate of conversion to grade C, or in the levels of progress, may not follow
the pattern of previous years as set-out in the transition matrix. Schools
should look to see how other schools have performed in the same subject.
4.1.13 When looking at the pupils achieving Grade C or above what significance
should be given to the Difference figures? A large positive or negative
difference would be an area worthy of further investigation, but for some
subjects/schools the data may not reveal anything particularly significant –
eg a small negative or positive figure is probably unlikely to indicate a
major issue – it is sensible to look at how other schools have performed in
the same subject to put results into context.
4.1.14 When I add up the difference data for the individual KS2 groups achieving
Grade C+, the figure does not agree with the ‘All Pupils’ total. What have
I missed? You have missed nothing: the analysis calculates the
percentage of pupils expected to reach level C or above, for presentation
this is rounded to a whole number. Totals may not agree due to rounding
therefore.
4.1.15 Some of the figures in this analysis are slightly different to figures that we
have calculated ourselves. Why is this? There may be slight differences
in the methodology for some of the calculations (eg the way that absent
pupils are counted) but the most likely reason is the matching of KS2 and
KS4 data – eg if data is held in school computer system that could not be
obtained from FFT. This does not invalidate either set of data although
they should point to the same issues. It is advisable not to ‘mix and
match’ data from different sources.
5.
5.1
5.2
UPDATED GCSE DATA/FURTHER INFORMATION
Towards the end of September I am expecting to receive the first feed of national
data, which will be used to provide a new more detailed analysis of KS4
performance including some national comparators for 2011.
Individual pupil data is not been supplied to schools as part of this analysis,
although background data can be made available on request.
Tony Verrier
Information Officer
Performance and Management Information Team
PMIT active site at: http://www.six.somerset.gov.uk/ist/
Appendix A – Subjects Analysed
The following Full GCSE Single Award subjects have been included in the comparison of
actual subject grades against the National Transition Matrix data.
GCSE Subject
Art & Design
Art & Design: Fine Art
Biology
Business Studies
Chemistry
Drama
D&T Food Technology
D&T Graphic Prods
D&T Product Design *
D&T Resistant Materials
D&T Textiles Technology
English/English Language & Literature
English Literature
French
Geography
German
History
Mathematics
Music
Sport/PE Studies
Physics
Religious Studies
Science: Additional
Science: Core
Spanish
Statistics
Notes:
(a) The results included in this analysis are limited to the specified GCSE subjects. This
means that where schools have reported results for other similar subjects they will not
be included in the analysis (e.g. Art & Design Photography is not part of the Art &
Design).
(b) Where a pupil's GCSE result is recorded as 'absent' or with a 'result to follow'
(grades X and Q) the result is not included in the analysis.
(c) Analysis is based on Full GCSEs only, meaning that short GCSEs or double award
subjects are not included even if they are in an equivalent subject.
(d) The data analysis relates to pupils in the 2013/14 Year 11 cohort only. It includes
GCSE subject results awarded in the June 2014 examinations and previous sessions. In
all cases of multiple entries the pupil's best-grade is used for analytical purposes.
(e) *Analysis for D&T Product Design uses the Graphic Productions transition matrix
Appendix B – Key Stage 2 Sub-Level Data
Pupils are placed bands according to their average Key Stage 2 level using data
obtained from the Fischer Family Trust website (FFT Live). For English, mathematics
and science the sub-levels are converted to points and the average point score is
converted back to the nearest KS2 level as shown in the example below. The analysis
presents data whole KS2 levels although the individual pupil data is held at sub-level
should further analysis be required.
(If a student does not have a level for each of the three subjects, the average point score
is calculated from those which are available).
Example:
Subject:
English:
Maths:
Science:
KS2 Sub-Level
5C
4B
4A
Total
Average
Equivalent Sub-Level
Level for inclusion in the tables
Points
31
27
29
87
29
4A
4
For this analysis the same KS2 score is used to plot each pupil’s progress as a measure
of their overall ability. This means that in the above example a pupil’s English progress
would be measured from level 4 (their overall level) not from level 5 (their English level).
PMIT_ACV/KS4 Analysis of Transition Matrix Comparison 2014.doc
September 2014