The Illusion of Knowing
by David Didau
@LearningSpy | http://www.learningspy.co.uk/
The author of ‘What if everything you knew about
education was wrong’
BROCHURE TITLE
About David Didau
In 2011, frustrated by the state of education
David Didau began to blog. His blog, The
Learning Spy, is ranked as the second most
influential education blog in the UK and has
won a number of awards. In addition to this
Didau has also authored a number of books
including ‘The Perfect English Lesson’, ‘The
Secret of Literacy’ and his most recent book
‘What if everything you knew about education
was wrong?’ which explores the idea that
much of what happens in schools is based on
unexamined assumptions.
As well as working as a freelance writer, speaker
and trainer, Didau is currently employed by
Swindon Academy as an internal consultant
advising on curriculum design, teaching and
literacy.
2 | The Illusion of Knowing
Contents
Contents
3
Summary
5
Introduction
4
Ignorance isn’t bliss; it’s scary
4
If we collect enough data, can we predict the future?
4
Can you have too much data?
6
Perverse incentives
8
What should we be collecting and analysing?
8
How can we guard against bad data?
9
O brave new world!
9
3
Summary
“The cost of bad
data is the illusion of
knowledge”
Stephen Hawking
Working groups are looking at reducing the administrative
burden on teachers. At the same time we are told that data
on learners is at the heart of helping them to achieve their
full potential. This means more data, doesn’t it? And more
data means even more administration? Not according to
David Didau who argues that we don’t need more data, we
need better data, and that with technology this does not
necessarily mean more workload.
Introduction
The more data schools collect, the better they
will understand their students, right? Well
maybe not. Much of the data schools collect
distorts how we think and warps decisions
we make about what to teach. The illusion of
knowing is comforting, but maybe we’d be
better off confronting the uncomfortable truth
that we know far less than we suppose.
As we find stuff out, we reveal new boundaries
to what we know. As the island of knowledge
grows, so does its shoreline and beyond that
shore is a vast ocean of ignorance. Nate Silver
said “Even if the amount of knowledge in the
world is increasing, the gap between what
we know and what we think we know may be
widening.”
Ignorance isn’t bliss; it’s scary
The vast, black ocean of our ignorance makes
us uncomfortable. Most of the time, we occupy
the centre ground of the island of knowledge in
which things are safe – only rarely do we venture
on to the shoreline to peer into the void. Much
easier to deal with, and concentrate on, what is
already known. And when we’re certain of what
we know we feel even more secure. Certainty
makes us feel safe; no matter how bad it is, we
can cope.
But there are some problems with certainty. On
the whole, we’d prefer people were wrong than
unsure. We punish politicians or school leaders
for acknowledging that they don’t know and are
much happier when they’re decisive, confident…
and mistaken. We prefer erroneous information
to no information at all.
Entertaining uncertainty is uncomfortable; we
fight to feel sure.
The thing about data is that it provides us with
a powerfully seductive sense of certainty. Data
4 | The Illusion of Knowing
occupies the high ground on our island of
knowledge; it sorts and quantifies what we know
into handy, bite-size chunks which tell us what to
believe and how to act.
We make decisions based on the data we collect
in the belief that, because it’s readily available,
it’s more likely to be accurate. Of course,
sometimes the information we’re able to draw to
mind might be accurate, but sometimes it won’t
be.
Validity and Reliability
We could inoculate ourselves with a better
understanding of some key statistical concepts.
For the data we produce to be useful it must be
both valid and reliable. Briefly, reliability relates
to the accuracy and precision of what’s being
measured, while validity refers to the depth and
scope of what’s being measured. Without these
considerations we cannot be sure the data
supports us in making the inferences we want to
make.
The reliability of data is just as important as
its validity. We need to trust that the data
we gather allows us to make meaningful
inferences about what’s going on in schools.
Too often, high stakes decisions are made on
insufficiently reliable evidence. Essentially,
the higher the stakes, the more important
precise measurement becomes. And if precise
measurement isn’t possible – as is all too
often the case – we should avoid rushing to
judgement.
If we collect enough data, can we predict the
future?
There are clear limits to the power of data, but
there seems to be a common misconception
that if only we could collect a large enough
sample size we would be able to predict
anything.
We could use data to make predictions by
imagining that our prediction is true and then
working out the probability of getting the results
we’ve got. Sadly though, we end up imagining
our data is precise and accurate, and then trying
to work out the probability that our prediction
is true.
a.I have made a prediction. What is
likelihood that my prediction is true
given the results I’ve observed?
b.I have observed some results. What
is the likelihood that my prediction is
true?
This confusion is at the heart of everything that
goes wrong. Let’s reframe the problem with
more familiar terms:
a.I ate some muesli this morning. What’s
the probability I had breakfast?
b.I ate breakfast this morning. What’s the
probability I had muesli?
Suddenly, it’s much easier to see where we’re
likely to go wrong.
Added to this is the fact that the future is a
moving target. We might be able to predict
the likelihood of a set of results if nothing else
changed, but that’s not
how reality works. No amount of data can
accurately describe the complexity of human
systems. Maybe the data we collect at Christmas
might help us to predict GCSE results with
a reasonable amount of accuracy, but data
collected 5 years previously is much less
reliable as the margin of error only grows with
time: inspection frameworks change; external
assessment are changed mid-year; teachers
leave, new teachers arrive; students experience
spurts and declines. Reality gets in the way of
our careful calculations.
Because we’re so poor at dealing with
uncertainty, we struggle to accurately forecast
the future. All prediction is based on probability;
these probabilities are risks which we calculate
by analysing historical data. Uncertainty though,
is different to risk. A risk describes the possible
range of all known outcomes and tries to
calculate the likelihood of each occurring. All too
often, we don’t even know that an outcome is
possible, let alone the probability of it occurring.
Data management systems can only tell us
what has happened in the past. If the future is
different to the past – as it inevitably will be –
Before we start collecting data, we
might do well to check its validity and
reliability. Ask yourself these 4 questions:
1. How much of what I want to measure is
actually being measured?
2. How much of what I did not intend to
measure is actually being measured?
3. What are the intended and
unintended consequences of
making these measurements?
4. What evidence do I have
to support my answers to the
first three questions?
5
Schools are awash with data.
We need to know
how to interpret data
clearly, appropriately
and fairly.
any forecast we make will be wrong.
We have school dashboards, the National Pupil
Database, Raise Online, School Performance
Tables, the Fischer Family Trust, all churning
out floods of figures which must be funnelled
appropriately. Many, if not most, schools these
days employ a full time Data Manager to nourish
and nurture all these numbers. As with so
much else in life, the having of a thing is not its
purpose. Analysing spreadsheets and graphs
can be like gazing, dumbly, into a crystal ball. We
need to know how to interpret what the data
tells us. And, perhaps more importantly, we
need to know what it can’t tell us.
If data is to be used to make decisions about
setting or grouping pupils according to ability,
diagnosing what individual pupils know,
promotion or retention, or measuring the
effectiveness of instruction.
For instance, numbers assigned to graded
lesson observations are entirely and utterly
subjective, but they provide the comfortiing
illusion of hard data. As Robert Coe, Professor of
Education at Durham University says,
“If we were to use the best classroom
observation ratings, for example, to identify
teachers as ‘above’ or ‘below’ average and
compare this to their impact on student learning
we would get it right about 60% of the time,
compared with the 50% we would get by just
tossing a coin.”
Many school leaders have been seduced into the
6 | The Illusion of Knowing
easy certainties of grading lesson observations,
aggregating the grades, and then proudly
declaring that teaching in their school is 80%
good or better. But all this means is that they
like 80% of their lessons. There is no objective
standard to compare this data against. Assigning
numerical values to our preferences and biases
gives them the power of data, but they’re still
just made up.
Now that Ofsted have accepted that attempting
to grade lessons is invalid and unreliable, many
school leaders are anxious about how to go
about holding teachers to account. This is an
understandable dilemma but one with no easy
answers. What we must remember is that while
its inescapably human to make judgements, we
should avoid high-stakes decisions based on
bad data.
Can you have too much data?
Data comes with other costs. When the
government issued its call for evidence
on teachers’ workload, there was an
unprecedented response. Thousands of
teachers poured out their angst at the
uneccessary and unproductive uses to
which their precious time was being put and
56% of respondents cited the recording,
inputting, monitoring and analysis of data as
the chief culprit.
We might mourn the
passing of national
curriculum levels, but
they are an example
of made up data at its
worst
To do this we need new
systems which work for
teachers rather than
creating work for them
to do.
Frustratingly, much of the data we so dutifully
collect tells us as little as lesson observation
grades. Teacher and statistician Jack Marwood
puts it starkly:
“Just about all of the internal progress-tracking
‘data’ and external test ‘data’ which is used
in English education could actually be more
correctly described as Cargo Cult data – that is
to say, it has the appearance of real, measurable
data, without fulfilling any of the requirements
of being statistically valid, approximatelymeasured, error-accepting, countable data from
which one could reasonably draw inferences.
The fundamental problem with education data
is that it cannot be subjected to close statistical
scrutiny.”
Contrary to what most people believe, teacher
assessments cannot reliably track pupil progress
and external tests can’t be used to measure
educational achievement. Even the most
accurate
assessment is guesswork, and the constraints
of designing and marking tests mean that our
best guesses are often wrong. Tests provide
a snapshot or sample of a pupil’s actual
knowledge, skills and understanding of a given
subject. On a different day, a pupil is highly likely
to get a different mark and quite likely to be
awarded a different grade. Dylan Wiliam warns
that up to 32% of students have been given the
wrong National Curriculum levels and urges us
to be aware that “the results of even the best
tests can be wildly inaccurate for individual
pupils, and that high-stakes decisions should
never be based on the results of individual
tests.”
Someone (often a teacher) is asked to assign a
numerical value to students’ work on a regular
basis. We then pore over these tea leaves
as if they are an objective reality instead of
someone’s best guess about how a student
may have performed in a particular task on a
particular day. And then we write reports saying
with absolute certainty that ‘Emily is a 4b in
writing’ and ‘Isaac is a 5c in maths’. But what
does this actually mean? What can we know
about what Emily or Isaac can actually do? These
numbers provide the illusion of knowledge. And
on this illusory foundation we build a house of
cards with which to hold schools and teachers to
account.
We stumble into the same sort of traps when
setting targets for students. We tell them they
need to know their target grades as if they
are cast iron certainties. But while they may
not be simply plucked from the air like lesson
observation grades, they’re based on statistical
probabilities that may have some validity when
applied to large cohorts but which are reduced
to meaningless nonsense when applied to
individuals.
Currently, many companies and organisations
are desperately seeking ways to replace levels
with… more levels. It doesn’t have to be this way
– the removal of statutory assessment levels
allows schools to focus on what children can
actually do.
7
What should we be collecting and analysing?
Valuable insights can
be gained by having a
system which is fit for
purpose and provides
a bespoke lens.
Perverse incentives
The problem for schools is that data
management systems end up wagging the
teaching dog. Mass collection is a wasteful fetish,
but in the brave new world of freedom from
government shackles on assessment, schools
are free to track and monitor what students can
actually do rather than quantify vague, generic
skills. When we try to work out whether students
fit the level descriptors for level 4 in writing we
might miss the fact that they don’t understand
subject-verb agreement; rather than collecting
bogus data on the percentage of students
achieving level 5 in mathematics we could
track whether students are able to accurately
compare negative numbers.
Often, our response to a perceived difficulty is to
offer incentives. Students misbehaving and not
working hard enough? Vivo Miles! Teachers not
working their fingers down to the bloody bone?
Performance Related Pay!
If no incentive is offered we’re in the terrifying
position of simply relying on altruism. But,
if people are properly incentivised, the
reasoning goes, they will act with motivation,
determination and efficiency. We tend,
unerringly, to respond to incentives by doing
what is in our best interests. We respond to the
letter of the initiatives rather than their spirit;
we ignore the intention and focus solely on the
incentive.
Because schools and teachers are judged on
the results of the children in their charge, a
lot of time is spent testing and recording how
well students can answer questions in external
written tests on which they will be examined,
8 | The Illusion of Knowing
The middle of the school year –
March 2nd – should be central to
every teacher’s understanding of the
relative abilities of the children in a
class. This is data worth using.
but this isn’t an outcome anyone really wants or
values, but it is a logical response to a perverse
incentive.
Economist, Rolf Dobrelli offers this advice: “Good
incentive systems comprise both intent and
reward.” Clearly this is sage advice, but how so
to do? When managing change, we tend to focus
on problems. We look at what is preventing
people from behaving in the way we want
them to behave rather than focusing on those
instances of success. The best way to change
behaviour is to focus on these positives instead
of all the frustrating negatives.
An accountability system that allows people
to either input numbers they’ve made up or
wrestle data into meaning something it was
never intended to mean is doomed to fail. But
what’s worrying is that many schools, teachers,
parents and children aren’t even aware of the
failure.
What should we be collecting and analysing?
One of the most important but widely neglected
data sets we should be collecting and analysing
is the age of students within a year group. This
is a useful, robust measurement which actually
affects what a child is able to do. Too often, high
ability groups are simply the older children in a
O brave new world!
class, with lower ability groups made up of the
younger children. But when it comes to learning
and progress we must be mindful that these are
too complex to be reduced to numbers. We can
certainly record indicators of children’s progress
against what teachers think they should have
learned based on the content that has been
covered. Clearly this is highly subjective, as long
as no-one tries to abuse this information by
feeding into high-stakes accountability systems,
then it might be useful.
It’s also possible to track the acquisition of the
fundamental knowledge vital for pupils to make
progress in education. For example, we can
monitor pupils’ knowledge of number bonds and
times tables, since this a clear and unambiguous
test of recall of factual information. We should
also track pupils’ knowledge of the alphabet,
phonemes and graphemes and punctuation
marks for those who have yet to master writing.
Attempting to track what children can do is
difficult, but monitoring progress is possible, and
teachers can assess whether pupils are using
certain key concepts correctly. What we can’t do
is assign numbers to make sense of what’s going
on.
O brave new world!
With the freedoms schools now have, it’s
increasingly possible to resist the accountability
club and concentrate on what you actually want
to know about the pupils in your school.
Valuable insights can be gained by having a
system which is fit for purpose and provides a
bespoke lens.
The market-leading behemoths with the
constant ‘add-ons’ and work arounds, just
How can we guard against bad data?
Next time someone shows you a spreadsheet,
try asking these questions:
•If the data is the solution, what’s the
problem?
•Is there a different way of interpreting the
data?
•How can I verify the quality of the data I’m
being shown, and what are the margins of
error?
increase workload. The prize is out there
for anyone flexible enough to provide data
managements systems that can integrate with
what schools really want and what they’re
already doing. Maybe this is an argument
for providing opportunities for schools and
teachers to design their own apps which can be
brought together under a powerful operating
system?
There are some amazing homemade systems
out there for collecting and analysing truly
useful data. Colin Hegarty’s award winning
website, Hegarty Maths, sequences all the skills
students need to be successful in mathematics,
provides instruction, assessment and accurate
tracking of what students can do, and
intervenes to either plug gaps or provide further
challenges.
At Michaela School in Brent, teachers ensure
students’ mastery of key curriculum content by
setting challenging multiple choice questions
and then giving students weekly quizzes in
each subject. Their bespoke data management
system tracks the time students spend
answering questions, identifies common
misconceptions, logs their progress and
challenges them to do even better.
Dr Chris Wheadon’s No More Marking system is
an exciting development in considering how we
can generate data which liberates rather than
constrains. Although still in the pilot phases
of development, the system asks teachers to
upload essays which are compared and placed
into a rank order. The system doesn’t rely on
computer programmes or complex algorithms,
instead teams of subject experts (PhD students
working for the sheer love of it) read a couple of
•What are the limitations of these data –
what don’t they show?
•How is the data likely to affect my decision
making? What would I do differently if I
didn’t have the data?
•How was the data acquired, and is it valid?
By being more critical of the data with which
we’re presented maybe, just maybe, we can
avoid some of the potential pitfalls associated
with the over abundance of data in schools
9
Section headline over one or two lines
essays and decide which one they like best. Each
essay is judged by a number of different experts
and their subjective opinions are aggregated.
There are no vague or over-complicated
markschemes to interpret and teachers can
select any scale – 1-20, 1-100 they wish; the
system will record the aggregate of the experts’
marks accordingly. This then allows teachers
and students to have meaningful discussions
about why an essay has scored a particular mark
to drive precise, generalisable feedback on how
performance might be improved.
The purpose behind the new breed of
management information systems is to put
schools in control of their data, not the other
way around. For instance, Advanced Learning’s
Progresso system helps schools put “the right
10 | The Illusion of Knowing
information in the hands of the right people at
the right time”, how many schools can say the
same of the system they use?
These examples underline that we should only
collect data we can use to give us insights and
to flag up problems we might otherwise miss,
but they also reveal the expertise and effort
required to produce a management system fit
for purpose. What schools really need are data
management tools to meet them where they
are and provide them with the wherewithal
to maintain and improve standards of pupil
achievement.
References
>> David Thomas, “Be scared of the myth of big
data” http://davidthomasblog.com/2015/09/
be-scared-of-the-myth-of-big-data/
>> Dylan Wiliam, Reliability, validity, and all
that jazz http://eprints.ioe.ac.uk/1156/2/
Wiliam2001Reliability3long.pdf (p 3)
>> Government response to the Workload
Challenge, February 2015 https://www.gov.
uk/ government/uploads/system/uploads/
attachment_data/file/415874/Government_
Response_to_ the_Workload_Challenge.pdf
>> hegartymaths.net
>> Nate Silver, The Signal and the Noise
>> nomoremarking.com
>> Robert Coe, Cesare Aloisi, Steve Higgins and
Lee Elliot Major, What makes great teaching?
Review of the underpinning research p. 3
>> Rolf Dobrelli, The Art of Thinking Clearly p.
57Marwood, Jack, Data by Numbers in Didau,
D,
>> Vivo Miles is a commercial rewards system
intended to work a little like Air Miles. Pupils
gain rewards for good behaviour and you
know what points make? That’s right. Prizes.
>> Marwood, Jack, Data by Numbers in Didau, D,
What if everything you knew about education
was wrong? (2015) p. 337-338
11
About Advanced
Advanced provides management solutions,
Facility and Progresso to schools around the
world. By basing its solutions on the latest
cloud technology and appreciating the everchanging nature of education, it works with
its customers to deliver systems which are
primarily focused on learners and improving
outcomes for them.
To get an insight into this approach and to
learn more about the work Advanced is doing
with some of the UK’s leading schools, please
contact us on 0330 060 2199 or learning.
[email protected]
More information
w oneadvanced.com
t +44(0) 8451 605 555
e [email protected]
Ditton Park, Riding Court Road, Datchet, SL3 9LL
Advanced Computer Software Group Limited is a company
registered in England and Wales under company number
05965280, whose registered office is Ditton Park, Riding Court
Road, Datchet, SL3 9LL. A full list of its trading subsidiaries is
available at www.oneadvanced.com/legal-privacy.
BROCHURE TITLE