Ensuring the Better Use of
Monitoring by Policymakers
Ken Guy
Wise Guys Ltd.
Barcelona, 4 May 2017
Introduction
• The topic I was asked to address today was “How to persuade decisionmakers to make use of monitoring information”
• I was also asked to illustrate my talk with “examples of monitoring being
taken into consideration”
• Unfortunately, examples of this nature are rare
• They undoubtedly exist, but the impacts of monitoring systems are often
hidden because monitoring inputs are rarely the only inputs into decisionmaking
• And even when they do play an important role, ‘data analysts’ often get
more credit than ‘data collectors’
• Most of the anecdotes I can draw upon concern evaluation impacts, not
impacts that are based solely on the fruits of comprehensive monitoring
systems
• For the most part, these are presented after generic suggestions pertinent
to the various stages involved in the process of utilising monitoring
information in decision-making environments
Monitoring as a Process
• Research and Design
• Data Collection
• Data Analysis
• Communication
• Presentation
• Use
Research and Design
• Make sure the system focuses on information that
policymakers need
• Involve policymakers in the research and design phases of
monitoring systems
• Involve policymakers from various vertical levels
• Involve policymakers from contiguous horizontal domains
• But anticipate beyond their perceived needs
• All monitoring systems are based on theories of
intervention, but try to make these as simple as possible
• Monitoring systems become misleading or redundant when
these theories differ from reality, so continuously assess
whether they describe reality adequately
Data Collection
• Let pragmatism decide the scope of data
collection
• Prioritise the collection of data on key
indicators
• Ensure that data entries are as up-to date as
possible
• At the margin, continuously explore the
possibility of developing and using new
indicators
Data Analysis
• Prioritise the analysis of key indicators and
hypothesised relationships
• Make sure data can be analysed in a responsive
and timely fashion
• Set up routines that can quickly output key
analytical results in standard formats
• Benchmark results with comparable systems
whenever possible
• At the margin, continuously explore new ways of
analysing data
Communication
• Communicate results as high and as low as
possible along vertical communication chains
• Communicate as broadly as possible at a
horizontal level
• Encourage open publication of data and
analyses
Presentation
• Make presentations as striking and
memorable as possible
• Deploy humour
• Exploit the power of imagery and graphics
• Never forget the appeal of comparison
The Appeal of Comparison
Though never confuse input indicators with result indicators
Deploying Humour
• Humorous Titles
– The Unbearable Lightness of Real-time Evaluation
– Four Evaluations and a Monitoring Exercise
– No Monitoring Please, We’re British
• Humorous Content
– Why is Innovation like Sex?
– Gone Fishing!
– A Salutary Tale
Additionality and Strategic Importance
Additionality
42%
58%
Pure Additionality
Process Additionality
Strategic Importance
51%
49%
Finnish Energy Programmes – 1990s
High Strategic
Importance
Low Strategic
Importance
Gone Fishing
• Where would you send your fleet to search for fish?
Isle of
Dreams
Treasure
Island
Fool’s
Paradise
Devil’s
Island
Finnish Energy Programmes – 1990s
Supplementary Funding
• Where would you send more ships if government matched
your extra funding?
Isle of
Dreams
Treasure
Island
Fool’s
Paradise
Devil’s
Island
Finnish Energy Programmes – 1990s
A Sensible Strategy?
• Does this distribution of extra ships make sense?
Isle of
Dreams
High
Strategic
Importance
Low
Strategic
Importance
Fool’s
Paradise
Process Additionality
Pure Additionality
31%
18%
11%
40%
Treasure
Island
Devil’s
Island
Additionality for industrialists in research projects
Finnish Energy Programmes – 1990s
Better!
• Does this distribution of extra ships make sense?
Isle of
Dreams
High
Strategic
Importance
Low
Strategic
Importance
Fool’s
Paradise
Process Additionality
Pure Additionality
15%
66%
3%
16%
Treasure
Island
Devil’s
Island
Additionality for industrialists in research projects
EU 4FP– 1990s
Workshop on the Scope and Focus of Joint
Programmes
• The aim of the exercise was to focus the attention of policymakers
on key hypotheses concerning the future of joint programming and
get them to discuss next steps
• The intention was to show them a series of slides that would focus
their attention on different hypotheses and then ask them a series
of questions
• For example, for a hypothesis concerning the scope and focus of
Joint Programmes, the questions were:
– Do you agree with the hypothesis?
– Are there some areas of Joint Programming where the scope is too
broad or unfocused?
– How could this be avoided?
– What steps need to be taken now?
• But the best laid plans of mice and men…….
The Scope and Focus of Joint
Programmes
Hypothesis
A greater focus on the scope of initiatives is
needed to make the best use of resources and
avoid obvious barriers, traps and pitfalls
• On occasion there are limited options
• And clear choices can be made
• But at other times there are too many balls to
juggle
• And different paths to consider
• Some of which are dead ends
• While others are littered with hidden barriers
• Traps
• And pitfalls
Caution!
• Humour can be a distraction that diverts
attention away from the main messages that
you are trying to convey
• Tastes in humour can also vary tremendously,
so assess the receptivity of your audiences
very carefully before deploying dubious
humour
Use
• Overcoming Resistance
– CS&HCI Evaluation
– Innopoli Evaluation
– SPRINT Evaluation
• Instant Gratification
– Swedish and Finnish Evaluations
• Demonstrating Patience
– FUPA Evaluation
Dealing with Complexity
Explaining oneself
In seventeen syllables
Is very
Diffic…
Thank You
with especial thanks to Spencer Tunick and John Cooper Clark