Assessing
Outcomes in a
Complex World
Bridget Rosewell
Volterra Consulting and Greater
London Authority
Outline
o Complex adaptive systems can generate
unexpected outcomes
o Managers and planners like predictable results
o These are conflicting pressures and create
tension
o The challenge is to make such tension creative
rather than destructive
o The opportunity is to make complexity analysis
exciting and accessible to a wider community
Unexpected outcomes
o Many model features can generate ranges of
outcomes, depending on how these are
structured
• Data uncertainties
• Feedback loops
• Step ordering
• Random effects from unknown variables
o Some complex systems analysts take the view
that a complete system would cease to produce
unexpected results.
o I view this as fundamentally failing to grasp the
limits of modelling
Planners’ preferences
o Planners like certainty, which leads them to
reject complex systems analysis
o They prefer command and control analysis where
their ‘direction’ is in the best interests of the
directed
o This may not in practice lead to best outcomes,
and especially prevents the emergence of new
solutions to any given problem.
o Planners may also fail to recognise the tensions
between different elements of the system
A housing example - Model
description
Greater London
Authority
Set borough
targets for
market &
affordable
London Housing
Strategy
Boroughs
Give permission
on the basis of
affordable %,
target &
experience with
developers
Planning
permissions
Homes
delivered
Developers
Make proposals
on basis of land,
profit & funding
Planning
applications
Banks
Agree to fund on
the basis of profit,
risk & experience
with developers
And a water industry example
Characteristics
o Mixed motivation for enterprises
o Some uncertainties in behavioural reactionFeedback
effects between outcomes and subsequent decision
making
o Identifies key relationships in collaboration with
practitioners – makes them think
o Identifies surprising outcomes which focus decision
makers minds
• Eg in water – customers motivations are key but not understood
• Eg in housing – rich parts of London always build enough in the
model but not in practice so stakeholders have not yet identified
the key relatiosnsips which their policies govern
A Cautionary Tale
o Transport infrastructure is analysed according to an
assessment of time savings.
o The models which take this approach have been an
effective export and experts and implementing them in
cities such as Shanghai
o The models do not include a feedback effect onto the
economy at present but some practitioners are working
to provide these – in a standard, optimising, traditional
framework
o Such models inevitably limit feedback and decompose
impacts in a linear way
o They are wrong especially when applied to large projects
and should not be allowed to gain ground.
Jubilee Line - example
o Original appraisal did not meet criteria
o As it currently exists would do so in spite of cost overruns
o Opened up a new area for business which has both over
and under performed
Jubilee Line Extension
o Original cost-benefit ratio of 0.95
o Overridden by Prime Minister Thatcher in order to
facilitate Docklands Development
o Cost over-run of 50% on a £3.5bn project, now used as a
basis of ‘optimism bias’ in public projects
o Current evaluation criteria would give it a cost benefit
ratio of 1.75, even after over-runs
Jubilee Line Extension
o 32,000 high value jobs additional in catchment in first
two years
o Supports 12m sq ft of commercial development in Isle of
Dogs
o Doubled rate of residential development and even faster
rate of population increase
o Estimated £20bn output increase, generating £8.5bn
taxes – more than enough to pay back
Crossrail
o Crossrail is a relatively major investment in London’s
infrastructure
o Without economic feedbacks – enabling more people to
work productively – the project seemed too expensive
o New modelling feedback frameworks were developed to
address this
o However, the traditionalists have worked hard to bring
these back into an equilibrium framework that is less
challenging.
Economic Benefits of Crossrail
Benefits
£bn
Conventional User Benefits
High Scenario
Welfare
Mid Scenario
GDP
12.8
Labour Force Participation
Move to more Productive Jobs
Welfare
4.8
Low Scenario
GDP
12.8
Welfare
4.8
Feb-95
GDP
12.8
Welfare
4.8
GDP
12.8
4.8
0.9
0.9
0.9
0.9
46.2
29.9
19.6
7.8
Pure Agglomeration
9.3
14.3
8.2
12.6
6.8
10.4
3.8
5.8
Imperfect Competition
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
Tax Implications
Wider Economic Benefits
Total (User and WEBs)
19.2
13.7
9.9
4.7
29
61.9
22.4
43.9
17.1
31.4
9
15
41.9
66.7
35.3
48.7
29.9
36.2
21.8
19.8
Conclusions
o Complex systems modelling does not sit easily
with a planning approach
o Using such modelling to explain a set of agents to
themselves and each other can help avoid this
problem
o Not forgetting that such explanations will in any
case change the way the system actually
operates
o Planners will often undermine uncertainties of
outcomes in order to create rules of behaviour
that they believe on principle (rather than
evidence) to be appropriate