Forum Forecasting Challenge 2017
A collaboration between
World Economic Forum and Good Judgment Inc
February 2017
PREFACE .................................................................................................................................................. 2
FORUM FORECASTING CHALLENGE ........................................................................................................... 3
GLOBAL GOVERNANCE .................................................................................................................................. 3
THE FOURTH INDUSTRIAL REVOLUTION ............................................................................................................. 4
FORECASTS ................................................................................................................................................. 5
QUESTION: BEFORE 1 JANUARY 2018, WILL A TAXI COMPANY PROVIDE RIDES FOR PAYING CUSTOMERS IN AN AUTONOMOUS
VEHICLE WITHOUT A BACKUP OPERATOR?..................................................................................................................... 5
QUESTION: BEFORE 30 APRIL 2017, WILL THE U.S. GIVE NOTICE OF INTENT TO WITHDRAW FROM THE UNITED NATIONS
FRAMEWORK CONVENTION ON CLIMATE CHANGE? ....................................................................................................... 6
QUESTION: WHEN WILL THE UNITED KINGDOM INVOKE ARTICLE 50 OF THE LISBON TREATY? .............................................. 7
STAY CONNECTED TO GOOD JUDGMENT INC ............................................................................................ 8
FOR INDIVIDUALS ......................................................................................................................................... 8
START KEEPING SCORE – GJOPEN.COM ....................................................................................................................... 8
FORECASTING WORKSHOPS ....................................................................................................................................... 8
FOR ORGANIZATIONS .................................................................................................................................... 8
SUPERFORECASTER ANALYTICS ................................................................................................................................... 8
SOLUTIONS ............................................................................................................................................................. 8
CONTACT GOOD JUDGMENT INC ...................................................................................................................... 8
Preface
Good Judgment Inc at Davos 2017
Good Judgment Inc launched the Forum Forecasting
Challenge 2017 in partnership with the World Economic
Forum (WEF). We hope forecasters enjoyed the challenge as
much as we enjoyed researching and drafting these
questions of relevance for the Fourth Industrial Revolution
and contemporary Global Governance.
Professor Tetlock addressed a question
on the challenges of forecasting.
WEF is known for predicting the year ahead and beyond,
with a view towards the major trends that will shape our
collective global future. Until now there has been limited
tracking on how accurate these forecasts are. This year, the
Forum Forecasting Challenge allows us to start building an
evidence-based track record for WEF participants, together
with friendly competition from the Good Judgment Inc
professional Superforecasters.
WEF forecasters also had the opportunity to complete an
abbreviated version of Good Judgment Inc’s proprietary
Forecasting Aptitude Inventory. This survey assesses the
personal attributes that characterize accurate forecasters.
This information, together with the forecasts on the
questions mentioned above, will help us to begin to better
understand how WEF participants see and approach the
future.
Good Judgment Inc is proud to be on the forefront of
forecasting research and its commercial application. In 2017
and beyond, we will continue to lead cutting-edge
forecasting research, including programs together with the
US Intelligence Advanced Research Projects Activity (IARPA).
We are bringing forecasting best practices to government,
market and civil society organizations on questions ranging
from geopolitical risk and early warning to technological
innovation and other economic and social trends. Through
tools such as Superforecaster® Analytics and training
workshops, we aim to bring scientifically validated
forecasting methodologies into mainstream use.
All forms of policy planning assume
forecasting. Anyone that has a policy
preference on anything is making an
implicit forecast, an implicit conditional
forecast.
So you are not going to get away from
forecasting.
The question is how explicitly you are
going to do it and if you are going to get
as much juice out of the system as you
can.
I think if you try to predict things that are
3 to 5 to 10 years out, virtually no-one
does appreciably better than chance.
Predicting things within a narrower
timeframe, with well-specified outcomes,
you can achieve incremental forecasting
accuracy. . . . . A 10% or 20 % increase in
accuracy matters a lot.
Professor Philip Tetlock
Good Judgment Inc Co-founder
WEF Davos, 18 January 2017
Our collaborations with the World Economic Forum, as well as with the Arab Strategy Forum and other
partners, are crucial in this endeavor to improve how our societies, governments and companies learn to
best approach the future.
How to improve forecasting - in a series of workshops - was the surprise success of Davos 2017. At
a time when the value of forecasting and opinion polling is under serious question, it was uplifting
to have such an insightful series.
Five things I learned from Davos 2017, 1 Huw van Steenis, Global Head, Strategy; Member of the
Executive Committee, Schroders Plc. 24 January 2017.
Good Judgment | Better Decisions
-2-
Forum Forecasting Challenge
The Good Judgment Inc Question Analytics team worked with WEF to break down key issues from the
agenda into questions that would be resolved in 2017 and indicate the longer-term trends underway.
The Forum Forecasting Challenge initially selected a small number of questions addressing the broad topics
below. Through our continued collaboration with WEF, we will build this into a more comprehensive cluster
of questions.
Global Governance
2016 brought surprise changes to global governance, challenging the reputation of pollsters, experts and
forecasters. For the Forum Forecasting Challenge, we selected two questions that forecast the impact and
speed of changes caused by the Brexit referendum and the new U.S. President.
Cluster
Themes
Good Questions
US pull out of UNFCC
US Politics
Syria peace plan
Economic sanctions on Iran
Global
Governance
Europe post
Brexit
French presidential election
UK Invoke Article 50
Figure 1 - Global Governance Questions. Shaded questions are not forecast in the initial challenge.
Good Judgment | Better Decisions
-3-
The Fourth Industrial Revolution
Our opening Fourth Industrial Revolution question looks at mobility and the launch of fully autonomous
taxis. As new services and technologies arising from the Fourth Industrial Revolution disrupt markets,
employment and supply chains, the speed of changes to mobility will have far reaching impact. Questions
around other themes such as Healthcare, Food Security, Environment and Resource Scarcity will be
developed in future forecasting challenges.
Cluster
Themes
Good Questions
Fully autonomous taxis
Fourth
Industrial
Revolution
Mobility
Drones for package delivery
Unmanned air delivery systems for delivering relief
aid
Figure 2 - Fourth Industrial Revolution Questions. Shaded questions are not forecast in the initial challenge.
The first industrial revolution pioneered the machine-assisted production line, and created
modern notions of the factory and the city. The second industrial revolution ushered in a
new age of modernization and the advent of computing. The third industrial revolution
digitalized that progress, and is still ongoing. But what will the Fourth Industrial
Revolution bring, and how will companies, individuals, society and governments adapt to
and harness the changes it will pioneer?
Alec Ash, Official Writer at the World Economic Forum Annual Meeting 2017 for WEF blog
Pioneering Change in the Fourth Industrial Revolution2. 27 January 2017.
Good Judgment | Better Decisions
-4-
Forecasts
Forecasting was open to attendees at Davos, plus participants in the WEF working groups from the 15th of
January until the 20th of February. WEF forecasters were able to submit a one-off probabilistic forecast as
well as qualitative comments on the reasoning behind their forecasts.
Superforecasters had access to the questions over the same period, having the ability to update their
forecasts. Superforecasters will continue to forecast the questions until resolution.
Question: Before 1 January 2018, will a taxi company provide rides for paying customers in an
autonomous vehicle without a backup operator?
A U.S. ride-sharing service has begun experimenting with driverless vehicles in both Pittsburgh and San
Francisco (CBS Pittsburgh 3, CBS San Francisco 4), and a firm in Singapore has also begun experimenting with
self-driving taxis, with its initial runs being free to select individuals (Bloomberg 5). However, in these
instances, a backup driver and/or engineer rides in the autonomous vehicle, and can override the
autonomous system if necessary. Services that operate along a fixed route (e.g. buses and shuttles) will not
count.
Responses
The median World Economic Forum participant said there is a 34% chance that a taxi company will reach this
threshold of automation in 2017. The median Good Judgment Superforecaster says there is just a 8%
chance.
Most WEF forecasters focused on the rapid development of the technology, and the desire of key tech
companies to be the first to use it:
•
•
•
“We are heading towards 4th industrial revolution, and we will get this result very soon.”
“It is highly likely that there will be early adopters among tech or taxi companies.”
“Uber, for example, has enough money to induce a small jurisdiction, desperate for cash, to act as a
guinea pig.”
Some WEF forecasters that gave a lower forecast focused on the legal and safety hurdles in the short-term,
as did most Good Judgment Superforecasters.
•
•
•
“It will be deemed too risky to allow self-driving cars in the wild.”
“Even if the technology is ready, safety tests and regulatory compliance will take time.”
“The cultural shift required to move away from a backup driver will take more than 1 year.”
34%
WEF Forecaster
8%
Superforecaster
Good Judgment | Better Decisions
-5-
Question: Before 30 April 2017, will the U.S. give notice of intent to withdraw from the United Nations
Framework Convention on Climate Change?
The UNFCCC entered into force for the United States in 1994 (UNFCC US 6, UNFCC Background 7). Presidentelect Donald Trump has cast uncertainty over the future of U.S. policy on climate change (The Guardian 8, OC
Register 9). Under Article 25 of the UNFCCC agreement, a party can withdraw from the Convention one year
after giving notification (UNFCC 10, UN Treaty Collection 11).
Responses
The median World Economic Forum participant said there is a 50% chance that the U.S. will give notice to
withdraw from the UNFCC in President Trump’s first 100 days. The median Good Judgment Superforecaster
says the odds are lower, at 15%.
About half of the WEF forecasters focused on the unpredictable nature of the new U.S. President:
•
•
•
“He appears willing to take bold moves that are unpopular with elites and established opinions.”
“If Trump can be president, anything is possible.”
“Anything involving Trump has 50% probability :)”
The other half of WEF forecasters found more pressing priorities for the first 100 days, as did most Good
Judgment Superforecasters.
•
•
•
“Our new president is looking to create a legacy for himself and will make rash decisions. However,
he has intelligent advisors that will advise him otherwise.”
“While unpredictable and anti-multilateral, the Trump administration will become bogged down with
other issues and climate change withdrawal will be subordinate to other events.”
“It’s more likely Trump and his cabinet will attempt to renegotiate the terms of our involvement
through the threat of withdrawing from this than actually doing so.”
For
Against
Good Judgment | Better Decisions
-6-
Question: When will the United Kingdom invoke Article 50 of the Lisbon Treaty?
To invoke Article 50 of the Treaty of Lisbon, a member state must notify the European Council of its
intention to withdraw from the European Union (Washington Post 12, Lisbon Treaty 13). Although a majority of
voters in the United Kingdom's June 2016 referendum on European Union membership voted to leave the
EU (BBC 14), it is unclear when Article 50 will be invoked or if it will govern the UK's withdrawal process (Wall
Street Journal 15, Reuters 16, Financial Times 17).
Responses
The median World Economic Forum participant said there is a 75% chance that the UK will invoke Article 50
before July. The median Good Judgment Superforecaster said the odds are even higher, at 95%.
Some WEF forecasters focused on the procedural requirements needed to move forward in the near term:
• “The Supreme Court of Britain might rule out this option and delay the process for a while.”
• “It might be delayed due to court issues and the time it takes to rebuild a team of negotiators.”
• “There’s an internal conflict within EU on how to proceed.”
Most WEF forecasters found the Government’s determination to meet its spring schedule to be the most
salient factor, as did most Good Judgment Superforecasters.
• “Theresa May seems determined to take this step, and no organised political force appears capable
of stopping her.”
• “Current course is the ‘hard Brexit’ one. For this course extending the time of invoking article 50
doesn’t make sense.”
• “The next general elections in the UK are three years from now. I doubt those who are behind Brexit
will want to risk losing Parliament.”
75%
WEF Forecasters
95%
Superforecasters
Good Judgment | Better Decisions
-7-
Stay connected to Good Judgment Inc
For Individuals
Start Keeping Score – GJOpen.com
Our public forecasting tournament at GJOpen.com features challenges for The Economist, The CFA Society,
Arab Strategy Forum and others.
Track your position on the forecasting leaderboard to see how your forecasting skills compare.
Forecasting Workshops
Our Superforecaster led workshops give participants the chance to understand their cognitive biases and
hone their forecasting skills in a collaborative group workshop.
Register your interest at goodjudgment.com/workshops
For Organizations
Good Judgment Inc works with some of the world’s most successful organizations helping leaders make
better decisions.
Superforecaster Analytics
Our Superforecaster Analytics provide accurate, calibrated quantified forecasts of organizations’ most
pressing questions.
Solutions
Our Solutions service helps organizations keep score and hone the forecasting talents of their internal
crowd.
Contact Good Judgment Inc
For more information on Good Judgment Inc, visit http://www.goodjudgment.com/
To discuss working with Good Judgment Inc, please contact Kate Patterson.
Email: [email protected]
Phone: (212) 328-0191
Copyright © 2017, Good Judgment Inc.
Good Judgment®, Superforecaster® and Superforecasting™ are trademarks of Good Judgment Inc
Good Judgment | Better Decisions
-8-
End Notes
1 https://www.weforum.org/agenda/2017/01/five-things-i-learned-at-davos/
2 https://www.weforum.org/agenda/2017/01/shaping-davos-pioneering-change-in-the-fourth-industrialrevolution/
3 http://pittsburgh.cbslocal.com/2016/09/14/uber-driverless-cars-hit-the-streets-of-pittsburgh/
4 http://sanfrancisco.cbslocal.com/2016/09/14/uber-self-driving-cars-passenger-pickup-pittsburgh-pennsylvania/
5 https://www.bloomberg.com/news/articles/2016-08-25/world-s-first-self-driving-taxis-debut-in-singapore
6 http://unfccc.int/tools_xml/country_US.html
7 http://unfccc.int/essential_background/convention/items/6036.php
8 https://www.theguardian.com/environment/2016/dec/02/trump-break-from-paris-climate-deal-unfccc-exit
9 http://www.ocregister.com/articles/trump-738643-donald-action.html
10 http://unfccc.int/files/essential_background/background_publications_htmlpdf/application/pdf/conveng.pdf
11 https://treaties.un.org/Pages/CNs.aspx?cnTab=tab1&clang=_en
12 https://www.washingtonpost.com/news/worldviews/wp/2016/06/24/article-50-the-untested-piece-of-e-ulegislation-that-will-control-britains-future/?wpisrc=nl_wv-draw6&wpmm=1
13 http://www.lisbon-treaty.org/wcm/the-lisbon-treaty/treaty-on-european-union-and-comments/title-6-finalprovisions/137-article-50.html
14 http://www.bbc.com/news/uk-politics-32810887
15 http://www.wsj.com/articles/after-brexit-vote-europes-leaders-debate-timing-of-u-k-s-departure-1466983952
16 http://uk.reuters.com/article/uk-britain-eu-scenarios-factbox-idUKKCN0ZC0WI
17 https://next.ft.com/content/d24b4dcc-3b9f-11e6-8716-a4a71e8140b0
Good Judgment | Better Decisions
-9-