Technological Forecasting & Social Change 78 (2011) 1487–1490
Contents lists available at SciVerse ScienceDirect
Technological Forecasting & Social Change
The Delphi technique: Past, present, and future prospects —
Introduction to the special issue ☆
Gene Rowe a,⁎, George Wright b
a
b
Gene Rowe Evaluations, 12, Wellington Road, Norwich NR2 3HT, UK
Durham Business School, Mill Hill Lane, Durham, DH1 3LB, UK
a r t i c l e
i n f o
Article history:
Received 22 August 2011
Received in revised form 5 September 2011
Accepted 6 September 2011
Available online 6 October 2011
a b s t r a c t
The Delphi technique has been around for over half a century, so now seems a proper time to
consider its past, present and future. This introduction characterises the papers in this special
Delphi issue, which include both conceptual and empirical works. It summarises the main lessons that have been learned from these for the conduct of the technique, and provides a call for
more and better empirical studies in the future.
© 2011 Elsevier Inc. All rights reserved.
Keywords:
Delphi
Evaluation
Participation
1. Introduction
The Delphi technique has been around for some time now. It is difficult to resist the urge here to recapitulate the story of its
origins, but we will: these details are noted in various papers in this special issue. One moment of history that is worth emphasising,
however, is that of 1975, when the first edition of Linstone and Turoff's [1] edited book on Delphi first appeared and brought notice of
the approach to a wider audience. Slowly at first, but at a seemingly growing rate, the technique has flourished, appearing in more and
more academic domains and being used for more and more purposes. As evidence of the impact of that work, Google Scholar reveals
that ‘The Delphi Method: Techniques and Applications’ has been cited over 2700 times (while it has undoubtedly been mis-cited
many more times too). Furthermore, as Delphi's ubiquity has grown, so has the method evolved, with the development of numerous
variants, so that it is perhaps better to talk of ‘Delphi techniques’ in the plural than in the singular. As such, though not quite at the
50th anniversary of publication of that seminal Delphi volume, now seems an apt moment to consider exactly how the method
has developed, into what new areas and forms, and to what ends (especially since such a review/compendium has not occurred in
this journal since one in 1975). Such consideration is the aim of this special issue.
2. The papers in this issue
The first paper in this issue by Roubelat and Marchais-Roubelat [2] appropriately delves the furthest back in history, to the original
Oracle at Delphi. The paper does not only provide a forecasting history lesson, however: importantly, it finds parallels between the
ancient approach and the modern-day namesake. Especially important is its consideration of the role of participants and the nature
of expertise, as well as the issue of trust. Bolger and Wright [3] also provide a broader context for considering Delphi — although rather
than historical, this focuses upon another research domain, namely social psychology. Their review and analysis discusses findings
from past research that have implications for how Delphi might work, that is, how the method functions and how it can be enhanced.
This research thus identifies a variety of important factors that need to be considered in deciding when and how to use Delphi.
☆ A special issue of technological forecasting and social change. Guest editors: Gene Rowe and George Wright.
⁎ Corresponding author. Tel.: + 44 1603 255125.
0040-1625/$ – see front matter © 2011 Elsevier Inc. All rights reserved.
doi:10.1016/j.techfore.2011.09.002
1488
G. Rowe, G. Wright / Technological Forecasting & Social Change 78 (2011) 1487–1490
Although this paper highlights many research questions that still need resolution, it provides a framework for conducting the necessary research — an issue to which we will return shortly.
Opposed to theoretical lessons (or lessons from other research areas), practical lessons from the conduct of ‘real’ Delphis are
elaborated in a number of case studies. Frewer and colleagues [4] discuss the recent novel use of Delphi in the agri-food area,
summarising results from a series of significant real-world exercises, in which the process was subtly amended, over the series,
to take on board practical lessons learned during previous exercises. The nature of expertise and recruitment were issues of prime
concern, as was the issue of language — given that many contemporary problems are global, and experts may thus speak different
languages and have differing levels of access to appropriate electronic media. The issue of identifying and selecting experts is also
covered in a case study by Goluchowicz and Blind [5], which looks at the area of ‘standardisation’. These authors provide a thorough analysis of the recruitment problem that may act as a good model for other researchers and practitioners.
Mateos-Ronco and Server [6] also describe a novel intervention – in the area of agricultural insurance – highlighting the benefits of
Delphi in terms of transparency and utility in an area that previously had limitations in these regards. And van de Linde and van der
Duin [7] describe Delphi use in an extremely important new domain, namely radicalisation and terrorism. The latter case also discusses the importance of identifying both consensus and dissensus, emphasising an important dimension in this methodological domain. Thus, while Delphi can help demonstrate and quantify consensus, it is often where consensus is not evident that the interesting
and important issues emerge. Finally here, Di Zio and Pacinelli [8] describe a truly novel approach that enables Delphi to be extended
to consider spatial problems where, for example, panellists may differ in their initial views of the location of an area of greatest seismic
risk.
A number of papers look at the role of Delphi, not as a standalone approach, but as a method that may be enhanced by other
approaches, or that may contribute as input to others. Bañuls and Turoff [9] discuss the role of cross-impact analysis as an adjunct
to Delphi in scenario development, and provide a full description to enable their approach to be replicated. On the other hand,
Nowack and colleagues [10] discuss how Delphi itself might provide input into scenario planning, and they review the studies
in which this has taken place. Similarly, Tapio et al [11] look at the use of disaggregative policy Delphis for scenario formation,
and pose more fundamental questions about the combining of qualitative and quantitative information. They argue that ‘mixed
methods’ are useful in order to form coherent scenarios. Landeta and colleagues [12] also describe using Delphi as part of a
broader process using Nominal Group Technique and focus groups. Each of these papers importantly points to limitations to Delphi
(for example, the relatively limited nature of interaction between participants) and indicates how these limitations – in terms of input
and output – might be addressed through the addition of alternative techniques. Using Delphi as only part of a wider process (with
qualitative and quantitative components) may well prove a means to enhance its utility; one can then avoid the question of ‘which
method?’ and ask instead ‘which methods, and in what order?’ This is a more sophisticated question that may well be apt for
many large, difficult, relatively well-financed real-world problems.
All of these papers – whether focused on discussing variations of Delphi, possible hybridisations, or in-depth case studies – touch
upon one or more methodological issue that can impact Delphi performance/utility. Other papers in this special issue essentially start
from a methodological problem per se, and from this draw implications for Delphi use more widely. Hussler et al [13] focus on the
issue of panel diversity, arguing for its benefits, and discussing the issue of who is an expert and who is not. They also provide a caveat
about the issue of self-interest and the potential biases this might lead to in the process, tieing-in with a paper by Ecken et al [14] on
the desirability of outcomes, and how a form of wishful thinking might lead to systematic bias in Delphi approaches. These latter authors suggest that a standard Delphi application may attenuate, but not totally ameliorate, the established phenomenon of desirability
bias, and they provide an interesting suggestion for how Delphi might be revised to deal with bias in forecasts that are perceived as
more or less desirable by individual panellists.
Bolger and colleagues [15] look at the issue of feedback and how this might impact on opinion change. In doing this, the authors follow the empirical framework already discussed by Bolger and Wright [3] and noted earlier. These authors argue that all
cues to majority opinion and confidence levels of panellists should be removed from Delphi feedback in order to let strong rationales for particular panellists' opinions exert a virtuous pull on opinion change. Gnatzy and colleagues [16] similarly provide an
empirical study that compares different Delphi formats — in this case, a conventional Delphi to a real-time version. Each of
these latter two studies finds only limited differences between compared versions, which implies that the generic Delphi approach is relatively robust (i.e. that even fairly significant variations matter little) and raises questions about how to conduct
and control Delphi applications in order to improve the practical efficacy of the Delphi process.
These observations lead on to two significant papers that address the crucial question of what we mean by the success of a Delphi
exercise. Hasson and Keeney [17] talk about ‘enhancing rigour’ in Delphi, and propose reliability, validity and trustworthiness as three
pertinent criteria for judging worth. They highlight the difficulties in using these criteria, particularly reliability, which has various
forms, some of which may seem confounded by the precise purpose of using Delphi (e.g. to establish consensus or highlight dissensus).
Parenté and Anderson-Parenté [18] also discuss the validity problem and provide one of the few retrospective attempts to validate the
results from a long-term Delphi forecast (see also [19]), and indeed, possibly the longest — of events 30 years post-forecast. Although
the latter authors conclude that they provide some evidence for the validity for the technique, we believe that the most important output from this paper is to demonstrate how difficult quarter-century-ahead validation is, and perhaps brings into question whether the
proven accuracy of a forecast is truly a good benchmark for forecasting performance — given that desired forecasts can often be encouraged to occur, and undesired ones pre-empted, and that the interpretation of whether a particular forecast turned out to be correct can
lie in large part on the interpretation of imprecise words used in the forecast formulation. This latter point brings us back to the first
paper of Roubelat and Marchais-Roubelat [2], who relate the story of Croesus — a ruler who interpreted the prediction that an empire
was about to fall (according to the Oracle at Delphi) as being that of his enemy… when it turned out to be his own.
G. Rowe, G. Wright / Technological Forecasting & Social Change 78 (2011) 1487–1490
1489
It is perhaps fitting that the last word in this issue has been reserved for Linstone and Turoff themselves. In their paper [20],
they provide some reflections on their long experience with the technique, and in particular, on the changing roles and forms
of the method as well as on what the future may hold for it. They especially note the opportunities that may arise with the electronic age, particularly in the context of greater citizen and stakeholder participation. Indeed, we concur: this social zeitgeist of
participation is an important and compelling one throughout many Western societies, where democratic principles almost
demand greater inclusivity in decision making, agenda setting, planning, and foresight, particularly in realms of high (or even
moderate) uncertainty in which human expertise has its limits [21], and lay perspectives have more than just token value. In
these circumstances, the question becomes: how does one include diverse perspectives from many people (a heterogeneity of
panel membership that various papers in this volume support)? Although many methods have been developed [22], most rely
upon small group processes that have been criticised for their lack of true representation of the general population. Furthermore,
with small group activities come all the potential social and political pressures and game playing that Delphi was partly developed
to overcome. In our view, therefore, we see Delphi as providing a potential structural antidote to the pathologies of the majority of
participation approaches (such as consensus conferences and citizens' juries) — and hence we too foresee much greater uptake of
the method in its various guises.
3. Concluding thoughts
The papers in this special issue bring together a wide variety of perspectives on the Delphi technique(s) from some of the most
important contributors to the thinking and development of the method of the last forty-fifty years. The ubiquity of application
that they hint at suggests that, in many domains, the method has filled a deep need of academics and practitioners for structured
ways of assessing and combining human judgement. The more exposure the method gains, the greater its uptake; in many cases,
Delphi seems to be a method that enables researchers to ask and answer questions that, previously, they did not know how to
address.
In this growth, we – as ‘supporters’ of the method (by which we mean, people who are aware of the potential uses and benefits
of the approach) – should be pleased. However, a great danger lies in the uncritical adoption of the method, or the careless
application of it. From our personal experience we are aware of how difficult it can be to do Delphi right, and how easy it is to
mis-frame questions, or fail in recruiting appropriate experts (whether true experts or lay citizens with relevant knowledge),
or in creating the right context to ensure participants continue through to the end of the process, take the task seriously, and
contribute to an output that is useful and will be used by exercise sponsors. Many of the papers here provide guidelines for
how to enhance use of the method, including:
Improving panellist recruitment and retention over Delphi rounds, through:
• Using a person-to-person cascade approach (i.e., “snowballing”) to secure easy agreement to panellist invitations (which will
also strengthen subsequent panellist retention) [4, 5]
• Making use of publically-available bibliographic information to identify potential expert panellists [5]
• Noting that self-rated experts tend to exhibit less drop-out over Delphi rounds than those who rate themselves as less-expert [5]
• Stressing the practical policy application of the Delphi yield to expert panellists to aid their retention [6]
• If panellists are widely scattered across the world, being aware of the de-motivating effect of poor internet speeds with complex
graphical content in communications [4]
• Using social rewards for recognition of participation — such as subsequently publishing panel membership listings [3].
Creating useful heterogeneity in panel membership through:
• Including experts and laypeople to increase the variety of viewpoints amongst first round opinions — even though the lay opinions
will be less stable and tend to reduce to expert viewpoints over rounds [13]
• Creating artificial heterogeneity in opinions at the first Delphi round – using role-playing, devil's advocacy, and dialectical inquiry –
and in this way, facilitating alternative framings [3].
Enhancing information exchange between panellists, through:
•
•
•
•
Removing any indicators of the prevalence of majority or minority opinions [3, 15, 16]
Removing any indication of panellists' confidence levels [3, 15]
Using rich qualitative feedback of panellists' rationales and reasoning behind their judgments [3, 10, 15]
Being alert to stability in dissensus over Delphi rounds — which could indicate the need to explore panellists' underpinning assumptions and logics in subsequent face-to-face meetings [7].
Improving question formulation through:
• Using an exploratory workshop to refine first-round Delphi questions [4]
• Using easy-to-answer questioning, preferably involving closed questions, for use in Delphi rounds [4]
• Using simple English expression in questions when panellists do not have English as a first language — but noting that exchange
of rationales and reasons between panellists will likely also be similarly restricted [4]
• Using Delphi questions with unambiguous wording such that subsequent evaluation of event outcomes is clear-cut [18].
1490
G. Rowe, G. Wright / Technological Forecasting & Social Change 78 (2011) 1487–1490
Considering combining Delphi with other techniques:
• Being aware of the benefits of technique combination to enhance panellist creativity and commitment [12]
• Being aware of the usefulness of Delphi as a means of eliciting group-based judgments for integration with other futures methodologies (as well as, for example, for establishing exogenous variables for econometric models, or in forming the coefficients
for an input-output table) [9-11]
• Measuring forecast desirability expressions of individual panel members and thus identifying potential optimism and pessimism
bias [14]
• Being aware that real-time Delphi and conventional Delphi produce similar results and that choice of the former approach has
potential benefits [16].
Many of these findings and observations have been made before — often in an off-hand manner, without particularly strong
empirical support. And this brings us to our final point: there is still a lack of good empirical studies that provide more rigorous
answers to ‘how to’ questions. When we first came to the subject 20 years ago, this was our initial insight [23], which we followed
up with a number of empirical studies that, we felt, were as important for showing how one might research Delphi processes as in
the results they provided per se [e.g. 24, 25]. Since then, however, we have sensed little increment in such empirical studies —
even though applications of method use have grown impressively. We continue to believe that it is especially important that decisions about whether to use Delphi (against using an alternative approach), decisions about which variant to use, and decisions
about how to precisely enact one's chosen variant, be informed by sound (social) science. In enacting this agenda, it is important
to be aware of research conducted in related literatures (as summarised here by Bolger and Wright [3]), as well as to appreciate
important concepts such as validity and reliability and what these mean (or should, or can, mean) in contexts in which Delphi
might be employed [17, 18]. Therefore our final urging is for researchers to take up this challenge of evidence, and ensure that
scientific evaluations of the foundations of the method – established half-a-century ago – are strengthened and deepened.
References
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
H.A. Linstone, M. Turoff, The Delphi Method: Techniques and Applications, Addison-Wesley, London, 1975.
F. Roubelat, A. Marchais-Roubelat, The Delphi method as a ritual: inquiring the Delphic Oracle, Technol. Forecast. Soc. Chang. 78 (2011) 1491–1499.
F. Bolger, G. Wright, Improving the Delphi process: lessons from social psychological research, Technol. Forecast. Soc. Chang. 78 (2011) 1500–1513.
L.J. Frewer, A.R.H. Fischer, M.T.A. Wentholt, H.J.P. Marvin, B.W. Ooms, D. Coles, G. Rowe, The use of Delphi methodology in agrifood policy development:
some lessons learned, Technol. Forecast. Soc. Chang. 78 (2011) 1514–1525.
K. Goluchowicz, K. Blind, Identification of future fields of standardisation: an explorative application of the Delphi methodology, Technol. Forecast. Soc.
Chang. 78 (2011) 1526–1541.
A. Mateos-Ronco, R.J.S. Server, Drawing up the official adjustment rules for damage assessment in agricultural insurance: results of a Delphi survey for fruit
crops in Spain, Technol. Forecast. Soc. Chang. 78 (2011) 1542–1556.
E. van de Linde, P. van der Duin, The Delphi method as early warning: linking global societal trends to future radicalization and terrorism in The Netherlands,
Technol. Forecast. Soc. Chang. 78 (2011) 1557–1564.
S. Di Zio, A. Pacinelli, Opinion convergence in location: a spatial version of the Delphi method, Technol. Forecast. Soc. Chang. 78 (2011) 1565–1578.
V.A. Bañuls, M. Turoff, Scenario construction via Delphi and cross-impact analysis, Technol. Forecast. Soc. Chang. 78 (2011) 1579–1602.
M. Nowack, J. Endrikat, E. Guenther, Review of Delphi-based scenario studies: quality and design considerations, Technol. Forecast. Soc. Chang. 78 (2011)
1603–1615.
P. Tapio, R. Paloniemi, V. Varho, M. Vinnari, The unholy marriage? Integrating qualitative and quantitative information in Delphi processes, Technol. Forecast.
Soc. Chang. 78 (2011) 1616–1628.
J. Landeta, J. Barrutia, A. Lertxundi, Hybrid Delphi: a methodology to facilitate contribution from experts in professional contexts, Technol. Forecast. Soc.
Chang. 78 (2011) 1629–1641.
C. Hussler, P. Muller, P. Ronde, Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy,
Technol. Forecast. Soc. Chang. 78 (2011) 1642–1653.
P. Ecken, T. Gnatzy, H. von der Gracht, Desirability bias in foresight: consequences for decision quality based on Delphi results, Technol. Forecast. Soc. Chang.
78 (2011) 1654–1670.
F. Bolger, A. Stranieri, G. Wright, J. Yearwood, Does the Delphi process lead to increased accuracy in group-based judgmental forecasts or does it simply
induce consensus amongst judgmental forecasters? Technol. Forecast. Soc. Chang. 78 (2011) 1671–1680.
T. Gnatzy, J. Warth, H. von der Gracht, I.-L. Darkow, Validating an innovative real-time Delphi approach — a methodological comparison between real-time
and conventional Delphi studies, Technol. Forecast. Soc. Chang. 78 (2011) 1681–1694.
F. Hasson, S. Keeney, Enhancing rigour in the Delphi technique research, Technol. Forecast. Soc. Chang. 78 (2011) 1695–1704.
R.J. Parenté, J.K. Anderson-Parenté, A case study of long-term Delphi accuracy, Technol. Forecast. Soc. Chang. 78 (2011) 1705–1711.
R. Amant, Comparison of Delphi Forecasting Studies in 1964 and 1969, Futures 2 (1970) 35–44.
H.A. Linstone, M. Turoff, Delphi: a brief look backward and forward, Technol. Forecast. Soc. Chang. 78 (2011) 1712–1719.
P. Goodwin, G. Wright, The limits of forecasting methods in anticipating rare events, Technol. Forecast. Soc. Chang. 77 (2010) 355–368.
G. Rowe, L. Frewer, A typology of public engagement mechanisms, Sci., Technol. Human Values 30 (2005) 251–290.
G. Rowe, G. Wright, F. Bolger, The Delphi technique: a re-evaluation of research and theory, Technol. Forecast. Soc. Chang. 39 (1991) 235–251.
G. Rowe, G. Wright, The impact of task characteristics on the performance of structured group forecasting techniques, Int. J. Forecast. 12 (1996) 73–89.
G. Rowe, G. Wright, A. McColl, Judgment change during Delphi-like procedures: the role of majority influence, expertise and confidence, Technol. Forecast.
Soc. Chang. 72 (2005) 377–399.
Gene Rowe is an independent research consultant (Gene Rowe Evaluations: email: [email protected]), who until recently was Head of Consumer Science of the
Institute of Food Research, Norwich, UK. Since gaining his PhD from the Bristol Business School, he has published around 80 journal articles and book chapters, and, with
George Wright, recently edited a special issue of the International Journal of Forecasting on group-based judgmental forecasting. His research interests focus on individual and group decision making (especially in forecasting contexts), human understanding of risk, and the evaluation of public participation exercises and methods.
George Wright is editor of the Journal of Behavioral Decision Making and an associate editor for Journal of Forecasting and for International Journal of Forecasting. He is
also an associate editor of Decision Support Systems. His publications on Delphi have appeared in Technological Forecasting and Social Change and in International
Journal of Forecasting. His general research interest is in the role and validity of judgement in anticipating the future.