Delphi method

The Delphi method is a systematic interactive forecasting method for obtaining forecasts from a panel of independent experts. The carefully selected experts answer questionnaires in two or more rounds. After each round, a facilitator provides an anonymous summary of the experts’ forecasts from the previous round as well as the reasons they provided for their judgments. Thus, participants are encouraged to revise their earlier answers in light of the replies of other members of the group. It is believed that during this process the range of the answers will decrease and the group will converge towards the "correct" answer. Finally, the process is stopped after a pre-defined stop criterion (e.g. number of rounds, achievement of consensus, stability of results) and the mean or median scores of the final rounds determine the results.

Delphi [pron: delfI] is based on well-researched principles and provides forecasts that are more accurate than those from unstructured groups. The technique can be adapted for use in face-to-face meetings, and is then called mini-Delphi or Estimate-Talk-Estimate (ETE). Delphi has been widely used for business forecasting and has certain advantages over another structured forecasting approach: prediction markets.

Free software for conducting Delphi studies is available at Principles of Forecasting.

History

The name "Delphi" derives from the Oracle of Delphi. The authors of the method were not happy with this name, because it implies "something oracular, something smacking a little of the occult". The Delphi method recognizes the value of expert opinion, experience and intuition and allows using the limited information available in these forms, when full scientific knowledge is lacking.

The Delphi method was developed, over a period of years, at the Rand Corporation at the beginning of the cold war to forecast the impact of technology on warfare. A number of events influenced the development. In 1944, General Arnold ordered the creation of the report for the U.S. Air Force on the future technological capabilities that might be used by the military. Two years later, Douglas Aircraft company started Project RAND to study "the broad subject of inter-continental warfare other than surface".

Different approaches were tried, but the shortcomings of traditional forecasting methods, such as theoretical approach, quantitative models or trend extrapolation, in areas where precise scientific laws have not been established yet, quickly became apparent. To combat these shortcomings, the Delphi method was developed in RAND Corporation during the 1950-1960s (1959) by Olaf Helmer, Norman Dalkey, and Nicholas Rescher.

The Delphi method was used by Rand Experts when they were asked to give their opinion on the probability, frequency and intensity of possible enemy attacks. Other experts could anonymously give feedback. This process was repeated several times until a consensus emerged.

Key characteristics

The following key characteristics of the Delphi method help the participants to focus on the issues at hand and separate Delphi from other methodologies:

Structuring of information flow

The initial contributions from the experts are collected in the form of answers to questionnaires and their comments to these answers. The panel director controls the interactions among the participants by processing the information and filtering out irrelevant content. This avoids the negative effects of face-to-face panel discussions and solves the usual problems of group dynamics.

Regular feedback

Participants comment on their own forecasts, the responses of others and on the progress of the panel as a whole. At any moment they can revise their earlier statements. While in regular group meetings participants tend to stick to previously stated opinions and often conform too much to group leader, the Delphi method prevents it.

Anonymity of the participants

Usually all participants maintain anonymity. Their identity is not revealed even after the completion of the final report. This stops them from dominating others in the process using their authority or personality, frees them to some extent from their personal biases, minimizes the "bandwagon effect" or "halo effect", allows them to freely express their opinions, encourages open critique and admitting errors by revising earlier judgments.

Role of the facilitator

The person coordinating the Delphi method can be known as a facilitator, and facilitates the responses of their panel of experts, who are selected for a reason, usually that they hold knowledge on an opinion or view. The facilitator sends out questionnaires, surveys etc. and if the panel of experts accept, they follow instructions and present their views. Responses are collected and analyzed, then common and conflicting viewpoints are identified. If consensus is not reached, the process continues through thesis and antithesis, to gradually work towards synthesis, and building consensus.

Applications

Use in forecasting

First applications of the Delphi method were in the field of science and technology forecasting. The objective of the method was to combine expert opinions on likelihood and expected development time, of the particular technology, in a single indicator. One of the first such reports, prepared in 1964 by Gordon and Helmer, assessed the direction of long-term trends in science and technology development, covering such topics as scientific breakthroughs, population control, automation, space progress, war prevention and weapon systems. Other forecasts of technology were dealing with vehicle-highway systems, industrial robots, intelligent internet, broadband connections, and technology in education.

Later the Delphi method was applied in other areas, especially those related to public policy issues, such as economic trends, health and education. It was also applied successfully and with high accuracy in business forecasting. For example, in one case reported by Basu and Schroeder (1977), the Delphi method predicted the sales of a new product during the first two years with inaccuracy of 3–4% compared with actual sales. Quantitative methods produced errors of 10–15%, and traditional unstructured forecast methods had errors of about 20%.

Appropriateness of healthcare

The RAND Corporation and others have extensively used the Delphi Method to apply expert knowledge to determining appropriateness of healthcare.

Delphi applications not aiming at consensus

Traditionally the Delphi method has aimed at a consensus of the most probable future by iteration. The Policy Delphi launched by Murray Turoff instead is a decision support method aiming at structuring and discussing the diverse views of the preferred future. The Argument Delphi developed by Osmo Kuusi focuses on ongoing discussion and finding relevant arguments rather than focusing on the output. The Disaggregative Policy Delphi developed by Petri Tapio uses cluster analysis as a systematic tool to construct various scenarios of the future in the latest Delphi round. The respondent's view on the probable and the preferable future are dealt with as separate cases.

Studies on validity

The Delphi method improves the agreement among experts beyond chance.

Delphi vs. Nominal Group Technique

One comparison found consensus was closer in the NGT than in the Delphi; no overall difference between groups in their concordance with research evidence; but the Delphi method was more reliable. In this study, the NGT had group meetings whereas the Delphi was done entirely independently.

Delphi vs. Prediction Markets

As can be seen from the Methodology Tree of Forecasting, Delphi has similar characteristics than prediction markets as both are structured approaches that aggregate diverse opinions from groups. Yet, there are differences that may be decisive for their relative applicability for different problems.

Some advantages of prediction markets derive from the possibility to provide incentives for participation.

1. They can motivate people to participate over a long period of time and to reveal their true beliefs.
2. They aggregate information automatically and instantly incorporate new information in the forecast.
3. Participants do not have to be selected and recruited manually by a facilitator. They themselves decide whether to participate if they think their private information is not yet incorporated in the forecast.

Delphi seems to have these advantages over prediction markets:

1. Delphi is easier to implement and to use since a broader range of problems can be formulated.
2. It is easier to reveal one's opinion in a questionnaire than to translate it into market prices.
3. It is easier to maintain confidentiality with Delphi.
4. Delphi is not vulnerable to manipulation by participants.
5. The transparent exchange of knowledge in Delphi allows participants to learn from each other or to introduce new ideas in the discussion.
6. Only 5 to 20 experts are necessary for conducting a Delphi.

Software

• Software to support Delphi over the Internet has been reported.
• Principles of Forecasting A free service to support Delphi forecasting and references are available on this site. However, neither software nor code are available.

External links

• Need of Forecasting in Online Business
• The Delphi Method: Techniques and Applications, edited by Harold A. Linstone and Murray Turoff — a comprehensive book on Delphi method (free download, 11Mb PDF, 618 pages)
• RAND publications on the Delphi Method Downloadable documents from RAND concerning applications of the Delphi Technique.