← Back to list
Project & Org Management
#델파이#전문가합의#기술예측#리스크식별#공수산정
Last updated · 2026-11-19

The Delphi Technique — Expert-Consensus-Based Forecasting and Decision Making

1. Overview

A. Definition

The Delphi Technique is a structured group communication and forecasting method that repeatedly administers anonymous questionnaires over several rounds to a panel of geographically dispersed experts, statistically summarizes the group's responses and feeds them back after each round, and has participants revise their answers, thereby gradually driving convergence toward consensus and a forecast.

The essence of Delphi is that it "gathers collective intelligence without putting the experts in one room." Face-to-face meetings are fast but vulnerable to the pathologies of groupthink — conformity to the majority (bandwagon), domination by the most vocal person, and rigidity in which people cannot change their stance for fear of losing face. By anonymizing responses and converting them into written iterations, Delphi shares the 'rationale' of an opinion while blocking the 'authority of the speaker'. As a result, participants revise their judgments only because of the arguments presented by peers, not social pressure, and as this process repeats, extreme values are filtered out and the distribution narrows.

A key point is that the goal of Delphi is not necessarily 'unanimity'. Failing to converge and instead splitting into two or more stable clusters (stable disagreement) is also a meaningful outcome, revealing that fundamental uncertainty or a clash of values exists around the issue. Delphi should therefore be understood not as an 'answer-generating machine' but as a device that structures and surfaces the distribution of expert judgment and its rationale in uncertain domains.

B. Background and Necessity

Delphi was developed in the 1950s–60s at the RAND Corporation in the United States by Olaf Helmer and Norman Dalkey for military technology forecasting (Project Delphi). The problem of the time was, for example, "how many atomic bombs would the Soviet Union need to paralyze U.S. industry" — a question with no historical data and no possibility of experiment, forcing reliance on expert estimation alone. In a domain where statistical models could not be used, the central motive was how to synthesize individual experts' intuition without bias, and the name was taken from the Oracle of Delphi in Greece, which foretold the future.

The arrival of Delphi also embodies a shift in perspective on 'what forecasting is'. Until then, forecasting the future was regarded as the exclusive province of quantitative models such as trend extrapolation, but as domains in which the past fails to explain the future — discontinuous innovation or social change — grew, a method to elevate expert judgment into systematic data became necessary. Delphi was an attempt to raise this judgment from 'individual intuition' to 'structured collective estimation', giving even subjective forecasts procedural rigor and reproducibility.

From the perspective of a professional engineer in information management, Delphi matters because a large share of IT projects' key decisions sit in judgment domains where quantitative data is scarce. Predicting the timing of a new technology's adoption, estimating the effort for a novel service with no comparable precedent, assessing the likelihood and impact of security threats, and prioritizing informatization policy cannot be answered from past records alone. Relying on a single authority here fully exposes you to individual bias, but structuring the judgment of many experts through Delphi lets you offset bias and make the estimate's confidence interval visible. Its practical applicability is especially high in risk identification, wideband effort estimation, and technology roadmap development.

C. Characteristics

Delphi is distinguished from other group techniques in three respects. First, anonymity separates the speaker from the opinion, removing the pressure of authority and conformity. Second, through iteration and controlled feedback, it is designed so opinions mature and converge over several rounds rather than a single aggregation. Third, it presents results as a statistical group response (median and quartiles), managing the degree of consensus and the distribution of disagreement quantitatively. Thanks to these three traits, Delphi — unlike impromptu brainstorming — becomes a reproducible, documented, structured forecasting procedure.

Another important characteristic is that it can be conducted asynchronously and remotely. Since participants need not gather at the same time and place, scarce experts from around the world can be assembled into a panel, with little scheduling burden. This is a practical strength of Delphi in global, distributed organizational environments where face-to-face techniques are virtually impossible, and it is maximized in the real-time e-Delphi discussed later.

2. The Four Core Principles and Structure of Delphi

flowchart TB
  FAC["Facilitator<br/>questionnaire design, anonymous aggregation, feedback"]
  subgraph PANEL["Expert Panel — anonymous, dispersed"]
    E1["Expert A"]
    E2["Expert B"]
    E3["Expert C"]
    EN["Expert N"]
  end
  FAC -- "distribute round questionnaire" --> PANEL
  PANEL -- "anonymous responses, rationale" --> AGG["statistical aggregation<br/>median, interquartile range(IQR)"]
  AGG -- "summary, minority-rationale feedback" --> FAC
  AGG -- "convergence test(IQR, consensus rate)" --> DEC{"converged?"}
  DEC -- "no: next round" --> FAC
  DEC -- "yes: stop" --> OUT["final consensus, forecast distribution"]
  style FAC fill:#e8f0fe,stroke:#2f6fed,stroke-width:2px
  style DEC fill:#fff4e5,stroke:#d97706,stroke-width:2px

Delphi's operation can be summarized by four principles. Each principle is a design prescription aimed at a specific pathology of face-to-face meetings, so it carries the philosophy of the technique rather than being a mere procedural rule. As the structure diagram shows, the facilitator does not debate directly with the panel but acts as a 'neural hub' mediating the cycle of distributing questionnaires → collecting anonymous responses → statistical aggregation → writing feedback, and this cycle repeats until it passes the convergence test.

A. Anonymity. All responses are collected and shared anonymously via the facilitator. Participants encounter only the 'claim and rationale' without knowing who the other is, so the influence of rank, personality, and loudness of voice on the conclusion is blocked. This is especially effective in organizations with pronounced hierarchy. For example, it is hard for staff to voice dissent in a threat-assessment meeting attended by the CISO, but in an anonymous Delphi they compete on argument alone, independent of rank.

B. Iteration & Controlled Feedback. Delphi usually proceeds over 2–4 rounds, and after each round the facilitator organizes a summary of the group's responses and the rationale of minority opinions and feeds them back. Participants read these and, in the next round, keep or revise their positions. The key is the modifier 'controlled'. The facilitator does not relay every utterance verbatim but edits around the issues and rationales to filter out noise. This feedback loop creates collective learning.

C. Statistical Group Response. The final result is presented not as a 'majority vote' but as distributional statistics such as the median and the interquartile range (IQR). The median rather than the mean is used to reduce the influence of extreme outliers, and the IQR is examined alongside it because the degree of consensus (dispersion) itself is made an object of management.

D. Expert Panel. Result quality depends on panel composition. Gathering only homogeneous experts converges quickly but leaves group bias intact, while an excessively heterogeneous panel does not converge. Typically around 10–18 members are recommended, and the crux is securing diversity of perspective and domain expertise at the same time. Accuracy does not rise in proportion to indiscriminately enlarging the panel; rather, operational burden and attrition grow, so finding a point that is 'diverse enough yet manageable' is the realistic design.

These four principles work interlockingly. Anonymity alone is no different from a one-off survey, and iteration alone merely accumulates noise. Only when a virtuous cycle holds — anonymity removes pressure, statistical feedback supplies the material for learning, iteration matures that learning, and panel diversity offsets bias — does Delphi produce 'collective intelligence' better than the simple sum of individual expert estimates. Bear in mind that this is an interdependent structure in which, if any one principle collapses (for example, the facilitator omits minority rationale), overall quality drops sharply.

Core principle Targeted pathology of face-to-face meetings Design prescription
Anonymity Authority, conformity, face-saving Anonymous responses via the facilitator
Iteration, feedback One-off judgment, rationale not shared Multiple rounds + minority-rationale feedback
Statistical response Domination by the loudest Distribution shown via median and IQR
Expert panel Sampling bias Balanced composition of diversity and expertise

The common goal these four principles pursue is a "democracy of rationale" — that is, placing the power to change the conclusion not in a person's status but in the persuasiveness of the argument presented. Hence the most important deliverable in Delphi is not the final figure but the accumulation of rationale that explains why it converged on that figure. If this record of rationale is poor, then even if the numbers converge cleanly, the method lacks reproducibility and accountability.

3. Procedure and Convergence Test

flowchart LR
  P0["0. define the problem<br/>goal, panel selection criteria"] --> R1["round 1(open-ended)<br/>elicit free opinions, items"]
  R1 --> R2["round 2(structured)<br/>rate, rank each item"]
  R2 --> FB["statistical feedback<br/>median, IQR, request outlier rationale"]
  FB --> R3["round 3(revise)<br/>re-rate after reading feedback"]
  R3 --> CHK{"convergence criteria met?<br/>IQR narrowed, consensus rate"}
  CHK -- "no" --> FB
  CHK -- "yes" --> FIN["compile results, report"]
  style P0 fill:#e8f0fe,stroke:#2f6fed,stroke-width:2px
  style FIN fill:#e7f6ec,stroke:#16a34a,stroke-width:2px

Stage 0 (preparation, problem definition). Clearly define the question to be solved, and fix in advance the panel selection criteria (expertise, experience, neutrality of interest) and the termination conditions. If termination conditions are not set beforehand, rounds expand indefinitely or end at the facilitator's discretion, damaging the objectivity of the result. If the question's scope is too broad or ambiguous at this stage, responses diverge in later rounds and never converge, so a design that decomposes the question into answerable units and agrees on definitions of terms determines success or failure.

Round 1 (open exploration). It usually begins in free-text form, broadly collecting items, scenarios, and risks to consider. For example, one asks something like "list the security issues that will pose the greatest threat to our service within the next three years." The facilitator removes duplicates and groups similar items to create a structured questionnaire for the next round.

Round 2 (structured evaluation). For the organized items, likelihood, impact, importance, and the like are rated and ranked on a Likert scale or numerically. After collection, the facilitator computes the median and IQR.

Round 3 onward (feedback-based revision). Along with the group statistics, participants — especially those who responded outside the IQR (upper and lower quartiles) — are asked to state their rationale. Other participants read this minority rationale and re-evaluate. In this process, a persuasive minority opinion can sometimes move the majority, and this is precisely the value of Delphi that distinguishes it from simple averaging. The core design is to 'request an explanation' rather than 'remove' extreme values, because sometimes the foresight of a minority corrects the average misjudgment of the majority.

Number of rounds and retention (attrition management). More rounds are not better. Empirically, most of the convergence gain is realized in 2–3 rounds, after which marginal utility drops sharply, so it is better to fix the maximum number of rounds in advance. As rounds lengthen, response fatigue causes participants to drop out, and mid-way attrition skews the sample toward a survivorship bias (a persistent minority). One should therefore track per-round response rates, manage retention with reminders, appropriate incentives, and simplified questionnaires, and explicitly note the limitation in interpreting results if response rates plummet.

Convergence test. When to stop is managed by quantitative criteria. Representatively, ① when the round-to-round decrease in IQR falls below a threshold, ② when the consensus rate (e.g., 70–80% of all respondents answering within a specific band) exceeds a standard, and ③ for rankings, a Kendall's coefficient of concordance (Kendall's W) of 0.7 or higher is taken as strong consensus. In addition, stability, where round-to-round response change becomes negligible, is used as a signal of convergence. Note that 'consensus = correct answer' does not hold; convergence may be a convergence of group bias, so the quality of the rationale must be examined alongside.

In practice, it is important to fix the convergence criteria in a document before project launch. If the criteria are set after the fact, it is hard to avoid the suspicion that the facilitator arbitrarily ended at the round where the desired conclusion emerged. Also, to prepare for the case where consensus ultimately fails to form, setting up a procedure that recognizes a "preserved disagreement" conclusion as an explicit deliverable, rather than forcing termination, increases the method's objectivity.

4. Types, Variants, and Comparison with Other Techniques

Delphi has evolved into several variants depending on purpose. Conventional Delphi focuses on numerical forecasting and consensus, while Policy Delphi aims not at consensus but at systematically surfacing opposing policy alternatives and their arguments, deliberately preserving disagreement. Real-time/e-Delphi uses a web platform to remove round boundaries, letting participants see the group statistics instantly and update their responses, shortening a process that took months to days. The Wideband Delphi famous in software engineering is a variant Barry Boehm applied to effort estimation, combining anonymous estimates with a facilitator-led discussion to share the rationale of the estimates. Agile's Planning Poker is also a lightweight descendant of Wideband Delphi.

As actual applications, in healthcare Delphi is used almost as a standard for clinical-guideline consensus, and in technology forecasting Japan's Ministry of Education has conducted a science and technology foresight survey (a panel of thousands of experts) by Delphi roughly every five years since 1971, reflecting it in national R&D strategy. In IT projects, it is common to estimate the effort of a new module by Wideband Delphi and decide a buffer (schedule slack) from the median and IQR of three or more independent estimates. For example, if three people estimated an unprecedented AI-integration module at 20, 35, and 60 person-days respectively, rather than using a simple average (about 38), one has them state in round 2 the reasons for the large estimate gap, surfacing the hidden risk of 'external-API uncertainty' and adding a matching buffer and a proof-of-concept (PoC) task.

Dimension Delphi Brainstorming Nominal Group Technique(NGT) AHP
Interaction Anonymous, written, remote Face-to-face, open Face-to-face(individual writing then discussion) Face-to-face/individual
Iteration Multiple rounds, repeated One-off Usually once Once(pairwise comparison)
Reaching consensus Statistical convergence Idea divergence Voting, rank aggregation Weight computation(consistency ratio)
Strength Bias blocking, remote-capable Fast divergence Equal participation Quantitative prioritization
Limitation Time, facilitator-dependent Conformity, domination Participant proximity needed Explosion of comparison pairs

As the table shows, technique choice is subordinate to purpose. When ideas must be poured out quickly, brainstorming fits; when the careful forecasts of geographically scattered experts must be gathered without bias, Delphi fits; and when quantitative prioritization is needed, AHP fits. In practice, a hybrid design that converges candidates with Delphi and then weights them with AHP is often used.

Conversely, one must also clearly recognize cases where Delphi is unsuitable. In an urgent situation requiring an immediate decision (incident response, crisis management), a Delphi taking several weeks does not fit, and using Delphi for a problem that has a verifiable correct answer and can be answered by experiment or measurement commits the error of substituting the weak evidence of expert opinion for strong fact. Also, when the issue is a clash of values and interests, so that factual convergence is impossible from the outset, one should switch to a design that clarifies the opposing structure, as Policy Delphi does, rather than forcibly squeezing out consensus.

5. Deep Dive — Digital Transformation and AI Integration Trends

The greatest weakness of traditional Delphi is its long duration (weeks to months) and high operational burden, and the fact that panel attrition grows as rounds lengthen. What emerged to resolve this is the web-based real-time Delphi, which automates questionnaires, aggregation, and feedback to dismantle round boundaries and sustain participation. Since responses are reflected in real-time statistics, participants can update their positions while viewing the group distribution at any time, and the facilitator's manual aggregation burden is greatly reduced.

Real-time Delphi, however, is not all advantages. As round boundaries disappear, there is a risk of premature convergence, in which participants conform too quickly to the group statistics, and the distribution of early respondents can act as an anchor for later respondents. Therefore, even in the real-time mode, the principles of traditional Delphi remain valid in that a design is needed that sufficiently exposes minority rationale and mitigates response-order effects.

Recently, integration with generative AI has drawn attention. There are growing attempts to use large language models (LLMs) to automatically cluster and summarize round-1 free responses into topics and items, or to structure minority rationale to assist feedback writing. Further, research on silicon sampling, which performs preliminary exploration using an LLM panel as virtual experts, is also underway. The points where AI can assist Delphi operations can be summarized as follows.

  • Round-1 response refinement: deduplicating free-text responses, clustering similar items, drafting a structured questionnaire.
  • Feedback-writing assistance: summarizing and classifying the rationale of quartile outliers to support the facilitator's feedback materials.
  • Pre-study: using an LLM virtual panel to discover issues and scenarios in advance, raising questionnaire-design quality.
  • Convergence monitoring: automatically aggregating and visualizing per-round changes in IQR, Kendall's W, and response rate.

That said, because AI risks converging on training-data bias as-is, the current cautious consensus is to confine it to a range that assists preprocessing and operations rather than replacing an actual expert panel. From the perspective of a professional engineer in information management, the task is to design such automation so it does not compromise audit trails (preserving response history and rationale) and methodological transparency.

6. Considerations and Implications

  • Application strategy — selectively deploy in data-absent domains: Delphi should be concentrated on judgment-intensive tasks where past records and experiment are impossible (new-technology forecasting, novel-service effort, threat-likelihood assessment), while domains with sufficient quantitative data should prioritize statistical techniques such as regression and simulation. Overusing it as an all-purpose tool lowers cost-effectiveness.

  • Trade-off — quality vs. speed, consensus vs. truth: Adding rounds raises convergence quality but increases duration and attrition, while cutting them yields only hasty consensus. Also, convergence is not the same as a correct answer and may be a convergence of group bias, so the quality of rationale must be evaluated in parallel with the consensus rate. Where preserving disagreement is the purpose, as in Policy Delphi, avoid forced convergence.

  • Facilitator risk management — neutrality and editorial power: Delphi results depend heavily on the facilitator's questionnaire design and feedback editing. If the facilitator steers toward a particular conclusion, the whole method is neutralized, so neutrality must be institutionally guaranteed and reproducibility secured through prior review of questionnaires, disclosure of feedback-editing standards, and preservation of raw response data.

  • Panel design — balancing diversity, expertise, and neutrality of interest: A homogeneous panel is fast but biased, while an overly heterogeneous panel diverges. Document the selection criteria in advance, control conflicts of interest (vendors, beneficiary departments), and secure an appropriate size (typically 10–18 members) and spare members against attrition.

  • Institutionalizing the operating procedure — guard against one-off eventization: Consuming Delphi as a one-off workshop for a particular project accumulates no know-how. Internalizing questionnaire templates, convergence criteria, a panel pool, and raw-response retention as a standard organizational process, and managing per-expert estimation accuracy by comparing past Delphi results with actual outcomes (calibration), improves the panel's calibration ability over successive rounds.

  • Ethics and governance — balancing participation and accountability: Anonymity is a device for eliciting candid opinions, but it can simultaneously conceal irresponsible responses. Anonymity is therefore about 'external disclosure', and the facilitator should retain response history so that it is internally traceable, and should agree in advance on panel-participation consent, data protection, and the scope of result use, thereby securing both the method's legitimacy and participants' trust.

  • Outlook and linked technologies: With real-time e-Delphi and generative-AI assistance, operating costs will fall and the scope of application will broaden. In practice, it is positioned as the front end of a hybrid decision pipeline that leads from qualitative convergence → quantitative prioritization and probabilistic analysis, combined as input to risk identification ([[it-project-risk-response]]), cross-validation of effort estimation ([[sw-cost-estimation]]), and with AHP ([[ahp]]) and Monte Carlo simulation ([[monte-carlo-simulation]]).

References


In one line: The Delphi Technique is a group decision-making method that repeatedly administers anonymous questionnaires to a dispersed expert panel over several rounds and feeds back statistical results each round, thereby structurally converging forecasts and consensus in data-scarce domains while blocking the bias of authority and conformity.