IM Engineer Study Notes
← Back to list
AI & Data
#환각#Hallucination#RAG#사실성#AI신뢰성
Last updated · 2026-10-22

LLM Hallucination and Mitigation Techniques

1. Overview

A. Definition and Background

LLM hallucination refers to the phenomenon in which a large language model generates content that is untrue or inconsistent with the evidence provided, fluently and confidently as if it were true. The key point is that this is not a mere typo or grammatical error but output that takes a plausible form yet lacks truthfulness in its content.

An LLM is a probabilistic language generator that has learned the "probability distribution of the next token" from a vast corpus. The model's objective function is aligned not with "tell the truth" but with "predict the most plausible next token from the learned distribution." Therefore, even when it meets a gap in its knowledge, the model tends to fill the blank by stitching together statistically natural sentences rather than staying silent or answering "I don't know." This structural characteristic is the root cause of hallucination.

The important point is that hallucination arises not from the model's "intent to lie" but as a result of optimizing only for formal plausibility without any internal representation of truthfulness. The model fundamentally lacks a mechanism to clearly distinguish what it knows from what it does not, so it answers questions it knows and questions it does not with equal fluency. The fundamental reason hallucination is hard to mitigate lies here as well — the root of the problem is inherent in the generative paradigm itself, not in a specific bug.

The reason hallucination has emerged as a serious problem is that the scope of LLM use has expanded beyond advisory and search assistance into high-stakes domains such as medicine, law, finance, and public administration. A fluent falsehood makes it hard for users to notice the error, leading directly to wrong decisions, legal liability, and damaged brand trust. In fact, the case of a lawyer who submitted fake precedents fabricated by ChatGPT to a court and was sanctioned (United States, 2023) shows that hallucination is not an abstract risk but a concrete loss.

OpenAI's "Why Language Models Hallucinate," published in September 2025, drew attention by reinterpreting hallucination not as a model defect but as a product of the incentives created by the training and evaluation system. If a model is scored on accuracy alone, then a strategy of guessing to get the answer right is more advantageous in points than answering "I don't know" to a question it does not know. In other words, as long as benchmarks penalize expressions of uncertainty and reward confident guesses, the model will learn to hallucinate. This suggests that addressing hallucination is a matter not only of algorithms but of evaluation design and governance.

B. Necessity and Characteristics

The reason hallucination mitigation is necessary is that reliability is the primary quality factor that governs the adoption and diffusion of an LLM service. No matter how high the productivity in summarization, translation, or coding, once a single fatal piece of false information occurs, the organization removes that system from operation. Especially in regulated industries, explainability, traceability, and verifiability are conditions of adoption, so a model that cannot guarantee the factuality of its output cannot get past the pilot stage.

The characteristics of hallucination can be summarized in three points. First, the coexistence of fluency and error: the sentences are smooth but the content is wrong, making the error hard to spot on the surface. Second, overconfidence: the model assigns high confidence even to wrong answers, lowering the user's guard. Third, nondeterminism and difficulty of reproduction: a different hallucination appears each time for the same question, making it hard to filter them all out with prior testing. These three combine to make hallucination a hard problem that is "detectable and mitigable but impossible to eliminate completely."

Therefore, the realistic goal is not the complete eradication of hallucination but its control to an acceptable level. Designing differentiated lines of defense according to risk, and managing the residual risk with human review, source attribution, and disclaimers — this risk-based approach is the professional-engineer standard.

Meanwhile, the view that sees hallucination only as "a bug to be removed" is one-sided. In tasks where creativity is the value, such as fiction, ideation, and brainstorming, free generation unbound by facts is actually a benefit. That is, hallucination and creativity are two sides of the same probabilistic generative capability, and the essence of the problem is "its being uncontrolled in contexts that demand factuality." Hence the goal of mitigation, too, should be set not as killing the generative capability but as context-aware control that adjusts the strength of factual grounding according to the nature of the task (fact-based vs. creative).

2. Types of Hallucination and How They Arise

The reason we first distinguish the types of hallucination is that only by identifying the type can we choose the right mitigation and evaluation metric. Even for the same "wrong answer," the prescription is the opposite depending on whether it conflicts with world facts or with the given document. Below, we divide hallucination along representative classification axes.

Hallucination is largely split in two by "against what is it wrong." One is factuality hallucination, which conflicts with facts about the world, and the other is faithfulness hallucination, which conflicts with the provided input, context, or source document. Fabricating content not in the source in a summarization task is the typical case of the latter. We also distinguish intrinsic hallucination, which distorts the input information itself, from extrinsic hallucination, which arbitrarily adds external information not in the input. This classification matters because an effective mitigation differs by type — factuality hallucination is well served by external-knowledge linkage (RAG), and faithfulness hallucination by grounding and constrained decoding.

The structure diagram below provides an overall view of where in the LLM pipeline hallucination originates. Each stage — data, training, inference, and evaluation — acts as a different source of hallucination.

flowchart TB
  subgraph DATA["Data stage"]
    D1["Corpus errors, bias, outdated info"]
    D2["Knowledge gaps(long-tail, recent events)"]
  end
  subgraph TRAIN["Training stage"]
    T1["Probabilistic objective(next-token prediction)"]
    T2["RLHF sycophancy incentive"]
  end
  subgraph INFER["Inference stage"]
    I1["Randomness of sampling decoding"]
    I2["Context-window limit, prompt ambiguity"]
  end
  subgraph EVAL["Evaluation stage"]
    E1["Accuracy-centric scoring(rewards guessing)"]
  end
  DATA --> OUT["Hallucinated output(fluent falsehood)"]
  TRAIN --> OUT
  INFER --> OUT
  EVAL -.provides incentive.-> TRAIN

On the data side, hallucination sprouts from errors, bias, and lack of timeliness in the training corpus. For long-tail knowledge the model never learned, or for recent events after its training point (after the knowledge cutoff), it has no basis and can only make things up. For example, "rare and frequently changing facts" such as a particular person's birth year or a new product's specifications are a habitual zone of hallucination. A fact that appears only once in the corpus has a weak statistical signal, so the model tends not to remember it stably and to substitute a different value of a similar pattern. This is why OpenAI's research links a substantial share of hallucination to "facts that appear rarely in the training data."

On the training side, the probabilistic objective explained earlier is the primary cause. On top of this, reinforcement learning from human feedback (RLHF) rewards "the answer the user prefers," which can bias the model toward producing a pleasing answer (sycophancy) over an accurate one. A representative phenomenon: when a user asks a question built on a wrong premise, the model goes along with that premise and confidently produces a wrong answer.

On the inference side, the probabilistic decoding introduced for diversity (temperature, top-p sampling) shakes factuality. Raising the temperature increases creativity but also the probability that a deviating token is chosen. In addition, the context-window limit causes information in the middle of a long document to be lost (lost-in-the-middle), and an ambiguous prompt triggers the model's arbitrary completion.

The evaluation side is the most easily overlooked source. In OpenAI's SimpleQA factuality measurement, a model that barely guessed and abstained on about half with "I don't know" (the gpt-5-thinking-mini line) had an error rate of about 26%, whereas an older model that barely abstained and mostly guessed (the o4-mini line) had similar accuracy but an error rate that shot up to about 75%. Looking at accuracy alone, the gap between the two seems small, but by "the proportion answered wrongly," it becomes clear that strategic guessing mass-produces hallucination. This clearly shows that hallucination is a matter not only of data, training, and inference but of what we score the model to reward.

In this way, hallucination is not a defect of a single stage but a systemic phenomenon produced by data, training, inference, and evaluation entangled like a chain. Therefore a local prescription that fixes only one point is prone to recurrence, and, as discussed later, multilayer defense spanning before, during, and after generation must go together with a redesign of the evaluation system for the effect to last.

3. Hallucination Mitigation — Multilayer Defense Before, During, and After Generation

The reason to view mitigation techniques by layer is that a prescription for a single stage alone cannot stop hallucination arising at another stage. Even if you inject good evidence (before generation), hallucination occurs if decoding wavers (during generation), and even if in-generation control is perfect, it is powerless if the knowledge was absent to begin with (a pre-generation gap). Therefore defense must be layered across the entire chain, not at a single point.

Hallucination is not solved by a single technique; it is effective to overlap lines of defense at the three layers of pre-generation (input augmentation), during generation (control), and post-generation (verification). The process diagram below shows the mitigation devices placed at each stage from when a query comes in to when a response goes out.

flowchart LR
  Q["User query"] --> PRE["Pre-gen: RAG retrieval, prompt design"]
  PRE --> GEN["During-gen: constrained decoding, CoT, uncertainty calibration"]
  GEN --> POST["Post-gen: self-verification(CoVe), factuality scoring"]
  POST --> GATE{"Pass confidence threshold?"}
  GATE -->|"Yes"| ANS["Provide response with sources"]
  GATE -->|"No"| ABSTAIN["Answer I-don't-know, hand to human review"]

The core of pre-generation defense is retrieval-augmented generation (RAG). When relevant evidence is retrieved from an external knowledge base or documents at query time and injected into the prompt, the model answers bound to the presented evidence rather than to its parametric memory. This directly attacks the knowledge-cutoff and long-tail problems and raises verifiability by presenting citations as well. However, if retrieval quality is poor it can instead reinforce hallucination with wrong evidence, so reranking and evidence-consistency checks must go together. The prompt design used alongside it also has meaning as a first line of defense — specifying role, scope, and evidence-use rules ("if it is not in the provided document, answer I don't know") can suppress a substantial part of the model's arbitrary completion.

During-generation defense controls the reasoning process itself. Inducing step-by-step reasoning with chain-of-thought (CoT) can reduce logical leaps, and constrained decoding, low temperature, and structured output (a JSON schema) suppress deviation. Furthermore, redesigning the incentive through uncertainty calibration so that the model answers "I don't know" when its confidence is low is the direction OpenAI's research proposes. That said, CoT increases cost as the reasoning length grows and sometimes has the adverse effect of a wrong intermediate reasoning justifying a plausible wrong answer, so it should be applied selectively according to task difficulty.

Post-generation defense verifies the output after the fact. Chain-of-Verification (CoVe) has the model itself create verification questions, cross-check the answer, and then revise it, while self-consistency draws multiple samples and stabilizes by majority vote. Representative detection techniques are SelfCheckGPT, which catches hallucination by the consistency of multiple responses without additional evidence, and FActScore, which breaks the output into atomic fact units and scores accuracy against evidence.

The last bastion is guardrails and human-in-the-loop (HITL) at the operational stage. Guardrails apply prohibition and factuality rules to the output to block or correct risky responses, and when confidence falls below the threshold they withhold the response and hand it to human review. In domains where the cost of error is high, such as medical consultation support, it is designed as "the model produces a draft and an expert approves it," filtering hallucination out before it reaches the end user. This is a device that absorbs the residual risk left by technical mitigation into organizational process.

The table below organizes representative mitigation and detection techniques by defense layer. Each technique differs in the hallucination type it attacks and in cost and limits, so they are combined to match the service's risk level.

Layer Representative technique Main target Limit, cost
Pre-gen RAG, knowledge-graph linkage Factuality(knowledge gap) Depends on retrieval quality, latency, infra
Pre-gen Prompt design, role setting Faithfulness, ambiguity Limited, variable effect
During-gen CoT, constrained decoding, low temperature Reasoning leaps, deviation Reduced diversity
During-gen Uncertainty calibration, abstain Overconfidence Lower recall(excessive refusal)
Post-gen CoVe, self-consistency Factuality, consistency Inference cost up several-fold
Post-gen SelfCheckGPT, FActScore Detection, scoring Reliability of automatic evaluation
Operations Guardrails, human review(HITL) Residual risk Cost, speed

4. Comparison — Why the Effect Differs by Technique

RAG and fine-tuning are often compared, but their points of attack differ. Fine-tuning is strong at adapting the model's style, format, and domain, but because it "memorizes" knowledge, it is weak on timeliness and verifiability. Conversely, RAG externalizes knowledge, so it is easy to update and allows source tracing, but it collapses outright when retrieval fails. Therefore, "frequently changing facts" are handled with RAG and "a consistent tone and format" with fine-tuning, while high-risk services combine the two — this is the practical standard.

There is also a fundamental cost-effectiveness difference between prompt-based techniques and retrieval/verification-based techniques. Prompt design (an instruction such as "if there is no evidence, answer I don't know") costs almost nothing and is immediately applicable, but because it depends on the model's voluntary compliance, its effect is unstable and it is easily broken by adversarial queries. By contrast, RAG and CoVe structurally enforce factuality through external evidence or additional reasoning, so the effect is large and stable, but they entail infrastructure, latency, and inference cost. The reason this difference arises is that the former "asks the model" whereas the latter "binds the model," and the more trust matters, the more reasonable it is to shift the weight from asking to binding.

The difference in effect among mitigation techniques ultimately comes from "where the root of hallucination is cut." In fact, in a 2025 experiment combining CoT, RAG, self-consistency, and self-verification (arXiv 2505.09031), a configuration adding self-verification to RAG and CoT greatly lowered the hallucination rate on HaluEval (to about 11% on GPT-3.5-Turbo) and showed the most stable performance on FEVER and TruthfulQA as well. This supports the intuition that the complementary effect is greatest when external-knowledge binding (RAG) and post-hoc verification are combined. However, since it is a single paper's result, the numbers should be taken as reference values, and there is a trade-off in which inference cost and latency rise several-fold as verification steps increase.

On the evaluation side, too, each benchmark measures a different target. TruthfulQA measures whether the model follows misconceptions people commonly believe, HaluEval measures the ability to identify hallucination samples, and FEVER measures fact verification between a claim and evidence. Hallucination cannot be concluded from a single score, so multiple metrics suited to the service's characteristics and real-data monitoring in operation must go together.

From an industrial-application perspective, too, the choice of mitigation strategy diverges by domain characteristics. For example, for a customer-support chatbot, binding FAQs, terms, and product manuals into a RAG knowledge base and enforcing the principle of "no answers without evidence" becomes the first line of defense. On the other hand, a code assistant can immediately confirm the factuality of generated code with the external verification tool of compilation and test execution, so tool use is a more powerful mitigation than self-verification. In this way, first finding "what means exist to verify facts cheaply in that domain" is the starting point of technique selection.

Hallucination must also consider the interaction among mitigation techniques. Even if RAG injects evidence, if the model ignores the evidence and prioritizes parametric memory, faithfulness hallucination remains, and overtrusting self-verification can create a secondary hallucination that packages a wrong answer as "passed verification." Therefore, rather than stacking each technique independently, including cross-verification between lines of defense — such as scoring evidence-response consistency with a separate model — in the design is what secures substantive reliability.

5. Deep Dive — Redesigning Evaluation Incentives and Linking to AI Governance

The latest discussion of the hallucination problem is shifting its center of gravity from algorithms to evaluation and institutional design. OpenAI's 2025 research points out that "what makes the model guess is the scoring method of the benchmark," and proposes a shift to a scoring system that gives no penalty to expressions of uncertainty but assigns a penalty to wrong answers and neutrality to abstention. In fact, the GPT-5-line reasoning models are introduced as a case of rebalancing abstention and accuracy, designed to lower the error rate at the cost of raising the rate of abstaining with "I don't know."

This perspective connects directly with the AI trustworthiness, AI ethics, and governance topics frequently covered in the professional-engineer exam. The EU AI Act imposes accuracy, robustness, and transparency requirements on high-risk AI, and domestically too, the AI Basic Act and the trustworthy-AI guidelines are trending toward explicitly naming the misinformation risk from hallucination as a management target. Therefore an organization should incorporate hallucination as a risk-management item across the whole model lifecycle, not as a mere technical bug, and embed hallucination monitoring, red-teaming, and incident-response procedures into the MLOps/LLMOps pipeline. As a likely exam direction, a question demanding integrated writing of technology plus management, such as "explain the architecture (RAG) and governance system for mitigating the hallucination risk of an LLM service," is probable.

The latest technology trends are also evolving to target hallucination mitigation. Reasoning models raise factuality by going through a longer internal thought process before producing an answer, and they reduce "memory-dependent guessing" by delegating computation, search, and DB lookup externally through tool use and function calling. In addition, a multi-agent verification structure in which several agents share the roles of generation, critique, and verification is spreading in high-risk applications. These commonly share the principle of "do not let the model answer alone from memory, but bind it to external evidence, tools, and peer verification," and this direction is expected to become the mainstream of hallucination-mitigation technology for the time being.

6. Considerations and Implications

First, risk-based differentiated design. Applying the highest level of verification to every response causes cost and latency to explode. It is reasonable to differentiate the intensity of applying RAG, CoVe, and human review according to the query's risk level (medicine/law vs. brainstorming), designing low-risk areas with light guardrails and high-risk areas with multilayer verification and mandatory source attribution.

Second, the trade-off between mitigation and usefulness. If you excessively induce "I don't know" to reduce hallucination, refusals increase and user utility plummets (excessive abstention). Conversely, chasing usefulness alone increases hallucination. The professional engineer must set the balance point of recall (how much it answers) and precision (how much it is right) to match the service goal, and handle that boundary operationally with uncertainty calibration.

Third, institutionalizing verifiability and accountability. Exposing source attribution, evidence links, and confidence scores in the UI to help users verify for themselves, and establishing an explanation, disclaimer, and audit-trail system that makes clear that final-judgment responsibility rests with the human, are essential. This is both regulatory compliance and the core of building trust.

Fourth, continuous operation and linked technologies. Because knowledge changes ceaselessly, RAG knowledge-base refresh, analysis of hallucination incident logs, and periodic red-teaming and benchmark re-measurement must be established as LLMOps routines. Also, since hallucination mitigation is maximized when combined with adjacent technologies such as RAG, knowledge graphs, multi-agent verification, and guardrails, it is desirable to design the defense layers together from the earliest stage of the overall AI architecture.

Along with this, addressing hallucination must encompass the human factor of user education and interface design. Because residual hallucination remains no matter how much it is mitigated, a design is needed that specifies confidence levels and evidence in the UI so users hold the awareness that "AI output is a draft requiring verification," and that induces direct checking of source originals for important decisions. Substantive safety is secured only when technical mitigation and user literacy are combined.

Fifth, a shift in evaluation culture. Even in an organization's internal model evaluation, chasing only "number one in accuracy" causes hallucination to recur in the field. Introducing a scoring standard that legitimately recognizes abstention and expressions of uncertainty, and a continuous measurement system based on real-use data, to correctly design the very incentive that evaluation creates, is the fundamental prescription.

References

In one line: LLM hallucination is a "fluent falsehood" that arises when the probabilistic language-generation structure combines with accuracy-centric evaluation incentives; rather than complete elimination, the professional-engineer standard is to control it to an acceptable level through multilayer defense — RAG (pre-generation), constrained decoding and uncertainty calibration (during generation), and verification chains and factuality scoring (post-generation) — together with risk-based governance.