IM Engineer Study Notes
← Back to list
AI & Data
#생성형 AI 평가#LLM Evaluation#LLM-as-a-Judge#RAG 평가#평가 지표#AI 품질관리
Last updated · 2026-09-30

Generative AI Evaluation and Quality Management with LLM-as-a-Judge

1. Overview

Generative AI evaluation is a lifecycle activity that measures and improves whether a system combining models, data, prompts, retrieval, and tools meets evidence-based criteria for accuracy, usefulness, safety, and reliability in its intended work.

Generative AI can produce different outputs for the same input, and fluent wording does not guarantee factual accuracy. A few demonstrations or a single benchmark score can therefore miss hallucinations, bias, and data leakage. Before discussing model scores, an information-management professional should define the business purpose, acceptable risk, users, and cost of failure. These criteria make trade-offs among accuracy, response time, and safety explicit.

The evaluation target is not only a foundation model. It is the application system, including datasets, prompts, retrievers, generators, tool calls, and user interfaces. Model replacement, knowledge-base updates, and prompt changes can all alter results. The same evaluation cases should be rerun, with results compared across versions.

Evaluation is both a release control and a development feedback loop. Operational data can reveal new failure cases that should be reviewed and added to future test sets. This note covers objectives, test-data design, quantitative and qualitative metrics, LLM-as-a-Judge, RAG and agent evaluation, and operational governance. For related material, see [[rag]] for RAG architecture and [[llmops]] for lifecycle operations.

2. Evaluation framework and lifecycle

A. Evaluation architecture

The quality of a generative AI service cannot be explained by comparing only its input and final answer. Input context, retrieval quality, prompt constraints, model output, and tool results can affect one another. Evaluation should therefore be decomposed into model, component, and business-scenario levels, then integrated into end-to-end quality.

flowchart LR
  A[Business goals and risk criteria] --> B[Evaluation dataset]
  B --> C[Model evaluation]
  B --> D[Component evaluation]
  C --> E[End-to-end scenario evaluation]
  D --> E
  E --> F[Human review and approval]
  F --> G[Deployment and monitoring]
  G --> H[Failure-case collection]
  H --> B

Business goals are the starting point for evaluation criteria. For internal-policy Q&A, evidence and freshness may be critical; for code generation, execution success and security defects may dominate. The required cases and thresholds change with context and potential harm, even when the model is unchanged.

An evaluation dataset structures inputs, expected outputs or grading criteria, risk levels, data provenance, and use scenarios. Tasks with one unambiguous answer can use gold labels; open-ended tasks such as summarization need rubrics and expert judgment. When data includes customer conversations or sensitive content, purpose limitation, access controls, de-identification, and retention must be designed separately.

Component evaluation narrows down where an error occurs and informs what to improve. If retrieval and generation failures are not distinguished, teams may pay to replace a model without fixing the underlying issue. End-to-end evaluation checks whether a real user request succeeds and whether the system safely abstains or hands off when it fails.

B. Evaluation procedure

First, express the evaluation objective as an observable business success condition. Instead of “a good answer,” specify, for example, “cite the relevant policy clause and abstain when no supporting evidence exists.” Include not only effectiveness but also latency, cost, human-review rate, and acceptable error risk.

Second, collect representative examples of real use along with boundary cases. Include ordinary requests, ambiguous questions, multilingual input, typos, long context, conflicting information, and adversarial prompts. Do not exclude rare but high-impact cases solely because they are infrequent; maintain a separate risk-based test group.

Third, select metrics and grading methods, then measure a baseline system. Record model, prompt, retrieval settings, embedding model, data snapshot, evaluation code, and randomness settings for reproducibility. Inspect confidence intervals, subgroup outcomes, and failure distributions as well as averages.

Fourth, classify errors, form an improvement hypothesis, and retest against the same evaluation set and an independent validation set. Repeatedly tuning on only one test set can overfit to its cases and reduce performance on new user inputs. Keep the validation set for fixed regression checks and review newly observed production cases before adding them.

Fifth, connect release thresholds with operational monitoring. A critical safety failure can be a release blocker rather than something that an average score is allowed to offset. Track answer quality, abstention rate, retrieval failures, cost, latency, and user reports; reevaluate when distributions shift.

3. Metrics and test data

A. Decomposing quality dimensions

Correctness measures whether an output matches facts or an answer standard. Accuracy, precision, recall, F1, and exact match can be used when labels are clear. For open-ended generation, do not rely on word overlap alone; evaluate task meaning, including factuality, relevance, and completeness.

Groundedness measures whether claims are supported by supplied documents or verifiable evidence. Faithfulness metrics check whether a RAG answer is consistent with its context, but the context itself may be stale or wrong. Groundedness therefore does not replace fact checking against current source material and provenance.

Usefulness and relevance measure whether the answer addresses user intent and includes necessary information. Safety evaluates harmful output, personal-data disclosure, bias, prompt injection, and risky tool execution. Operational quality includes success rate, p95 latency, token cost, retry rate, human handoff, and service availability.

B. Metric selection and interpretation

Evaluation area Example metrics Interpretation cautions
Correctness Accuracy, F1, exact match Accuracy can mislead on imbalanced classes
Retrieval Context precision/recall, ranking metrics Check relevant-document availability and rank
Generation Relevance, completeness, faithfulness No single score represents every quality dimension
Safety Policy-violation, toxicity, leakage rates Segment by severity and attack type
Agents Tool-selection accuracy, execution success Check arguments, permissions, and side effects
Operations p95 latency, cost/request, handoff rate Segment service levels by user and business task

Context precision is the share of retrieved material relevant to the query; recall indicates how much relevant material was not missed. Higher precision can reduce noise but may omit important evidence, while higher recall can give the generator more irrelevant context. Tune top-k, chunking, reranking, and generation prompts together for the target workload.

Quantitative metrics can conflict. For example, making an answer shorter can reduce verbosity but omit a necessary procedural step. A metric registry should document the definition, unit, data scope, formula, owner, threshold, and known limitations.

A test set is not simply a copy of live traffic. Check for personal data, copyright issues, and overrepresentation of particular customers or regions. Stratify inputs by distribution, language, difficulty, and risk. Synthetic data can fill rare-case gaps, but may reproduce the generating model’s biases; review it with experts and real cases.

4. LLM-as-a-Judge

A. Principle and use

LLM-as-a-Judge uses a separate language model to score an answer against a rubric, reference answer, or pairwise comparison. It can reduce the cost of reading every answer and score large volumes of open-ended results in a consistent format. It is useful for dimensions such as relevance, groundedness, and completeness that simple string matching cannot capture.

Clearly separate the question, context used by the system, generated answer, grading criteria, and optional reference answer. Constrain output to a structured schema such as a rating, pass/fail, and supporting evidence rather than free-form prose. Preserve the input version, judge model and prompt, grading rationale, and agreement with human judgments—not just the score.

sequenceDiagram
  participant D as Evaluation data
  participant S as Target system
  participant J as Judge model
  participant H as Expert review
  D->>S: Query and fixed context
  S-->>D: Answer and trace data
  D->>J: Answer, rubric, and evidence
  J-->>D: Score and rationale
  D->>H: Sample and disagreement cases
  H-->>D: Gold label and error class
  D->>S: Improvement request and regression test

B. Bias and calibration

A judge is itself a probabilistic model and does not guarantee correct labels. Verbosity bias may favor longer answers, position bias may favor the first or last response, and style bias may favor certain wording. Self-preference and conflicts between safety policy and task-specific criteria also require testing.

Start calibration with a small, expert-labeled set. Report agreement, false positives, false negatives, grade-level confusion, and subgroup disagreement. Then refine the rubric to address observed error patterns. Even high overall agreement can hide failures concentrated in a critical risk group.

Pairwise comparison can reduce differences in absolute score scales by judging two answers against the same criteria. Use order-swapped repeats or blind evaluation to reduce position effects. Fix the question, answer length, and context; record model versions and sampling parameters.

A judge should extend human review, not replace it. Maintain domain-expert review and an appeal path for high-impact decisions, legal or medical judgments, discrimination, and actual-harm assessments. If the automated score is misaligned with the business objective, teams may optimize the score instead of the outcome. Review evaluation criteria regularly to detect this form of metric gaming.

5. Evaluating RAG and agent systems

RAG evaluation diagnoses retriever and generator components separately, then verifies the usefulness of the complete answer. Retrieval tests cover document recall, ranking, duplication, freshness, and access-control filtering. Generation tests cover relevance, faithfulness, completeness, citation accuracy, and abstention when evidence is missing.

For example, an internal-policy assistant can retrieve the right clause but misinterpret its effective date. Retrieval has succeeded, yet the service has failed. Conversely, an answer can be faithful to retrieved text that is itself an obsolete policy. Keep document version, effective date, access rights, and source identifiers with the evaluation data.

Agent evaluation also covers intermediate actions, not just the final sentence. Test request classification, planning, tool selection, argument formation, result interpretation, approval requests, error recovery, and termination conditions. When an agent writes to an external system, verify the impact of incorrect calls and the ability to undo them.

Success rate alone is insufficient for multi-step workflows. Also measure unnecessary tool calls, tokens and cost, completion time, retries, loops, permission violations, and human handoffs. When a tool fails or evidence is insufficient, safe termination may be preferable to forcing completion.

6. Comparisons and application cases

Method Strength Limitation and fit
Rules and string matching Fast, deterministic, reproducible Weak for open-ended semantic answers
Traditional quantitative metrics Useful for scale and trend analysis Depends on reference answers and label quality
LLM judge Scalable semantic scoring Requires bias checks and human calibration
Expert review Deep contextual and risk judgment Cost, speed, and reviewer disagreement
Live experiments Observes actual behavior and satisfaction Requires risk controls; beware sample and causal bias

These methods complement rather than replace one another. Before deployment, combine automated regression tests with expert review of a sample. In production, use safe online metrics and user reports together. Interpret quality, risk, and cost as a balance rather than compressing every outcome into one number.

Case 1 — Internal knowledge search: include questions that mix recently revised and retired policies. Test retrieval freshness and citation correctness. When the right clause cannot be found, verify that the system abstains and routes the user to an owner rather than inventing an answer. If recall is high but obsolete text is often returned, improve document lifecycle controls and reranking.

Case 2 — Customer support: stratify refund, contract, and privacy inquiries by intent. Include multilingual wording, ambiguity, and adversarial input. Measure policy compliance, answer correctness, agent handoff, and latency together. Set a human-approval boundary for high-risk refund decisions. Customer feedback is a useful signal, but a sample containing only complaints can understate overall satisfaction.

Case 3 — Code-generation agent: validate build and unit-test success, security findings, tool-argument accuracy, and repository change scope rather than code-text similarity. Run in a restricted test environment. Check separately for secret disclosure, dangerous commands, and newly introduced external dependencies. Because multiple implementations can pass the tests, combine execution-based evaluation with expert code review.

7. Advanced topic: combining benchmarks and risk-based evaluation

Public benchmarks help compare models and understand baseline capabilities, but cannot substitute for an organization’s use context. A high benchmark score does not ensure success on local data, terminology, permissions, or safety requirements. Combine public baselines with domain test sets and regression cases selected from real usage logs.

Stanford CRFM’s HELM takes a broad approach to comparing models across scenarios and multiple metrics. Its example shows why evaluation should define both the scenario range and the measured dimensions, rather than focusing on accuracy alone. However, public benchmark scores should not be used directly as business release thresholds.

The Generative AI Profile for the NIST AI RMF connects trustworthy evaluation to lifecycle risk management. It emphasizes validity of evaluation methods and fit to intended use, and recommends both automated evaluation and human oversight. It also encourages testing with data and conditions similar to the intended deployment environment. This broadens evaluation beyond model performance to privacy, bias, security, provenance, and social impacts.

In practice, eval-driven development reruns evaluation when models or application components change. Teams add cases when prompt failures are discovered and use continuous integration for regression tests and release gates. An LLM judge is first checked for agreement with human labels before being automated within a limited scope.

8. Considerations and implications

A. Business- and risk-based metrics: Do not impose an identical scorecard on every service. Set minimum thresholds according to failure costs and user impact. When efficiency and safety conflict, business owners should state the acceptable risk explicitly.

B. Representative data and leakage prevention: Reflect real input distributions, languages, regions, user groups, and boundary cases. Protect personal data and intellectual property. Separate training or prompt-tuning examples from evaluation cases and control access to the test set.

C. Validity and reproducibility: Preserve versions of models, data, prompts, and evaluation code as a single experiment record. Report sampling uncertainty and evaluator disagreement. Explain the scope and limits of results instead of presenting uncertain scores as proof of quality.

D. Appropriate human/automation balance: Expand automated scoring for repetitive, low-risk tasks. Keep expert review and appeal paths for high-risk or ambiguous cases. Involve users and affected groups as well as the people who designed the rubric.

E. Judge governance: Manage judge bias, version changes, cost, and data-processing location. Recalibrate periodically against human judgments. Because a judge may share errors with the target model, combine independent models, rule-based checks, and external evidence.

F. Operational feedback and accountability: Classify failure reports and human corrections, then feed them into the evaluation set. Minimize sensitive log collection and limit it to the stated purpose. Connect evaluation results to change approval, rollback, user notices, and incident response.

Generative AI evaluation is not a one-time benchmark to advertise model performance. It is a continuing control system that aligns business value with acceptable risk. The information-management professional should integrate data, measurement, people, and operations to establish what the organization means by “good.”

References

In one line: Trustworthy generative AI requires layered evaluation that reflects real work, human-calibrated automation, and continuous feedback from operational failures.