IM Engineer Study Notes
← Back to list
AI & Data
#모델위험관리#MRM#AI거버넌스#모델검증#모델리스크#MLOps#NISTAIRMF#생성형AI
Last updated · 2026-09-27

AI Model Risk Management (MRM)

1. Overview

A. Definition

AI Model Risk Management (MRM) is the discipline of identifying and controlling losses caused by model error, misuse, change, or data bias through validation, approval, monitoring, and lifecycle controls.

An AI model generalizes data and produces probabilistic results, so successful code execution does not prove that an operational decision is valid. Weak representation or a badly defined target can produce a wrong conclusion even when the computation is correct. After deployment, customer behavior, policy, markets, and equipment conditions may change and invalidate relationships learned during training. MRM is therefore not a one-time approval; it is a closed loop for the model's purpose, limits, permitted use, changes, and retirement.

Model risk is broader than a defect in the model. Bad input data, an unsuitable purpose, operational drift, supplier changes, and automation bias are also model-risk sources. Generative AI adds hallucination, prompt injection, unsupported reasoning, tool misuse, and sensitive-data reproduction. A Professional Engineer should assess quantitative performance together with business impact, regulation, security, explainability, and recoverability.

B. Background and Need

AI is used in lending, insurance, hiring, healthcare assistance, and manufacturing inspection where errors can cause material harm. Average accuracy is not enough; group-level errors, reasons for rejection, appeals, and human review must also be checked.

The AI supply chain is complex. Organizations combine internally trained models with open-source models, external APIs, embedding models, retrieval systems, datasets, plugins, and agent tools. Without version and usage tracking, the cause of a change and the boundary of responsibility are unclear.

Models can degrade after a technically successful deployment. Data drift changes the input distribution, while concept drift changes the relationship between inputs and the target. The organization needs re-training, scope reduction, and shutdown criteria.

Responsible AI requires evidence rather than declarations. An inventory, risk tier, data lineage, validation report, approval record, operating metrics, and incident records must be connected for auditability and reproduction.

2. Goals and Governance Structure

The first MRM goal is fitness for purpose. The model must meet the needs of its declared use, stay within its prohibited-use boundary, and remain consistent with business rules. The second is independent validation. An independent reviewer reproduces data, methods, results, and limitations instead of accepting the development team's claim. The third is traceability and recoverability. The exact combination of model, data, code, prompt, and policy must be recorded so that a safe version or manual process can be restored.

flowchart TB
  B[Board and executives] --> P[Risk appetite and policy]
  P --> C[Model risk committee]
  C --> O[Model owner and business owner]
  C --> V[Independent validation and audit]
  O --> D[Development, data, and operations]
  D --> E[Inventory, validation, and release evidence]
  V --> E
  E --> M[Performance, fairness, and drift monitoring]
  M --> C

Executives define risk appetite and approval or shutdown authority. The model risk committee standardizes risk tiers, validation independence, exception approval, and stop criteria. The model owner manages purpose, inputs, outputs, users, limitations, and operations; the data owner manages quality, rights, retention, and lineage. The independent validator reviews assumptions, data, implementation, performance, fairness, security, and operational controls.

Role Responsibility Evidence
Executives Decide risk appetite and stop authority Policy and tolerance
Model owner Own purpose, scope, change, and operations Model card and registry
Data owner Own source, quality, privacy, and representation Datasheet and lineage
Validator Reproduce results and test limitations Validation report
Operations Deploy, monitor, respond, and roll back Runbook and metrics
Audit/compliance Check policy and evidence sufficiency Audit and remediation

Roles must have real authority, not only an organization-chart label. An approver needs the power and expertise to stop release, and a model owner needs access to versions and logs for incident analysis. Small organizations may combine roles, but a high-impact model should not be developed, approved, and independently validated by one person.

3. Risk Taxonomy and Lifecycle

Risk should not be reduced to a single score. Separate input, model, output, business, operations, and supply-chain layers, and consider likelihood, impact, detectability, and recovery time.

A. Input and Data Risk

Data risk includes missing values, duplicates, errors, representation, label consistency, leakage, privacy, and usage rights. If training data differs from the operational population, strong test performance can still result in field errors. Check under-represented regions, genders, ages, devices, or customer groups and do not hide gaps behind an average.

B. Model and Method Risk

Model risk comes from a poor algorithm choice, overfitting, false assumptions, leakage, unstable hyperparameters, and irreproducible training. More complex models may improve performance while increasing explanation, validation, and operational costs. Generative models add failure modes caused by stochastic generation, long context, retrieval, and tool combinations.

C. Output and Business Risk

Output risk grows when a business process accepts a result without review. Even a “recommendation” becomes an automated decision if staff routinely approve it without meaningful checks. Specify intended use, human review, appeal, fallback, and blocking rules for high-impact outputs.

D. Operations and Supply Chain Risk

Different libraries, hardware, or permissions between training and production can prevent reproduction. External APIs and pretrained models introduce provider changes, outages, pricing, and regional data-processing risks. Record model, prompt, index, tool, and policy versions together.

flowchart LR
  A[Purpose and impact] --> B[Inventory data and model]
  B --> C[Risk tier and validation plan]
  C --> D[Build, train, and test]
  D --> E[Independent validation]
  E --> F{Approval criteria met?}
  F -- No --> G[Mitigate, retrain, or reduce scope]
  G --> D
  F -- Yes --> H[Phased deployment]
  H --> I[Operational monitoring]
  I --> J{Drift, incident, or change?}
  J -- No --> I
  J -- Yes --> K[Revalidate, stop, or roll back]
  K --> C

The key lifecycle principle is to reassess risk whenever a meaningful change occurs. A changed distribution, purpose, provider, or tool can change risk even when no application code changes. Conversely, applying a high-impact process to every low-risk wording change makes controls ceremonial, so validation scope must be proportional.

4. Registration, Validation, and Approval

A. Inventory and Risk Tiers

Record identifier, owner, purpose, inputs, outputs, users, decision impact, data class, provider, version, environment, and expiry date. Separate experiments from production models so abandoned models do not become invisible attack surfaces.

Combine impact, autonomy, data sensitivity, external exposure, recoverability, and regulatory relevance to set a tier. Internal document classification may be low risk, while the same technique used for lending or safety shutdown is high risk.

Tier Example Minimum control
Low Internal search or deduplication Basic tests and owner approval
Medium Customer recommendation or forecasting Independent sample test and drift monitoring
High Finance, hiring, or medical assistance Independent validation and human approval
Very high Safety shutdown or rights restriction Restricted use and executive approval

B. Validation Scope and Methods

The validator first reads the requirements and data-generation process, then checks whether the chosen metrics represent the business risk. Review accuracy, F1, or AUC together with threshold curves, confusion matrices, group gaps, cost, latency, and robustness. Include regression, boundary, adversarial, and distribution-shift tests, and check that test data is not duplicated in training data.

A validation report states purpose, data, method, environment, results, limitations, residual risk, approval conditions, and revalidation interval. Poor results do not always require abandonment; threshold changes, scope limits, human review, additional data, or an alternative model may reduce risk. Safety, privacy, and regulatory minimums must not be offset by a higher performance score.

C. Approval and Exceptions

Approval means that risk is acceptable for a defined use, not that the model is universally “good.” Conditional approval should state limited users, shadow mode, manual checks, expiry, and stop criteria. An exception is time-limited evidence with reason, risk, compensating controls, approver, and expiration date.

5. Monitoring and Change Management

Collect model, data, business, and control metrics separately. Model metrics include precision, recall, calibration, and accuracy; data metrics include missingness, distribution shift, new categories, and freshness. Business metrics include approval, complaint, rework, and manual-escalation rates; control metrics include access violations, policy blocks, missed reviews, and rollback time.

Data drift changes inputs; concept drift changes the input-target relationship. Do not replace a model only because the distribution moved; check actual error and business impact. Conversely, a stable average can hide a fairness regression in one group and should still trigger review.

Link changes to models, data, code, features, prompts, indexes, policies, and external APIs as one change unit. Low-risk changes may use automated tests and approval, but a new purpose, input, output, or tier requires impact assessment comparable to a new model. Deploy through shadow mode, internal users, limited traffic, and full traffic, with automatic stop conditions for error, cost, and safety events.

6. Comparison and Cases

A. MRM, MLOps, and AI Governance

MLOps is the automation base for repeated training, deployment, and operation. AI governance defines acceptable purposes, risk, and responsibility. MRM connects the two by independently testing assumptions and performance and translating operating risk into approval, restriction, and rollback.

Aspect MRM MLOps AI governance
Question Under what conditions can this model be trusted? How can it be reproduced and deployed? What is allowed and who is responsible?
Activity Validation, approval, monitoring, exceptions Pipelines, registry, automation Principles, policy, committee, impact
Failure response Restrict, revalidate, roll back Fix pipeline or redeploy Change policy, responsibility, remedy

B. Financial Advisory Assistant

Suppose an assistant provides product conditions and draft answers for financial counselors. Average answer accuracy can still hide a wrong fee based on an old policy. Attach effective dates, product identifiers, and evidence passages, and require counselor review for new products or exceptions.

Test normal questions, similar product names, ineligible customers, and questions absent from the knowledge base. Monitor unsupported-answer rate, counselor edits, complaints, product-level errors, and masking failures. When errors rise, inspect index freshness, document filtering, permissions, and approvals before simply retraining.

C. Manufacturing Inspection

A vision model on a production line faces drift from lighting, cameras, raw materials, and suppliers. Measure miss rates and false-alarm cost by product, equipment, and shift rather than relying on one average. Use operator confirmation for safety-relevant judgments and return to sampling inspection when conditions exceed limits.

7. Advanced Topic: Standards and Generative AI

The NIST AI RMF functions Govern, Map, Measure, and Manage provide a useful structure for MRM policy, context, validation, and response. Govern sets roles and risk appetite; Map identifies context and impact; Measure tests performance, fairness, security, and uncertainty; Manage prioritizes mitigation, suspension, and recovery. ISO/IEC 42001 can connect model validation to organization-wide training, audit, and continual improvement.

Generative AI cannot be validated only through final text. Check retrieval freshness and grounding, prompt and policy precedence, tool permissions, agent state transitions, cost, and latency. Email, payment, and permission changes must pass a policy engine, human approval, transaction limits, and audit logging even when the model proposes them.

Link prompt, embedding, retrieval-index, and tool-schema versions to the model registry. This makes it possible to reproduce why the same model changed its answer and to detect safety regression before release.

8. Considerations and Implications

A. Risk-based differentiation

Apply the strictest process to high-impact models and lighter controls to low-impact work. Use impact, autonomy, sensitivity, exposure, and recoverability to set tiers.

B. Practical independence

Independence means authority to challenge and stop a release, not merely a different department name. Cross-review, external experts, and automated reproduction can supplement scarce staff, while accountability remains internal.

C. Data and model lineage

Saving only a model file cannot reproduce a result. Store dataset, labels, features, code, libraries, hardware, prompts, policies, APIs, and execution time together.

D. Explanation and remedy

An explanation should provide relevant factors, evidence, uncertainty, and an appeal path rather than expose every internal parameter. Check that a person affected by an automated result can actually obtain review and correction.

E. Operational resilience

Prepare manual procedures, a prior version, rule-based fallback, retention, and notification before stopping a model. Exercise rollback against database schemas and external contracts.

F. Cost and performance

Risk-based routing can choose model size, test frequency, human review, and retention. Do not trade away safety minimums for cost optimization.

G. Professional Engineer roadmap

Start with a production inventory, owners, purposes, and risk tiers. Then establish independent validation templates, a registry, lineage, approval, and exception processes. Next connect quality, fairness, security gates, and monitoring to MLOps or LLMOps. Finally feed incidents, complaints, and drift into revalidation and improve response with policy-as-code and rollback.

References

In one line: AI Model Risk Management turns model, data, business, and supply-chain risk into lifecycle governance through independent validation, approval, monitoring, change control, and recoverability.