IM Engineer Study Notes
← Back to list
Security & Privacy
#IDS#IPS#침입탐지#이상탐지#NGIPS
Last updated · 2026-10-02

Intrusion Detection and Prevention Systems (IDS/IPS)

1. Overview

A. Definition and Background

An Intrusion Detection System (IDS) is a security system that collects and analyzes traffic and events occurring on networks and hosts to detect signs of attacks, misuse, and policy violations and raise alerts, while an Intrusion Prevention System (IPS) additionally performs active response that blocks and isolates detected attacks in real time. In short, an IDS is a detector that "watches and alerts," while an IPS is a blocker that "watches and stops," and an IPS is often described as "an evolved IDS with in-line blocking capability added."

The fundamental background behind IDS/IPS lies in the recognition that "control over opening and closing passages alone cannot stop intrusions" — a limitation of firewalls. Traditional network firewalls allow or block access based on source/destination IP and port. However, when an attack payload is hidden within traffic passing through an already-permitted port (e.g., port 443 for web services), the firewall cannot distinguish it from normal traffic. In other words, a firewall controls "who enters through which door" but cannot see "what that person does once inside." Intrusion detection was born to monitor this blind spot — malicious activity occurring inside permitted communication.

Historically, IDS traces its theoretical roots to the intrusion detection model proposed by Dorothy Denning in 1980, and it became widespread in the 1990s as open-source signature-based IDSs such as Snort spread. However, early IDSs could only "detect and alert" on attacks, requiring humans to respond after the fact, exposing the limitation of responding only after an attack had already succeeded. As networks grew faster and automated attacks unfolded within seconds, the need for in-line active response that blocks at the moment of detection grew, leading to the emergence of IPS in the 2000s. Today it has evolved into next-generation IPS (NGIPS) combining application identification, user awareness, and threat intelligence, showing a trend of being absorbed as a core function of next-generation firewalls (NGFW).

B. Necessity

Modern breaches tend to move laterally inside after breaching the external perimeter and lie dormant for long periods. A single perimeter firewall cannot detect attacks that have already entered internally or insider misuse, so IDS/IPS that continuously monitor traffic and host events become an essential layer of Defense in Depth. On the regulatory side as well, ISMS-P, PCI-DSS, and the Electronic Financial Supervisory Regulations require the operation of network intrusion detection/prevention measures and log retention, and IDS/IPS detection logs serve as key evidence for root-cause analysis and accountability when incidents occur.

In fact, breach investigation reports in the security industry have repeatedly pointed out that the dwell time from when an attacker infiltrates internally until detection reaches tens of days. Defense that only guards the perimeter cannot capture the reconnaissance, privilege escalation, and data exfiltration taking place internally during this long dormancy period. IDS/IPS monitor precisely this "time after passing the perimeter," shortening dwell time and providing an opportunity to sever the kill chain at an intermediate stage before it completes — positioning them, beyond mere alerting devices, as a core means of shortening detection and response time.

C. Core Characteristics

The characteristics of IDS/IPS can be condensed into three. First is in-depth analysis at the content and behavior level, reassembling payloads beyond packet headers and interpreting protocol context to judge whether an attack is occurring. Second is the division of roles between detection (IDS) and blocking (IPS): detection-only appliances observe broadly without availability burden, while blocking appliances respond immediately on the path but bear the risk of cutting off even normal communication on false positives. Third is dependence on continuous rule/model updates: since new attacks constantly emerge, signature updates and anomaly baseline retraining determine operational success. Combining these three, IDS/IPS functions not as "an appliance you install and forget" but as "a control completed through operation."

2. Overall Structure and Deployment Modes of IDS/IPS

IDS/IPS should be understood not as a single algorithm but as a pipeline of data collection (sensor) → analysis (engine) → response → management. Below is a structural diagram showing the overall components and data flow.

flowchart LR
  SRC["Traffic·Host events"] --> SEN["Sensor: collect·normalize"]
  subgraph ENG["Analysis engine"]
    MIS["Misuse detection: signature"]
    ANO["Anomaly detection: behavior·statistics"]
  end
  SEN --> ENG
  ENG --> DEC["Decision: normal·attack·suspicious"]
  DEC -->|IDS| AL["Alert·logging"]
  DEC -->|IPS| BLK["Block·session kill·isolate"]
  AL --> MGR["Management console·SIEM linkage"]
  BLK --> MGR
  MGR --> TI["Threat intelligence·rule update"]
  TI --> ENG

The starting point of this structure is the Sensor. The sensor sniffs packets on a network segment or collects host logs/system calls, and before analysis it reassembles fragmented packets and normalizes them at the protocol level. Since attackers try to evade detection by chopping packets into small fragments or manipulating TTL, the quality of the sensor's reassembly/normalization becomes the hidden foundation governing all detection accuracy above it. The analysis engine then performs misuse detection and anomaly detection in parallel to make a decision; the IDS responds with alerts/logging and the IPS with session blocking, reset, or isolation, and all results converge to the management console and SIEM for use in correlation analysis and rule updates.

A. Network-based (NIDS/NIPS) and Host-based (HIDS/HIPS)

The first design axis running through IDS/IPS is "where do we observe." Network-based (NIDS/NIPS) places sensors on network segments and monitors all passing packets. One unit can cover many hosts, giving broad visibility without affecting host performance, but the inside of encrypted traffic cannot be seen without decryption, and in switched environments a copy of the traffic must be received through a mirror port (SPAN) or network TAP. On large-traffic segments, the sensor's processing performance becomes a bottleneck and some packets may be dropped, so performance design matched to line speed is important.

Host-based (HIDS/HIPS) installs agents on individual servers/endpoints to monitor file integrity changes, registry/system calls, login history, and process behavior. Since it observes actual execution at the endpoint where encryption is removed, it accurately detects internal misuse and privilege escalation the network cannot see, but agents must be deployed/managed per host and consume host resources, and if an attacker takes over a host, the agent itself can be neutralized. Representative examples in the HIDS category include the file integrity checking tools OSSEC and Wazuh, and Windows event-based detection.

In practice, the two are combined complementarily. For example, NIDS/NIPS broadly monitor traffic at the DMZ segment and internal core switches, while HIDS is added to high-importance hosts such as payment/authentication servers to precisely reinforce internal tampering the network cannot see. Recently, host-based detection has evolved into EDR (Endpoint Detection and Response) and network-based detection into NDR (Network Detection and Response), converging into XDR that integrates the two.

B. Passive (IDS) Deployment and In-line (IPS) Deployment

The second axis is "does it intervene in the traffic path." In passive deployment, the sensor receives only a mirrored copy from outside the traffic path and detects/alerts. It adds no delay to the original traffic and an appliance failure does not directly cause service interruption, so there is no availability burden, but it only "discovers" attacks without "blocking" them, creating a time gap between detection and response. A typical IDS uses this mode.

In in-line deployment, the sensor sits on the traffic path so all packets pass through the appliance, allowing it to immediately drop a session judged to be an attack or terminate it with a TCP reset. The IPS takes this mode. In return, the appliance becomes a single point of failure (SPOF) and a latency factor, and if a false positive occurs, even normal communication is blocked, escalating into a service outage. For this reason, in-line appliances must decide in advance whether to pass traffic (fail-open) or block it (fail-close) during a failure, matched to service characteristics, and must be designed together with redundancy and bypass switches. A public portal where availability is paramount chooses fail-open, while a financial/classified network chooses fail-close — reflecting the organization's priorities.

C. Detection and Response Processing Flow

Below is a detailed process diagram of how one piece of suspicious traffic proceeds from sensor to decision and response.

sequenceDiagram
  participant N as Attacker·Insider
  participant S as Sensor NIPS
  participant E as Analysis engine
  participant T as Target server
  participant M as SIEM·Admin
  N->>S: Packet inflow session
  S->>S: Reassembly·normalization
  S->>E: Pass normalized stream
  E->>E: Signature matching + anomaly scoring
  alt Judged as attack - in-line
    E-->>N: Session block·TCP Reset
    E->>M: Alert·log - rule ID·payload
  else Normal
    S->>T: Forward traffic
  end
  M->>E: Update rules·thresholds after correlation

3. Detection Techniques — Misuse Detection and Anomaly Detection

The heart of IDS/IPS is "on what basis is something judged to be an attack," and this splits into two broad branches.

A. Misuse / Signature-based Detection

Misuse detection takes the approach of "block what is known to be bad," registering the characteristics (signatures, patterns, rules) of already-analyzed attacks in a database and judging traffic as an attack when it matches. For example, a Snort rule describes conditions such as a specific string, port, and direction, like alert tcp any any -> 192.168.0.0/24 80 (content:"/etc/passwd"; ...). This approach offers high detection accuracy for known attacks, few false positives, and clear decision rationale, so it is widely used as the first line of defense in field IDS/IPS.

However, misuse detection can inherently only detect "attacks seen in the past," so its structural limitation is the false negative of missing new, variant, and zero-day attacks not in the signatures. Attackers bypass signatures with small variations such as encoding the payload or reordering it, and preventing this requires endlessly adding signatures, bloating the ruleset and degrading performance. When the WannaCry ransomware spread rapidly in 2017, the concentration of damage during the brief gap before the signature for its SMB vulnerability (EternalBlue) was distributed is a representative case demonstrating the time-lag limitation of misuse detection.

B. Anomaly-based Detection

Anomaly detection takes the opposite approach of "establish a baseline of normal and suspect what deviates from it." It learns the usual traffic volume, protocol distribution, access time zones, and user behavior to build a normal profile (baseline), and assigns an anomaly score to behavior that deviates statistically significantly. The greatest value of this approach is that it can detect new and zero-day attacks without signatures, as well as insider anomalous behavior. For example, if an account that normally connects only in small amounts during business hours transmits large volumes of data externally at dawn, anomaly detection catches this even though it does not match any signature.

The weakness of anomaly detection is that it has many false positives. It easily mistakes situations where normal traffic changes abruptly (access surges from a promotion, a new service deployment) for attacks, and if the baseline training data already contains attacks, the error of learning attacks as normal also occurs. Therefore anomaly detection must be operated in a learning mode that observes without blocking for a certain initial period to refine the baseline, with conservative threshold tuning being essential. Recently it has been advanced into UEBA (User and Entity Behavior Analytics) applying machine learning/deep learning to capture subtle anomalies across multidimensional features, but it also carries the homework of the black-box problem, where the decision rationale is hard to explain, and securing explainability (XAI).

C. Comparison of the Two Techniques and IDS/IPS Types

Field IDS/IPS run both techniques in parallel, blocking known threats quickly and accurately with misuse detection and supplementing unknown threats with anomaly detection. The tables below summarize the key axes in comparison, but the reasons the differences arise must be understood together.

Category Misuse detection (signature) Anomaly detection (behavior)
Decision basis Match of known attack pattern Deviation from normal baseline
New·zero-day Cannot detect (false negative) Detectable
False positive rate Low High
Rationale explanation Clear (rule ID) Hard (statistics·model)
Operational burden Continuous rule updates Baseline learning·tuning
Comparison axis IDS IPS
Main function Detect·alert Detect·block
Deployment Passive (mirror·TAP) In-line
Latency·SPOF None Present
False-positive impact Alert overload Blocking normal communication
Representative products Snort (IDS mode), Zeek Suricata, NGIPS, Snort (in-line)

IPS is not always superior to IDS. Its blocking capability is powerful, but a single false positive leads directly to a service outage, so before detection reliability is sufficiently verified it is safer to operate in IDS mode, which only detects without blocking. In other words, IDS and IPS are not in a generational-replacement relationship but should be accurately seen as options to operate the same engine in different modes according to availability/security priorities.

D. Detection Performance Metrics — A Quantitative Understanding of False Positives and Negatives

Operational success of IDS/IPS ultimately comes down to "how detection accuracy is measured and managed," so confusion-matrix-based metrics must be understood quantitatively. Judging an attack as an attack is a true positive (TP), mistaking normal for attack is a false positive (FP), and missing an attack is a false negative (FN). The key metrics are detection rate (Recall = TP/(TP+FN)), which sees how well actual attacks are not missed, and Precision = TP/(TP+FP), the proportion of actual attacks among alerts, with the F1 harmonic mean of the two used as a balance measure.

For example, assume an environment with 1 million sessions per day of which 1,000 are actual attacks, and an IPS with 99% detection rate and 0.1% false-positive (FP) rate. It catches 990 attacks (TP=990) but false-alarms on about 999 of the 999,000 normal sessions, which is 0.1% (FP≈999). Here precision is 990/(990+999)≈49.7%, meaning "half of the alerts are false alarms." This is the notorious base-rate fallacy in security. Since normal traffic overwhelmingly outnumbers attacks, no matter how low the false-positive rate appears, the absolute number of false positives overwhelms the number of attacks and plunges the operations team into "alert fatigue." Therefore the goal of IPS tuning should not simply be to raise the detection rate, but to prioritize alerts on a risk basis and raise precision through SIEM correlation analysis, leaving only the alerts that actually require response.

4. Deep Dive — NGIPS, Evasion Techniques, Practical Application

Traditional IPS relied on ports, protocols, and signatures, leaving them vulnerable to attacks that bypassed via non-standard ports or were varied at the application layer. The next-generation IPS (NGIPS) that emerged to overcome this integrates ① application awareness (App-ID) that identifies the actual application regardless of port, ② user awareness applying policy per user rather than per IP, ③ threat intelligence linkage reflecting external reputation and IoCs (indicators of compromise) in real time, and ④ sandboxing that runs suspicious malicious files in an isolated environment to observe them. Today NGIPS is often provided as an integrated module of next-generation firewalls (NGFW) rather than as a standalone appliance, in a trend where firewall, IPS, and application control converge in a single device.

Meanwhile, attackers employ various evasion techniques to avoid detection. Representative ones include fragment attacks that chop packets to interfere with reassembly, techniques that manipulate TTL to make the reassembly results differ between the sensor and the target server, techniques that multiply-encode/obfuscate the payload, and methods that encrypt traffic so the sensor cannot see the content at all. In particular, as most of today's traffic is encrypted with TLS, an IPS needs SSL/TLS decryption (securing visibility) to see the content, but this incurs performance burden along with a new risk of exposing plaintext sensitive information at the decryption point. This is why the decryption scope, key management, and privacy impact assessment must be designed together.

The open ecosystem of IDS/IPS is also mature. Snort, which has become a de facto industry standard, enabled rapid response to new threats with its rule syntax and community rulesets; Suricata, which handles high-speed lines with a multithreaded architecture, shares the same rule syntax while adding automatic protocol identification, file extraction, and TLS metadata logging; and Zeek (formerly Bro) is widely used for threat hunting with analysis centered on network behavior logs rather than signatures. In such an environment where commercial and open source coexist, a rule management system (Policy as Code) that version-controls, tests, and deploys rulesets like code governs operational quality.

As a practical application case, the financial sector places NIPS in-line at the Internet segment and internal network boundary to block external attacks, overlays HIPS on internal critical servers to detect privilege escalation and file tampering, and operates a multilayered system that gathers all alerts into a SIEM and responds automatically with SOAR playbooks (IP blocking, session isolation). Likely exam directions include: ▲discuss the trade-off between misuse detection and anomaly detection and the situational choice between them; ▲describe the deployment and operational differences between IDS and IPS; ▲present the impact of increasing encrypted traffic on IPS and the response; ▲compare IPS, WAF, EDR, and firewalls and present a defense-in-depth design.

5. Considerations and Implications (Professional Engineer Perspective)

To successfully establish IDS/IPS, one must approach it from the perspective of "operational capability and process" rather than "appliance deployment." A professional engineer's answer should discuss the following trade-offs and strategies together.

  • Balancing false positives and negatives (security-availability trade-off): The stronger blocking is set, the fewer false negatives, but false positives cut off normal communication, causing service outages and "alert fatigue" for the operations team. New rules must always be observed sufficiently in IDS (detection-only) mode to tune out false positives before being transitioned step by step to IPS (blocking) mode, and one must recognize that IDS/IPS is an operational control premised on continuous tuning, not "install and done."

  • Encrypted traffic and securing visibility: In today's world where most traffic is encrypted, an IPS cannot inspect payloads without decryption. Since full decryption incurs performance and privacy burdens, a design is needed that selectively decrypts only high-importance segments, protects keys with an HSM, and supplements non-decrypted segments with JA3 fingerprinting and metadata-based encrypted traffic analytics (ETA).

  • Performance, scalability, and SPOF response: Since in-line IPS must process without interruption at line speed, packet drops and delays occur when processing performance is exceeded. Redundancy, auto-scaling, hardware bypass, and fail-open/fail-close policies must be clearly defined to match service characteristics, and this is a managerial judgment about whether to prioritize availability or security.

  • SIEM/SOAR linkage and response automation: IDS/IPS alerts are not complete in themselves; they must be collected and correlation-analyzed by a SIEM and automated for blocking, isolation, and ticket issuance with SOAR playbooks to reduce the time from detection to response (MTTR). Mapping alerts to the MITRE ATT&CK framework allows systematic identification and reinforcement of detection gaps by attack stage.

  • Related technologies and outlook: IDS/IPS form defense in depth by dividing roles with firewalls, WAF, EDR, and NDR, and are evolving into XDR integrating host and network detection and ML-based UEBA. However, as automated attack variation using generative AI increases, the defensive side must strengthen adaptive detection, and securing explainability (XAI) in response to the black-box problem of AI-based detection remains a future task. Also, in a zero-trust architecture, combination with microsegmentation that monitors even internal East-West traffic beyond perimeter-centric IPS becomes important.

References

In one line: IDS/IPS is a defense-in-depth layer that monitors attacks inside permitted traffic that firewalls cannot see, through a sensor → analysis (misuse·anomaly detection) → response pipeline; IDS handles passive detection·alerting and IPS handles in-line blocking, evolving into NGIPS·XDR·UEBA, with balancing false positives/negatives, encryption visibility, SPOF, and SIEM/SOAR linkage as the core operational tasks.