IM Engineer Study Notes
← Back to list
AI & Data
#Apache Spark#RDD#DataFrame#분산처리#구조적 스트리밍#레이크하우스
Last updated · 2026-10-31

Unified Distributed Data Processing with Apache Spark

1. Overview

A. Definition

Apache Spark is a unified distributed data processing engine that spreads large data across cluster nodes, reuses intermediate results in memory wherever possible, and optimizes every operation through a directed acyclic graph (DAG) built on lazy evaluation.

The essence of Spark is the unification claim that "one engine handles batch, interactive query, streaming, and machine learning." Developers manipulate distributed collections as if they were a single collection through the RDD (Resilient Distributed Dataset), DataFrame, and Dataset abstractions, and the engine translates that declaration into an execution plan, handling partitioning, scheduling, shuffling, and failure recovery. Thus developers build data pipelines scaled to hundreds or thousands of nodes without implementing node communication or failure retry themselves.

Spark began at UC Berkeley's AMPLab in 2009, became an Apache top-level project in 2014, and positioned its core value as overcoming the iterative-computation inefficiency of MapReduce's disk-centric batch model through memory-based processing. Today Spark is the de facto standard processing engine for data lakehouses, ETL/ELT, feature engineering, and log analytics.

B. Background and Need

MapReduce was an innovation that simplified large-scale batch work, but it necessarily wrote each stage's (Map→Reduce) intermediate result to a distributed file system (HDFS) and read it back in the next stage. This disk round trip is fine for a single-pass aggregation, but for iterative algorithms that traverse the same data dozens of times (gradient descent, PageRank, K-means) or interactive analysis where a human varies queries, the I/O cost accumulates and runtime grows unrealistically.

Real data organizations ran a different engine per workload (MapReduce for batch, Impala for query, Storm for stream, Mahout for ML), paying the cost of data movement, duplicated storage, and operational complexity. Spark emerged to solve both pains at once with "a single abstraction that keeps intermediate results in memory" and "a unified stack that runs SQL, stream, and ML on the same engine."

Especially in the era of data lakehouses and generative AI, the need grew to transform structured, semi-structured, and unstructured data in one engine and generate large-scale features to feed model training pipelines. Spark fits this demand of converging "heterogeneous data and heterogeneous workloads" into one processing layer.

C. Core Design Goals

Spark's design goal is not merely fast execution. The aim is to jointly achieve speed through memory reuse, fault tolerance via lineage, programming simplicity through declarative APIs, unification that accommodates multiple workloads on one engine, and automatic optimization through the optimizer.

First, it caches data in memory to eliminate the cost of repeated access. Second, instead of replicating data, it recovers from failures using lineage that records "how it was produced," reducing storage overhead. Third, with lazy evaluation it first grasps the whole operation graph and then applies optimizations such as shuffle minimization and predicate pushdown. These three axes underpin Spark's performance and stability.

2. Core Abstractions and Overall Architecture

A. RDD, DataFrame, and Dataset Abstractions

Spark's first-generation abstraction is the RDD, an immutable distributed collection split into partitions, transformed by functional operators such as map, filter, and reduceByKey. An RDD remembers through lineage "how each transformation was derived from its parent," so even if some partitions are lost to a node failure, only that portion is recomputed to recover. This is the core principle of tolerating failure without replication.

The second-generation abstraction, the DataFrame, is a table-shaped distributed collection with rows and columns (schema) that internally receives optimization from the Catalyst optimizer and the Tungsten execution engine. Because a DataFrame lets the user declare "what they want" while the engine decides "how to execute," it is generally faster than writing the same logic in low-level RDDs. A Dataset adds compile-time type safety on top, as a JVM-only (Scala/Java) abstraction.

In practice, DataFrame/Dataset should be preferred unless special low-level control is needed. For example, when aggregating revenue per merchant from tens of billions of daily transaction logs, a DataFrame's groupBy().agg() has the optimizer automatically insert partial aggregation before the shuffle, greatly reducing network transfer. Implementing the same logic directly in RDDs forces the developer to hand-code this optimization.

B. Overall Execution Architecture

flowchart LR
    subgraph Driver["Driver"]
        SC["SparkContext / SparkSession"]
        CAT["Catalyst optimizer"]
        DAG["DAG scheduler"]
    end
    subgraph CM["Cluster manager (YARN / K8s / Standalone)"]
        RES["Resource allocation"]
    end
    subgraph Workers["Worker nodes (Executor)"]
        E1["Executor 1<br/>Task / cache"]
        E2["Executor 2<br/>Task / cache"]
        E3["Executor 3<br/>Task / cache"]
    end
    SRC["Data source (HDFS / S3 / Kafka / JDBC)"] --> SC
    SC --> CAT --> DAG
    DAG -->|request resources| RES
    RES -->|launch Executor| E1
    RES -->|launch Executor| E2
    RES -->|launch Executor| E3
    DAG -->|dispatch Task| E1
    DAG -->|dispatch Task| E2
    DAG -->|dispatch Task| E3
    E1 -->|result / status| SC
    E2 -->|result / status| SC
    E3 -->|result / status| SC

A Spark application works by having the driver execute user code while building a logical plan, requesting resources from the cluster manager (YARN, Kubernetes, Standalone) to launch Executors, and then dispatching tasks to each Executor. The driver is the brain of the whole computation, handling scheduling and result collection, while the Executors handle actual per-partition computation and in-memory cache. Because the whole application terminates if the driver dies, the driver's memory and HA configuration become important management points in operations.

C. Lazy Evaluation and DAG Execution

Spark operations split into transformations and actions. Transformations such as map, filter, and join do not execute immediately but only accumulate lineage; the full plan executes only when an action such as count, collect, or save is called. Thanks to this lazy evaluation, the engine can view the entire operation chain and then perform optimizations such as column pruning, predicate pushdown, and fusing consecutive narrow transformations.

flowchart TD
    READ["1. read data read()"] --> T1["2. filter (narrow dependency)"]
    T1 --> T2["3. select (narrow dependency)"]
    T2 --> SH["4. groupBy / join (wide dependency / shuffle)"]
    SH --> T3["5. agg aggregation"]
    T3 --> ACT["6. write / collect (action)"]
    ACT -.trigger.-> JOB["create Job"]
    JOB --> ST["split Stage (by shuffle boundary)"]
    ST --> TASK["create Task (one per partition)"]

At execution, Spark splits one action into a Job, splits it into Stages at the boundaries where a shuffle (wide dependency) occurs, and splits each Stage into as many Tasks as there are partitions. A narrow dependency, where one partition depends on only one parent partition, can be pipelined without a shuffle, while a wide dependency triggers a shuffle that redistributes data across many partitions. A large part of performance tuning is precisely the work of reducing this shuffle count and shuffle data volume.

3. Unified Library Stack and Processing Models

A. Spark SQL with Catalyst and Tungsten

Spark SQL unifies SQL and the DataFrame API on the same execution engine. A query is transformed by the Catalyst optimizer through the stages of logical plan → optimized logical plan → multiple physical plans → cost-based selection, while Tungsten raises CPU and memory efficiency with whole-stage code generation and off-heap memory management. For example, applying WHERE dt='2026-10-31' to 1 TB of Parquet logs stored in S3 has Catalyst push the predicate down to the source scan stage (predicate pushdown) and use partition pruning to read only the needed date folder, cutting actual I/O to the level of a few GB.

B. Structured Streaming

Structured Streaming models a stream as an "unbounded, ever-growing table," letting the DataFrame code used in batch be reused almost unchanged on a stream. It runs in micro-batch or continuous processing mode, handles late-arriving data with event-time windows and watermarks, and provides exactly-once processing semantics with checkpointing and a write-ahead log (WAL). For instance, a pipeline that aggregates payment events flowing in from Kafka over five-minute windows to detect anomalous transactions can be implemented with the same aggregation code as the nightly batch—this is the heart of operational simplification.

C. MLlib and GraphX

MLlib provides distributed machine learning algorithms for classification, regression, clustering, and collaborative filtering, plus a pipeline API, distributing training even over feature data of several TB. GraphX (and GraphFrames) distributes graph operations such as PageRank and connected-component search. Since deep-learning training itself fits GPU-dedicated frameworks better than Spark, a common practice is to perform large-scale preprocessing and feature generation in Spark and hand training over to a separate GPU cluster.

D. Memory Management and Shuffle

Executor memory is jointly managed across an execution region and a storage cache region (Unified Memory Management), and the two borrow space from each other as needed. In the shuffle stage, map output is written to disk and then sent to the reduce side, so if data skew (values concentrated on a particular key) occurs, some tasks delay the entire Stage. To mitigate this, Spark 3.x's AQE (Adaptive Query Execution) automatically adjusts the number of shuffle partitions from runtime statistics, splits skewed joins, and converts small joins to broadcasts.

4. Comparison and Application Cases

Spark's position becomes clear through comparison with MapReduce and pure stream engines. The table below is a supplementary summary; the reasons for the differences are explained in the following sentences.

Category MapReduce Apache Spark Apache Flink
Processing model Disk-based batch Memory-based batch + micro-batch stream Native stream (batch as a special case)
Intermediate result Written to HDFS Kept in memory, optional disk State-based
Iterative computation Slow (I/O per stage) Fast (cache reuse) Fast
Latency High (minutes) Medium (seconds to minutes) Low (ms to seconds)
Main use Large-scale batch Unified batch / ETL / ML / near-real-time Low-latency real time

The reason Spark is several to tens of times faster than MapReduce on iterative computation is that it replaced disk round trips with memory reuse. By contrast, the reason Flink has the edge in ultra-low-latency streaming is that Spark's traditional micro-batch inherently carries a delay equal to the batch boundary, whereas Flink makes record-level native streaming and sophisticated state management its default. Thus the practical implication follows that Spark is favorable if "unifying everything into one" is the goal, while Flink is favorable if "millisecond-level event processing" is the goal.

As a concrete application case, large commerce firms at home and abroad aggregate tens of billions of daily click and order logs with Spark batch to generate features for product recommendation, while simultaneously computing real-time inventory and fraud-payment metrics with Structured Streaming in a Lambda-like structure. In finance, cases are reported of settling and aggregating risk over several TB of transaction data nightly with Spark SQL and stabilizing batch completion time by having AQE automatically absorb the shuffle skew that surges at month-end peaks. Telecom carriers preprocess the CDRs (call detail records) of tens of millions of subscribers with Spark to refresh MLlib-based churn-prediction features daily.

5. Advanced: Spark in the Lakehouse and AI Era, and Recent Trends

Recently Spark has evolved beyond a mere processing engine into the processing layer of the data lakehouse. Combined with open table formats such as Delta Lake, Apache Iceberg, and Hudi, it supports ACID transactions, schema evolution, and time travel, thereby implementing on object storage the reliable updates that were once possible only in a data warehouse. For example, performing a MERGE (upsert) into an Iceberg table with Spark manages concurrent write conflicts through snapshot isolation, and a faulty load can be rolled back to a previous snapshot.

On the operations side, the trend of Spark on Kubernetes replacing YARN is distinct. Combined with Kubernetes dynamic resource allocation, it automatically returns idle Executors to cut cloud cost, and complements stability with techniques that separately store shuffle data in external storage when a spot instance is reclaimed. In addition, serverless Spark (managed services) and interactive notebook environments (Databricks, EMR, Dataproc) have become common, lowering the burden of cluster operations.

Entering the AI era, Spark's role as the preprocessing hub of generative-AI pipelines has grown. Large-scale LLM training-data curation such as deduplication, filtering, and tokenization of billions of documents, batch embedding generation, and chunk preprocessing for RAG are performed with Spark. In recent versions, accessibility has improved through Spark Connect (a client-server separation architecture) that lets a lightweight client connect to a remote Spark cluster, and the pandas API on Spark that runs single-node pandas code in a distributed manner. That said, a division of roles that delegates vector similarity search or GPU-intensive inference to dedicated engines is still recommended.

6. Considerations and Implications

From a professional engineer's perspective, adopting and operating Spark should comprehensively consider the following.

First (adoption strategy), the choice of "unification vs. specialization" must come first. Spark is strong at bundling batch, ETL, near-real-time, and ML into one, but if millisecond-level real time is central it is reasonable to run Flink alongside, and if ultra-low-latency OLAP query is central, a distributed SQL engine. Spark should be placed as the central engine based on the organization's dominant workload while clearly distinguishing complementary engines, so as to avoid the inefficiency of excessive unification.

Second (trade-off), the advantage and cost of memory reuse must be viewed in balance. Caching accelerates iterative computation but excessive caching induces GC burden and OOM (out of memory). Overuse of driver collect, unrealistic partition counts, and data skew are typical causes of failure, so AQE, broadcast joins, proper partitioning, and skew-key salting should be reflected at the design stage.

Third (cost and operations), cloud cost governance is important. Cost can be lowered greatly by reducing scan volume with Spark on Kubernetes dynamic allocation, use of spot instances, and data skipping (partitioning, Z-Ordering). From a FinOps perspective, per-job cost visibility must be combined with automatic scaling policies for sustainable operations.

Fourth (data reliability and governance), combined with lakehouse table formats, ACID, schema evolution, data lineage, and quality validation should be embedded in the pipeline. Pursuing only processing speed while neglecting data quality and governance risks mass-producing "fast but wrong results."

Fifth (related technologies and outlook), Spark forms the processing backbone of the modern data platform by combining with Kafka (ingestion), open table formats (storage), MLOps/feature stores (utilization), and orchestration (Airflow). Going forward, lightweighting based on Spark Connect, interoperability with vector and AI workloads, and serverless adoption will deepen, and the professional engineer must position Spark not as an "all-purpose engine" but as a "unified processing layer" and design its division of roles with specialized engines from an architectural perspective.

References

In one line: Apache Spark is a distributed data processing platform that unifies batch, SQL, streaming, and ML on one engine through RDD/DataFrame abstractions, lineage-based fault recovery, lazy evaluation with DAG optimization, and memory reuse, and it is evolving into the processing backbone of lakehouses and AI preprocessing.