Data Virtualization

1. Overview

A. Definition

Data Virtualization is a data integration technology and architectural pattern that unifies heterogeneous, physically dispersed data sources into a single logical virtual data layer through real-time connection and querying — without replicating or moving the data — and presents it to consumers as a single view.

The essence of data virtualization is not "bringing data together and merging it," but "merging it virtually at query time while leaving it where it resides." Whereas traditional integration used ETL (Extract-Transform-Load) to extract and transform source data and load it into a data warehouse (DW) as a physical copy, data virtualization defines only metadata-based virtual views and pulls the actual data from the sources at the moment a consumer queries, combining and processing it on the fly. Data virtualization is therefore not a technology for expanding storage but one that provides an abstraction layer for access and integration; consumers treat everything as a single dataset through standard SQL or REST/GraphQL without needing to know whether a source is Oracle, cloud object storage, or a SaaS API.

For this reason data virtualization is commonly understood as a key means of implementing a Logical Data Warehouse, and it is not mutually exclusive with physical integration (DW, data lake); rather, it sits as a higher layer that bundles and abstracts them together.

B. Background and Necessity

First, the explosive dispersion of data and the shift to multi-cloud. Enterprise data has scattered across on-premises DWs, multiple public clouds, SaaS (e.g., Salesforce, SAP), data lakes, and operational DBs, and a strategy of gathering it all into one physical store caused replication cost, loading latency, and integrity damage from duplication. An approach that provides a unified view without moving the data became necessary.

Second, the growing demand for real-time freshness. Batch ETL loads on a cycle of hours to a day, so data ages by the loading delay. For tasks that need the current state of the source right now, such as real-time risk monitoring or a 360-degree customer view, virtualization that queries sources directly at query time is more suitable.

Third, the cost and governance burden of data replication. When the same data is scattered across many copies, not only storage cost but also governance questions grow — "which copy is authoritative" and "how far have copies of personal data spread." Virtualization minimizes copies so that policy can be enforced consistently from a single point of access.

Fourth, self-service analytics and time-to-value. Developing a new ETL pipeline for every new analytic request takes weeks, but a virtual view can be provided within days from a metadata definition alone, improving business agility.

2. Architecture and Core Components

A data virtualization platform can be understood as a structure that stacks logical layers of connect → abstraction (virtual views) → query optimization → delivery on top of diverse physical sources. The conceptual diagram below shows the overall structure from source to consumption.

graph TD
    subgraph SRC["Data sources (heterogeneous, dispersed)"]
        S1["Relational DB<br/>(Oracle·MySQL)"]
        S2["Cloud DW<br/>(Snowflake·BigQuery)"]
        S3["Data lake<br/>(S3·HDFS)"]
        S4["SaaS/API<br/>(Salesforce·REST)"]
    end
    subgraph DV["Data virtualization layer"]
        C["Connection/adapter (Connector)"]
        B["Base View abstraction"]
        D["Derived View/semantic model"]
        O["Query optimization engine"]
        G["Security·governance·cache"]
    end
    subgraph CON["Data consumers"]
        U1["BI/reporting"]
        U2["Data analytics/AI"]
        U3["Applications (API)"]
    end
    S1 --> C
    S2 --> C
    S3 --> C
    S4 --> C
    C --> B --> D --> O --> G
    G --> U1
    G --> U2
    G --> U3

A. The connection/adapter layer is the set of connectors that attach to heterogeneous sources. It absorbs each source's drivers (JDBC/ODBC), APIs, and file formats and converts them into a common internal representation of the platform. Thanks to this layer, upper layers need not know the physical differences between sources, and adding a new source does not require changing upper views. In practice, dozens of source types are absorbed through pre-built connectors, and when no connector exists, a generic REST/SQL adapter extends coverage.

B. The abstraction (virtual view) layer is the heart of data virtualization. On top of base views that mirror source tables 1:1, it stacks derived views that apply joins, filters, transformations, and aggregations, and finally builds a semantic model (business view) organized in business terminology. For example, if a "Customer_Order_Unified" view that joins Oracle's CUST table and Snowflake's ORDERS is defined, consumers query only this single view without knowing the physical locations. Since views have no physical loading, definition changes take effect immediately, and even when a source schema changes, only the view mapping needs fixing, isolating the consumer from the impact.

C. The query optimization engine is the core that governs performance. A virtual view query is executed distributed across multiple sources, so performance plummets if large volumes of data are pulled over the network. To prevent this, pushdown optimization performs filters, aggregations, and joins at the source side as much as possible, and joins across sources are ordered on a cost basis. The effect of pushdown is quantified with numbers in the comparison section below.

D. The security·governance·cache layer maximizes the advantage of a single point of access. Row- and column-level access control, data masking, and lineage tracking are applied uniformly at the virtual layer, and repeated queries are accelerated by caching.

3. Query Processing Flow and Optimization

How a query against a virtual view is decomposed, optimized, and executed is central to understanding performance. The sequence diagram below shows the process by which a consumer query is handled in the virtualization engine.

sequenceDiagram
    participant U as Consumer·BI
    participant E as Virtualization engine
    participant O as Optimizer
    participant A as Source A·DB
    participant B as Source B·DW
    U->>E: Virtual view query SQL
    E->>O: Logical query decomposition
    O->>O: Pushdown·join-order decision
    O->>A: Delegated filter/aggregate query
    O->>B: Delegated filter/aggregate query
    A-->>O: Reduced result set
    B-->>O: Reduced result set
    O->>E: Cross-source join·post-processing
    E-->>U: Return unified result

Query processing begins with decomposing the SQL the consumer submits into a logical query tree. The optimizer analyzes this tree to decide which operations to push down to which source, in what order to join across sources, and how to combine intermediate results in memory. The core principle is minimizing data movement. Filters and aggregations should be performed first at the source to reduce the number of rows crossing the network.

For example, consider a query that joins a 100-million-row transaction table (source B) with a 100-thousand-row customer table (source A) to compute monthly totals for customers in a specific region. Without pushdown, all 100 million rows must be brought into the engine to join and aggregate, but if the regional filter and monthly aggregation are delegated to source B, only a result reduced to the order of thousands of rows is transmitted, so network transfer volume and response time can improve dozens of times. Thus the performance of virtualization depends on "how much work is pushed down to the sources," and the optimization effect is limited if a source does not support aggregation or if join-key statistics are inaccurate.

Cache strategy must also be considered. For views that are queried often but impose heavy load on sources, results are stored and reused in a query cache or a materialized cache, while for views where freshness is important, caching is turned off or given a short TTL. In other words, virtualization is not "fully copy-free" but a technology that tunes the trade-off between performance and freshness on a per-view basis.

4. Comparison with Traditional Integration (ETL/DW)

The difference between data virtualization and physical integration is not a mere technology choice but stems from a difference in design philosophy over when and where data is combined. ETL/DW combines data in advance (at load time) and keeps copies, so it is strong at complex large-scale aggregation and historical analysis but entails loading latency and replication cost. Conversely, virtualization combines at query time, so it is strong at freshness and agility but is dependent on source performance and availability and can be disadvantageous for very large repeated aggregations.

Category	Data virtualization	ETL/data warehouse
Combination time	Query time (on-demand)	Load time (in advance)
Physical copy	Minimal (cache only)	Full replication
Data freshness	High (real-time)	Delayed by batch cycle
Build speed	Fast (view definition)	Slow (pipeline development)
Large repeated aggregation	Relatively disadvantageous	Advantageous
Source load	Incurred per query	Once at load

In practice, the two are designed as complementary rather than opposing. For example, three years of historical structured data is physically loaded into a DW, the latest operational data and SaaS data are connected in real time via virtualization, and the two are joined at the virtual layer to provide "historical trend + real-time status" as one view — a widely used hybrid configuration. That is, virtualization does not so much replace the DW as fill the freshness and scope the DW cannot cover.

5. Use Cases and Industry Application

Data virtualization delivers real value in industries where integration is repeatedly needed while replication is burdensome.

A. A 360-degree customer view and regulatory reporting in finance. Banks have accounts, cards, loans, and investment systems dispersed across different DBs. Physically integrating them would require enormous ETL and duplicate storage, but virtually joining each system around a customer identifier can provide a real-time unified customer view. In fact, commercial platforms such as Denodo hold many cases of virtually combining multiple sources in financial risk and compliance reporting to shorten reporting cycles.

B. Real-time visibility in manufacturing and supply chains. Manufacturing has ERP (SAP), MES, and IoT sensor data existing heterogeneously. When sensor time series are in a data lake and materials and orders are in ERP, binding these at the virtualization layer to provide "current inventory + production status + sensor anomalies" on a single dashboard can pull a consolidated report that took hours down to the minute scale.

C. Addressing data-movement constraints in public and healthcare sectors. Personal and medical information is often legally restricted from being copied outside its source. Allowing only virtual queries while applying access control and masking at the source, without moving the data, enables analytical use while complying with regulation.

What these three cases have in common is that virtualization is chosen "when the cost and risk of gathering data exceed the value of integration," whereas DW loading remains advantageous when stable large-scale batch analysis is the main purpose.

6. In Depth: Latest Trends and Position within Data Architecture

Data virtualization has recently been re-examined within higher-level architectural discourses such as data fabric and data mesh. A data fabric is a concept that makes integration intelligent with active metadata and AI automation, and virtualization is used as the enforcement means by which that fabric "integrates access without moving data." In a data mesh too, combined with the federation approach in which consumers connect and use domain-specific data products, virtualization provides the query layer that crosses domain boundaries.

Technically, the rise of open-source distributed SQL engines is prominent. Trino (formerly PrestoSQL) and Starburst, which commercializes it, as well as Dremio specialized in lakehouse querying, provide federated queries that span data lakes, DWs, and DBs, and broadly belong to the lineage of data virtualization query engines. They are also evolving in the direction of combining with lakehouse table formats (Apache Iceberg, Delta Lake) to impose transactions and schema evolution, via a virtual layer, even on raw data in object storage.

As for likely exam directions, the following may be frequently covered: ① comparison of data virtualization with ETL/DW and hybrid design, ② its role as a means of implementing a logical data warehouse, ③ its relationship with data fabric and mesh, and ④ performance optimization principles and trade-offs such as pushdown and caching. When composing an answer, it is effective to first present the essence of "real-time integration without replication," build the skeleton with an architecture layer diagram and an ETL comparison table, and then discuss performance limits together with their remedies (cache, pushdown) in a balanced narrative.

7. Considerations and Implications

Data virtualization is not an end in itself but a matter of a design decision that divides its role with physical integration across the overall integration strategy.

First, a fit-for-purpose application strategy. Virtualization suits integrations where freshness and agility matter and data volume is moderate, but analytic workloads that repeatedly aggregate multiple TB or more favor DW loading. A hybrid criterion mixing virtualization and physical integration by workload characteristic must be defined first.

Second, the performance·availability trade-off. Virtual queries are dependent on source performance and the network, and each query imposes load on the source. Therefore, querying an operational DB directly can harm operational performance, so replicas, caches, and concurrency limits must be designed together, and pushdown feasibility must be verified per source.

Third, the opportunity and responsibility of centralized governance·security. The virtual layer becomes a single control point that can enforce access control, masking, and lineage in one place, but at the same time, if this layer is breached, it becomes a single point of failure and risk that exposes all sources. Least privilege, audit logging, encryption, and availability redundancy must be accompanied without fail.

Fourth, organizational·operational maturity and related technologies. Virtualization delivers maximal value when it works together with a metadata catalog, data quality, and master data management (MDM). Also, without a naming and version-management scheme that prevents a proliferation of views and an operational process that absorbs source schema changes, it can degenerate into "virtual spaghetti," so it must be adopted in stages together with a data governance system.

Fifth, the outlook. As multi-cloud and lakehouses become ubiquitous, the cost of gathering data in one place grows, and the demand for metadata-based virtual access increases. Virtualization is expected to grow in strategic importance as the enforcement engine of data fabric and mesh and as a real-time supply layer for AI training data.

References

Gartner, "Data Virtualization" (Information Technology Glossary), https://www.gartner.com/en/information-technology/glossary/data-virtualization
Denodo, "What is Data Virtualization?", https://www.denodo.com/en/data-virtualization/overview
Trino Project Documentation, https://trino.io/docs/current/
AWS, "What is Data Virtualization?", https://aws.amazon.com/what-is/data-virtualization/

In one line: Data virtualization integrates dispersed heterogeneous data in real time into a virtual layer at query time without replication or movement, providing a single view — gaining freshness, agility, and centralized governance while offsetting dependence on source performance and the limits of large-scale repeated aggregation through caching, pushdown, and hybrid design.