Data fabric
Data fabric is an architecture that uses metadata, automation, and integration technology to connect and provide unified access to data across disparate systems, without necessarily moving it all into one place.
Overview
Data fabric describes a technical architecture designed to provide a consistent, unified way of accessing and managing data that's spread across many different systems—on-premises databases, multiple clouds, SaaS applications, and data warehouses/lakes—without requiring all of it to be physically consolidated first. It relies heavily on metadata (about what data exists, its structure, its lineage, its quality) and increasingly on automation, including machine learning, to actively discover, connect, and recommend data across those systems.
The core idea is that in most real organizations, data will always be scattered across multiple systems for legitimate reasons (different tools for different jobs, acquisitions, legacy systems that are too costly to migrate), so rather than fighting that reality by trying to centralize everything, a data fabric weaves a connective layer over the top that makes the scattered data usable as if it were unified.
Key characteristics
- Metadata-driven: an active, continuously updated metadata layer (not just a static catalog) drives discovery, integration, and even recommendations about how data should be used.
- Automated integration: connections between systems are automated and often intelligent, reducing the manual engineering work of building point-to-point pipelines between every pair of systems.
- Unified access: users and applications query or consume data through a consistent layer, regardless of where the underlying data physically lives.
- Governance built in: because the fabric mediates access, policies (access control, masking, compliance) can be applied consistently across all connected systems from one place.
Data fabric vs. data mesh
Data fabric and data mesh both address the pain of siloed, fragmented data at scale, but from different angles: data fabric is primarily a technology-driven integration architecture (often vendor-led, emphasizing automation and metadata), while data mesh is primarily an organizational and ownership model (domain teams owning their own data as a product). Some organizations combine ideas from both—decentralized ownership per data mesh, connected via fabric-style integration and metadata tooling.
Where DuckDB fits
DuckDB is a query engine, not a data fabric platform—but it's commonly used as one of several execution engines that a data fabric or federated query layer might invoke, particularly for analytical queries over local files or embedded use cases, since it can query Parquet, CSV, and other formats directly from object storage without a separate ingestion step.
Related terms
Data mesh is a decentralized approach to data architecture in which individual business domains own and serve their own data as a product, rather than a central team owning all data pipelines.
metadata →Metadata is information that describes other data.
Data catalog →A data catalog is a centralized inventory of an organization's datasets, tracking metadata like schema, location, lineage, and ownership so people and systems can discover, understand, and access data.
Data governance →Data governance is the set of policies, roles, and processes an organization uses to manage the availability, quality, security, and proper use of its data.
storage layer →The storage layer refers to the component of a data system responsible for persistently storing and managing data.
Lambda architecture →Lambda architecture is a data processing design that runs a batch layer and a speed layer in parallel to balance completeness and accuracy against low-latency results, merging both into a serving layer.
FAQS
No, though many vendors market products as implementing data fabric capabilities. It's better understood as an architectural approach—unifying access to distributed data via metadata and automation—that different tools implement in different ways.
A warehouse or lake is a place where data is physically stored and processed. A data fabric is a connective layer that can span multiple warehouses, lakes, and other systems at once, without requiring the underlying data to be moved into a single location.
