metadata
Metadata is information that describes other data.
Metadata is information that describes other data. In the context of data analytics and engineering, metadata provides context about datasets, including details like column names, data types, creation dates, update frequencies, and data lineage. It's crucial for data governance, discovery, and understanding the structure and meaning of data assets. Tools like Amundsen and DataHub help organizations manage and explore metadata at scale. For example, in a DuckDB table, metadata might include the schema definition, table constraints, and column statistics. You can query metadata in DuckDB using system tables and functions like:
Copy code
-- View table metadata
DESCRIBE mytable;
-- Get column statistics
SELECT * FROM pragma_table_info('mytable');
-- View system-wide metadata
SELECT * FROM duckdb_tables();
Effective metadata management is essential for maintaining data quality, ensuring compliance, and enabling efficient data discovery and analysis across an organization.
Related terms
DuckLake is an open table format, created by the DuckDB team, that stores lakehouse metadata in a transactional SQL database instead of files — while keeping the actual data as Parquet files in object storage.
Table format →A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
Data fabric →Data fabric is an architecture that uses metadata, automation, and integration technology to connect and provide unified access to data across disparate systems, without necessarily moving it all into one place.
Data catalog →A data catalog is a centralized inventory of an organization's datasets, tracking metadata like schema, location, lineage, and ownership so people and systems can discover, understand, and access data.
Data lakehouse →A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance features of a data warehouse.
SUMMARIZE →SUMMARIZE is a powerful DuckDB command that automatically generates descriptive statistics and metadata about your data.