Lance format
Lance is a modern, open-source columnar file format designed for machine learning and vector-search workloads, offering fast random access, versioning, and native support for embeddings alongside tabular data.
Overview
Lance is a columnar data format created by LanceDB, designed as an alternative to Parquet for workloads that mix traditional tabular analytics with machine learning data — particularly vector embeddings, images, and other unstructured blobs. It aims to keep Parquet-like scan performance for analytical queries while adding fast random access to individual rows, which columnar formats built purely for sequential scans generally handle poorly.
Why it was built
Parquet and ORC were designed for large, sequential analytical scans, which makes point lookups (fetching a single row or embedding by ID) slow. ML workflows — like retrieving a specific vector for a nearest-neighbor search, or randomly sampling training examples — need efficient random access as well as bulk scans. Lance's on-disk layout and indexing support both patterns, alongside built-in versioning so datasets can evolve (add columns, update rows) without full rewrites.
Typical use cases
Lance is used as the storage layer for vector databases and ML feature stores, where a single dataset holds both structured metadata columns and high-dimensional embedding columns, and needs to serve both analytical filtering and vector similarity search efficiently.
Relationship to the wider lakehouse ecosystem
Lance occupies a different niche than general-purpose lakehouse formats like Apache Iceberg or Delta Lake, which are optimized for large-scale batch analytics on tabular data rather than embeddings and random row access. It's most often encountered inside ML-specific tools like LanceDB rather than general SQL analytics stacks. DuckDB does not have native Lance support; teams typically export Lance data to Parquet, or use DuckDB's vector similarity search (VSS) extension for embedding search on data already stored in DuckDB or Parquet.
Related terms
A vector database stores high-dimensional numeric vectors (typically embeddings) and provides fast similarity search over them, powering use cases like semantic search and retrieval-augmented generation.
Embeddings →Embeddings are dense numeric vector representations of data — text, images, audio, or other objects — learned so that semantically similar inputs end up close together in vector space.
ORC →ORC (Optimized Row Columnar) is a columnar file format originally built for Hadoop and Hive, designed for fast reads, high compression, and efficient predicate pushdown on large analytical datasets.
Parquet →Apache Parquet is a columnar storage file format designed for efficient data processing and analytics.
storage →DuckDB uses an efficient columnar storage format optimized for analytical queries.
Parquet vs CSV →Parquet is a compressed, columnar binary file format optimized for analytics, while CSV is a plain-text, row-based format with no built-in schema or compression — Parquet is generally faster and smaller for analytical workloads, while CSV is simpler and universally readable.
FAQS
Lance is used for storing machine learning data, particularly vector embeddings alongside structured metadata, where workloads need both fast analytical scans and fast random row access — a pattern general-purpose columnar formats like Parquet handle less efficiently.
Parquet is optimized for large sequential scans in analytical queries. Lance adds efficient random access and built-in versioning on top of a columnar layout, which better suits point lookups and updates common in ML and vector-search workloads.
Not natively. Lance is primarily used within ML-focused tools like LanceDB. To analyze Lance data in DuckDB, teams typically export it to Parquet, or use DuckDB's own vector similarity search extension for embedding data already stored in Parquet or DuckDB tables.
