Vector Technologies for AI: Extending Your Existing Data Stack

2025/03/28 - 17 min read

The database landscape has reached 394 ranked systems across multiple categories—relational, document, key-value, graph, search engine, time series, and the rapidly emerging vector databases. As AI applications multiply quickly, vector technologies have become a frontier that data engineers must explore.

The essential questions to be answered are: When should you choose specialized vector solutions like Pinecone, Weaviate, or Qdrant over adding vector extensions to established databases like PostgreSQL or MySQL? What fundamental differences exist between AI-focused vector databases and analytical vector engines like DuckDB or DataFusion? And perhaps most importantly—do we really need separate systems for these workloads?

This article explores vector databases, their differences from vector engines, and how to integrate them into your existing data engineering landscape. You'll learn each technology's unique benefits, practical applications, and guidance on when to use a vector engine versus a vector database, not to create yet another parallel data stack.

Vector Engine vs. Vector Database

Let's start with a Vector, what is it? A vector, in contrast to its database, is a mathematical term for an ordered array of numbers. In the data space, we call a vector a fixed-size array of numerical values representing a point in multi-dimensional space, such as AI embeddings that capture semantic meaning. Or a batch of values processed simultaneously using CPU optimizations.

Next, not all vector databases are the same. There are two distinct vectorized types: the engines and the databases. Vector engines are fast for everyday jobs and integrated into databases such as DuckDB. However, there are also AI vector databases that store embeddings for AI workloads. Do we need them both?

Generally, vector-based engines leverage column-oriented architecture to perform analytical operations efficiently across large datasets, applying the same operation to many values simultaneously—vectorized execution. When working with embeddings and vector similarity search, the column orientation is particularly valuable as it allows for efficient parallel processing of high-dimensional vector data. However, specialized vector databases may add additional indexing structures optimized specifically for similarity searches.

What is a Vector Engine?

Examples of vector engines include DuckDB¹, Photon Engine, and DataFusion. These are general-purpose analytical engines that have vector processing capabilities built in. They excel at traditional analytical workloads and can handle vector operations efficiently.

These engines are optimized for column-oriented operations, SIMD (Single Instruction, Multiple Data) instructions, and in-memory analytical processing. Instead of processing data row by row (scalar processing), these engines process data in chunks or "vectors" of values. For example, they might apply an operation to 1024 values at once rather than one at a time. This chunking method is more efficient for CPUs to process because it can take better advantage of modern hardware capabilities, such as cache utilization and vectorized instructions.

But how do vectorized engines work? What makes vectorized execution particularly efficient is its ability to optimize three critical aspects of modern CPU architecture:

CPU cache hierarchy optimization: Processes data in chunks that fit well in CPU caches (L1, L2, L3), minimizing costly data movement between CPU and RAM. These hierarchical cache levels provide increasingly larger but slower storage, making efficient use with access times ranging from 1-4ns (L1) to 10-40ns (L3), compared to 50-150ns for main memory. This approach can perform 10-100x faster than traditional processing for analytical workloads.
Batch processing: Reduces computational overhead by handling hundreds or thousands of values per function call, spreading the cost of operations across many data items.
Memory latency hiding: Generates multiple parallel memory requests during complex operations like hash joins, allowing the CPU's out-of-order execution capabilities to work on other data while waiting for memory fetches. This out-of-order execution enables the CPU to continue processing instructions that are ready instead of stalling on memory-dependent operations.

vector1 Different execution types: row-based vs. column-based vs. vectorized

The above diagram illustrates how vectorized engines utilize the CPU cache hierarchy more efficiently than row-by-row processing. The vectorized approach takes advantage of the faster L1 and L2 caches, whereas row-by-row processing often results in cache misses that force the CPU to retrieve data from slower memory tiers.

In comparison, relational databases such as PostgreSQL, MySQL, or SQLite process each row sequentially. However, they come with extensions to make them behave more like vector databases.

NOTE: Where does the name 'Vector Engines' come from? Vector engines get their name from vectorized execution, a processing technique. While the data isn't stored as mathematical vectors (like embeddings), these engines process data in vector-like batches. That's where the name comes from—and it makes them efficient for analytical workloads, including working with actual vector data like embeddings.

What is a Vector Database?

Vector databases, on the other hand, are specifically designed to store, index, and query high-dimensional vector embeddings, often created by AI models. These databases optimize for approximate nearest neighbor (ANN) search, similarity matching, storage of embedding vectors, and integration with AI/ML workflows. They focus on vector searches, document storage, full-text search, metadata filtering, and multi-modal as opposed to vectorized SQL query execution.

Examples include Pinecone, Weaviate, Qdrant, Chroma, Milvus, and Zilliz.

How Vector Embeddings Work for AI

Vector embeddings are used for Large Language Models (LLMs) and AI workloads. Generally, the process of loading data into vector databases and making it useful for AI analytics involves several key considerations. A simplified version of creating embeddings from content looks like this:

This gives us a better understanding of the differences between a vector engine and a vector database, as the latter works with actual vectors stored in the database, unlike the row-based or columnar-based data we typically store.

INFO: Further Terminologies Before we go any further, let's align on terminology, as it's evolving rapidly in the AI space:

(Columnar-) Vectorized Engine, Vectorized Query Engine, or Execution Query Engine: An engine that processes data in chunks. Sometimes also called embeddable database engines.
Vector database: A database used to store vector embeddings.
Retrieval-Augmented Generation (RAG): An architecture that uses vector databases to retrieve knowledge from embedded vectors.
AI Agents: Systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.
Agentic Systems (or workflows): Systems where LLMs and tools are orchestrated through predefined code paths.²
OLAP: A fast, columnar storage format optimized for analytical queries.

The Current Vector Technology Landscape

The vector technology landscape is evolving non-stop, driven by the expansion of AI applications and the need for efficient data processing. We're witnessing a diversification between purpose-built vector databases designed specifically for AI embedding storage and traditional database systems rapidly adding vector capabilities to remain relevant.

On one side, dedicated vector databases like Pinecone, Weaviate, Qdrant, Zilliz, and Chroma have positioned themselves as specialized solutions for AI workloads, particularly for RAG applications. These solutions offer optimized vector indexing structures (like Hierarchical Navigable Small World (HNSW)) and similarity search algorithms out of the box. These dedicated databases are split between open-source options (Chroma, Qdrant, Milvus) and commercial offerings (Pinecone, Weaviate). Meanwhile, established database providers like PostgreSQL, Redis, Elasticsearch, and even ClickHouse have added vector search capabilities to their existing systems, blurring the lines between dedicated and adapted solutions.

The landscape is further complicated by the AI agent ecosystem explosion, where vector databases are just one component of a complex stack that includes vertical agents, frameworks, model serving, and more. Beyond the engines shown in our timeline below, emerging solutions like Blaze, Quokka, and SingleStore are further diversifying the options available to data engineers.

vector3 Comparing the most prominent vector databases with engines, and see them in the context of AI frameworks and Vector Extension. By no means complete, but a first overview | Image by the Author

This rapid evolution raises important questions for data engineers: are specialized vector databases a temporary solution that will eventually be absorbed by existing database technologies? Or will the specialized optimization paths of dedicated vector engines and databases continue to provide value as AI workloads grow in complexity and scale?

Key Differences: When to Use Each

A vector engine's key purpose is general analytical processing, whereas a vector database specializes in AI embedding storage and similarity search. They have different query types. For example, DuckDB uses SQL and analytical queries, while vector databases focus on queries such as vector similarity search and semantic search.

The architectures differ as well, with vector engines focusing on processing efficiency and vector databases on vector indexing and retrieval. So far, vector databases with embeddings have been chosen as the backbone for AI use cases with LLMs.

Relational databases such as Postgres with pgvector, MySQL with HeatWave, or SQLite with sqlite-vss are integrating these capabilities as well with vector extensions. Another approach is to use DuckDB, which has a blazingly fast vector engine but lacks the native storage format of a vector. However, there are Array and List data types, which can be used to store and process vector embeddings as well. Plus, it has a Vector Similarity Search Extension that adds indexing support to accelerate vector similarity search queries using DuckDB's fixed-size ARRAY type. MotherDuck also adds Search in DuckDB.

So the question remains: how long until general databases catch up, and will the need for dedicated vector databases persist? Time will tell, but given the history of similar specifications for databases such as time series, document, or graph databases that have integrated series, JSONs, or graphs in relational databases, we know that we might always need both. Like always, sometimes you need just a very narrow use case, and then the dedicated database makes more sense.

The choice of whether to use vector engines or vector databases follows similar reasoning and is highly dependent on the specific use cases. The optimal solution varies based on workload characteristics, existing infrastructure, and organizational expertise.

Vectorization and vector-based engines matter because of their incredible performance with data workloads. The simplest argument to use is speed, and the second, smaller impact, is storage optimization, which means less storage is needed. For example, DuckDB databases are tiny and can handle millions of rows. This is more a side effect of hardware optimization.

Don't Build a Parallel Stack: Integrate Vectors into Data Engineering Workflow

Integration into the enterprise data landscape and well-functioning data engineering workflow is key to success with AI in the long run. Because it's changing so fast, it's even more important that we take a step back and think about how vector embeddings and their AI use cases fit in.

The key is to not repeat ourselves.

Integration into the Data Engineering Lifecycle

The data engineering lifecycle defines the end-to-end data engineering process, addressing all different components. When integrating vector operations into this lifecycle, we should aim to enhance rather than duplicate existing infrastructure. Vector operations should complement, not replace, your well-established data engineering practices.

Just as we don't replace existing data connectors (like ODBC/JDBC) with each new technology wave, we shouldn't create an entirely separate infrastructure for AI workloads. Instead, we should leverage existing tools like orchestration, scheduling, and processing frameworks while adding vector capabilities where they provide clear benefits.

This approach prevents duplication, maintains consistency, and leverages your team's existing expertise. The goal should be to add vector storage and processing capabilities within your existing data engineering cycle, not to build a parallel system.

NOTE: Example 1 - E-Commerce Recommendation Pipeline Fintech Company X enhanced their existing Airflow-orchestrated data pipeline by integrating vector embeddings without duplicating infrastructure. They added a single embedding generation step that processes product descriptions with OpenAI's embedding model, storing results in their existing PostgreSQL database using pgvector. Their recommendation service now queries these embeddings for similarity search while continuing to leverage their established monitoring, versioning, and deployment workflows. NOTE: Example 2 - Healthcare Document Processing Healthcare Provider Y integrated RAG capabilities into their existing document processing pipeline using MotherDuck as their cloud data warehouse. Rather than creating a separate AI infrastructure, they extended their ETL process to generate embeddings during the standard transformation phase. Clinical notes are processed and embedded, with these embeddings stored in MotherDuck as List data types alongside structured data. This approach leverages MotherDuck's scalability for large document collections while maintaining centralized governance controls and enabling semantic search capabilities through SQL-based similarity functions for their clinical decision support system.

Don't Repeat Yourself with AI Data Pipelines

By integrating it into the existing data stack, we can follow the rules of Don't Repeat Yourself (DRY). As the vector database and AI tooling landscape expands rapidly, organizations face the temptation to build parallel systems instead of extending their established data platforms as we had with so many other cycles (Machine Learning, DevOps, or data engineering itself). This approach creates unnecessary silos, duplicates functionality, and ultimately increases technical debt. Rather than reimagining the entire data engineering lifecycle for AI, it would be smarter to find ways to leverage existing orchestration tools, scheduling frameworks, and processing pipelines while simply adding vector capabilities where they provide clear value.

The cautionary path with frameworks like LangChain illustrates this principle. What began as an abstraction layer for LLM applications has, for many teams, become a redundant orchestration tool that sounds and feels a lot like an ETL tool or orchestrator that teams already have in place. I'm not saying it's a bad practice; it has its place too, but engineering teams reported that after initial adoption, they discovered that 95% of their work remained in prompt engineering and data formatting—tasks these frameworks don't meaningfully simplify. The pattern repeats across the industry: new tools emerge promising integration, and teams adopt them, seeking quick solutions, only to later dismantle them when realizing they've created more complexity without addressing the core challenges of working with foundation models.

Instead of multiple specialized tools for AI workloads, the sustainable path forward integrates vector operations directly into existing data platforms. As someone with two decades working with data, this lesson stands out above all others—avoid creating parallel systems for what could be extensions of existing ones. This approach preserves hard-won expertise, maintains consistency across systems, and avoids the maintenance burden of parallel infrastructures.

Data engineers already excel at building reliable pipelines, transforming data, and ensuring consistency—these skills transfer directly to AI workflows when the right integrations are in place. By focusing on adapting proven orchestrators and query engines rather than adopting entirely new frameworks, organizations can achieve better ergonomics across their entire data platform while allowing AI engineers to focus on their core competencies.

When Not to Use a Vector Database

Lastly, let's explain when it is better not to use a vector database altogether. Vector technologies are evolving rapidly to keep up with the growing AI requirements. That's why we've already seen dedicated file systems emerge for DeepSeek's storage, such as 3FS and SmallPond, which shows that it's changing fast. Although these address limitations at a massive scale that most of us will probably never experience.

I'd say the more traditional limitations are the bottlenecks in integrating vector technologies into an organization's current data architecture instead of adding another siloed stack. The challenge lies in scaling up the architecture to integrate AI use cases within the existing orchestration framework while maintaining the speed and flexibility that AI requires.

There is also the saying, why your vector database should not be a vector database:

A specialty vector database will lead to the usual problems we see (and solve) repeatedly with our customers who use multiple specialty systems.
Redundant data, excessive data movement, lack of agreement on data values among distributed components, extra labor expense for specialized skills, extra licensing costs, limited query language power, programmability and extensibility, limited tool integration, and poor data integrity and availability compared with a true DBMS.

These are all valid points as to why you might want a dedicated vector database. And also showcases that we must find a way to integrate into the data engineering lifecycle.

DuckDB: The Vector-Powered Swiss Army Knife

Let's finish this off with the bridge between a vector engine and the toolkit for data engineering: DuckDB.

It's safe to say that DuckDB has conquered the data world. Almost everyone who has used it becomes an immediate fan. Guess what? DuckDB is an in-memory vector engine that speaks SQL. With its versatility, it has been used in at least five distinct categories: interactive data apps, on-demand pipeline compute engines, lightweight SQL analytics solutions, secure enterprise data handlers, and zero-copy SQL connectors. DuckDB is a Swiss army knife that is useful for almost everything except for extremely large datasets.

So with that in mind, how do we best integrate vector engines like DuckDB or vector embeddings into our data engineering work?

DuckDB is in a unique position to bridge the gap between data engineering and AI workflows with a fast, analytical database that is heavily based on vectorization and offers super fast response times that make AI more useful. While vector databases primarily use vector embedding representations of textual data to enable vector search capabilities, DuckDB provides the Array and List data types, which can store and process vector embeddings in DuckDB or MotherDuck to enable vector search. It's fast, open-source, and free to use - making it an attractive option for data engineers looking to integrate AI capabilities without creating redundant infrastructure.

It's not the solution for everything, and there are many more, but it's a frictionless start without interrupting the data architecture. Also, DuckDB runs anywhere with a small standalone binary.

Building a Sustainable Vector Strategy in your Data Platform

We initially learned the difference between a vector database and a vector engine and understood when to use a vector database. Second, we learned the criticality of integrating vector databases into the data engineering workflow without building a parallel data stack, reducing maintenance and governance work.

I hope this article gave you some insights into how vector databases work, why we are using them, and how they are different from vector engines. It also explains how vector engines are different from columnar systems and why DuckDB might be a good option to bridge the gap to some features before diving in neck deep.

Technology, specifically in the AI domain, is rapidly changing, and new technologies are being presented, as we've seen in the current vector technology landscape. If we keep all of this in mind, we can build a more efficient data flow with fewer intermediate copied datasets and hopefully fewer siloed data stacks, which will, therefore, also result in a better overall solution.

As discussed in this article, recent developments indicate that vectorized execution engines for analytical processing and specialized vector databases for AI embeddings together represent the future of fast data processing. While established database systems will gradually incorporate these capabilities, the transition takes time—just as we've seen with specialized workloads like GIS that are still being integrated into mainstream databases. By prioritizing integration over isolation, data engineers can harness the power of vector technologies while building cohesive, maintainable data platforms that stand the test of time.

Further Reads

Whitepapers around DuckDB and embeddable:

Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask
DuckDB: an Embeddable Analytical Database
MotherDuck: DuckDB in the cloud and in the client: A paper that introduces the hybrid query processing and 1-5-Tier Architecture.
Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age. What DuckDB uses for parallelism.

Vector Search three-part series:

Part: 1: Building Vector Search in DuckDB
Part 2: Developing a RAG Knowledge Base with DuckDB
Part 3: Introducing the embedding() function: Semantic search made easy with SQL!

1. Although it happens to be a database as well, but runs well as an engine too most of the time -- you don't need storage, you can use it as a zero-copy tool to read and query parquet files

2. This workflow and agent differentiation is how Anthropic draws the between.