A vector database stores high-dimensional numeric representations of content — embeddings — and finds the ones closest to a query vector. The underlying idea is that semantically similar content produces nearby vectors, so “nearest neighbour” becomes “most similar in meaning”.
The category attracted enormous attention alongside large language models. Some of the use cases are genuinely strong; others are solved better by tools that have existed for twenty years.
What it actually does
An embedding model converts text, images or audio into a list of several hundred to several thousand numbers. Content with similar meaning produces vectors that are close together under a distance measure such as cosine similarity.
Searching exhaustively through millions of vectors is too slow, so these systems use approximate nearest neighbour indexes that trade a small amount of recall for very large speed gains. That trade-off — approximate, not exact — is worth understanding, because it means results can occasionally miss a genuinely close match.
Use cases that hold up
Retrieval for language models
The dominant use. Documents are chunked, embedded and stored; a user question is embedded and used to retrieve relevant chunks, which are passed to a language model as context. This is what makes a model able to answer questions about your own documents. See fine-tuning versus RAG for when this is the right approach.
Semantic search where vocabulary varies
Keyword search fails when the user and the document use different words for the same thing. A support search for “can’t log in” should find an article titled “authentication failures”. Vector search handles this naturally.
Recommendation and similarity
“More like this” over products, articles or media, based on content rather than on behaviour. Particularly valuable for cold-start items with no interaction history.
Deduplication and clustering
Finding near-duplicate support tickets, job listings or product records where exact matching fails because the text differs slightly. This is an unglamorous and genuinely high-value application.
Image and audio search
Multimodal embedding models let you search images by text description or find visually similar images, which has no reasonable keyword-based equivalent.
Where it is the wrong tool
Exact lookups. Finding a record by ID, email or SKU. Use an index.
Structured filtering. “Orders over $500 in March from customers in Germany” is a SQL query. Vector search answers it badly or not at all.
Search where precision matters most. Legal, medical and compliance search where missing an exact term is unacceptable. Keyword search with carefully curated synonyms often outperforms semantic search here.
Small corpora. Below a few thousand documents, computing similarity in memory on every query is entirely feasible and much simpler.
Hybrid search is usually better than either
In practice, combining keyword and vector search outperforms both. Keyword search handles exact terms, product codes, names and rare words that embedding models represent poorly. Vector search handles paraphrase and concept matching.
The standard approach runs both and merges the result lists, commonly with reciprocal rank fusion. A reranking model applied to the merged top results improves relevance further at modest cost. Teams that start with pure vector search and add keyword matching after user complaints usually find the hybrid was the answer all along.
Do you need a dedicated database?
Three options, in increasing order of commitment.
An extension to your existing database. Postgres with a vector extension handles millions of vectors adequately and keeps everything in one system, with real transactions, joins to your relational data and one thing to operate. For the majority of applications this is the right answer.
Vector capability in an existing search engine. If you already run a search cluster, it probably supports vectors now, and hybrid search comes naturally.
A dedicated vector database. Justified at very large scale — hundreds of millions of vectors — or where specific index types, high write throughput or advanced filtering performance matter. It is another system to run, secure and back up.
Start with the extension. Moving later is a data migration, which is tractable; operating an unnecessary distributed system for two years is not.
The parts that actually determine quality
Retrieval quality depends far more on these than on which database you chose.
- Chunking. How documents are split. Chunks too large dilute the embedding; too small lose context. Splitting on semantic boundaries beats fixed character counts.
- The embedding model. Domain-appropriate models materially outperform general ones on specialist content.
- Metadata filtering. Restricting by tenant, date or document type before similarity search improves both relevance and speed — and in multi-tenant systems it is a security requirement, not an optimisation.
- Reranking. A cross-encoder applied to the top 50 results reliably improves the top 5.
- Evaluation. A labelled set of queries with expected results, run on every change. Without it, tuning is guesswork.
Operational considerations
Embeddings must be regenerated when the model changes, which for a large corpus is a substantial batch job and a cost. Index builds are memory-hungry. And updating or deleting vectors is more expensive than inserting them in several index types, which matters for content that changes often.
Plan for re-embedding from the start: keep the source content, record which model version produced each vector, and make the pipeline re-runnable.
A practical evaluation approach
Retrieval quality is measurable, and measuring it changes the conversation from opinion to iteration.
Assemble 50 to 100 real questions with the documents that should be retrieved for each. Twenty minutes of a subject expert’s time produces something more useful than any amount of tuning by feel.
Then track two numbers. Recall at k: how often the correct document appears in the top k results. And mean reciprocal rank: how high up it appears. Recall tells you whether the information was available at all; rank tells you whether it was prominent enough to be used.
Run this on every change — a new embedding model, a different chunk size, added metadata filters, a reranker. Most tuning produces a mixed result, improving some queries and harming others, and only a fixed test set reveals that.
Chunking, in more detail
Chunking has more effect on retrieval quality than almost any other choice, and it is usually done with a default setting nobody revisited.
Fixed-size chunks with overlap are the common default and are reasonable for homogeneous prose. They cut mid-sentence and mid-table, which damages both the embedding and the readability of what is passed to the model.
Structure-aware chunking splits on headings, paragraphs or list boundaries, keeping semantically coherent units together. For documentation with clear headings this is noticeably better.
Two refinements are worth the effort. Prepend the document title and section heading to each chunk, so an isolated paragraph carries its context into the embedding. And store a pointer to neighbouring chunks, so the retrieved passage can be expanded with surrounding text before being passed to the model.
Frequently asked questions
Is a vector database required for RAG?
Some vector store is required. It does not have to be a dedicated product — a Postgres extension or an existing search engine is often sufficient.
How many vectors before a dedicated system is needed?
A general-purpose database with a vector extension is comfortable into the low millions. Beyond that, or with demanding latency requirements, dedicated systems start to earn their cost.
What dimension should embeddings be?
Determined by the model. Larger dimensions capture more nuance and cost more to store and search; several models now support reducing dimensions with modest quality loss.
How do I handle multi-tenant data?
Filter by tenant as part of the query, ideally enforced at the storage layer rather than in application code. Treat cross-tenant leakage as the primary risk in the design.
