Surprise 1: The Database Is Only Half the Battle
The first reality check for many is that a vector database, by itself, does very little. Its performance is almost entirely dependent on the quality of the vector embeddings you put into it. An embedding is a numerical representation of data—like text
or an image—generated by a machine learning model. Choosing the right embedding model is arguably more critical than choosing the database itself. A generic model might fail to capture the nuances of your specific domain, leading to irrelevant search results. Think of it like a library; the database provides the shelves, but the embedding model is the librarian who decides how to categorize the books. If the categorization is poor, you'll never find what you're looking for, no matter how efficient the shelving system is.
Surprise 2: 'Similarity' Is a Complicated Guess
In demos, vector search seems magical, finding concepts, not just keywords. In production, that magic is revealed to be a series of complex trade-offs. Vector databases don't perform an exact, exhaustive search across millions of records for every query; doing so would be far too slow. Instead, they use Approximate Nearest Neighbor (ANN) algorithms. These algorithms are incredibly fast because they intelligently skip most of the data to find results that are probably the closest matches. This introduces a direct trade-off between speed, cost, and accuracy (or 'recall'). Tuning these systems requires developers to decide how much accuracy they're willing to sacrifice for lower latency and cost, a balancing act that is far from the simple plug-and-play experience many expect.
Surprise 3: Your Keywords Still Matter—A Lot
A common misconception is that vector search replaces keyword search entirely. Many teams are surprised to find that relying solely on vector similarity can lead to poor results, especially for queries involving specific names, codes, or jargon. An embedding might place 'CR-V' near 'compact SUV' but miss a document that only contains the exact product code. The most robust systems don't choose one over the other; they use 'hybrid search'. This approach combines the contextual understanding of vector search with the precision of traditional keyword matching (like BM25), and then re-ranks the combined results to get the best of both worlds. This means that old-school search technology is often a necessary component of a state-of-the-art vector search system.
Surprise 4: The Costs Are Hiding in Plain Sight
The price tag of a vector database isn't just the hosting fee. The real costs often emerge from the surrounding pipeline. First, there's the computational cost of generating embeddings. Running data through a powerful embedding model can be resource-intensive, especially if you have millions of documents to process and keep updated. Second, there's the memory cost. High-dimensional vectors—which often provide better accuracy—consume significant RAM. A system with a million 1536-dimensional vectors requires a substantial amount of memory just to hold the index, leading to unexpectedly high infrastructure bills. As data scales, these operational costs related to embedding generation and memory usage often dwarf the database subscription itself.
Surprise 5: Keeping Data Fresh Is a Major Headache
In a traditional database, updating a record is a simple command. In a vector search system, it's a multi-step process. When source data changes, you can't just update a field; you must regenerate the vector embedding and then update the vector index. This synchronization process can become a significant engineering challenge, especially in real-time applications. If a user changes their profile or a product description is edited, how quickly does that change reflect in the search results? Building a reliable pipeline to detect changes, re-embed content, and efficiently update the vector index without downtime or performance degradation is a complex task that many practitioners underestimate when starting out.
