Lecture
A vector database , vector store , or vector search engine — is a database that stores and retrieves vector representations of data in a vector space . Vector databases typically use approximate nearest neighbor search algorithms, so that users can search for records that are semantically similar to a given input, unlike traditional databases, which mainly search for records by exact match. Use cases for vector databases include similarity search , semantic search , multimodal search , recommendation systems , object detection , and retrieval-augmented generation (RAG).
Vector representations are mathematical representations of data in a multidimensional space. In this space, each dimension corresponds to a feature of the data, with the number of dimensions ranging from a few hundred to tens of thousands depending on the complexity of the data being represented. Each data item is represented by a single vector in this space. Words, phrases, or entire documents, as well as images, audio, and other data types, can be vectorized.
A vector can express anything: a word, a product, an image, and even a user's past interactions with your application.
For example, if you're recommending products to visitors, you might represent them with a vector of hundreds of dimensions, each expressing a separate aspect of a given product (color, size, price, seasonality, and so on).
In that case, you can compare such vectors to find similar products, even if they don't contain the same keywords.
These feature vectors can be computed from the source data using machine learning methods, such as feature extraction algorithms, word embeddings, or deep learning networks. The goal is for semantically similar data items to have feature vectors that are close to one another.
Among the most important methods for similarity search in high-dimensional vectors are:
and combinations of these methods.
In recent benchmarks, HNSW-based implementations have shown some of the best results. Conferences such as the International Conference on Similarity Search and Applications (SISAP) and the Conference on Neural Information Processing Systems (NeurIPS) have hosted competitions on vector search over large databases.
In general terms, the workflow of a vector database looks like this:
In the first stage, you convert your data (text, images, and so on) into vectors, using a suitable model such as OpenAI or Cohere, or a library such as SentenceTransformers.
The database will store the vectors in a way that's convenient for subsequent querying. This is often done using a technique called «approximate nearest neighbors» (ANN), so that the database doesn't have to compare every vector in the dataset against every other one.
You can picture this mechanism as a giant map, where the closer two points are, the greater the similarity between them. The database divides such maps into regions so it can quickly find the points closest to a specific query.
When a query is executed, your input is converted into a vector, and the database searches for the region of the map to which the resulting vector belongs. Once it has found the right region, it can quickly find the vectors located near yours.
This is a highly simplified picture, but it helps convey the underlying principle of how vector databases work.
Next, let's look at how vector databases measure similarity. Here are two popular approaches:
But there are many other distance metrics as well. The final choice will depend on your data and the problem you're solving.
Below I'll briefly describe a common problem with vector databases known as the «curse of dimensionality».
When working with high-dimensional vectors, it can be difficult to search and compare them efficiently. On top of that, you may run into problems with overfitting and noise.
You can address such difficulties using dimensionality reduction techniques, such as principal component analysis (PCA) or t-SNE. Applying them lets you reduce the number of dimensions while preserving the most important information.
For example, you could reduce a 1000-dimensional vector down to 100 dimensions
Vector databases are used in a wide range of machine learning applications, including similarity search , semantic search , multimodal search , recommendation systems , object detection , and retrieval-augmented generation .
A particularly common use case for vector databases is retrieval-augmented generation (RAG), a method for improving the responses of large language models on domain-specific topics . The retrieval component in RAG can be any search system, but it is most often implemented as a vector database. Text documents describing the domain of interest are collected, and for each document or section of a document a feature vector (known as an «embedding») is computed, typically using a deep learning network, and stored in the vector database along with a reference to the document. When a user submits a query, a feature vector for the query is computed, and the database is queried to retrieve the most relevant documents. These are then automatically added to the large language model's context window, and the large language model proceeds to generate a response to the query, taking this context into account.
Comments