What is a Vector Database?

A Vector Database is a database designed to store, manage, and index high-dimensional vector embeddings (generated by deep learning models like OpenAI Ada or BERT). It allows systems to perform rapid Approximate Nearest Neighbor (ANN) searches based on semantic meaning rather than exact keyword matches.

Core Concepts & Mathematics

• Machine learning models represent text, images, or audio as arrays of numbers (floats) called embeddings. These vectors occupy a high-dimensional space (typically 768 to 1536 dimensions).

Distance & Similarity Metrics

• To find the “most similar” items, databases measure the spatial distance between query vector $\mathbf{A}$ and database vectors $\mathbf{B}$:

1. Cosine Similarity

• Measures the angle between two vectors, ignoring magnitude.
• $\text{Similarity}=\cos (\theta )=\frac{\mathbf{A}\cdot \mathbf{B}}{\Vert \mathbf{A}\Vert \Vert \mathbf{B}\Vert }=\frac{\sum a_i{b}_i}{\sqrt{\sum a_i^2}\sqrt{\sum b_i^2}}$
• Value range: [-1, 1] (where 1 means identical direction).

2. Dot Product

• Measures both direction and magnitude. Optimal if vectors are normalized to unit length (length = 1), as it simplifies to Cosine Similarity.
• $\mathbf{A}\cdot \mathbf{B}=\sum a_i{b}_i$

3. Euclidean Distance (L2)

• Measures the straight-line distance between two coordinates.
• $d(\mathbf{A},\mathbf{B})=\sqrt{\sum (a_i-b_i{)}^2}$
• Value range: [0, ∞) (where 0 means identical coordinates).

Approximate Nearest Neighbor (ANN) Indexing

• Finding the exact nearest neighbors requires calculating the distance between the query vector and every single vector in the database (K-Nearest Neighbors, $O(N)$ complexity). This does not scale. Vector databases trade a tiny margin of accuracy for $O(\log N)$ speed using ANN indexing:

1. Hierarchical Navigable Small World (HNSW)

• HNSW is a graph-based indexing algorithm inspired by SkipLists.
• • Structure: It builds a multi-layered graph. The top layers contain a sparse graph with long-distance connections (for fast routing across the vector space). Lower layers contain increasingly dense graphs with shorter connections (for fine-tuning precision).
• • Search Path: A query starts at the top layer, jumps greedily to the node closest to the query, shifts down a layer, and repeats until it reaches the bottom layer to find the closest neighbors.
• • Trade-off: Very fast queries ($O(\log N)$) and high accuracy, but consumes large amounts of memory (RAM) because the graph must reside in memory.

2. Inverted File Index with Product Quantization (IVF-PQ)

• IVF-PQ is a clustering and compression-based indexing algorithm.
• • IVF (Inverted File): Uses K-Means clustering to partition the vector space into $C$ clusters. During queries, it computes the distance to the cluster centroids, selects the closest centroids, and only searches vectors within those specific clusters (pruning the search space).
• • PQ (Product Quantization): Compresses high-dimensional vectors by breaking them into smaller sub-vectors, quantizing them to a codebook index. This reduces memory footprint by up to 95%.
• • Trade-off: Low memory footprint (can run on disk-backed architectures), but slower queries and lower search accuracy compared to HNSW.

Vector Engine Comparison

Implementation: Cosine Similarity

​

import math
from typing import List
 
def dot_product(v1: List[float], v2: List[float]) -> float:
    return sum(x * y for x, y in zip(v1, v2))
 
def magnitude(v: List[float]) -> float:
    return math.sqrt(sum(x * x for x in v))
 
def cosine_similarity(v1: List[float], v2: List[float]) -> float:
    """
    Computes the cosine similarity between two float vectors.
    Assumes vectors are of equal length.
    """
    mag1 = magnitude(v1)
    mag2 = magnitude(v2)
    
    if mag1 == 0 or mag2 == 0:
        return 0.0
        
    return dot_product(v1, v2) / (mag1 * mag2)
 
# Example Usage
if __name__ == "__main__":
    # Semantic representation vectors (e.g. "king" and "queen" vs "computer")
    vec_king = [0.15, 0.88, 0.45]
    vec_queen = [0.17, 0.85, 0.49]
    vec_computer = [0.72, 0.11, 0.05]
    
    sim_monarchs = cosine_similarity(vec_king, vec_queen)
    sim_tech = cosine_similarity(vec_king, vec_computer)
    
    print(f"Similarity (King, Queen): {sim_monarchs:.4f}")   # Expected close to 1
    print(f"Similarity (King, Computer): {sim_tech:.4f}")     # Expected close to 0

// JS magnitude and cosine score calculator
function cosineSimilarityJS(v1, v2) {
    let dot = 0;
    let m1 = 0;
    let m2 = 0;
    for (let i = 0; i < v1.length; i++) {
        dot += v1[i] * v2[i];
        m1 += v1[i] * v1[i];
        m2 += v2[i] * v2[i];
    }
    if (m1 === 0 || m2 === 0) return 0;
    return dot / (Math.sqrt(m1) * Math.sqrt(m2));
}

​

More Learn

• Hierarchical Navigable Small World (HNSW) Original Paper — Yu. A. Malkov et al.
• Pinecone Vector Indexing Guide — Complete handbook on vector similarity.