### Trade-offs: Vector Databases vs. BM25 Keyword Search In a product recommendation engine, the core trade-off between Vector Databases (Dense Retrieval) and BM25 (Sparse Retrieval) comes down to **semantic understanding vs. exact token matching**, alongside significant differences in infrastructure cost. #### 1. Relevance and Recall * **Vector Databases (Semantic):** Represent products and queries as dense embeddings (e.g., 768-dimensional floats). They excel at capturing intent, synonyms, and cross-modal relationships (e.g., matching a query for "beach outfit" to "linen shorts" even if no words overlap). However, they struggle with "out-of-vocabulary" exact matches, such as specific model numbers, SKUs, or rare brand names. * **BM25 (Lexical):** Uses term frequency-inverse document frequency (TF-IDF) mechanics. It guarantees precision for exact keyword matches (e.g., "Sony WH-1000XM5" or "size 10 red"). However, it suffers from the "vocabulary mismatch" problem—if a user searches for "warm winter coat" and the product description only says "insulated parka," BM25 returns a score of zero. #### 2. Computational Cost and Memory Footprint * **Vector Databases:** Very memory-intensive. To achieve sub-100ms latency, Approximate Nearest Neighbor (ANN) indices like HNSW (Hierarchical Navigable Small World) must reside entirely in RAM. Updating an HNSW index is computationally expensive, making real-time inventory updates difficult. * **BM25:** Extremely lightweight. It relies on inverted indices, which compress well and can be partially stored on disk (SSD) while relying on the OS page cache for hot terms. Real-time updates (e.g., changing price or stock status) are near-instantaneous. #### 3. Cold Start and Personalization * **Vector Databases:** Naturally support personalization. You can average a user's historical interaction vectors to create a "User Embedding" and query the product space directly without a text query. * **BM25:** Poor at personalization unless heavily engineered with manual query-expansion rules (e.g., appending a user's favorite brands as boosted optional terms to their search). --- ### Summary Comparison Table (Scale: 50M Products) *Assumptions: 50M products, average 200 words per product description, 768-dimensional float32 embeddings.* | Metric / Feature | Vector Database (HNSW) | BM25 (Inverted Index) | | :--- | :--- | :--- | | **Primary Strength** | Semantic similarity, intent, personalization | Exact matches (SKUs, brands, specs) | | **Primary Weakness** | Hallucinates on exact SKUs; high RAM cost | Zero recall on synonyms/concepts | | **Est. Storage / RAM** | **~220 GB – 300 GB RAM** (uncompressed) | **~30 GB – 50 GB Disk** (~16 GB RAM cache) | | **P99 Latency (50M items)** | 15ms – 40ms | 5ms – 15ms | | **Ingestion/Update Speed** | Slow (CPU-bound graph rebuilding) | Fast (Segment merging) | | **Compute Cost** | High (Requires GPU for encoding + High-RAM nodes) | Low (Standard CPU + NVMe SSD nodes) | --- ### Recommended Architecture: 50M Items @ <100ms Latency For a catalog of 50 million items, relying solely on one method will yield poor business results. The industry standard is a **Multi-Stage Hybrid Retrieval & Ranking Architecture**. To keep costs reasonable while hitting the sub-100ms SLA, you must use **Scalar Quantization (SQ8)** for the vectors and a unified search engine (like Elasticsearch, OpenSearch, or Vespa) to avoid network hop penalties. #### Concrete Infrastructure Sizing (The "Unified Hybrid" Cluster) Instead of maintaining a separate vector DB and text DB, use **OpenSearch or Vespa** to house both the BM25 inverted index and the HNSW vector index on the same shard. * **Vector Math:** 50M items $\times$ 768 dimensions $\times$ 4 bytes (float32) = 153.6 GB. With HNSW graph overhead ($\sim 1.5\times$), this requires $\sim 230$ GB of pure RAM. * **Optimization:** Apply **INT8 Scalar Quantization (SQ8)**. This reduces the vector size by 75% with $<1\%$ recall loss. Vector RAM drops to **~60 GB**. * **Cluster Specs:** **4 Data Nodes** (e.g., AWS `r6gd.4xlarge` – 16 vCPU, 128 GB RAM, NVMe SSD). Total cluster RAM: 512 GB (plenty of headroom for the 60GB vectors, 50GB BM25 index, JVM heap, and OS file cache). * **Sharding:** 16 primary shards with 1 replica (32 shards total, 8 shards per node). Each shard searches ~3.1M items. #### The Sub-100ms Execution Pipeline (Total Time: ~65ms) **Stage 1: Query Processing & Encoding (Latency: ~10ms)** * Incoming request hits the Recommendation API. * User context and query are converted into a 768d vector using a distilled model (e.g., `bge-base` or a custom Two-Tower model) running on an **NVIDIA T4/L4 GPU** via Triton Inference Server using ONNX Runtime / TensorRT. **Stage 2: Parallel Hybrid Retrieval & Pre-Filtering (Latency: ~25ms)** The engine executes a single network request to the 4-node cluster, running two queries in parallel per shard, **pre-filtered** by hard attributes (e.g., `in_stock: true`, `region: US`): 1. **BM25 Query:** Retrieves top 250 items based on keyword/attribute matching. 2. **ANN Vector Query:** Searches the SQ8 HNSW index (`ef_search=100`) to retrieve the top 250 semantically similar items. **Stage 3: Score Normalization & Fusion (Latency: ~5ms)** * Because BM25 scores (unbounded) and Cosine Similarity scores (0 to 1) cannot be added directly, combine the two lists of 250 items using **Reciprocal Rank Fusion (RRF)**. * Deduplicate and select the **Top 100 candidates**. **Stage 4: Lightweight Cross-Encoder / GBDT Re-ranking (Latency: ~20ms)** * Retrieval optimizes for *recall*; re-ranking optimizes for *precision and business logic*. * Pass the Top 100 candidates into a fast **LightGBM / XGBoost** model (hosted in-memory on the API layer). * This model incorporates real-time features that cannot be indexed in HNSW: historical Click-Through Rate (CTR), profit margin, dynamic pricing, and user-specific affinity scores. * Sort and return the final **Top 20 recommendations** to the user. **Stage 5: Network Overhead (Latency: ~5ms)** * gRPC serialization/deserialization between internal microservices. **Total P99 Latency:** $10 + 25 + 5 + 20 + 5 = \mathbf{65\text{ms}}$ (Well within the 100ms budget).
