AI
Why Lexical Search Breaks on Natural Language Queries
Elasticsearch builds on Apache Lucene’s inverted index, a token-to-document map that Richmond described as
“effectively a glorified hashmap. It has”
the positions and document references needed for fast lookups. Since 2015, both Lucene and Elasticsearch use BM25, a probabilistic ranking model from Robertson and Spark Jones that scores documents by query-term frequency regardless of word proximity. BM25 handles exact product lookups cleanly. Type “the latest film in the Skywalker series” instead of “Rise of Skywalker” and the right document lands in the middle of results. That vocabulary mismatch is the core limit of lexical search.
How Vector Embeddings Capture Semantic Meaning
Vector search converts documents and queries into embeddings, numerical arrays of 256 to 1,024 dimensions that position semantically similar content close together. Elasticsearch uses Hierarchical Navigable Small Worlds (HNSW) for nearest-neighbor lookup. The default similarity metric is cosine. Bit vectors use L2 Euclidean distance. Picking an embedding model starts with off-the-shelf options on Hugging Face, filtered by license and whether the environment needs to be air-gapped. A query for “solo” resolves closest to “Han Solo” because the vectors cluster by semantic attributes, not keyword overlap. That is what lexical search cannot do.
Vector Search Performance Tradeoffs and Quantization
High-dimensional vectors carry real costs. Embeddings up to 1,024 dimensions multiply storage when the corpus reaches millions of documents. Indexing is slower because embeddings must be generated first. HNSW also trades accuracy for speed by skipping comparisons in sparser upper graph layers. Quantization is the main mitigation. Richmond described it as compression for vectors: split the vector into sub-vectors, cluster around a centroid, then encode. Elasticsearch supports product quantization and Better Binary Quantization (BBQ), which converts floats to a binary representation while preserving enough accuracy to keep results usable.
Combining Lexical and Vector Search with Hybrid Scoring
Exact matches still matter. A search for “Han Solo” by name needs lexical precision. A search for “a reliable ship captain” needs semantic context. Linear combination blends both by applying explicit boost factors (Richmond used 0.4 for BM25, 0.6 for KNN) but finding those numbers requires repeated experiments with no principled stopping point. Reciprocal rank fusion (RRF) removes that burden by reranking results by position using a default k-factor of 60. After retrieval, a reranking model can rescore results against a supplied judgment list without touching the underlying query.
Why LLMs Still Need Search: Hallucination and Context Engineering
LLMs hallucinate when context is missing.
“LLMs are kind of like my six-year-old.”
Richmond’s point: a six-year-old with no context invents answers. An OpenAI paper from the previous month found that training incentives reward giving an answer over saying “I don’t know.” RAG is the fix: inject your own data into the prompt so the model reasons from supplied facts. Multi-agent systems add a second constraint: agents sharing context must not share the wrong context. Richmond cited her banking background, noting that a buyside agent with access to sellside data is a compliance violation, not a search misconfiguration.
Notable Quotes
effectively a glorified hashmap. It has Carly Richmond · ▶ 5:30
LLMs are kind of like my six-year-old. Carly Richmond · ▶ 30:02
context is going to be king to make sure Carly Richmond · ▶ 36:11
Key Takeaways
- BM25 fails on synonym-rich or semantic queries because it relies on exact token matches.
- Hybrid search with reciprocal rank fusion removes the need to hand-tune score boost factors.
- RAG is how you give an LLM the proprietary or recent context it needs to stop hallucinating.