Vector Search vs Hybrid Search using LanceDB
Learn the key differences between vector search and hybrid search in RAG applications. Use cases, performance tradeoffs, and when to choose each.
In this cookbook, we’ll explore the differences between pure vector search and hybrid search approaches that combine vector embeddings with metadata filtering. We’ll see how structured metadata can dramatically improve search relevance and precision beyond what vector similarity alone can achieve.
When users search for products, documents, or other content, they often have specific attributes in mind. For example, a shopper might want “red dresses for summer occasions” or a researcher might need “papers on climate change published after 2020.” Pure semantic search might miss these nuances, but metadata filtering allows you to combine the power of vector search with explicit attribute filtering.
Like always, we’ll focus on data-driven approaches to measure and improve retrieval performance.
Requirements
Before starting, ensure you have the following packages installed:
Setup
Start by setting up the environment:
The Dataset
In this cookbook, we’ll work with a product catalog dataset containing fashion items with structured metadata. The dataset includes:
- Basic product information: titles, descriptions, brands, and prices
- Categorization: categories, subcategories, and product types
- Attributes: structured characteristics like sleeve length, neckline, and fit
- Materials and patterns: fabric types and design patterns
Here’s what our taxonomy structure looks like:
Having well-structured metadata enables more precise filtering and can significantly improve search relevance, especially for domain-specific applications where users have particular attributes in mind. This data might come from manual tagging by product managers or automated processes with LLMs.
Let’s first load the dataset from Huggingface:
Prepare DataFrame for LanceDB
We’ll use a Pandas DataFrame as the ingest interface.
For simplicity, use description
as the “text” field, although you could concatenate title/description/etc.
Generate Embeddings (OpenAI)
Now, let’s create embeddings for our product descriptions. We’ll use OpenAI’s text-embedding-3-large model:
Combine all text fields into a single searchable text field
We’ll create a single text field that combines the product name, description, and category. This will allow us to perform a single search over all relevant text content:
Ingest Data into LanceDB
We’ll use LanceDB to store our product data and embeddings. LanceDB makes it easy to experiment, as it provides both vector and hybrid search capabilities within one single API.
Generating Synthetic Data
When you don’t have production data to start with, you can generate synthetic data to simulate a real-world scenario. We already have the ‘output’, which is the clothing item we just embedded. We now want to generate synthetic queries that would be relevant to the clothing item.
In this case, we’ll use GPT-4 to generate realistic user queries that would naturally lead to each product in our catalog. This gives us query-product pairs where we know the ground truth relevance.
Let’s visualize what this looks like:
Hybrid Search in LanceDB
LanceDB makes it easy to combine vector search with full-text search in a single query. Let’s see how this works with a practical example:
title | brand | description | category | subcategory | product_type | price | _relevance_score |
---|---|---|---|---|---|---|---|
Elegant Wedding Guest Dress | Zara | A stunning formal dress perfect for wedding ceremonies and receptions | Women | Dresses | Formal Dresses | 189.99 | 0.87 |
Floral Maxi Dress for Special Occasions | H&M | Beautiful floral pattern dress ideal for weddings and formal events | Women | Dresses | Maxi Dresses | 149.50 | 0.82 |
Satin Wedding Guest Jumpsuit | ASOS | Sophisticated alternative to dresses for wedding guests | Women | Jumpsuits | Formal Jumpsuits | 165.75 | 0.79 |
Men’s Formal Wedding Suit | Hugo Boss | Classic tailored suit perfect for wedding guests | Men | Suits | Formal Suits | 399.99 | 0.71 |
Beaded Evening Gown | Nordstrom | Elegant floor-length gown with beaded details for formal occasions | Women | Dresses | Evening Gowns | 275.00 | 0.68 |
Implementing Different Search Methods
To properly compare different search approaches, we’ll implement three search functions:
These functions provide a clean interface for our three search methods:
- Semantic search: Uses only vector embeddings to find similar products
- Lexical search: Uses only BM25 text matching (similar to what traditional search engines use)
- Hybrid search: Combines both approaches for potentially better results
Note that we sanitize the query text to remove characters that might break the full-text search functionality. This is an important preprocessing step when working with user-generated queries.
Evaluation Metrics
To objectively compare our search methods, we’ll use two standard information retrieval metrics:
- Recall: The proportion of relevant items successfully retrieved
- Mean Reciprocal Rank (MRR): How high relevant items appear in our results
The evaluate_search function runs all three search methods on each query and calculates both metrics. This gives us a nice view of how each method performs across our test set.
Prepare Evaluation Data
Assuming your synthetic queries are a list of dicts with "query"
and "id"
.
Run the Experiment
Now we can run the experiments. The code does the following:
- Tests each search method with different numbers of results (k=3, 5, and 10)
- Aggregates the metrics by calculating the mean recall and MRR for each method
- Organizes the results in a DataFrame for easy comparison
k | method | recall | mrr |
---|---|---|---|
3 | semantic | 0.906 | 0.816 |
3 | lexical | 0.937 | 0.815 |
3 | hybrid | 0.916 | 0.848 |
5 | semantic | 0.937 | 0.823 |
5 | lexical | 0.969 | 0.822 |
5 | hybrid | 0.948 | 0.860 |
10 | semantic | 0.974 | 0.828 |
10 | lexical | 0.984 | 0.824 |
10 | hybrid | 0.990 | 0.868 |
Conclusion
Our evaluation demonstrates that hybrid search consistently outperforms both pure vector search and lexical search across all tested k values. Key findings:
- Hybrid search achieves the highest MRR scores, showing that combining semantic understanding with keyword matching places relevant results higher in the result list.
- Lexical search performs surprisingly well on recall, reminding us that traditional keyword approaches remain valuable for explicit queries.
- Vector search provides a solid baseline but benefits significantly from the precision that text matching adds.
As k increases, recall improves across all methods, but hybrid search maintains its advantage in ranking relevant items higher. These results highlight that the best search approach depends on your specific data and user query patterns. For product search where users combine concepts (“casual”) with attributes (“red”), hybrid search offers clear advantages.
I hope this analysis helps you make informed decisions about the best approach for your own use case. Remember to:
- Test multiple retrieval strategies on your specific data
- Measure performance with appropriate metrics
- Consider the trade-offs between implementation complexity and performance gains
For the full notebook, check it out on: GitHub.