Hybrid Search Explained: Combining Vector Search, Keywords, and Metadata Filters

Hybrid Search Explained: Combining Vector Search, Keywords, and Metadata Filters

Hybrid search combines semantic vector retrieval, exact keyword retrieval and metadata filtering so a search system understands intent without losing precision. In a messaging app it lets a user find "the invoice Priya sent in the Design Team chat last week" even when the message says "payment document", while restricting results to chats the user may read, a sender and a date range.

Why one method is not enough

A hybrid pipeline runs dense and sparse retrieval in parallel over the same eligible set, the one defined by metadata filters applied during retrieval, and fuses the two rankings. Filters are not another score; they decide which documents may appear at all.

Fusion: scores versus ranks

The familiar mental model is final = α·vector + (1 − α)·keyword, where α = 0 is keyword-only and α = 1 is vector-only. It only works once both signals are on a comparable scale. Cosine similarity and BM25 are not, and BM25 scores vary with the query and the corpus. Reciprocal rank fusion (RRF) sidesteps scaling by combining ranks: each document scores Σ 1/(k + rank). Weighted RRF keeps the intuition of α by weighting each list's contribution: weights [α, 1 − α], available in Qdrant from 1.17. Values around 0.6–0.75 are a common starting point for conversational search and a poor one for ID lookups. Tune α, k and the candidate depth per search surface on a labelled evaluation set. RRF scores are ranking signals, not probabilities.

Reference implementation: Qdrant, FastEmbed, weighted RRF

Dense embeddings capture meaning, BM25 sparse vectors (with Qdrant's IDF modifier) capture exact terms, and Qdrant applies the workspace, permission and structured filters inside both prefetches before fusing them. Several details matter in a real deployment:

Index what users search for. Embedding only the message body leaves filenames such as invoice-october.pdf or INV-2026-1042.pdf unreachable, even though exact filenames are the very case sparse retrieval is meant to cover. The indexed text includes the attachment name; OCR text and summaries can join it, while permissions and timestamps stay as payload.

Permissions by membership, resolved at query time. Copying an allowed_user_ids list onto every message multiplies storage by channel size, and every membership change means rewriting thousands of points, so revocations lag. Here each message carries its channel, and the service filters on the channels the user can read, fetched from the membership service per request. A requested channel only narrows that set, and both dependencies fail closed until wired.

α is real. The narrative recommends a tunable α, but a plain-RRF implementation ignores it. Weighted RRF makes the request's alpha actually change the ranking, with None meaning unweighted RRF.

Lifecycle and correctness. The lifespan replaces the deprecated @app.on_event. Payload indexes are created even when the collection already exists. A delete path exists, because deleted or revoked messages must leave the index, not just the UI. Date windows are validated and half-open, so adjacent windows never double-count a message. Embedding lists are zipped strictly, and the collection name is versioned with the model and text recipe.

# hybrid_search_service.py -- dense + BM25 sparse retrieval in Qdrant, fused with weighted RRF
# pip install "qdrant-client[fastembed]>=1.16" fastapi uvicorn   (server: Qdrant >= 1.17)
from __future__ import annotations
import logging
import os
import uuid
from contextlib import asynccontextmanager
from dataclasses import dataclass
from typing import Annotated, Any, Iterable, Protocol
from fastapi import Depends, FastAPI, HTTPException, Request
from pydantic import BaseModel, Field, model_validator
from qdrant_client import QdrantClient, models
log = logging.getLogger("hybrid-search")
COLLECTION = os.getenv("QDRANT_COLLECTION", "messages_v2")   # version per model/text recipe
PAYLOAD_INDEXES = {"workspace_id": models.PayloadSchemaType.KEYWORD,
                   "channel_id": models.PayloadSchemaType.KEYWORD,
                   "sender_id": models.PayloadSchemaType.KEYWORD,
                   "has_attachment": models.PayloadSchemaType.BOOL,
                   "sent_at_epoch": models.PayloadSchemaType.INTEGER}
class Embedder(Protocol):
    def embed(self, texts: Iterable[str]) -> Iterable[Any]: ...
    def query_embed(self, text: str) -> Iterable[Any]: ...
@dataclass
class Search:
    client: QdrantClient
    dense: Embedder            # e.g. fastembed.TextEmbedding(...), 384-d for all-MiniLM-L6-v2
    sparse: Embedder           # fastembed.SparseTextEmbedding("Qdrant/bm25"); needs IDF modifier
    dense_dim: int
    def ensure_collection(self) -> None:
        if not self.client.collection_exists(COLLECTION):
            self.client.create_collection(
                COLLECTION,
                vectors_config={"dense": models.VectorParams(size=self.dense_dim,
                                                             distance=models.Distance.COSINE)},
                sparse_vectors_config={"sparse": models.SparseVectorParams(
                    modifier=models.Modifier.IDF)})
        for field, schema in PAYLOAD_INDEXES.items():   # idempotent; also covers old collections
            self.client.create_payload_index(COLLECTION, field, schema, wait=True)
    def index(self, messages: list[dict], batch_size: int = 64) -> None:
        for start in range(0, len(messages), batch_size):
            batch = messages[start:start + batch_size]
            # Index what users search for: the body *and* the attachment name, so
            # "INV-2026-1042.pdf" is reachable by both retrievers.
            texts = [" ".join(filter(None, (m["text"], m.get("attachment_name")))) for m in batch]
            points = [models.PointStruct(
                id=str(uuid.uuid5(uuid.NAMESPACE_URL, f'{m["workspace_id"]}:{m["id"]}')),
                vector={"dense": [float(x) for x in d],
                        "sparse": models.SparseVector(indices=[int(i) for i in s.indices],
                                                      values=[float(v) for v in s.values])},
                payload={k: m.get(k) for k in ("id", "workspace_id", "channel_id", "sender_id",
                                               "sent_at_epoch", "has_attachment",
                                               "attachment_name", "text")})
                for m, d, s in zip(batch, self.dense.embed(texts), self.sparse.embed(texts),
                                   strict=True)]
            self.client.upsert(COLLECTION, points=points, wait=True)
    def delete(self, workspace_id: str, message_ids: list[str]) -> None:
        """Deleted or revoked messages must leave the index, not just the UI."""
        self.client.delete(COLLECTION, points_selector=[
            str(uuid.uuid5(uuid.NAMESPACE_URL, f"{workspace_id}:{m}")) for m in message_ids],
            wait=True)
    def query(self, text: str, *, workspace_id: str, readable_channels: list[str],
              f: "SearchRequest", candidate_k: int = 50) -> list[dict]:
        channels = set(readable_channels)
        if f.channel_id:                       # a requested channel narrows, never widens
            channels &= {f.channel_id}
        if not channels:
            return []
        must = [models.FieldCondition(key="workspace_id", match=models.MatchValue(value=workspace_id)),
                models.FieldCondition(key="channel_id", match=models.MatchAny(any=sorted(channels)))]
        if f.sender_id:
            must.append(models.FieldCondition(key="sender_id", match=models.MatchValue(value=f.sender_id)))
        if f.has_attachment is not None:
            must.append(models.FieldCondition(key="has_attachment",
                                              match=models.MatchValue(value=f.has_attachment)))
        if f.sent_after_epoch is not None or f.sent_before_epoch is not None:
            must.append(models.FieldCondition(key="sent_at_epoch", range=models.Range(
                gte=f.sent_after_epoch, lt=f.sent_before_epoch)))       # half-open window
        flt = models.Filter(must=must)
        dense_q = [float(x) for x in next(iter(self.dense.query_embed(text)))]
        sp = next(iter(self.sparse.query_embed(text)))
        sparse_q = models.SparseVector(indices=[int(i) for i in sp.indices],
                                       values=[float(v) for v in sp.values])
        # alpha weights dense vs keyword *ranks* (weighted RRF), so no score normalisation.
        rrf = models.Rrf(weights=[f.alpha, 1 - f.alpha]) if f.alpha is not None else models.Rrf()
        res = self.client.query_points(
            COLLECTION,
            prefetch=[models.Prefetch(query=dense_q, using="dense", filter=flt, limit=candidate_k),
                      models.Prefetch(query=sparse_q, using="sparse", filter=flt, limit=candidate_k)],
            query=models.RrfQuery(rrf=rrf), limit=f.top_k, with_payload=True)
        return [{**(p.payload or {}), "rank_score": p.score} for p in res.points]
class SearchRequest(BaseModel):
    query: str = Field(min_length=1, max_length=1000)
    channel_id: str | None = Field(default=None, max_length=200)
    sender_id: str | None = Field(default=None, max_length=200)
    has_attachment: bool | None = None
    sent_after_epoch: int | None = Field(default=None, ge=0)
    sent_before_epoch: int | None = Field(default=None, ge=0)
    alpha: float | None = Field(default=None, ge=0.0, le=1.0)   # None: plain RRF
    top_k: int = Field(default=10, ge=1, le=100)
    @model_validator(mode="after")
    def window(self) -> "SearchRequest":
        if (self.sent_after_epoch is not None and self.sent_before_epoch is not None
                and self.sent_after_epoch >= self.sent_before_epoch):
            raise ValueError("sent_after_epoch must be before sent_before_epoch")
        return self
class Principal(BaseModel):
    workspace_id: str
    user_id: str
def get_principal() -> Principal:
    raise HTTPException(401, "authentication_not_configured")   # wire verified JWT/session
def readable_channels(p: Annotated[Principal, Depends(get_principal)]) -> list[str]:
    raise HTTPException(503, "membership_service_unavailable")  # resolve per request, fail closed
@asynccontextmanager
async def lifespan(app: FastAPI):
    from fastembed import SparseTextEmbedding, TextEmbedding
    app.state.search = Search(
        client=QdrantClient(url=os.getenv("QDRANT_URL", "http://localhost:6333"),
                            api_key=os.getenv("QDRANT_API_KEY"), timeout=5),
        dense=TextEmbedding(os.getenv("DENSE_MODEL", "sentence-transformers/all-MiniLM-L6-v2")),
        sparse=SparseTextEmbedding("Qdrant/bm25"), dense_dim=int(os.getenv("DENSE_DIM", "384")))
    app.state.search.ensure_collection()
    yield
app = FastAPI(title="Hybrid Search API", lifespan=lifespan)
@app.post("/v1/search")
def search(req: SearchRequest, request: Request, p: Annotated[Principal, Depends(get_principal)],
           channels: Annotated[list[str], Depends(readable_channels)]):
    try:
        results = request.app.state.search.query(req.query.strip(), workspace_id=p.workspace_id,
                                                 readable_channels=channels, f=req)
    except Exception:
        log.exception("search_failed workspace=%s", p.workspace_id)
        raise HTTPException(503, "search_temporarily_unavailable") from None
    return {"results": results, "count": len(results)}

Tested with Qdrant's in-process mode (qdrant-client 1.20) and deterministic stand-in embedders. The tests cover: a message whose body says only "Approved." found by its attachment ID; synonym recall with channel and attachment filters; workspace isolation; membership ACLs that a requested channel cannot widen; α = 0 versus α = 1 changing the ranking; half-open windows and invalid windows; idempotent re-indexing; deletion; and an API that returns 401 or 503 until identity and membership are wired. Real FastEmbed models download on first use and were not exercised; warm them at deploy time and verify that identifiers such as INV-2026-1042 survive the BM25 tokeniser, or copy extracted IDs into a keyword payload field for exact matching.

Run Qdrant locally with docker run -p 6333:6333 -v "$PWD/qdrant_storage:/qdrant/storage" qdrant/qdrant. Natural-language dates such as "last week" are not parsed by the service: resolve them in the client or query planner to explicit, timezone-aware epoch boundaries and pass them as filters.

Real-world use case: enterprise messaging

An employee asks: "Find the contract Ravi shared in the Legal channel yesterday." The system resolves the user's readable channels and intersects them with Legal, filters by sender and a date window, runs dense search over message text, attachment names and OCR text, and runs keyword search for exact terms such as contract, MSA, NDA or a document ID. It then fuses both rankings and returns messages with attachments. This avoids two classic failures: vector search returning a similar message from the wrong channel, and keyword search missing a message that says "agreement" instead of "contract".

Common mistakes

Filtering after retrieval. Fetching the top 100 semantic matches and then removing out-of-scope or unauthorised ones can leave too few results. Filter inside each retriever.

Treating metadata as text. Do not embed "channel: design-team"; enforceable constraints belong in indexed payload filters.

One α everywhere, or a weighted sum of raw scores. Product-code lookups need keyword weight and exploratory queries need vector weight. Mixing raw BM25 and cosine scores without normalisation is meaningless; use (weighted) RRF or normalise explicitly, for example with distribution-based score fusion (DBSF).

Embedding the wrong content, or forgetting re-embedding. Index body, attachment names and OCR text, and re-embed into a new versioned collection when the model or text recipe changes.

Missing tenant isolation or stale ACLs. Every query needs the workspace filter, and permissions must be fresh enough that revocations take effect promptly.

No evaluation set. Without labelled queries, tuning α, candidate depth and reranking is guesswork.

Production hardening and checklist

Identity and access: derive workspace and user from verified JWT or session identity, never from the request body, and keep the source system's permission checks authoritative. Reliability: run Qdrant as a managed or clustered service with TLS, API keys, network restrictions, backups and resource alerts. Bound timeouts, concurrency and retries, and ingest through background workers with idempotent upserts, dead-letter handling and the deletion pipeline. Quality: track Recall@k, Precision@k, MRR or nDCG, zero-result rate, over-filtering, latency percentiles and permission-boundary tests on labelled queries that include paraphrases, exact identifiers, multilingual messages and attachment-only matches, and log filters, retrieved IDs and clicks (not message bodies) for offline evaluation. Model operations: pin model and package versions, warm models, benchmark embedding throughput, and version collections when models change. all-MiniLM-L6-v2 is English-centric, so multilingual workspaces need a multilingual model. Privacy: deleted messages, revoked attachments and removed users must disappear from results promptly; test it.

Takeaway. Filters define who may see what; dense and sparse retrieval find candidates inside that set; rank fusion combines them without pretending their scores share a scale. Index the text users actually search for, keep permissions fresh, and tune fusion on real queries.

References: Qdrant, Hybrid queries (qdrant.tech/documentation/search/hybrid-queries/): RRF, parameterised RRF (v1.16), weighted RRF (v1.17), DBSF (v1.11). Qdrant FastEmbed (qdrant.tech/documentation/embeddings/fastembed/).

Back to all blogs