Tokenization, Embeddings, and Vector Databases: The Building Blocks of Modern AI

Modern AI systems turn unstructured text into searchable meaning in three connected steps. Tokenization breaks text into the units a model reads, an embedding model turns a piece of text into a fixed-length vector whose distance to other vectors reflects semantic relatedness, and a vector database stores those vectors with metadata and retrieves the nearest ones at scale. In a telecom company, these components power a support assistant that finds the right troubleshooting article for a customer's network or billing issue.

Running telecom scenario

A telecom operator has thousands of help-center articles, outage bulletins, plan documents, and troubleshooting scripts. Customers ask questions such as “My mobile data is not working after roaming,” “Why is my bill higher this month?” and “My fiber internet keeps disconnecting at night.” Keyword search fails when customer wording differs from documentation wording; semantic search can retrieve the right article even when no exact terms match.

Install the dependencies. Pin versions in real projects, because tokenizer and model behaviour change between releases; the examples below rely on encode_query/encode_document, added in Sentence Transformers 5.0. PyTorch is installed as a dependency of sentence-transformers.

# Pin versions so tokenizer and model behaviour is reproducible
pip install "transformers>=4.44" "sentence-transformers>=5.0" \
    "qdrant-client>=1.12" numpy pandas
# Optional: a local Qdrant server instead of in-memory mode
docker run -p 6333:6333 -v "$PWD/qdrant_storage:/qdrant/storage" qdrant/qdrant

1. Tokenization

Conceptual explanation

Tokenization converts raw text into tokens — words, subwords, punctuation, or special symbols, depending on the tokenizer — and maps each to an integer ID. Transformer models consume these IDs, not raw characters or sentences. Subword tokenization lets a model handle rare words, product names, typos, and multilingual vocabulary without an enormous word-level vocabulary.

Splits are tokenizer-specific, so print them rather than assume them. all-MiniLM-L6-v2 uses BERT’s lower-casing WordPiece vocabulary: continuation pieces are prefixed with ## (an uncommon word such as “disconnecting” may come out as disconnect + ##ing), and every input is wrapped in [CLS] … [SEP]. GPT-style byte-level BPE tokenizers instead fold the leading space into the token (" not"). The two conventions should not be mixed in one example.

Runnable Python code

The snippet prints tokens and IDs, then provides a token-aware chunker. It deliberately does not truncate at an arbitrary length: the original version used max_length=32, which silently threw away everything after the 32nd token — in testing, a 540-token ticket kept only 32 tokens. Chunking on token boundaries with overlap keeps every part of a long ticket or transcript searchable. Plain Python lists are returned, so PyTorch is not needed for tokenization.

from transformers import AutoTokenizer

MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
# The model's max_seq_length: longer input is silently truncated
MAX_TOKENS = 256

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

telecom_texts = [
    "My mobile data is not working after roaming.",
    "Why is my bill higher this month?",
    "My fiber internet keeps disconnecting at night.",
]

for text in telecom_texts:
    # Plain Python lists: no PyTorch needed just to tokenize
    ids = tokenizer(text, add_special_tokens=True)["input_ids"]
    print("\nOriginal:", text)
    print("Token count (incl. [CLS]/[SEP]):", len(ids))
    print("Tokens:", tokenizer.convert_ids_to_tokens(ids))
    print("Token IDs:", ids)

def chunk_by_tokens(text: str, max_tokens: int = MAX_TOKENS,
                    overlap: int = 32) -> list[str]:
    """Split text into overlapping chunks that each fit the model's window."""
    budget = max_tokens - tokenizer.num_special_tokens_to_add()
    if not 0 <= overlap < budget:
        raise ValueError("overlap must be smaller than the token budget")
    offsets = tokenizer(text, add_special_tokens=False,
                        return_offsets_mapping=True)["offset_mapping"]
    chunks, start = [], 0
    while start < len(offsets):
        end = min(start + budget, len(offsets))
        chunks.append(text[offsets[start][0]:offsets[end - 1][1]])
        if end == len(offsets):
            break
        start = end - overlap
    return chunks

Real-world use case

A support assistant receives “My 5G is not working while travelling abroad.” Tokenization does not understand the message; it only produces IDs. The embedding model built on those tokens is what places the query near roaming, APN, and international-plan articles. Token counts still matter operationally: they decide whether a transcript must be chunked, how much an external API call costs, and whether the prompt fits the generation model’s context window.

Common mistakes and production notes

Assuming one word is one token. Uncommon words, product codes, and non-English text often split into several pieces; budget with real token counts.

Silent truncation. all-MiniLM-L6-v2 truncates input beyond 256 word pieces (it was trained on 128), and nothing warns you. Chunk long content, and monitor how often inputs hit the limit.

Tokenizer mismatch. Embedding and generation models usually use different tokenizers. Always count and chunk with the tokenizer of the model that will receive the text, and record its name, version, maximum length, and special-token settings.

Unnormalized and sensitive text. Normalize phone numbers, IMEIs, plan codes, and error strings consistently, and redact PII such as MSISDNs, account IDs, and addresses before any text leaves your boundary.

Untested vocabulary. Test with slang, typos, emojis, and telecom terms (eSIM, VoLTE, APN, MVNO). Note that all-MiniLM-L6-v2 is an English model; multilingual traffic needs a multilingual embedding model.

2. Embeddings

Conceptual explanation

An embedding is a fixed-length numeric vector that represents the meaning of a text. Texts with similar meaning produce vectors that are close under a similarity measure such as cosine similarity; OpenAI’s documentation, for example, describes embedding distance as a measure of relatedness. “Mobile data not working abroad” and “no internet connection while roaming” may land near each other despite sharing few words — but whether they do depends on the model, so verify with representative queries.

Some models are asymmetric: E5, BGE, and Qwen3 embeddings expect a query prefix that differs from the document prefix. Sentence Transformers 5 handles this with encode_query and encode_document, which apply the model’s configured prompts and are identical to encode for symmetric models such as MiniLM. Using them by default means a later model swap does not silently degrade retrieval.

Runnable Python code

Embeddings are normalized to unit length, so a plain dot product is cosine similarity; scikit-learn’s cosine_similarity would only re-normalize vectors that are already normalized. The title is embedded together with the body because it carries strong retrieval signal.

import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

telecom_articles = [
    {"id": "KB-001", "title": "Mobile data not working while roaming",
     "text": "If mobile data stops working abroad, check that data roaming "
             "is enabled, verify your international plan, and restart "
             "your device."},
    {"id": "KB-002", "title": "Understanding your monthly bill",
     "text": "Your bill may increase due to prorated charges, plan "
             "upgrades, roaming usage, premium services, or taxes."},
    {"id": "KB-003", "title": "Fiber internet frequent disconnections",
     "text": "Frequent fiber disconnections can be caused by router "
             "overheating, loose cables, local outages, or Wi-Fi "
             "interference."},
    {"id": "KB-004", "title": "Set up eSIM on your phone",
     "text": "To install an eSIM, scan the QR code from your telecom "
             "account, follow the installation prompts, and set the eSIM "
             "as your data line."},
    {"id": "KB-005", "title": "No mobile coverage at home",
     "text": "Poor indoor coverage can result from building materials, "
             "distance from a tower, or a temporary network issue."},
]
articles_df = pd.DataFrame(telecom_articles)

# Embed title + body: the title carries strong retrieval signal.
doc_texts = (articles_df["title"] + ". " + articles_df["text"]).tolist()

# encode_document / encode_query apply the model's query and passage
# prompts when it defines them (E5, BGE, Qwen3); for all-MiniLM-L6-v2
# they are identical to encode().
article_embeddings = model.encode_document(
    doc_texts, batch_size=32, normalize_embeddings=True,
    show_progress_bar=False)
print("Embedding matrix shape:", article_embeddings.shape)  # (5, 384)

query = "I have no internet on my phone when I travel overseas."
query_embedding = model.encode_query(query, normalize_embeddings=True)

# Vectors are unit length, so a dot product IS cosine similarity.
articles_df["similarity"] = article_embeddings @ query_embedding
print("\nQuery:", query)
print(articles_df.sort_values("similarity", ascending=False)[
    ["id", "title", "similarity"]])
print("\nFirst 8 dimensions:", np.round(query_embedding[:8], 6))

Real-world use case

A customer writes “I have no internet on my phone when I travel overseas.” Without the word “roaming,” embedding search may still rank KB-001 (Mobile data not working while roaming) first. The exact ranking depends on the model and corpus, so measure it on labelled queries rather than assuming it. An agent — or an assistant using retrieval-augmented generation — then works from the right troubleshooting steps instead of unrelated help pages.

Common mistakes and production notes

Embedding whole documents. Long articles exceed the model window and blur several topics into one vector. Chunk semantically, keeping headings, article IDs, and source URLs on every chunk.

Mixing models. Vectors from different models, versions, or dimensions are not comparable. Use one approved model per collection, store its name, dimension, normalization setting, and creation time, and re-embed the corpus whenever the model or chunking strategy changes.

Wrong metric. For normalized vectors, dot product and cosine give identical rankings. The real error is using a raw dot product on unnormalized vectors from a model trained for cosine, which lets vector length — often correlated with text length — dominate the score.

Treating scores as probabilities. Similarity scores are not calibrated and their range differs per model; set any “confident match” threshold from labelled data, per model.

Similarity is not correctness. A plausible but outdated article can rank highly; combine retrieval with metadata such as product, region, language, and freshness, and evaluate with labelled telecom queries. Monitor embedding latency, cost, failure rate, and stale-content rate, and keep PII out of embedded text.

3. Vector databases

Conceptual explanation

A vector database stores high-dimensional embeddings and retrieves the nearest neighbours of a query vector, usually through an approximate index such as HNSW. In Qdrant, a point is a record consisting of an ID, a vector, and an optional JSON payload; similarity search finds points in a collection by vector proximity and can be restricted by payload filters. A telecom knowledge base stores the embedding of each article chunk as the vector; the article ID, title, URL, product, region, language, version, and effective date as payload; and queries filter on fields such as product = mobile, region = India, language = en.

Runnable Python code

The example uses Qdrant’s in-memory mode, so it runs without a server. Compared with a naive version, it derives the vector size from the model instead of hard-coding 384, creates payload indexes for filter fields before ingesting, embeds all documents in one batched call, uses deterministic UUIDs so re-ingesting an article overwrites rather than duplicates it, waits for the write to be searchable, applies a score threshold with a fallback, and checks approximate-search recall against exact search.

import uuid

from qdrant_client import QdrantClient, models
from sentence_transformers import SentenceTransformer

MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
COLLECTION = "telecom_support_kb"
MIN_SCORE = 0.30  # placeholder: calibrate on labelled queries per model

model = SentenceTransformer(MODEL_NAME)
dim = model.get_sentence_embedding_dimension()  # 384; never hard-code it

# In-memory mode is exact brute force, for demos only. In production:
# client = QdrantClient(url="https://...", api_key=...)
client = QdrantClient(":memory:")

if not client.collection_exists(COLLECTION):
    client.create_collection(
        collection_name=COLLECTION,
        vectors_config=models.VectorParams(
            size=dim, distance=models.Distance.COSINE))
    # Index filter fields BEFORE ingesting so HNSW builds filter-aware links
    for field in ("product", "region", "language"):
        client.create_payload_index(
            COLLECTION, field_name=field,
            field_schema=models.PayloadSchemaType.KEYWORD)

documents = [
    {"article_id": "KB-001", "chunk": 0, "product": "mobile",
     "title": "Mobile data not working while roaming",
     "text": "If mobile data stops working abroad, enable data roaming, "
             "confirm your international plan, and restart your device."},
    {"article_id": "KB-002", "chunk": 0, "product": "billing",
     "title": "Understanding your monthly bill",
     "text": "Your monthly bill can increase because of prorated charges, "
             "plan changes, roaming usage, premium services, or taxes."},
    {"article_id": "KB-003", "chunk": 0, "product": "broadband",
     "title": "Fiber internet frequent disconnections",
     "text": "Frequent fiber disconnections may be due to router "
             "overheating, loose cables, local outages, or Wi-Fi "
             "interference."},
]

def point_id(article_id: str, chunk: int) -> str:
    """Deterministic ID: re-ingesting a chunk overwrites, never duplicates."""
    return str(uuid.uuid5(uuid.NAMESPACE_URL, f"kb/{article_id}#{chunk}"))

# One batched call instead of one model call per document
vectors = model.encode_document([d["text"] for d in documents],
                                normalize_embeddings=True)
client.upsert(
    collection_name=COLLECTION,
    points=[
        models.PointStruct(
            id=point_id(d["article_id"], d["chunk"]),
            vector=v.tolist(),
            payload={**d, "region": "global", "language": "en",
                     "version": "2026-10-01",
                     "embedding_model": MODEL_NAME})
        for d, v in zip(documents, vectors)
    ],
    wait=True,  # return only once the points are searchable
)

query = "My phone has no data connection while I am abroad."
query_vector = model.encode_query(query, normalize_embeddings=True).tolist()
mobile_only = models.Filter(must=[
    models.FieldCondition(key="product",
                          match=models.MatchValue(value="mobile"))])

result = client.query_points(
    collection_name=COLLECTION, query=query_vector,
    query_filter=mobile_only, limit=3, score_threshold=MIN_SCORE,
    with_payload=True)

if not result.points:
    print("No confident match: fall back to keyword search or an agent.")
for p in result.points:
    print(f"{p.score:.4f}  {p.payload['article_id']}  {p.payload['title']}")

# Recall check: compare approximate (HNSW) results with exact search.
exact = client.query_points(
    collection_name=COLLECTION, query=query_vector,
    query_filter=mobile_only, limit=3,
    search_params=models.SearchParams(exact=True))
ann_ids = {p.id for p in result.points}
exact_ids = {p.id for p in exact.points if p.score >= MIN_SCORE}
print("recall@3:", len(ann_ids & exact_ids) / max(len(exact_ids), 1))

Two behaviours are specific to local mode: payload indexes have no effect and search is always exact, so the recall check is only meaningful against a Qdrant server. With Distance.COSINE, Qdrant normalizes vectors on upload, so client-side normalization is harmless but not required. For large corpora, use upload_points with a batch size rather than one upsert call.

Real-world use case

When a customer says “My phone has no data connection while I am abroad,” the contact-center assistant:

Redacts PII and embeds the query with the same model used for the documents.

Searches the telecom_support_kb collection, filtered to mobile-product articles the customer is entitled to see.

Drops results below the calibrated score threshold and falls back to keyword search or a human agent if nothing remains.

Returns the roaming troubleshooting article to the agent, or passes it to an LLM to generate a grounded answer with a citation.

Measured against a baseline, this kind of retrieval can shorten handle time and reduce outdated answers — provided articles are versioned and stale chunks are deleted when an article changes.

Common mistakes and production notes

Schema and metadata. Define collection names, vector dimension, distance metric, and payload schema up front. Store document ID, source URL, chunk text, language, product, region, version, embedding model, and ingestion timestamp for filtering, tracing, and access control.

Filtering without indexes. On a Qdrant server, filters on unindexed fields still work but get slower as the collection grows; create payload indexes before ingestion, and enable strict mode to reject unindexed filters.

Unvalidated ANN. Approximate search trades recall for speed; benchmark recall against SearchParams(exact=True) and measure latency, precision, and cost at realistic volume.

Exact identifiers. Error codes, plan IDs, and device models are matched poorly by dense vectors alone. Use hybrid search: Qdrant can combine dense vectors with BM25 sparse vectors through prefetch and reciprocal-rank fusion (FusionQuery(fusion=Fusion.RRF)), and recent versions can also score text with a BM25-enabled full-text index.

Lifecycle and security. Implement incremental updates, deletions, and reindexing; monitor zero-result, low-confidence, stale-result, and high-latency queries; and protect vector data with encryption, role-based access, audit logs, and PII redaction — embeddings of confidential text can leak information and belong under the same controls as the text.

Measuring retrieval quality

Every section above says to evaluate on labelled queries; this is the minimal harness for doing it. Write queries in customer wording, record which article should answer each, and track hit rate@k (did a correct article appear in the top k?) and mean reciprocal rank (how high did it appear?). qdrant_search assumes the section 3 code is saved as qdrant_example.py. Run the harness on every model, chunking, or index change, and treat a drop as a failing test.

from collections.abc import Callable

def evaluate_retrieval(
    search: Callable[[str, int], list[str]],
    labelled: dict[str, set[str]],
    k: int = 5,
) -> dict[str, float]:
    """Hit rate@k and MRR@k for a search function returning article IDs."""
    if not labelled:
        raise ValueError("need at least one labelled query")
    hits, rr = 0, 0.0
    for query, relevant in labelled.items():
        ranked = search(query, k)[:k]
        for rank, article_id in enumerate(ranked, start=1):
            if article_id in relevant:
                hits += 1
                rr += 1 / rank
                break
    n = len(labelled)
    return {f"hit_rate@{k}": hits / n, f"mrr@{k}": rr / n}

# Labelled queries written in CUSTOMER wording, not documentation wording
labelled = {
    "No internet on my phone when I travel overseas": {"KB-001"},
    "Why did my bill go up this month?": {"KB-002"},
    "Wifi drops every night": {"KB-003"},
    "How do I activate a digital SIM?": {"KB-004"},
}

def qdrant_search(query: str, k: int) -> list[str]:
    from qdrant_example import COLLECTION, client, model  # section 3
    vector = model.encode_query(query, normalize_embeddings=True).tolist()
    hits = client.query_points(COLLECTION, query=vector, limit=k).points
    return [p.payload["article_id"] for p in hits]

# print(evaluate_retrieval(qdrant_search, labelled, k=5))
# Re-run on every model, chunking, or index change; block regressions in CI.

How the pieces fit together

A complete retrieval pipeline runs in one direction: the customer query is redacted for PII, embedded by the same model that embedded the documents (the model tokenizes internally, with the same tokenizer used at ingestion), searched in the vector database with metadata filters, thresholded, and the ranked chunks go to an agent or a grounded LLM response. Tokenization prepares text, embeddings create semantic representations, and vector databases make those representations searchable at production scale. Together they enable telecom applications such as support assistants, network-incident search, plan recommendation, churn-risk analysis, and internal agent knowledge retrieval.

Takeaway: Use one embedding model and one chunking strategy per collection, measure retrieval on labelled queries, and treat scores, filters, and freshness as part of the answer — not just the nearest vector.

Back to all blogs