How Transformers Work - Explained With a Simple Python Example

Transformers are the architecture behind modern large language models. Their central idea is self-attention: each token builds its representation by weighing other tokens in the sequence. In the decoder-only models behind today’s chatbots, those are only the tokens before it. This article explains the mechanism and builds it in NumPy, from one attention head to a complete decoder block.

The problem

In “The bank refused the loan because the money was not available,” the word bank could mean a financial institution or a riverbank. A model cannot resolve it from the word alone; it needs context such as loan and money. Older recurrent models passed information along one word at a time, which made long-range relationships harder to learn and training hard to parallelize. Self-attention lets tokens interact directly and lets training process all positions at once. The price is that standard attention compares every pair of tokens, so compute and memory grow with the square of the sequence length.

Conceptual explanation

Tokenization: text is split into tokens — usually subwords, not whole words.

Embedding: each token ID becomes a learned vector.

Position information: attention by itself ignores order, so models add it — learned or sinusoidal position embeddings, relative positions, or rotary embeddings (RoPE) applied inside attention, depending on the architecture.

Self-attention: each token produces a query (what it is looking for), a key (what it offers), and a value (what it passes on). Queries are compared with keys; the scores become weights; the output is a weighted sum of values.

Masking: encoder models such as BERT attend in both directions. Decoder-only LLMs such as GPT apply a causal mask, so each token sees itself and earlier tokens only; that is what lets them be trained to predict the next token and generate text left to right.

Multi-head attention: several heads run in parallel on smaller slices of the vector (d_model / heads each), with separate learned projections, and their outputs are concatenated. Heads can learn different patterns, though these are rarely cleanly interpretable.

Residuals, layer normalization, and feed-forward network: each block adds attention output and a position-wise feed-forward output back onto its input, with layer normalization keeping training stable. Dozens of such blocks are stacked.

The core formula, with M the mask (0 where allowed, −∞ where blocked):

Attention(Q, K, V) = softmax(QKᵀ / √dₖ + M) V

Dividing by √dₖ keeps the scores from growing with vector size, which would otherwise push softmax towards one-hot weights and tiny gradients.

Runnable Python example

The earlier version of this example implemented single-head attention correctly but drew the wrong lesson from it. Its weights were random, yet it presented the “attention from bank” row as showing which words the model considers relevant: with untrained weights that row is noise. Its sentence contained “the” twice and used no position information, so both copies produced identical outputs, which is exactly the order-blindness it described but never noticed. It also had no causal mask while presenting attention as how LLMs work, and its explanation of positional encoding appeared twice. The version below makes each of those points measurable, then assembles a full decoder block. Sections 1–3 saved together form attention_demo.py.

1. Attention, masking, and positions

"""Self-attention from scratch in NumPy (random, untrained weights).
Save sections 1-3 together as attention_demo.py."""
import numpy as np

np.set_printoptions(precision=3, suppress=True)
rng = np.random.default_rng(42)
D_MODEL = 16

def softmax(x: np.ndarray, axis: int = -1) -> np.ndarray:
    x = x - x.max(axis=axis, keepdims=True)   # stable: avoids overflow
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

def causal_mask(n: int) -> np.ndarray:
    """True where attention is allowed: each position sees itself and
    earlier positions only (decoder-only LLMs such as GPT)."""
    return np.tril(np.ones((n, n), dtype=bool))

def attention(Q, K, V, mask=None):
    """softmax(Q K^T / sqrt(d_k)) V, with optional boolean mask."""
    d_k = Q.shape[-1]
    scores = Q @ K.swapaxes(-1, -2) / np.sqrt(d_k)
    if mask is not None:
        scores = np.where(mask, scores, -np.inf)  # blocked -> weight 0
    weights = softmax(scores)
    return weights @ V, weights

def sinusoidal_positions(n: int, d: int) -> np.ndarray:
    """Fixed sine/cosine position vectors (original Transformer paper)."""
    pos = np.arange(n)[:, None]
    i = np.arange(0, d, 2)[None, :]
    angles = pos / np.power(10_000, i / d)
    pe = np.zeros((n, d))
    pe[:, 0::2], pe[:, 1::2] = np.sin(angles), np.cos(angles)
    return pe

# Word-level tokens for readability; real models use subword tokenizers
sentence = "the bank refused the loan because the money was not available"
tokens = sentence.split()
vocab = {t: i for i, t in enumerate(dict.fromkeys(tokens))}
embeddings = rng.normal(0, 0.5, (len(vocab), D_MODEL))  # learned in practice
W_q, W_k, W_v = (rng.normal(0, D_MODEL ** -0.5, (D_MODEL, D_MODEL))
                 for _ in range(3))

def embed(words, with_positions=True):
    X = embeddings[[vocab[w] for w in words]]
    return X + sinusoidal_positions(len(words), D_MODEL) if with_positions \
        else X

2. What attention can and cannot see

def self_attend(words, with_positions=True, causal=False):
    X = embed(words, with_positions)
    mask = causal_mask(len(words)) if causal else None
    return attention(X @ W_q, X @ W_k, X @ W_v, mask)

# 1. Without positions, attention cannot tell repeated words apart
out, _ = self_attend(tokens, with_positions=False)
the = [i for i, t in enumerate(tokens) if t == "the"]
print("identical outputs for every 'the', no positions:",
      all(np.allclose(out[the[0]], out[i]) for i in the[1:]))
out, _ = self_attend(tokens, with_positions=True)
print("identical outputs for every 'the', with positions:",
      all(np.allclose(out[the[0]], out[i]) for i in the[1:]))

# 2. ...or word order: reordering the input just reorders the output
a, _ = self_attend(["the", "bank", "refused"], with_positions=False)
b, _ = self_attend(["refused", "bank", "the"], with_positions=False)
print("order-blind without positions:", np.allclose(a, b[::-1]))

# 3. Causal mask: no token may look ahead
_, w = self_attend(tokens, causal=True)
print("weights on future tokens:", w[np.triu_indices(len(tokens), 1)].max())
bank = tokens.index("bank")
print("'bank' attends to:", {t: round(float(x), 3) for t, x in
                             zip(tokens[:bank + 1], w[bank, :bank + 1])})

3. A complete decoder block

This block has four causal attention heads of 4 dimensions each, residual connections, pre-layer-norm, and a GELU feed-forward network — the structure repeated in GPT-style models, minus training, learned norm parameters, dropout, and the final vocabulary projection.

def layer_norm(x, eps=1e-5):
    mu, var = x.mean(-1, keepdims=True), x.var(-1, keepdims=True)
    return (x - mu) / np.sqrt(var + eps)   # learned scale/shift omitted

def gelu(x):
    return 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x**3)))

class DecoderBlock:
    """One pre-norm decoder block: x + MHA(LN(x)), then x + FFN(LN(x))."""

    def __init__(self, d_model=D_MODEL, n_heads=4, d_ff=64, seed=0):
        assert d_model % n_heads == 0, "d_model must split evenly"
        r = np.random.default_rng(seed)
        self.h, self.d_head = n_heads, d_model // n_heads
        init = lambda *s: r.normal(0, s[0] ** -0.5, s)
        self.W_qkv = init(d_model, 3 * d_model)
        self.W_o = init(d_model, d_model)
        self.W_1, self.W_2 = init(d_model, d_ff), init(d_ff, d_model)

    def heads(self, x):  # (n, d_model) -> (h, n, d_head)
        return x.reshape(x.shape[0], self.h, self.d_head).transpose(1, 0, 2)

    def __call__(self, x):
        n = x.shape[0]
        q, k, v = np.split(layer_norm(x) @ self.W_qkv, 3, axis=-1)
        # each head: d_k = d_model / n_heads, its own causal attention
        out, _ = attention(self.heads(q), self.heads(k), self.heads(v),
                           causal_mask(n))
        x = x + out.transpose(1, 0, 2).reshape(n, -1) @ self.W_o
        return x + gelu(layer_norm(x) @ self.W_1) @ self.W_2

block = DecoderBlock()
y = block(embed(tokens))
print("block output shape:", y.shape)

Tests confirmed the implementation against a naive loop version, with and without the mask, and checked that softmax stays stable on very large scores. They also showed that changing a later token never changes the block’s outputs for earlier positions, while unmasked attention does let later tokens influence earlier ones, and that a model width not divisible by the head count is rejected.

A real-world use case

A support assistant for an online academy receives: “I completed the Python module, but my certificate was not generated. What should I do?” A trained transformer builds contextual representations in which “module” relates to the course, “certificate” to completion, and “not generated” to the problem, while “What should I do?” marks a request for help. The surrounding application uses those representations — through a classifier, retrieval, or generation — to route the request or answer it. Attention supplies the context; training supplies the knowledge; the application supplies the decisions.

Common mistakes

Thinking attention understands meaning by itself. It is a weighting mechanism; meaning comes from training, and attention guarantees neither facts nor reasoning.

Reading meaning into random weights. Attention patterns from an untrained model, like the ones above, show the mechanics only.

Ignoring tokenization. Words often split into several subwords, which affects context length, cost, and behaviour.

Forgetting position information. Without it, “dog bites man” and “man bites dog” look the same to attention, as the demo shows.

Assuming LLMs see the whole sentence at every token. Decoder-only models attend backwards only; bidirectional attention belongs to encoder models.

Treating attention weights as explanations. They show what was mixed, not why the model decided; do not present them as definitive interpretability evidence.

Expecting more context to always help. Longer inputs cost quadratically more compute in standard attention; relevant context beats more context.

Production checklist

Define the task (classification, extraction, summarization, retrieval, generation, or conversation) and choose a model size that balances quality, latency, memory, and cost. Inspect how domain terms, product names, and multilingual text tokenize, set maximum input lengths, and truncate safely. Mask sensitive data before it reaches external APIs; validate inputs; guard outputs for unsafe content, hallucinations, and invalid structure. Measure quality with task metrics, evaluation sets, and human review; monitor tokens, latency, errors, cost, and drift; version prompts, models, and retrieval settings; plan timeouts, retries, fallbacks, caching, and human escalation; and test for prompt injection, data leakage, and abuse.

Takeaway: Self-attention lets each token build a context-aware representation by mixing information from the tokens it is allowed to see, with positions telling it where they are. Everything that makes the result useful comes from training, and from the engineering around the model.

Back to all blogs