10 Pandas Mistakes That Slow Down Data Pipelines

Introduction

Most slow Pandas pipelines are not slow because Pandas is slow. They are slow because of a handful of habits that force Python-speed work, waste memory, or move far more data than the job needs. Fixing them routinely turns hour-long jobs into minutes without changing infrastructure.

Pandas is the default tool for tabular data in Python. It is expressive, forgiving and ideal for exploration. That forgiveness is also the problem: code that runs fine on a 10,000-row sample in a notebook often collapses when the same logic meets 50 million rows in a nightly job. Memory spikes, workers get killed, and SLAs slip.

This article covers the ten mistakes that most often cause that collapse. For each one you will see what the mistake looks like, why it is expensive, and what to do instead, with code you can drop into your own pipelines. The fixes are ordered roughly by how often they appear in real codebases, not by difficulty.

Examples target pandas 2.2 or later and have been checked against pandas 3.x. Where behavior differs between 2.x and 3.x, the text says so.

Measure before you optimize

Before changing anything, find out where the time and memory actually go. A few tools cover most needs:

%timeit and %%time in Jupyter for quick comparisons of two approaches.

df.info(memory_usage="deep") and df.memory_usage(deep=True) to see true memory per column, including Python string objects.

cProfile or pyinstrument to find which pipeline step dominates wall-clock time.

memray or tracemalloc to find peak memory, which is usually what crashes jobs. tracemalloc only sees memory allocated through Python's allocator (NumPy reports to it, Arrow buffers do not), so prefer memray for Arrow-backed data.

import logging
import time
from contextlib import contextmanager

log = logging.getLogger("pipeline")

@contextmanager
def timed(step_name):
    start = time.perf_counter()
    try:
        yield
    finally:
        log.info("step=%s seconds=%.2f", step_name, time.perf_counter() - start)

with timed("load"):
    df = load_events(path)
with timed("clean"):
    df = clean_events(df)
with timed("features"):
    df = build_features(df)

A simple step timer like this, logged on every run, will often point straight at one of the ten mistakes below. Sending the timings through logging rather than print puts them in your job logs, where you can track them over time and alert on regressions. Speed-up figures in this article are typical orders of magnitude; your results depend on data shape and hardware, so always confirm with your own measurements.

Mistake 1: Looping over rows with iterrows() and apply()

Row-wise iteration is the single most common cause of slow Pandas code, and replacing it with vectorized operations typically gives a 50–500x speed-up.

What it looks like

# Version A: explicit loop
for i, row in df.iterrows():
    df.at[i, "total"] = row["price"] * row["qty"] * (1 - row["discount"])

# Version B: looks idiomatic, but is still a loop
df["tier"] = df.apply(
    lambda r: "gold" if r["spend"] > 1000 else ("silver" if r["spend"] > 200 else "bronze"),
    axis=1,
)

Why it is slow

Pandas stores each column as a contiguous array of a single type. Vectorized operations run over those arrays in compiled C code. iterrows() throws that away: for every row it builds a new Series object, boxes each value into a Python object, and runs your logic in the Python interpreter. apply(axis=1) does essentially the same thing behind a friendlier interface. On a few million rows, that per-row overhead adds up to minutes.

The fix: think in columns

Arithmetic is the easy case: operate on whole columns.

df["total"] = df["price"] * df["qty"] * (1 - df["discount"])

For conditional logic, use np.where for two branches and np.select for several.

import numpy as np

conditions = [df["spend"] > 1000, df["spend"] > 200]
choices = ["gold", "silver"]
df["tier"] = np.select(conditions, choices, default="bronze")

For binning numeric values into ranges, pd.cut is clearer still:

df["tier"] = pd.cut(df["spend"], bins=[-np.inf, 200, 1000, np.inf],
                    labels=["bronze", "silver", "gold"])

For lookups against a small mapping, use map with a dictionary rather than a lambda; Pandas handles dictionary lookups efficiently. Keys missing from the dictionary become NaN, so check for unmapped values or add an explicit fallback such as .fillna("OTHER").

region = {"IN": "APAC", "SG": "APAC", "US": "AMER", "DE": "EMEA"}
df["region"] = df["country"].map(region)

When you truly need a loop

Some logic is genuinely sequential, such as a running balance that resets on certain events. In that case, extract the NumPy arrays and loop over those, ideally compiled with Numba. If you must iterate in pure Python, itertuples() is several times faster than iterrows() because it yields lightweight named tuples instead of Series.

import numpy as np
from numba import njit

@njit(cache=True)
def running_balance(amounts, resets):
    out = np.empty(amounts.shape[0], dtype=np.float64)
    bal = 0.0
    for i in range(amounts.shape[0]):
        bal = 0.0 if resets[i] else bal + amounts[i]
        out[i] = bal
    return out

df["balance"] = running_balance(
    df["amount"].to_numpy(dtype="float64", na_value=0.0),
    df["reset"].to_numpy(dtype=bool),
)

Convert to NumPy with an explicit dtype. Numba cannot compile over object arrays, which is what to_numpy() returns for nullable or Arrow-backed columns that contain missing values. cache=True stores the compiled function on disk so scheduled jobs do not pay the compilation cost on every run.

Mistake 2: Reading more data than you need

The cheapest data to process is data you never load. Selecting columns and filtering rows at read time often cuts load time and peak memory by more than half.

What it looks like

df = pd.read_csv("events_2026.csv")          # 80 columns, 40M rows
df = df[["user_id", "ts", "event", "amount"]]
df = df[df["event"] == "purchase"]

The pipeline uses four columns and a fraction of the rows, but it pays to parse, type-infer and hold all 80 columns in memory first. Peak memory is set by the full file, not by what you keep.

The fix: push selection into the reader

For CSV, use usecols and declare dtype up front so Pandas does not have to infer types (which can also trigger mixed-type warnings and extra passes). Give the timestamp layout with date_format too. On pandas 2.x and later, engine="pyarrow" uses a multi-threaded parser that is often several times faster on large files (it does not support chunksize). CSV readers cannot skip rows by value, so row filters still run after the load, which is one more reason to prefer Parquet.

df = pd.read_csv(
    "events_2026.csv",
    usecols=["user_id", "ts", "event", "amount"],
    dtype={"user_id": "int64", "event": "category", "amount": "float64"},
    parse_dates=["ts"],
    date_format="%Y-%m-%d %H:%M:%S",
    engine="pyarrow",
)

For Parquet, you can push down both column selection and row filters, so unneeded row groups are skipped entirely on disk.

df = pd.read_parquet(
    "events/",
    columns=["user_id", "ts", "event", "amount"],
    filters=[("event", "==", "purchase"), ("ts", ">=", pd.Timestamp("2026-10-01"))],
)

Filter values must match the column's type. If ts is timezone-aware, comparing it with a naive timestamp raises an error, so pass pd.Timestamp("2026-10-01", tz="UTC") instead.

For databases, filter and aggregate in SQL. A WHERE clause and a GROUP BY run on the database server, close to the data, and send back only the result.

from sqlalchemy import text

query = text("""
    SELECT user_id, DATE(ts) AS day, SUM(amount) AS revenue
    FROM events
    WHERE event = :event AND ts >= :start
    GROUP BY user_id, DATE(ts)
""")
daily = pd.read_sql(query, engine, params={"event": "purchase", "start": "2026-10-01"})

Bind values as parameters instead of writing them into the SQL string. That prevents SQL injection when values come from configuration or user input, and lets the database reuse query plans. Date functions such as DATE() differ between database dialects, so adjust them to your engine.

Chunking for files that do not fit

If a file is too large to load even after column selection, process it in chunks and aggregate as you go. Only the aggregates stay in memory.

totals = []
for chunk in pd.read_csv("events_2026.csv", usecols=["user_id", "amount"],
                         chunksize=2_000_000):
    totals.append(chunk.groupby("user_id")["amount"].sum())

revenue = pd.concat(totals).groupby(level=0).sum()

Chunking works well for associative aggregations such as sums, counts, minimums and maximums. A mean can be rebuilt from chunked sums and counts. For medians, distinct counts or joins across the whole dataset, it is usually time to consider DuckDB or Polars (see Mistake 10).

Mistake 3: Accepting default dtypes

Pandas picks safe, generous types by default: int64, float64 and, before pandas 3.0, Python object for strings. Choosing tighter types commonly shrinks a DataFrame by 50–90%, and smaller data is faster to sort, group, join and serialize.

What it looks like

df = pd.read_csv("orders.csv")
df.info(memory_usage="deep")
# country     object    <- 2M copies of ~20 distinct strings
# status      object    <- "pending", "shipped", "delivered"
# qty         int64     <- values 1 to 50
# price       float64
# memory usage: 1.4 GB

Why it matters

An object column stores a pointer to a separate Python string for every row. Each string carries roughly 50 bytes of overhead before its characters, and operations on the column run at Python speed. Numeric defaults waste space too: a quantity that never exceeds 50 fits in one byte, not eight.

Pandas 3.0 changed the string default. Text columns are now inferred as the dedicated str dtype, which uses PyArrow storage when PyArrow is installed and falls back to object storage when it is not (missing values are NaN either way). On 3.x with PyArrow, much of the object-string overhead goes away without any change to your code. On 2.x, or without PyArrow, the advice below applies in full. On every version, categoricals remain the better choice for low-cardinality columns.

The fix: choose types deliberately

Categoricals for low-cardinality strings. A category column stores each distinct value once and keeps small integer codes per row. Group-bys and comparisons on categories are also much faster.

for col in ["country", "status", "channel"]:
    df[col] = df[col].astype("category")

A practical rule: use category when the number of distinct values is well below the row count, say under 5–10% of it.

Arrow-backed strings for high-cardinality text. string[pyarrow], available since pandas 1.3 and much improved in 2.x, stores text in compact contiguous buffers and runs string operations in C++. It uses the same storage as pandas 3.0's default str dtype. It is a strong default for IDs, emails and free text.

df["email"] = df["email"].astype("string[pyarrow]")

# Or for the whole read (every column becomes an Arrow type):
df = pd.read_csv("orders.csv", dtype_backend="pyarrow", engine="pyarrow")

Arrow-backed columns behave slightly differently from NumPy-backed ones in some operations (missing values are pd.NA, and a few methods return different types), so run your tests before switching a whole pipeline to dtype_backend="pyarrow".

Downcast numerics. Use the smallest integer and float types that safely hold your values.

df["qty"] = pd.to_numeric(df["qty"], downcast="integer")    # -> int8
df["price"] = df["price"].astype("float32")

Be careful with float32 for money or large identifiers: it holds about seven significant digits. Keep float64 (or integer cents) for financial totals, and never store IDs as floats.

Nullable integers. An integer column with a single missing value silently becomes float64 in classic Pandas. Use Int64 (capital I) or Arrow integers to keep integers integers. The same applies to a schema: a column declared as "int16" makes the read fail if it ever contains a missing value, so declare it as "Int16" when missing values are legitimate.

Make it a contract

The most robust approach is to define a schema once and apply it at every read, so types never drift between runs.

ORDERS_SCHEMA = {
    "order_id": "int64",
    "customer_id": "int64",
    "country": "category",
    "status": "category",
    "qty": "int16",
    "price": "float64",
    "email": "string[pyarrow]",
}
DATE_COLS = ["created_at"]

df = pd.read_csv(
    "orders.csv",
    usecols=[*ORDERS_SCHEMA, *DATE_COLS],
    dtype=ORDERS_SCHEMA,
    parse_dates=DATE_COLS,
    date_format="%Y-%m-%d %H:%M:%S",
)

Listing the columns in usecols makes the read fail loudly if an expected column disappears upstream, instead of letting the problem surface three stages later. For stronger guarantees such as value ranges, uniqueness and nullability, validate the frame with a library such as Pandera before passing it to the next stage.

Mistake 4: Growing a DataFrame inside a loop

Appending to a DataFrame one piece at a time turns a linear job into a quadratic one. Collect pieces in a list and concatenate once at the end.

What it looks like

result = pd.DataFrame()
for path in daily_files:                     # 365 files
    day = pd.read_parquet(path)
    result = pd.concat([result, day])        # copies everything so far

Or, row by row:

rows = pd.DataFrame(columns=["id", "score"])
for rec in api_records:
    rows.loc[len(rows)] = [rec["id"], rec["score"]]

Why it is slow

A DataFrame is not a list. Its columns are fixed-size arrays, so every concat or loc enlargement allocates new arrays and copies all existing data into them. On iteration 300 you copy 299 days of data to add one more. Total work grows with the square of the number of pieces, and peak memory briefly doubles on each step. (DataFrame.append was removed in Pandas 2.0 partly to discourage this pattern.)

The fix: accumulate, then build once

frames = [pd.read_parquet(path) for path in daily_files]
result = pd.concat(frames, ignore_index=True)

For records, build plain Python dictionaries or lists and create the DataFrame in a single call. Python lists are designed for cheap appends.

records = [{"id": r["id"], "score": r["score"]} for r in api_records]
rows = pd.DataFrame.from_records(records)

If every piece shares a schema and lives in one directory, let the reader do the stitching: pd.read_parquet("daily/") reads a whole folder of Parquet files as one dataset.

A related trap: adding columns one by one

Inserting hundreds of new columns in a loop (df[f"lag_{k}"] = ...) fragments the DataFrame's internal blocks and triggers a PerformanceWarning. Build the new columns in a dictionary and attach them in one go.

lags = {f"lag_{k}": df["value"].shift(k) for k in range(1, 101)}
df = pd.concat([df, pd.DataFrame(lags, index=df.index)], axis=1)

Mistake 5: Using CSV between pipeline stages

CSV is a fine exchange format for humans and external partners, but a poor format for data passing between your own pipeline steps. Switching intermediates to Parquet typically makes reads and writes several times faster and files several times smaller, and it preserves types.

What it looks like

# stage_1.py
cleaned.to_csv("s3://bucket/stage1/cleaned.csv", index=False)

# stage_2.py
cleaned = pd.read_csv("s3://bucket/stage1/cleaned.csv")
cleaned["created_at"] = pd.to_datetime(cleaned["created_at"])   # types lost
cleaned["country"] = cleaned["country"].astype("category")       # again

Why it is slow

CSV is text. Every write converts each number and date into characters, and every read parses them back, which is CPU-heavy. CSV carries no schema, so the reader must infer types, and all the careful dtype work from Mistake 3 is lost between stages. It is row-oriented, so you cannot read a few columns without scanning the whole file. And it compresses less effectively than columnar formats.

The fix: Parquet for intermediates

Parquet is a compressed, columnar, typed binary format. Categories, timestamps with time zones, nullable integers and Arrow strings all round-trip intact, and readers can load only the columns they need.

# stage_1.py
cleaned.to_parquet("s3://bucket/stage1/cleaned.parquet", index=False,
                   compression="zstd")

# stage_2.py: types come back exactly as written
cleaned = pd.read_parquet("s3://bucket/stage1/cleaned.parquet",
                          columns=["customer_id", "created_at", "amount"])

For large datasets, write partitioned Parquet so downstream jobs can skip whole directories:

events.to_parquet(
    "s3://bucket/events/",
    partition_cols=["event_date"],
    existing_data_behavior="delete_matching",
)

existing_data_behavior="delete_matching" (passed through to PyArrow) replaces only the partitions being written. Without it, rerunning a day adds a second set of files next to the first and silently duplicates data. Reading and writing s3:// paths requires s3fs (or another fsspec filesystem) to be installed.

Choosing a format

Keep CSV at the edges of the system, where people or external tools need it, and use Parquet everywhere in between.

Mistake 6: Slow group-by patterns

Pandas' built-in aggregations (sum, mean, count, nunique, max and friends) run in optimized Cython. Passing a Python lambda to groupby().apply() or agg() drops back to a Python call per group, which can be 10–100x slower when there are many groups.

What it looks like

# Python function called once per customer (millions of calls)
stats = df.groupby("customer_id").apply(
    lambda g: pd.Series({
        "orders": len(g),
        "revenue": g["amount"].sum(),
        "avg_basket": g["amount"].mean(),
    })
)

# Lambda where a built-in exists
df.groupby("store")["amount"].agg(lambda s: s.sum())

The fix: named aggregations with built-ins

Named aggregation lets you express many outputs in one pass, each using a fast built-in.

stats = df.groupby("customer_id").agg(
    orders=("order_id", "count"),
    revenue=("amount", "sum"),
    avg_basket=("amount", "mean"),
    last_order=("created_at", "max"),
)

If you need a derived metric, compute the built-in parts first, then combine the resulting columns with vectorized arithmetic.

stats["revenue_per_order"] = stats["revenue"] / stats["orders"]

For conditional aggregates, such as revenue only from returned items, create a helper column first instead of filtering inside a lambda, and include it in the same named aggregation so the data is grouped only once.

stats = (
    df.assign(returned_amount=df["amount"].where(df["is_returned"], 0))
      .groupby("customer_id", sort=False)
      .agg(
          orders=("order_id", "count"),
          revenue=("amount", "sum"),
          returned_revenue=("returned_amount", "sum"),
      )
)

Use transform for group-level features on each row

A common pattern is to aggregate, then merge the result back to the original rows. transform does both in one step and keeps the original index.

# Slower: aggregate, then join back
means = df.groupby("store")["amount"].mean().rename("store_mean")
df = df.merge(means, on="store")

# Faster and simpler
df["store_mean"] = df.groupby("store")["amount"].transform("mean")

Pass the aggregation name as a string ("mean"), not a lambda, so Pandas uses the fast path.

Small settings that add up

observed=True with categorical keys. Pandas 3.0 made this the default. On 2.x, pass it explicitly: without it you get a row for every combination of categories, even ones that never occur, which can explode output size (pandas 2.1 and later also emit a FutureWarning).

sort=False when you do not need sorted group keys; it skips a sort of the result.

Group on fewer, tighter keys. Grouping on category or integer keys is much faster than grouping on object strings.

Mistake 7: Inefficient merges and exploding joins

Joins are where pipelines most often blow up in both time and memory, usually because of duplicate keys, mismatched key types, or joining before filtering.

What it looks like

# Join everything, then filter
full = orders.merge(customers, on="customer_id").merge(products, on="sku")
result = full[full["country"] == "IN"][["order_id", "sku", "amount", "segment"]]

Three hidden problems are common here. The join carries every column of three tables before most are discarded. If customers has accidental duplicate customer_id rows, every matching order is duplicated, and a many-to-many match can multiply rows silently. And if customer_id is int64 in one table and object (string) in the other, the merge raises a ValueError, often after the expensive steps before it have already run.

The fix: filter, trim, validate, then join

Filter and select columns first. Reduce both sides to only the rows and columns the result needs.

in_customers = customers.loc[customers["country"] == "IN", ["customer_id", "segment"]]
orders_small = orders[["order_id", "customer_id", "sku", "amount"]]

result = orders_small.merge(in_customers, on="customer_id", how="inner")

Validate cardinality. The validate argument raises an error if the relationship is not what you expect, catching duplicate-key explosions before they reach production.

result = orders_small.merge(in_customers, on="customer_id",
                            how="inner", validate="many_to_one")

Align key dtypes. Make sure join keys share the same type on both sides, ideally integer or a categorical with identical categories.

left, right = orders["customer_id"].dtype, customers["customer_id"].dtype
if left != right:
    raise TypeError(f"customer_id dtype mismatch: {left} vs {right}")

Use an explicit check rather than assert, because Python removes assert statements when it runs with the -O flag.

Use indicator=True when debugging. It adds a _merge column showing whether each row matched on the left, right or both, which quickly reveals key mismatches.

Lookups: map instead of merge

When you need a single column from a small lookup table, map with a Series indexed by the key is lighter than a full merge.

segment_by_id = customers.set_index("customer_id")["segment"]
orders["segment"] = orders["customer_id"].map(segment_by_id)

map requires the lookup keys to be unique and raises InvalidIndexError otherwise, which doubles as a cheap duplicate check. Keys with no match become NaN.

Joining on a sorted index

For repeated joins on the same key, setting that key as a sorted index and using join can be faster than repeated merge calls on columns. For time-based matching, such as attaching the latest price at or before each trade, use pd.merge_asof rather than a cross join followed by filtering. Both inputs must be sorted on the key, and the by= argument restricts matches to the same instrument or customer.

Mistake 8: Unnecessary copies and memory churn

Many Pandas operations return a new DataFrame. Long scripts that keep every intermediate alive, or that copy defensively at every step, can hold several full copies of the data at once, and peak memory is what kills jobs.

What it looks like

raw = pd.read_parquet("events.parquet")      # 6 GB
df1 = raw.copy()
df2 = df1.dropna(subset=["user_id"])
df3 = df2[df2["amount"] > 0]
df4 = df3.reset_index(drop=True)
df5 = df4.rename(columns=str.lower)
# raw, df1 ... df5 may all still be in memory

Another variant is chained assignment. On pandas 2.x it raises the familiar SettingWithCopyWarning, which often leads people to sprinkle .copy() everywhere to silence it. On pandas 3.0 it never modifies the original and emits a ChainedAssignmentError warning instead:

df[df["country"] == "IN"]["discount"] = 0.1  # 2.x: may modify a temporary copy; 3.x: never updates df

The fix: chain, reuse names, and assign with loc

Write pipelines as method chains or reassign the same name. Intermediates without a name can be freed as soon as the next step finishes.

df = (
    pd.read_parquet("events.parquet", columns=COLUMNS)
      .dropna(subset=["user_id"])
      .query("amount > 0")
      .rename(columns=str.lower)
      .reset_index(drop=True)
)

Method chains also make each step easy to read and test. pipe lets you insert your own functions into the chain.

df = df.pipe(clean_events).pipe(add_time_features).pipe(add_customer_features)

Assign with .loc in one step. It modifies the original DataFrame unambiguously, with no warning and no defensive copy.

df.loc[df["country"] == "IN", "discount"] = 0.1

Turn on copy-on-write. Copy-on-write makes every derived object behave like a copy while sharing memory until something is actually modified. It removes the ambiguity behind chained-assignment bugs and avoids many real copies. It is always on in pandas 3.0, where the option below is deprecated and has no effect. On pandas 2.x, enable it explicitly to get the same behavior and prepare for the upgrade. Under copy-on-write, defensive .copy() calls are unnecessary and can be removed.

pd.options.mode.copy_on_write = True  # pandas 2.x only

Release big objects explicitly. In long-running scripts or notebooks, del large intermediates you no longer need.

Do not trust inplace=True

Many people use inplace=True hoping to save memory. In most cases Pandas still creates a new object internally and then swaps it in, so there is little or no saving, and it blocks method chaining. PDEP-8 formalized this. pandas 3.1 deprecates inplace for methods that cannot truly work in place, such as dropna, rename, sort_values and reset_index, and plans to remove it in pandas 4.0. It stays for the few methods that can, such as fillna and replace. Prefer reassignment: df = df.dropna().

Mistake 9: Parsing strings and dates the slow way

Text cleaning and timestamp parsing are often the hidden hot spots in ingestion code. Using vectorized accessors and explicit formats instead of per-element Python functions routinely gives 10x or larger gains.

Dates: always tell Pandas the format

from datetime import datetime

# Slow: per-element parsing with a Python function
df["ts"] = df["ts_str"].apply(lambda s: datetime.strptime(s, "%Y-%m-%d %H:%M:%S"))

# Works, but the format is guessed from the first value
df["ts"] = pd.to_datetime(df["ts_str"])

# Fast and explicit: vectorized with a declared format
df["ts"] = pd.to_datetime(df["ts_str"], format="%Y-%m-%d %H:%M:%S")

Since pandas 2.0, to_datetime infers one format from the first non-null value and applies it to every row. Inference is fast, and the row-by-row guessing of pandas 1.x is gone. The risk is in the guess itself. If the first value is 03/04/2026, pandas assumes month-first, and a later 13/04/2026 then fails to parse, or quietly becomes NaT under errors="coerce". An explicit format removes the guess and documents the contract. If the data has a few bad values, errors="coerce" turns them into NaT so you can count and handle them instead of crashing. Log or alert on that count rather than letting rows disappear silently.

For ISO 8601 strings, format="ISO8601" is both fast and robust. For epoch timestamps, parse the integers directly: pd.to_datetime(df["epoch_ms"], unit="ms", utc=True).

Pandas 3.0 also changed the default resolution. Strings now parse to microseconds (datetime64[us]) rather than nanoseconds, so astype("int64") on a parsed column returns values 1,000 times smaller than before. If downstream code expects nanosecond epochs, call .dt.as_unit("ns") first.

Date parts: use the .dt accessor

# Slow
df["hour"] = df["ts"].apply(lambda t: t.hour)

# Fast
df["hour"] = df["ts"].dt.hour
df["weekday"] = df["ts"].dt.dayofweek
df["month_start"] = df["ts"].dt.to_period("M").dt.start_time

to_period drops time-zone information (with a warning). For tz-aware timestamps, convert to the reporting time zone and remove the zone first, for example df["ts"].dt.tz_convert("Asia/Kolkata").dt.tz_localize(None).

Strings: use the .str accessor and regex once

# Slow: Python per element
df["domain"] = df["email"].apply(lambda e: e.split("@")[1].lower())

# Fast: vectorized
df["domain"] = df["email"].str.split("@", n=1).str[1].str.lower()

# Even better for patterns: one regex extraction
df[["user", "domain"]] = df["email"].str.extract(r"^([^@]+)@(.+)$")

Combine .str methods with the Arrow-backed string[pyarrow] dtype from Mistake 3, and most string operations run in compiled code end to end. On pandas 3.0 with PyArrow installed, text columns already use this storage by default.

Clean categories before you expand them

If a column has a few hundred distinct messy values (for example, inconsistent country spellings), clean the distinct values once and map them back, rather than cleaning millions of rows.

cats = df["country_raw"].astype("category")
cleaned = cats.cat.categories.str.strip().str.upper().str.replace(".", "", regex=False)
mapping = dict(zip(cats.cat.categories, cleaned))
df["country"] = cats.map(mapping).astype("category")

The cleaning runs only on the few hundred distinct categories, and map on a categorical Series applies the mapping per category rather than per row. Unlike cat.rename_categories, which raises an error if two old names clean to the same new name, this also works when cleaning merges variants such as "U.S." and "US". The principle is the same: work on the small set of distinct values, not on every row.

Mistake 10: Forcing Pandas past its limits

Pandas is eager, mostly single-threaded and in-memory. Once data approaches the RAM of one machine, or a step is dominated by heavy joins and aggregations, the right fix is often a different engine for that step, not more tuning.

Signs you have hit the ceiling

Peak memory is regularly more than about half of the machine's RAM, even after fixing dtypes and column selection.

One CPU core is pegged at 100% while the others sit idle.

The job is dominated by large joins, distinct counts or window functions across tens or hundreds of millions of rows.

You are chunking data by hand and stitching results together with increasingly fragile code.

The team is reaching for bigger instances every quarter to keep up.

Better tools for specific jobs

These tools interoperate through Apache Arrow, so you do not need to rewrite the whole pipeline. A common, pragmatic pattern is to let DuckDB or Polars do the heavy lifting and hand a much smaller result to Pandas for the final steps.

import duckdb

# DuckDB scans Parquet in parallel and returns only the aggregate
con = duckdb.connect(config={"threads": 8, "memory_limit": "16GB"})
daily = con.execute(
    """
    SELECT customer_id, DATE_TRUNC('day', ts) AS day,
           COUNT(*) AS orders, SUM(amount) AS revenue
    FROM read_parquet('events/*.parquet')
    WHERE ts >= $start
    GROUP BY ALL
    """,
    {"start": pd.Timestamp("2026-01-01")},
).df()  # a Pandas DataFrame, now small enough to be comfortable

An explicit connection with threads and memory_limit keeps DuckDB inside the container's resources. When it reaches the limit it spills to disk instead of being killed, and the bound parameter keeps values out of the SQL string.

The same step in Polars, using a lazy query so filters and column selection are pushed down into the scan:

from datetime import datetime

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("ts") >= datetime(2026, 1, 1))
    .group_by("customer_id", pl.col("ts").dt.truncate("1d").alias("day"))
    .agg(pl.len().alias("orders"), pl.col("amount").sum().alias("revenue"))
    .collect(engine="streaming")
    .to_pandas()
)

On Polars 1.x, collect(engine="streaming") runs the query in batches, so it can handle inputs larger than memory. to_pandas() requires PyArrow.

The goal is not to abandon Pandas. It is to use Pandas where it shines, on data that fits comfortably in memory and needs expressive manipulation, and to delegate the rest.

Quick-reference checklist and conclusion

Use this table as a code-review checklist for any Pandas pipeline headed to production.

A sensible order of attack

If you inherit a slow pipeline, work through it in this order. It usually delivers the most improvement for the least risk.

Measure. Add step timers and check peak memory so you know which stage to fix first.

Load less. Apply column selection, row filters and an explicit schema at read time (Mistakes 2 and 3).

Remove Python loops. Replace iterrows, apply(axis=1) and per-element lambdas (Mistakes 1, 6 and 9).

Fix the plumbing. Switch intermediates to Parquet and stop growing DataFrames in loops (Mistakes 4 and 5).

Tame joins and memory. Validate merges, chain operations and enable copy-on-write, or upgrade to pandas 3.0 where it is always on (Mistakes 7 and 8).

Change engines only if needed. If the measurements still show a ceiling, move the heaviest step to DuckDB or Polars (Mistake 10).

Conclusion

None of these fixes requires exotic knowledge. They come down to three habits: work on whole columns instead of individual rows, move and store only the data and precision you need, and know when a step has outgrown a single-threaded, in-memory tool. Teams that bake these habits into code reviews and shared pipeline templates rarely need heroic optimization later. Their Pandas code simply stays fast as the data grows.

Back to all blogs