Pandas vs. NumPy: When to Use Each in an ML Workflow

Introduction

Use Pandas to understand and shape your data; use NumPy to compute on it. That one sentence covers most decisions in a machine learning workflow, but the details are where projects win or lose time, memory and correctness.

A typical script reads a file into a DataFrame, cleans and engineers features, then hands a NumPy array to scikit-learn, XGBoost or PyTorch. Because the two libraries interoperate so smoothly, few practitioners ask which one should own each step, and the result is code that works on a laptop sample but crawls, or silently misaligns rows, on production data. This article gives you a mental model of each library, maps them to every stage of an ML workflow, covers safe conversion and common anti-patterns, and closes with a decision framework. All code targets pandas 3.0, NumPy 2.x and scikit-learn 1.8 or later.

How the two libraries relate

Pandas is built on top of NumPy, so the question is rarely "either/or"; it is which layer of abstraction fits the task. NumPy provides the ndarray: a fixed-size, homogeneous, n-dimensional block of memory with a single dtype, operated on by compiled C loops. Pandas adds labels (a row index and column names) and heterogeneity (each column has its own dtype). Columns are backed by NumPy arrays or, since pandas 2.0, Apache Arrow arrays. In pandas 3.0 text columns default to a dedicated str dtype, Arrow-backed when PyArrow is installed.

import numpy as np
import pandas as pd

arr = np.array([[5.1, 3.5], [4.9, 3.0]])           # numbers, positions only
df = pd.DataFrame(arr, columns=["sepal_len", "sepal_wid"],
                  index=["flower_a", "flower_b"])
df["species"] = ["setosa", "setosa"]               # mixed types now allowed
df.to_numpy()                                      # dtype=object: speed lost

Converting a mixed-type DataFrame produces an object array, which discards most of NumPy's speed. Think of NumPy as a math engine ("compute this on these numbers") and Pandas as a data workbench ("which customer, on which date, with which missing values"). Meaning matters most early in an ML pipeline; raw speed matters most later. Where you hand off between them is a key design decision.

NumPy's core strengths

NumPy wins when data is purely numeric, the shape is regular, and the job is computation rather than bookkeeping.

Vectorization and broadcasting. Whole-array operations run in C, typically 10–100× faster than an equivalent Python loop. Broadcasting stretches smaller arrays across larger ones without materialising copies; it underpins normalisation, distance computations, attention masks and loss functions. Use the Generator API rather than the legacy np.random.rand, and guard against zero-variance columns.

rng = np.random.default_rng(seed=42)
X = rng.random((1_000_000, 20), dtype=np.float32)

mu, sd = X.mean(axis=0), X.std(axis=0)
X_std = (X - mu) / np.where(sd == 0, 1, sd)   # safe for constant columns

# Pairwise squared Euclidean distances (500 queries x 10,000 points)
Q, P = rng.random((500, 64)), rng.random((10_000, 64))
d2 = (Q**2).sum(1)[:, None] + (P**2).sum(1)[None, :] - 2 * Q @ P.T
d2 = np.maximum(d2, 0)          # clip tiny negatives caused by float rounding

N-dimensional data. DataFrames are 2-D; arrays handle images (batch, height, width, channels), time-series windows and embeddings. PyTorch, TensorFlow and JAX expose NumPy-like APIs (JAX most closely), and the Python Array API standard is narrowing the remaining differences.

Linear algebra and random numbers. np.linalg (SVD, eigen-decomposition, least squares) and np.random.Generator are the building blocks of classical ML. Prefer np.linalg.solve or lstsq over explicit inverses, which are slower and numerically less stable.

idx = rng.permutation(len(X_std))                     # reproducible shuffle
sample = X_std[idx[:5000]]                            # random, not first, 5,000 rows
U, S, Vt = np.linalg.svd(sample, full_matrices=False) # PCA via SVD

Memory control. A million float32 values occupy about 4 MB. You choose the dtype, slices are views that share memory, and np.load(..., mmap_mode="r") memory-maps arrays larger than RAM. float32 halves memory versus float64 with negligible accuracy loss for most models, though some scikit-learn estimators upcast to float64 internally. NumPy is the ecosystem's common currency: SciPy expects arrays, scikit-learn converts to them, and torch.from_numpy shares memory with no copy.

Pandas' core strengths

Pandas wins when data is tabular, messy, mixed-type, or must stay attached to IDs, timestamps and categories.

Automatic alignment. Operations between Series align on the index, not position, eliminating off-by-one bugs after a filter or sort. The same division on raw arrays would silently pair the wrong users.

revenue = pd.Series({"u1": 120, "u2": 80, "u3": 45})
visits  = pd.Series({"u3": 9, "u1": 12, "u2": 4})    # different order
revenue / visits                                      # aligned by user id

Missing data. isna, fillna, dropna, interpolate and nullable dtypes (Int64, boolean) treat missingness as first-class. NumPy only has np.nan for floats and NaT for datetime64/timedelta64, so an integer array with a gap must become float.

I/O, relational operations and time series. read_csv, read_parquet and read_sql handle parsing, type inference and compression. groupby, merge, pivot_table and melt cover most relational feature engineering, and a datetime column unlocks resampling (resample("1h")), time-based rolling windows, lags and time zones. The category dtype stores repeated strings as integer codes, and describe, value_counts and corr make Pandas the natural EDA environment.

features = (
    tx.groupby("customer_id", as_index=False)
      .agg(n_orders=("order_id", "nunique"),
           avg_basket=("amount", "mean"),
           last_order=("ts", "max"))
      .merge(profiles, on="customer_id", how="left", validate="one_to_one")
)
validate="one_to_one" makes the merge raise if profiles has duplicate keys, instead of silently multiplying rows.

Head-to-head comparison

Stage by stage: which library owns each step

In a typical tabular project Pandas owns ingestion, EDA, cleaning and most feature engineering; NumPy (or tensors) owns training and the online inference path; evaluation uses both. Keep one deliberate handoff in between.

1. Ingestion: Pandas

Control types at read time, the cheapest place to save memory. Parquet preserves its schema, so read only the columns and rows you need; for CSV, declare dtypes up front.

df = pd.read_parquet(
    "events.parquet",
    columns=["user_id", "ts", "country", "amount", "churned"],
    filters=[("ts", ">=", pd.Timestamp("2024-01-01", tz="UTC"))],  # pushdown
)
df = df.astype({"country": "category", "amount": "float32"})

# CSV equivalent: types set during parsing, no second pass
# pd.read_csv("events.csv", dtype={"country": "category", "amount": "float32"},
#             parse_dates=["ts"], usecols=[...])

Use NumPy for ingestion only when data is already numeric and array-shaped: .npy feature matrices, image tensors or exported embeddings.

2–3. EDA, cleaning and validation: Pandas

EDA answers questions (null counts, class balance, outliers, leakage) in one line each. Cleaning (deduplication, type fixes, filtering invalid rows) is label-aware and keeps each row traceable for auditing.

df.info()
df["churned"].value_counts(normalize=True)
df.groupby("country", observed=True)["churned"].mean().sort_values()

df = df.drop_duplicates(subset=["user_id", "ts"])
df = df[df["amount"] >= 0]                          # rule-based: safe pre-split
df = df[df["ts"] >= pd.Timestamp("2024-01-01", tz="UTC")]  # tz-aware compare

Only stateless, rule-based cleaning belongs here. Anything learned from the data, such as a 99th-percentile clipping threshold or a median for imputation, must be fitted on the training split only, ideally inside a scikit-learn Pipeline. Computing quantile(0.99) on the full dataset before splitting is data leakage.

4. Feature engineering: Pandas first, NumPy for the math

Relational and temporal features (per-entity aggregates, lags, rolling statistics, date parts) are natural in Pandas. NumPy ufuncs work directly on columns and preserve the index. Note that rolling(7) means 7 rows, not 7 days; for a calendar window use a time offset.

df = df.sort_values(["user_id", "ts"])
# 7-day time window per user. Positional assignment is safe because
# groupby(sort=True) emits rows in the same (user_id, ts) order as df.
df["amount_7d_mean"] = (df.set_index("ts")
                          .groupby("user_id", sort=True)["amount"]
                          .rolling("7D").mean()
                          .to_numpy())
df["log_amount"] = np.log1p(df["amount"])            # NumPy ufunc on a Pandas column
df["hour"] = df["ts"].dt.hour
df["is_weekend"] = df["ts"].dt.dayofweek >= 5

Switch to NumPy for custom numeric logic over large arrays (pairwise distances, kernels, signal processing). Use np.where and np.select instead of apply for conditional logic.

5. Modeling and training: the handoff

Two production rules apply at the split. First, split rows, not anonymous arrays, so predictions can be joined back. Second, when one entity contributes many rows (here, many events per user), split by group or by time; a random row split puts the same user in train and test and inflates scores.

from sklearn.model_selection import GroupShuffleSplit

feature_cols = ["amount", "log_amount", "amount_7d_mean",
                "hour", "is_weekend", "country"]
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_pos, test_pos = next(gss.split(df, groups=df["user_id"]))
df_train, df_test = df.iloc[train_pos], df.iloc[test_pos]    # index preserved

For forecasting-style problems use a time cut-off or TimeSeriesSplit instead. When you do need raw arrays (custom training loops, PyTorch), convert once with explicit dtypes: df_train[num_cols].to_numpy(dtype=np.float32), then torch.from_numpy(...). Save the column list with the model; it is the only record of which array column is which.

6. Evaluation: NumPy for metrics, Pandas for analysis

Compute metrics on arrays, then bring predictions back into a DataFrame to slice errors by segment. Guard segments that contain a single class, which make AUC undefined.

from sklearn.metrics import roc_auc_score

proba = model.predict_proba(df_test[feature_cols])[:, 1]
print("AUC:", roc_auc_score(df_test["churned"], proba))

def safe_auc(g):
    return roc_auc_score(g["y"], g["p"]) if g["y"].nunique() == 2 else np.nan

results = df_test[["user_id", "country"]].assign(y=df_test["churned"], p=proba)
results.groupby("country", observed=True)[["y", "p"]].apply(safe_auc)

7. Inference and serving: one feature contract

Building a DataFrame per request costs tens to hundreds of microseconds, which matters only on very tight latency budgets. The larger production risk is training/serving skew: hand-assembling an array online duplicates feature logic and silently breaks when the column order changes. Prefer serving the same fitted Pipeline with a one-row DataFrame built from the saved column list. Drop to a raw float32 array only for a model trained on arrays, with order validated against the saved list, or export to ONNX for low-latency runtimes.

# req_features: dict produced by the same feature module used in training
row = pd.DataFrame([req_features], columns=feature_cols)
score = model.predict_proba(row)[0, 1]

The handoff: converting between them safely

Convert once, at a deliberate boundary, with explicit dtypes and a saved column order. Prefer .to_numpy() over .values: it is explicit, accepts dtype, na_value and copy, and handles extension types consistently.

X = df[num_cols].to_numpy(dtype=np.float32, na_value=np.nan)

Dtype. Every column must be numeric; one stray string column turns the array into object.

Column order. Select by an explicit list, never by df.drop(columns=...) alone.

Missing values. XGBoost, LightGBM, HistGradientBoosting* and (since scikit-learn 1.4) random forests accept NaN; linear models, SVMs and neural networks do not.

Views and read-only arrays. Under pandas 3.0 copy-on-write, .to_numpy() on a single-dtype frame can return a read-only view: in-place edits raise ValueError, and torch.from_numpy warns. Pass copy=True if you will mutate the array or hand it to a framework that writes to it.

When predictions or embeddings come back as arrays, re-attach labels immediately: pd.Series(pred, index=df_test.index, name="pred").

A Pipeline as the contract

The cleanest handoff passes DataFrames into a scikit-learn Pipeline that selects columns by name, learns preprocessing only on training folds and converts to arrays internally. Note that OneHotEncoder must use sparse_output=False when pandas output is enabled; the default sparse output raises a ValueError.

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

num_cols = ["amount", "log_amount", "amount_7d_mean", "hour"]
pre = ColumnTransformer(
    [("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                       ("scale", StandardScaler())]), num_cols),
     ("cat", OneHotEncoder(handle_unknown="infrequent_if_exist", min_frequency=20,
                           sparse_output=False), ["country"]),
     ("bool", "passthrough", ["is_weekend"])],
    remainder="drop", verbose_feature_names_out=False,
).set_output(transform="pandas")

model = Pipeline([("pre", pre),
                  ("clf", HistGradientBoostingClassifier(random_state=42))])
model.fit(df_train[feature_cols], df_train["churned"])

Persist the fitted Pipeline (with joblib or skops) together with pinned library versions; the object itself is the contract between training and serving.

Performance considerations

The biggest gap is not Pandas versus NumPy; it is vectorized code versus Python-level loops. For a column transform on a million rows, from slowest to fastest (profile your own case):

df.iterrows() loop: seconds to tens of seconds.
df.apply(func, axis=1): still a Python call per row.

Series.map / Series.apply with a Python function: per element, somewhat faster.

Vectorized Pandas column operations: milliseconds, typically 100× faster than steps 1–3.

The same operation on NumPy arrays: usually under 2× faster again, since Pandas delegates to NumPy.

Numba-compiled loops over arrays: fastest for stateful custom logic that cannot be vectorized.

# Slow: row-wise apply
df["risk"] = df.apply(lambda r: r.amount * 2 if r.is_weekend else r.amount,
                      axis=1)
# Fast: vectorized
df["risk"] = np.where(df["is_weekend"], df["amount"] * 2, df["amount"])

Where Pandas overhead dominates: thousands of tiny operations in a loop, df.loc[i, "col"] row access (extract the column to an array once), very wide numeric matrices (hold as one 2-D array), and repeated .to_numpy() calls inside a training loop.

Memory tactics, roughly by impact: downcast float64 to float32 and int64 to smaller ints; use category for low-cardinality strings; install PyArrow so pandas 3.0's default str dtype is Arrow-backed and compact; read only needed columns (columns=, usecols=); and rely on copy-on-write, now the only mode in pandas 3.0, to avoid defensive .copy() calls. Measure with df.memory_usage(deep=True).sum() and arr.nbytes.

Common pitfalls and anti-patterns

Leakage and skew deserve emphasis because they produce wrong answers, not slow ones: the model looks better in validation than in production, or scores inputs incorrectly without raising an error. Both are prevented by keeping preprocessing inside a fitted Pipeline and treating the feature list as a versioned artifact.

Beyond the duo: when to look elsewhere

Polars. Rust DataFrame library on the Arrow memory format, with a lazy query engine and multi-threading; often several times faster than Pandas for large single-machine feature engineering. Converts to NumPy and Pandas when needed.

Apache Arrow. Shared columnar format behind pandas' Arrow dtypes (dtype_backend="pyarrow"), enabling low-copy interchange with Polars and DuckDB, and faster Pandas–Spark transfer.

DuckDB. In-process SQL over Parquet files and DataFrames; excellent for heavy joins and aggregations.

Dask and Modin. Parallelize Pandas/NumPy APIs across cores (Modin) or a cluster with out-of-core support (Dask).

CuPy and RAPIDS cuDF. GPU counterparts to NumPy and Pandas; cudf.pandas accelerates existing Pandas code with no code changes.

xarray and tensor libraries. xarray adds labels to n-D arrays; PyTorch, TensorFlow and JAX take over NumPy's role from the data loader onward.

The principle carries over: use a labelled, relational tool for shaping data, and a dense array tool for computing on it.

Decision framework and conclusion

Shape in Pandas, compute in NumPy. Keep meaning attached while transforming; drop to arrays when you only need numbers.

Hand off once, ideally through a fitted scikit-learn Pipeline that also serves in production.

Vectorize before you optimize. Removing Python loops matters far more than switching libraries.

Keep labels recoverable. Version feature lists and keep row indices so arrays map back to the real world.

Outgrow deliberately. Move to Polars, DuckDB, Dask or GPU tools when profiling shows a real ceiling.

Pandas and NumPy are not competitors; they are two layers of the same stack. Pandas is where data makes sense to people, NumPy is where it makes sense to machines. A well-designed ML workflow uses each where it is strongest and crosses the boundary between them once, on purpose.

Back to all blogs