Overfitting, Underfitting, and the Bias-Variance Tradeoff - With Visual Intuition

A good model learns real patterns without memorizing noise. In e-commerce and retail, that distinction decides whether a demand forecast, churn model, or price-elasticity model performs well next week or only on last quarter’s training data. scikit-learn’s documentation describes underfitting as low training and validation performance, and overfitting as high training performance with low validation performance. Every number and figure below comes from running the code shown.

The core idea

Bias is systematic error from a model too simple to represent the pattern; variance is how much the fitted model changes when trained on a different sample of the same process, typically because it is too flexible. Increasing complexity usually lowers bias and raises variance, so the goal is the best validation performance, not perfect training performance.

1. Overfitting

Problem statement

An online retailer forecasts daily demand for a product category. A highly flexible model trained on a few weeks of history fits it almost perfectly but predicts poorly, because it learned random spikes, stockouts, and one-off marketing effects rather than repeatable demand drivers.

Conceptual explanation

Overfitting means learning noise and quirks of the training set instead of relationships that generalize — treating a viral social-media moment, a price error, or a warehouse delay as a stable pattern. Low-volume items are especially exposed, because random variation is large relative to the number of observations. Regularization is the standard remedy: it lets a flexible model keep its capacity while penalizing the wild coefficients that produce wiggles.

Runnable Python code

The demo fits polynomials of increasing degree to 21 training days and scores them on the 40 days in between. The earlier version of this example did not actually show overfitting: with its random split and 39 training points, the degree-15 model scored *lower* test error (4.65) than degree 4 (4.71). It also labelled randomly chosen points “later test data”. The version below uses fewer, evenly spaced training days, labels the held-out points honestly, and adds a ridge-regularized degree-15 model.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.metrics import root_mean_squared_error
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

rng = np.random.default_rng(11)
days = np.linspace(0, 10, 61)
true_demand = 50 + 6 * days + 12 * np.sin(days)  # trend + seasonal wave
demand = true_demand + rng.normal(0, 6, days.size)
X = days.reshape(-1, 1)

# Every third day trains; the days in between are held out. Held-out
# points lie between training points, so this measures interpolation,
# not forecasting (section 3 handles time properly).
train = np.zeros(days.size, dtype=bool)
train[::3] = True

def poly_model(degree, regressor):
    # Scale AFTER expansion too: x**15 dwarfs x, and ridge penalizes
    # every coefficient equally only if the columns share a scale.
    return make_pipeline(StandardScaler(),
                         PolynomialFeatures(degree, include_bias=False),
                         StandardScaler(), regressor)

models = {
    "degree 1": poly_model(1, LinearRegression()),
    "degree 4": poly_model(4, LinearRegression()),
    "degree 15": poly_model(15, LinearRegression()),
    "degree 15 + ridge": poly_model(15, RidgeCV(np.logspace(-6, 3, 40))),
}
grid = np.linspace(0, 10, 400).reshape(-1, 1)
fig, axes = plt.subplots(1, 4, figsize=(16, 4), sharey=True)
rows = []
for ax, (name, model) in zip(axes, models.items()):
    model.fit(X[train], demand[train])
    rows.append({
        "model": name,
        "train_rmse": root_mean_squared_error(demand[train],
                                              model.predict(X[train])),
        "held_out_rmse": root_mean_squared_error(demand[~train],
                                                 model.predict(X[~train]))})
    ax.scatter(days[train], demand[train], s=18, label="training days")
    ax.scatter(days[~train], demand[~train], s=18, marker="x",
               label="held-out days")
    ax.plot(grid, model.predict(grid), color="red", label="model")
    ax.set(title=name, xlabel="day", ylim=(20, 140))
axes[0].set_ylabel("units sold")
axes[0].legend()
fig.tight_layout()
fig.savefig("overfitting_underfitting_visual.png", dpi=150)
plt.close(fig)
print(pd.DataFrame(rows).set_index("model").round(1))

The second StandardScaler after the polynomial expansion matters: without it, the columns range from x to x¹⁵ and a ridge penalty shrinks them unevenly. In a test with that scaler omitted, cross-validated ridge chose a very large penalty and underfit (held-out RMSE about 15).

Use case and common mistakes

A retailer forecasting a limited-edition sneaker may extrapolate a celebrity-driven spike, overstock, tie up working capital, and be left with unsold stock when the trend fades. Typical causes: judging only training accuracy, using hundreds of features for a small dataset, splitting time series randomly, tuning on the test set until it becomes another training set, and letting the model memorize one-off events instead of handling them deliberately. Keep separate training, validation, and untouched test sets; prefer simpler models unless complexity improves out-of-time validation; control complexity with regularization, feature selection, early stopping, or ensembles; backtest across promotions, holidays, and quiet periods; compare live error with offline validation; and record each model’s data range, features, hyperparameters, and evaluation results in a registry.

2. Underfitting

Problem statement

A retailer predicts 30-day churn from a single feature such as total purchases last year. The model scores a lapsed high-value customer and a weekly buyer alike, so retention campaigns miss the customers most likely to leave.

Conceptual explanation

Underfitting is high bias: the model makes systematic errors even on its own training data, because it lacks either the inputs or the flexibility to represent the relationship. The diagnostic signature is training performance that is itself poor and close to validation performance. The fix is better features or a more expressive model; more rows do not help, and the learning curve below shows why.

Runnable Python code

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import (StratifiedKFold, learning_curve,
                                     train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Abstract synthetic features; the churn names are illustrative only
X, y = make_classification(
    n_samples=1_200, n_features=8, n_informative=5, n_redundant=1,
    class_sep=1.1, weights=[0.75, 0.25], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)

def logistic():
    return make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

candidates = {  # name: (model, columns)
    "linear, one feature": (logistic(), [0]),
    "linear, all features": (logistic(), slice(None)),
    "boosted trees, all features": (
        HistGradientBoostingClassifier(random_state=42), slice(None)),
}
rows = []
for name, (model, cols) in candidates.items():
    model.fit(X_train[:, cols], y_train)
    rows.append({
        "model": name,
        "train_auc": roc_auc_score(
            y_train, model.predict_proba(X_train[:, cols])[:, 1]),
        "test_auc": roc_auc_score(
            y_test, model.predict_proba(X_test[:, cols])[:, 1])})
print(pd.DataFrame(rows).set_index("model").round(3))

# Learning curve: more rows barely help a model that is too simple
cv = StratifiedKFold(5, shuffle=True, random_state=42)
for name, model, cols in [("linear, one feature", logistic(), [0]),
                          ("boosted trees", HistGradientBoostingClassifier(
                              random_state=42), slice(None))]:
    sizes, _, valid = learning_curve(
        model, X[:, cols], y, cv=cv, scoring="roc_auc",
        train_sizes=np.linspace(0.1, 1.0, 5))
    curve = "  ".join(f"{n}:{v:.3f}" for n, v in zip(sizes,
                                                       valid.mean(axis=1)))
    print(f"{name:>20} validation AUC by training rows -> {curve}")

The one-feature model’s validation AUC stayed at 0.649 whether it trained on 96 or 960 rows; boosted trees climbed from 0.842 to 0.955 over the same range. Two further lessons: the all-feature linear model, which the earlier version presented as the “better” model, is itself still underfitting this non-linear data (training AUC only 0.85); and the boosted model has the largest train-test gap yet by far the best test score. A gap is a warning sign to investigate, not a verdict — the choice is made on validation performance.

Use case and common mistakes

A better-specified churn model can find customers whose sessions, purchase frequency, and basket value have recently declined, so the CRM team targets them with a relevant offer instead of discounting loyal customers. Common causes of underfitting are missing behavioural signals, overly rigid models where behaviour has thresholds, and annual aggregates that hide a recent drop. Start from domain features (recency, frequency, monetary value, engagement, returns, support contacts), compare a simple baseline with a more flexible model under the same protocol, use imbalance-aware metrics such as PR-AUC or lift, validate on a later period, confirm every feature is available at scoring time, and prove value with an A/B test.

3. The bias-variance tradeoff

Problem statement

A product-level demand model that is too simple under-forecasts promotions and holidays; one that is too complex chases daily noise. The team needs a principled way to choose complexity that minimizes future error.

Conceptual explanation

For squared error, the expected prediction error at a point splits exactly into three parts, averaged over all the training sets the model might have seen:

Expected MSE = Bias² + Variance + Irreducible noise

Retail data always carries irreducible noise — weather, competitor moves, supply disruptions — that no model removes. Because bias and variance are defined across training sets, the clearest way to see them is to simulate many training sets from a known process and measure both directly.

Runnable Python code: measuring bias and variance

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

def true_demand(x):
    return 50 + 6 * x + 12 * np.sin(x)

NOISE_SD, N_TRAIN, N_REPEATS = 6.0, 21, 300
rng = np.random.default_rng(7)
x_train = np.linspace(0, 10, N_TRAIN)        # same days every time...
x_eval = np.linspace(0.25, 9.75, 200)        # ...evaluated in between

rows = []
for degree in range(1, 13):
    preds = np.empty((N_REPEATS, x_eval.size))
    for r in range(N_REPEATS):               # ...but fresh noise each time
        y = true_demand(x_train) + rng.normal(0, NOISE_SD, N_TRAIN)
        model = make_pipeline(StandardScaler(),
                              PolynomialFeatures(degree, include_bias=False),
                              StandardScaler(), LinearRegression())
        preds[r] = model.fit(x_train[:, None], y).predict(x_eval[:, None])
    bias_sq = np.mean((preds.mean(axis=0) - true_demand(x_eval)) ** 2)
    variance = np.mean(preds.var(axis=0))
    # Direct check: error against brand-new noisy observations
    y_new = true_demand(x_eval) + rng.normal(0, NOISE_SD,
                                             (N_REPEATS, x_eval.size))
    rows.append({"degree": degree, "bias_sq": bias_sq,
                 "variance": variance,
                 "bias_sq+var+noise": bias_sq + variance + NOISE_SD ** 2,
                 "measured_mse": np.mean((preds - y_new) ** 2)})

results = pd.DataFrame(rows).set_index("degree")
print(results.round(1))

ax = results[["bias_sq", "variance", "bias_sq+var+noise"]].plot(
    marker="o", logy=True, figsize=(8, 5))
ax.axhline(NOISE_SD ** 2, linestyle="--", color="grey",
           label="irreducible noise")
ax.set(xlabel="polynomial degree (model complexity)",
       ylabel="squared error (log scale)",
       title="Bias-variance decomposition")
ax.legend()
ax.figure.savefig("bias_variance_decomposition.png", dpi=150,
                  bbox_inches="tight")
plt.close(ax.figure)

Over 300 simulated training sets, squared bias fell from 65.7 at degree 1 to almost zero by degree 6, while variance rose steadily from 3.6 to 23.2 at degree 12. Their sum plus the noise (36) matched the error measured against fresh observations to within 1% at every degree, and total error was lowest around degrees 6–7. The curve is U-shaped here; it is not guaranteed to be in general.

Runnable Python code: choosing complexity for a forecast

For real forecasting, complexity must be chosen on forward-chaining splits. The earlier version did this with polynomials in the day number, which cannot forecast: a fitted polynomial shoots off once it passes the last training day. Validation RMSE reached about 2 × 10¹⁰ at degree 15, and the method chose degree 1 even though the true curve was quadratic — it was measuring extrapolation blow-up, not the bias-variance tradeoff. The version below uses features that stay bounded in the future (trend, weekday dummies, and yearly Fourier terms), varies the number of Fourier harmonics as the complexity knob, and compares against a seasonal-naive baseline.

import numpy as np
import pandas as pd
from sklearn.linear_model import Ridge
from sklearn.metrics import root_mean_squared_error
from sklearn.model_selection import TimeSeriesSplit

# Two years of daily demand: trend + weekly + yearly pattern + noise
rng = np.random.default_rng(29)
dates = pd.date_range("2024-11-01", periods=730, freq="D")
t = np.arange(len(dates))
doy = dates.dayofyear.to_numpy()
demand = (200 + 0.05 * t
          + 25 * np.isin(dates.dayofweek, [5, 6])        # weekend lift
          + 30 * np.sin(2 * np.pi * doy / 365.25)
          + 10 * np.cos(4 * np.pi * doy / 365.25)
          + rng.normal(0, 12, len(dates)))

def design(n_harmonics):
    """Trend + weekday dummies + n yearly Fourier pairs. Unlike powers of
    time, these terms stay bounded when the model forecasts ahead."""
    cols = {"trend": t / len(t)}
    for d in range(1, 7):
        cols[f"dow_{d}"] = (dates.dayofweek == d).astype(float)
    for k in range(1, n_harmonics + 1):
        cols[f"sin_{k}"] = np.sin(2 * np.pi * k * doy / 365.25)
        cols[f"cos_{k}"] = np.cos(2 * np.pi * k * doy / 365.25)
    return pd.DataFrame(cols, index=dates)

# Five forward-chaining folds, each forecasting the next 4 weeks
cv = TimeSeriesSplit(n_splits=5, test_size=28)
rows = []
for k in [0, 1, 2, 4, 8, 16, 32, 64]:
    X = design(k)
    train_err, valid_err = [], []
    for tr, va in cv.split(X):
        model = Ridge(alpha=1e-3).fit(X.iloc[tr], demand[tr])
        train_err.append(root_mean_squared_error(
            demand[tr], model.predict(X.iloc[tr])))
        valid_err.append(root_mean_squared_error(
            demand[va], model.predict(X.iloc[va])))
    rows.append({"harmonics": k, "n_features": X.shape[1],
                 "train_rmse": np.mean(train_err),
                 "valid_rmse": np.mean(valid_err),
                 "valid_rmse_sd": np.std(valid_err)})

# Seasonal-naive baseline: same weekday last week
naive = [root_mean_squared_error(demand[va], demand[va - 7])
         for _, va in cv.split(design(0))]
results = pd.DataFrame(rows).set_index("harmonics")
print(results.round(2))
print(f"seasonal-naive baseline RMSE: {np.mean(naive):.2f}")
print("chosen harmonics:", results["valid_rmse"].idxmin())

Training error fell with every added harmonic, while validation error bottomed out at two harmonics — the true structure of the simulated data — and rose again as the model began fitting noise, all while beating the naive baseline. Validation RMSE varied by about ±2 across folds, so differences smaller than that are not meaningful.

Use case and common mistakes

A grocery chain forecasting fresh produce faces both failures: a high-bias model predicts average demand and runs out before weekends; a high-variance model overreacts to one rainy day and wastes perishables. Avoid selecting complexity on training error, random cross-validation for time series, changing many things at once, ignoring that under- and over-forecasting cost different amounts for perishables and durables, and assuming one global model suits fast-moving, seasonal, and long-tail products alike. Start from a naive baseline, select on forward-chaining validation, evaluate by product segment, include business metrics (stockouts, waste, inventory turns, forecast bias), monitor prediction intervals, retrain on a cadence and after major assortment, pricing, or promotion changes, and document the chosen complexity and expected error range.

Practical decision guide

Takeaway: Do not try to eliminate bias or variance. Choose the least complex model that wins on honest validation — forward-chaining for anything over time — and confirms its value on unseen periods and real business decisions.

Back to all blogs