How to Handle Imbalanced Data Without Fooling Yourself

Imbalanced data is not solved by making the classes look equal. It is solved by evaluating on the right population, avoiding leakage, choosing a model that ranks well, and turning scores into actions with a threshold or capacity rule that reflects the business decision. In food delivery the rare event might be a late order, a cancellation, fraud, or churn; blindly balancing the data can produce a model that looks excellent offline and disappoints in production.

The running example predicts whether a customer will churn in the next 30 days. Churners are the minority class, and finding them early lets the growth team send targeted retention offers. Every number quoted below comes from running the code in this article.

Setup: food-delivery dataset

The setup creates a synthetic dataset with about 10% churn. The columns are artificial numeric features with food-delivery names, so use it to demonstrate evaluation workflow, not to draw conclusions about real churn. It also defines two helpers used throughout: a scaler-plus-logistic-regression pipeline and an exact top-k selector. Save it as churn_data.py.

# TunedThresholdClassifierCV needs scikit-learn 1.5+
pip install "scikit-learn>=1.5" imbalanced-learn pandas numpy matplotlib
# Save as churn_data.py: every later section imports from it.
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

SEED = 7
CAPACITY = 0.10  # the team can contact 10% of customers per cycle

X, y = make_classification(
    n_samples=10_000, n_features=10, n_informative=6, n_redundant=2,
    weights=[0.90, 0.10], class_sep=0.85, random_state=SEED)

feature_names = [
    "orders_last_30d", "avg_order_value", "days_since_last_order",
    "app_sessions_last_30d", "avg_delivery_time", "late_order_rate",
    "refund_rate", "promo_usage_rate", "tenure_days", "support_tickets"]
customers = pd.DataFrame(X, columns=feature_names)
customers["churned_next_30d"] = y

# Random stratified split for the demo. With real customers, split by
# time: train on earlier observation dates, test on later ones.
X_train, X_test, y_train, y_test = train_test_split(
    customers[feature_names], customers["churned_next_30d"],
    test_size=0.25, stratify=customers["churned_next_30d"],
    random_state=SEED)

def logistic(**kwargs):
    """Scaler + logistic regression, refit from scratch on every call."""
    return make_pipeline(StandardScaler(),
                         LogisticRegression(max_iter=1000, **kwargs))

def flag_top_fraction(scores, fraction=CAPACITY):
    """Flag exactly the top `fraction` of scores (ties cannot over-select)."""
    scores = np.asarray(scores)
    k = int(round(fraction * len(scores)))
    flags = np.zeros(len(scores), dtype=int)
    flags[np.argsort(-scores, kind="stable")[:k]] = 1
    return flags

if __name__ == "__main__":
    print("Train churn rate:", round(y_train.mean(), 4),
          " Test churn rate:", round(y_test.mean(), 4))

Keep validation and test data at the natural, production-like class rate. Resampling an evaluation set changes precision and every other prevalence-sensitive metric. Real churn data should also be split by time — train on earlier observation dates, test on later ones — because customer behaviour drifts and a random split leaks the future into training.

1. Start with the real problem

Problem statement

The growth team wants to find customers likely to churn so it can send a win-back offer. It can contact only 10% of customers per cycle, so it needs to find churners without wasting offers on customers who would have stayed anyway.

Concept

Before touching the data, define what the positive class means (churned_next_30d = 1), when the prediction is made (the start of the 30-day window), what action follows (an offer, a push notification, a retention manager), and what false positives and false negatives cost. Evaluate on the same population and prevalence the model will face in production, and exclude customers with no opportunity to churn, such as users who installed the app but never ordered. Keep the label definition identical in training and production — “no order in 30 days” in one and “45 days” in the other silently changes the target.

2. Do not trust accuracy

Problem statement

A stakeholder asks: “Our churn model has 90% accuracy — why are we not retaining more customers?”

Concept and code

With 90% non-churners, predicting “no churn” for everyone scores about 90% accuracy. Here the gap is worse than it sounds: the logistic model at its default 0.5 cut-off beat the dummy by half a point of accuracy (0.901 vs 0.896) while catching 11 of 259 churners. The confusion matrix makes that visible; accuracy hides it.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, recall_score

from churn_data import X_test, X_train, logistic, y_test, y_train

rate = y_train.mean()
print(f"Churn rate: {rate:.1%}   imbalance {(1 - rate) / rate:.1f}:1")

dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
baseline = logistic().fit(X_train, y_train)
for name, model in [("all-negative dummy", dummy),
                    ("logistic @ 0.5", baseline)]:
    pred = model.predict(X_test)
    print(f"{name:>18}: accuracy {accuracy_score(y_test, pred):.3f}  "
          f"recall {recall_score(y_test, pred, zero_division=0):.3f}")
    print(confusion_matrix(y_test, pred))

Operations cares about “how many churners did we catch?” and “how many offers did we waste?”. Show accuracy only next to the confusion matrix, report minority-class precision and recall, and track prevalence in every evaluation period — accuracy comparisons across periods are meaningless when the churn rate moves.

3. Use stratified splits and cross-validation

Problem statement

The team wants a reliable performance estimate without folds that contain too few churners.

Concept

Stratified splitting keeps the class ratio equal in every split and fold, which reduces the variance of recall and PR-AUC estimates; plain KFold on a rare class can produce folds with almost no positives. Report the mean and spread across folds, never the best fold, keep the test set untouched until final evaluation, and keep the scaler inside the pipeline so it is fitted per fold.

Code

from sklearn.model_selection import StratifiedKFold, cross_validate

from churn_data import SEED, X_train, logistic, y_train

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=SEED)
scoring = ["precision", "recall", "f1", "roc_auc", "average_precision"]

cv_results = cross_validate(logistic(), X_train, y_train, cv=cv,
                            scoring=scoring, n_jobs=-1)
for name in scoring:
    scores = cv_results[f"test_{name}"]
    print(f"{name:>17}: {scores.mean():.3f} ± {scores.std():.3f}")

The output is a warning in itself: precision 1.000 ± 0.000 with recall 0.036. At the default threshold the model flags only a handful of customers per fold, so “perfect precision” is an artefact of the cut-off, not a good model. Threshold-free metrics (ROC-AUC 0.66, PR-AUC 0.40 here) describe ranking quality; precision and recall describe one operating point.

4. Resample only inside the training pipeline

Problem statement

The team wants to try oversampling to show the model more churners, without leaking test information into training.

Concept

Resampling must happen after the split and inside each cross-validation fold. If SMOTE or random oversampling runs on the full dataset first, synthetic points interpolated from test rows end up in training and the evaluation folds are artificially balanced. The imbalanced-learn documentation lists this as a common pitfall and recommends its Pipeline, which applies samplers during fit only.

Code

The snippet runs both versions so the leak is measurable rather than asserted.

from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline as make_imb_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.preprocessing import StandardScaler

from churn_data import SEED, X_train, logistic, y_train

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=SEED)

# Correct: SMOTE is a pipeline step, so it runs on each training fold only
smote_model = make_imb_pipeline(
    StandardScaler(), SMOTE(random_state=SEED),
    LogisticRegression(max_iter=1000))
honest = cross_val_score(smote_model, X_train, y_train, cv=cv,
                         scoring="average_precision")

# Wrong: resample first, then cross-validate. Test folds now contain
# synthetic neighbours of training rows and a 50% churn rate.
X_leaky, y_leaky = SMOTE(random_state=SEED).fit_resample(X_train, y_train)
leaky = cross_val_score(logistic(), X_leaky, y_leaky, cv=cv,
                        scoring="average_precision")

print(f"PR-AUC, SMOTE inside pipeline:  {honest.mean():.3f}")
print(f"PR-AUC, SMOTE before CV (leak): {leaky.mean():.3f}")

The leaky version reported PR-AUC 0.685; the honest pipeline 0.245. Part of the gap is leakage and part is that the leaky folds have 50% churn, which inflates any precision-based metric — both are reasons never to evaluate on resampled data. Note also that the honest SMOTE result is lower than the unweighted model’s 0.40: on this data, SMOTE made ranking worse. SMOTE interpolates between neighbours, so it needs numeric, meaningfully scaled features (use SMOTENC or an encoding strategy for categoricals), and its synthetic rows are not real customers.

5. Class weights: what they do and do not fix

Problem statement

The team wants to penalize missed churners more heavily without creating synthetic customers.

Concept

Class weighting scales the loss so mistakes on the minority class cost more; class_weight="balanced" weights classes inversely to their frequency and leaves the data itself unchanged. For logistic regression its main effect is to shift predicted probabilities upward, which moves the 0.5 cut-off — something a tuned threshold does more directly. It rarely improves the ranking, and it destroys calibration.

Code

from sklearn.metrics import average_precision_score, precision_score
from sklearn.metrics import recall_score

from churn_data import X_test, X_train, logistic, y_test, y_train

for name, model in [("unweighted", logistic()),
                    ("balanced", logistic(class_weight="balanced"))]:
    model.fit(X_train, y_train)
    pred = model.predict(X_test)  # default 0.5 cut-off
    score = model.predict_proba(X_test)[:, 1]
    print(f"{name:>10}: precision {precision_score(y_test, pred):.3f}  "
          f"recall {recall_score(y_test, pred):.3f}  "
          f"PR-AUC {average_precision_score(y_test, score):.3f}  "
          f"mean predicted P {score.mean():.2f}")

Recall at 0.5 rose from 0.042 to 0.645, which is why weighting looks like a fix. But PR-AUC fell from 0.455 to 0.247, and the mean predicted churn probability became 0.47 for a population that churns 10% of the time. Across five random seeds of this dataset, balanced weights and SMOTE never improved PR-AUC or precision in the top 10%; the seed used here is the worst case. Treat weighting as one candidate to compare under the same protocol, tune weights by cross-validation if you use them, and never stack weighting, oversampling, and threshold tuning without measuring the combined effect.

6. Tune the decision threshold

Problem statement

The retention team can contact at most the top 10% highest-risk customers, and each offer has a cost. The default 0.5 cut-off matches neither constraint.

Concept

A classifier produces a score; the threshold turns it into an action. The right rule depends on campaign capacity, offer cost, expected retention uplift, and customer experience. There are two honest options. A fixed threshold is chosen on data the test set never sees — scikit-learn’s TunedThresholdClassifierCV (version 1.5+) does this with internal cross-validation on the training data and a business scorer. A capacity rule contacts exactly the top k% of each scored batch; it needs no labels to apply, so it can be evaluated on the test set directly.

Code

import pandas as pd
from sklearn.metrics import make_scorer, precision_score, recall_score
from sklearn.model_selection import TunedThresholdClassifierCV

from churn_data import (SEED, X_test, X_train, flag_top_fraction, logistic,
                        y_test, y_train)

# Assumptions to replace with finance and experiment results
VALUE_PER_SAVED_CHURNER = 40.0  # margin kept x uplift from the offer
OFFER_COST = 8.0                # cost of every offer sent

def campaign_profit(y_true, y_pred):
    true_pos = ((y_true == 1) & (y_pred == 1)).sum()
    return VALUE_PER_SAVED_CHURNER * true_pos - OFFER_COST * y_pred.sum()

# Policy A: a fixed threshold chosen by internal cross-validation on the
# TRAINING data to maximise profit. The test set is used once, below.
tuned = TunedThresholdClassifierCV(
    logistic(), scoring=make_scorer(campaign_profit), cv=5,
    random_state=SEED).fit(X_train, y_train)

# Policy B: fixed capacity, contact the top 10% of each scored batch
scores = tuned.estimator_.predict_proba(X_test)[:, 1]
policies = {"default 0.5": (scores >= 0.5).astype(int),
            f"tuned {tuned.best_threshold_:.2f}": tuned.predict(X_test),
            "top 10%": flag_top_fraction(scores)}

rows = []
for name, flags in policies.items():
    rows.append({"policy": name, "flagged": int(flags.sum()),
                 "precision": precision_score(y_test, flags,
                                              zero_division=0),
                 "recall": recall_score(y_test, flags),
                 "profit": campaign_profit(y_test.to_numpy(), flags)})
print(pd.DataFrame(rows).set_index("policy").round(3))

Under the stated cost assumptions, the default threshold flagged 11 customers (profit 352), the tuned threshold of 0.21 flagged 191 (precision 0.52, profit 2,432), and the top-10% rule flagged exactly 250 (precision 0.44, profit 2,360). The earlier version of this section computed the cut-off with np.quantile(scores, 0.90) and >=, which over-selects whenever scores tie — in a tie-heavy test it flagged three times the capacity — and it tabulated thresholds directly on the test set, the practice it warned against. Select fixed thresholds on training or validation data, confirm once on test, monitor the actual flag rate against campaign capacity, and re-tune when prevalence, offer cost, or budget changes.

7. Evaluate with PR-AUC, not ROC-AUC alone

Problem statement

The team needs a threshold-independent metric that reflects performance on the rare churn class.

Concept

Average precision (AP) summarizes the precision-recall curve and is commonly called PR-AUC, although it is a step-wise sum rather than trapezoidal area. With rare positives, PR summaries reveal differences that ROC-AUC hides: in the comparison below, class weighting nudged ROC-AUC up (0.689 → 0.696) while cutting AP nearly in half. AP’s chance level equals the prevalence, so it cannot be compared across datasets or periods with different churn rates, and it says nothing about calibration.

Code

import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import PrecisionRecallDisplay, precision_recall_curve

from churn_data import X_test, X_train, logistic, y_test, y_train

scores = logistic().fit(X_train, y_train).predict_proba(X_test)[:, 1]

display = PrecisionRecallDisplay.from_predictions(
    y_test, scores, name="Churn model", plot_chance_level=True)
display.ax_.set_title("Precision-recall curve: 30-day churn")
display.figure_.savefig("churn_precision_recall_curve.png", dpi=150)
plt.close(display.figure_)

# "If we want to catch 70% of churners, how many contacts are wasted?"
precision, recall, _ = precision_recall_curve(y_test, scores)
best_at_70 = precision[recall >= 0.70].max()
print(f"Average precision: {display.average_precision:.3f}  "
      f"(chance level = prevalence {y_test.mean():.3f})")
print(f"Best precision at >= 70% recall: {best_at_70:.3f}")
print(f"Share of contacts wasted: {1 - best_at_70:.1%}")
assert np.isfinite(best_at_70)

The curve answers the operational question directly: to catch 70% of churners with this model, the best achievable precision is 0.148 — about 85% of offers would go to customers who were not going to churn. Report AP with prevalence and with precision and recall at the deployed threshold, and track it by city, cuisine, platform, and segment, re-evaluating after major changes in promotions, delivery operations, or customer mix.

8. Validate calibration and business impact

Problem statement

The model says a customer has a 35% chance of churning. Is that a trustworthy probability, and does contacting high-risk customers actually reduce churn?

Concept

A model can rank well and still be badly calibrated. Calibration checks whether predicted probabilities match observed frequencies, which matters whenever scores feed expected-value calculations or are shown as probabilities. Class weights and resampling deliberately distort probabilities, so recalibrate after using them, with cross-validation on training data rather than the test set.

Code

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.calibration import CalibratedClassifierCV, CalibrationDisplay
from sklearn.metrics import brier_score_loss

from churn_data import X_test, X_train, logistic, y_test, y_train

models = {
    "unweighted": logistic(),
    "balanced weights": logistic(class_weight="balanced"),
    # Recalibrate with internal CV on training data, never on the test set
    "balanced + sigmoid": CalibratedClassifierCV(
        logistic(class_weight="balanced"), method="sigmoid", cv=5),
}

fig, ax = plt.subplots(figsize=(6, 5))
rows = []
for name, model in models.items():
    scores = model.fit(X_train, y_train).predict_proba(X_test)[:, 1]
    CalibrationDisplay.from_predictions(y_test, scores, n_bins=10,
                                        strategy="quantile", name=name,
                                        ax=ax)
    rows.append({"model": name, "brier": brier_score_loss(y_test, scores),
                 "mean_predicted": scores.mean(),
                 "observed_rate": y_test.mean()})
ax.set_title("Calibration: predicted vs observed churn")
fig.savefig("churn_calibration_curve.png", dpi=150, bbox_inches="tight")
plt.close(fig)
print(pd.DataFrame(rows).set_index("model").round(3))

The unweighted model was well calibrated (Brier 0.080, mean prediction 0.105 against an observed 0.104). The balanced model predicted 0.47 on average — customers it scored around 0.44 actually churned about 6% of the time — and its Brier score nearly tripled to 0.225. Sigmoid calibration restored the mean to 0.104 and the Brier score to 0.087. A retention team acting on uncalibrated scores would over-invest in offers for customers who were never likely to leave.

Calibration still says nothing about impact. A churn score predicts who will leave, not who an offer will keep: some high-risk customers leave regardless and some low-risk ones only stay because of the offer. Estimate incremental retention with a randomized holdout or A/B test, consider uplift modelling once you have experimental data, and track offers sent, redemption, churn, revenue, and cost per retained customer.

Complete comparison framework

Comparing models with predict() at 0.5, as the earlier version did, rewards whichever model happens to inflate its probabilities: weighted and SMOTE models look like recall winners only because their scores are shifted. A fair comparison uses threshold-free metrics plus the same operating policy for every model — here the top 10% the team can actually contact — and includes the Brier score for calibration. It should also include a stronger model family, because imbalance tricks are rarely the biggest lever.

import pandas as pd
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline as make_imb_pipeline
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (average_precision_score, brier_score_loss,
                             precision_score, recall_score, roc_auc_score)
from sklearn.preprocessing import StandardScaler

from churn_data import (SEED, X_test, X_train, flag_top_fraction, logistic,
                        y_test, y_train)

candidates = {
    "logistic": logistic(),
    "logistic + SMOTE": make_imb_pipeline(
        StandardScaler(), SMOTE(random_state=SEED),
        LogisticRegression(max_iter=1000)),
    "logistic, balanced": logistic(class_weight="balanced"),
    "gradient boosting": HistGradientBoostingClassifier(random_state=SEED),
}

def evaluate_model(name, model):
    scores = model.fit(X_train, y_train).predict_proba(X_test)[:, 1]
    flags = flag_top_fraction(scores)  # same operating policy for all
    return {"model": name,
            "roc_auc": roc_auc_score(y_test, scores),
            "pr_auc": average_precision_score(y_test, scores),
            "precision@10%": precision_score(y_test, flags),
            "recall@10%": recall_score(y_test, flags),
            "brier": brier_score_loss(y_test, scores)}

results = pd.DataFrame([evaluate_model(n, m) for n, m in candidates.items()])
print(results.set_index("model").round(3))

Neither rebalancing technique helped the business metric; a model that could capture the non-linear structure nearly doubled precision in the contacted group with no rebalancing at all. On real data the gap will differ, but the protocol — same split, same policy, threshold-free metrics, calibration — is what keeps the comparison honest.

Practical decision guide

Takeaway: Split first (by time for real data), resample only inside training folds, evaluate on the natural class distribution, compare models under the same operating policy, choose thresholds without the test set, check calibration, and prove business impact with an experiment.

Back to all blogs