Model Versioning, Data Versioning, and Experiment Tracking

Model Versioning, Data Versioning, and Experiment Tracking

Why reproducibility matters in banking ML systems: from experiment traceability to controlled production operation

Core principle. A production prediction must be traceable to an immutable model version, its exact training data and feature definitions, code, environment, configuration, approval evidence, and the policy decision that consumed its score.

Scope. The examples use synthetic transactions and demonstrate engineering patterns; they must not be connected to live card authorisation, core banking or customer data. Real deployment needs the bank's architecture, security, privacy, model-risk, compliance, resilience and change controls, independent validation and integration testing. Regulatory references are examples, not legal advice. In the US, SR 26-2 (17 April 2026, issued by the Federal Reserve, OCC and FDIC) supersedes SR 11-7 and SR 21-8 and is described as most relevant to Fed-regulated organisations with over $30 billion in assets; banks in India map controls to applicable RBI directions, the Digital Personal Data Protection framework and payment-system rules instead.

Introduction: what reproducibility means

In banking, a model that cannot be reproduced is hard to trust, audit, debug, validate or defend. Reproducibility means reconstructing the full development and release context: data snapshot, transformations, feature definitions, code commit and whether the working tree was clean, dependency environment, parameters, seeds, model artifact, threshold or policy configuration, evaluation, approvals and deployment identity.

The running example is card-fraud scoring. The model produces a risk score; a separately governed decision layer combines it with authorisation rules, customer context, limits, step-up authentication and approved policy. A score alone is never an authorisation decision. Reproducibility lets the bank answer:

Which dataset snapshot, extraction window, labels and transformations trained the deployed model?

Which commit, package versions, configuration and runtime produced the artifact?

Which model version and threshold or policy configuration produced a particular score or action?

Were candidates compared on the same time-consistent validation and test periods?

Can the previously approved model and its compatible runtime be restored quickly?

Can an independent validator reproduce the reported results and challenge the assumptions?

The three practices are distinct but linked. Data versioning identifies the exact data state; experiment tracking records each run's inputs and outputs; model versioning manages the deployable artifact and its lifecycle. The registry links a model version to its run, and the run links to its data snapshot.

1. Model versioning

A fraud team improves recall, but the candidate also challenges or declines more legitimate transactions. The team must know which exact model and policy versions are active, what evidence supported approval, and how to restore the previous release.

Registry aliases such as champion are convenient mutable pointers, not identities. At release, resolve the alias to a concrete version, record it and its digest in the deployment manifest, and deploy by that version. Never overwrite an approved artifact; treat threshold, calibration and policy changes as controlled releases; store artifacts in encrypted, access-controlled storage with WORM retention where required; verify checksums or signatures before loading; and keep a tested rollback package of model, dependencies, feature contract and policy.

Deserialisation is code execution. Pickle and joblib files can run arbitrary code on load. In the current MLflow 3.x release (3.17), scikit-learn models are saved in the skops format by default outside Databricks; the pipeline below requests it explicitly so the behaviour does not depend on where it runs.

2. Data versioning

A quarterly extract changes the model's results. The cause might be a new extraction window, late fraud labels, schema drift, a currency-unit change, duplicates or a pipeline change, and if inputs and lineage were overwritten, the prior run cannot be reconstructed. A dataset version should record:

Source system, extraction job and time, business-date range, event-time semantics and timezone.

Schema and data contract: units, currency, nullability, ranges, code sets and definitions.

Row and duplicate counts, missingness, label prevalence and validation results.

A content checksum or partition manifest, transformation version and reference-data versions.

Label maturity date, label source, adjudication policy, chargeback handling and known delay.

Classification, permitted purpose, legal basis, retention, residency, masking and access policy.

Fraud evaluation must be chronological. Random splits put future transactions into training and leak through rolling aggregates and delayed labels. Use point-in-time-correct joins, split on timestamps rather than row positions (so identical timestamps never straddle a boundary), and leave an embargo gap between periods. Validation must fail safely: check required columns first, then types, finiteness, duplicates, ranges, units, timestamps and label domain, and keep PAN, CVV, secrets and unnecessary identifiers out of features, logs and experiment parameters.

A local folder plus a checksum is useful for teaching but is not immutability: a privileged user can replace both data and metadata. Banks store snapshots in governed object storage or a versioned lakehouse with retention lock, access logging, encryption, backup and independently protected manifests. DVC manages references to remote storage; it does not supply the bank's governance controls.

3. Experiment tracking

Two data scientists compare a logistic-regression baseline with a random forest. The comparison is only fair if both use the same data windows, features, preprocessing, labels and evaluation design. Record for every run:

Run and parent IDs, owner, purpose, source commit and clean-tree flag.

Dataset version and checksum, period boundaries, embargo, feature schema and transformation version.

Algorithm, hyperparameters, seeds, model-selection rule, threshold method and calibration.

Python and library versions, container digest, hardware where relevant, known reproducibility limits.

Validation and test metrics, confusion matrix, prevalence, segment analysis and cost estimates.

Model artifact, evaluation report, explanation artifacts and approval references.

For imbalanced fraud data, accuracy is misleading. Use average precision, recall at an approved false-positive rate, precision, error counts, calibration, expected cost, review capacity and segment performance. The test set is not a tuning set: both the model choice and the threshold are made on validation data, and the held-out out-of-time test period is evaluated once. Choosing between algorithms by test results leaks the test set into development as surely as tuning a threshold on it.

4. Integrated reproducible reference pipeline

The script consolidates the three practices. It publishes a validated, content-addressed snapshot and verifies its checksum on every read. Re-running with identical data reuses the snapshot, while different bytes under the same version are refused: the original version check made every second run fail with FileExistsError. Snapshots copy the source bytes instead of re-serialising a DataFrame, so the checksum describes the real input, and each writer gets a unique staging directory. Partitions are cut on time with a 24-hour embargo. Both candidates are compared on validation average precision. The threshold is the highest-recall point at or below the validation false-positive limit, searched over every distinct score with roc_curve; a fixed 0.05–0.95 grid can miss the feasible region entirely and abort. The run logs the dataset as an MLflow input, library versions, the commit with a dirty-tree flag, and both validation and test metrics. It then registers a new model version tagged candidate_not_approved in a SQLite-backed registry.

Illustrative only. The synthetic fraud mechanism and the 2% validation false-positive limit are not bank policy, acceptance criteria or evidence of production performance.

"""Reproducible fraud-model development: synthetic data only, not a bank deployment.
Python 3.11+.  pip install "mlflow>=3.4,<4" skops scikit-learn pandas numpy
Run: python reproducible_fraud_pipeline.py   View: mlflow ui --backend-store-uri sqlite:///mlflow.db
"""
from __future__ import annotations
import hashlib
import json
import os
import platform
import shutil
import subprocess
import tempfile
from pathlib import Path
from typing import Any
import mlflow
import numpy as np
import pandas as pd
import sklearn
from mlflow import MlflowClient
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, confusion_matrix, roc_auc_score, roc_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
ROOT = Path(os.getenv("PIPELINE_ROOT", Path(__file__).resolve().parent))
SNAPSHOT_DIR = ROOT / "data" / "snapshots"
FEATURES = ["transaction_amount", "hours_since_last_txn", "customer_tenure_months",
            "merchant_risk_score", "is_foreign_country"]
TARGET, SEED = "is_fraud", 42
REQUIRED = ["transaction_id", "event_time", *FEATURES, TARGET]
MAX_VALIDATION_FPR = 0.02          # illustrative; the bank approves its own limit
EMBARGO = pd.Timedelta(hours=24)   # gap between periods so look-ahead features cannot leak
REGISTERED_MODEL = "card_fraud_reference"
def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as stream:
        for chunk in iter(lambda: stream.read(1 << 20), b""):
            digest.update(chunk)
    return digest.hexdigest()
def source_state() -> dict[str, Any]:
    """Real commit plus a dirty flag: a commit ID means nothing if the tree had edits."""
    def git(*args: str) -> str:
        return subprocess.check_output(["git", *args], cwd=ROOT, text=True,
                                       stderr=subprocess.DEVNULL).strip()
    try:
        return {"code_commit": git("rev-parse", "HEAD"),
                "code_dirty": bool(git("status", "--porcelain"))}
    except (OSError, subprocess.CalledProcessError):
        return {"code_commit": os.getenv("GIT_COMMIT", "unavailable"), "code_dirty": None}
def make_synthetic_transactions(rows: int = 20_000, seed: int = SEED) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    amount, hours = rng.exponential(120, rows), rng.exponential(12, rows)
    merchant_risk, foreign = rng.uniform(0, 1, rows), rng.binomial(1, 0.08, rows)
    logit = (-5.0 + 0.003 * np.minimum(amount, 1500) + 1.4 * merchant_risk
             + 0.9 * foreign - 0.025 * np.minimum(hours, 48))   # deliberately simple
    return pd.DataFrame({
        "transaction_id": [f"t{i:07d}" for i in range(rows)],
        "event_time": pd.date_range("2025-01-01", periods=rows, freq="5min", tz="UTC")
                        .strftime("%Y-%m-%dT%H:%M:%SZ"),
        "transaction_amount": amount.round(2) + 0.01,
        "hours_since_last_txn": hours.round(3),
        "customer_tenure_months": rng.integers(1, 240, rows),
        "merchant_risk_score": merchant_risk.round(4),
        "is_foreign_country": foreign,
        TARGET: rng.binomial(1, 1 / (1 + np.exp(-logit))),
    })
def validate_transactions(frame: pd.DataFrame) -> None:
    if missing := sorted(set(REQUIRED) - set(frame.columns)):
        raise ValueError(f"missing required columns: {missing}")
    if frame.empty:
        raise ValueError("dataset is empty")
    if frame["transaction_id"].isna().any() or frame["transaction_id"].duplicated().any():
        raise ValueError("transaction_id must be present and unique")
    times = pd.to_datetime(frame["event_time"], utc=True, errors="coerce", format="ISO8601")
    if times.isna().any():
        raise ValueError("event_time contains invalid or missing timestamps")
    for column in [*FEATURES, TARGET]:
        values = pd.to_numeric(frame[column], errors="coerce").to_numpy(dtype=float)
        if not np.isfinite(values).all():
            raise ValueError(f"{column} contains null, non-numeric or non-finite values")
    for column in (TARGET, "is_foreign_country"):
        if not frame[column].isin([0, 1]).all():
            raise ValueError(f"{column} must contain only 0 or 1")
    if (frame["transaction_amount"] <= 0).any() or not frame["merchant_risk_score"].between(0, 1).all():
        raise ValueError("transaction_amount must be > 0 and merchant_risk_score in [0, 1]")
def publish_snapshot(source: Path, version: str) -> dict[str, Any]:
    """Content-addressed, write-once snapshot. Re-publishing identical bytes is a no-op;
    different bytes under an existing version are refused. Production storage also needs
    retention lock and an independently protected manifest."""
    if not version or not all(c.isalnum() or c in "._-" for c in version):
        raise ValueError("version may contain only letters, digits, dot, underscore, hyphen")
    validate_transactions(pd.read_csv(source))
    digest = sha256_file(source)
    destination = SNAPSHOT_DIR / version
    SNAPSHOT_DIR.mkdir(parents=True, exist_ok=True)
    staging = Path(tempfile.mkdtemp(prefix=f".{version}.", dir=SNAPSHOT_DIR))  # unique per writer
    try:
        shutil.copyfile(source, staging / "transactions.csv")         # exact bytes, not a re-dump
        frame = pd.read_csv(staging / "transactions.csv")
        meta = {"dataset_name": "synthetic_card_transactions", "dataset_version": version,
                "sha256": digest, "row_count": len(frame),
                "label_prevalence": float(frame[TARGET].mean()),
                "event_time_min": str(frame["event_time"].min()),
                "event_time_max": str(frame["event_time"].max()),
                "sensitivity": "synthetic_non_personal_data"}
        (staging / "metadata.json").write_text(json.dumps(meta, indent=2))
        try:
            staging.rename(destination)  # atomic; fails if another writer got there first
        except OSError:
            if not destination.exists():
                raise
        existing = json.loads((destination / "metadata.json").read_text())
        if existing["sha256"] != digest:
            raise FileExistsError(f"dataset version {version} exists with different content")
        return existing | {"path": str(destination / "transactions.csv")}
    finally:
        shutil.rmtree(staging, ignore_errors=True)
def load_snapshot(meta: dict[str, Any]) -> pd.DataFrame:
    path = Path(meta["path"])
    if sha256_file(path) != meta["sha256"]:   # verify on every read, not only on write
        raise RuntimeError(f"snapshot {meta['dataset_version']} failed checksum verification")
    frame = pd.read_csv(path)
    frame["event_time"] = pd.to_datetime(frame["event_time"], utc=True, format="ISO8601")
    return frame.sort_values(["event_time", "transaction_id"], ignore_index=True)
def chronological_partitions(frame: pd.DataFrame, embargo: pd.Timedelta = EMBARGO):
    """Split on timestamps, not row positions, so equal timestamps never straddle a
    boundary, and drop an embargo period after each cut."""
    t = frame["event_time"]
    train_end, valid_end = t.quantile(0.60), t.quantile(0.80)
    parts = {"train": frame[t < train_end],
             "validation": frame[(t >= train_end + embargo) & (t < valid_end)],
             "test": frame[t >= valid_end + embargo]}
    for name, part in parts.items():
        if part[TARGET].nunique() < 2:
            raise ValueError(f"{name} partition must contain both classes")
    return parts["train"], parts["validation"], parts["test"]
def metrics_at_threshold(y_true, scores, threshold: float) -> dict[str, float]:
    tn, fp, fn, tp = confusion_matrix(y_true, scores >= threshold, labels=[False, True]).ravel()
    return {"threshold": float(threshold), "roc_auc": float(roc_auc_score(y_true, scores)),
            "average_precision": float(average_precision_score(y_true, scores)),
            "precision": tp / (tp + fp) if tp + fp else 0.0, "recall": tp / (tp + fn),
            "false_positive_rate": fp / (fp + tn), "tp": int(tp), "fp": int(fp),
            "fn": int(fn), "tn": int(tn)}
def choose_threshold(y_true, scores) -> float:
    """Highest recall with validation FPR <= limit, searched over every distinct score
    (a fixed 0.05-0.95 grid can miss the feasible region entirely)."""
    fpr, tpr, thresholds = roc_curve(y_true, scores)
    ok = np.flatnonzero((fpr <= MAX_VALIDATION_FPR) & np.isfinite(thresholds))
    if ok.size == 0:
        raise RuntimeError("no threshold satisfies the validation FPR constraint")
    return float(thresholds[ok[np.argmax(tpr[ok])]])
def candidates() -> dict[str, Any]:
    return {
        "logistic_regression": make_pipeline(
            StandardScaler(), LogisticRegression(class_weight="balanced", max_iter=1000)),
        "random_forest": RandomForestClassifier(  # trees need no scaling
            n_estimators=250, max_depth=8, min_samples_leaf=5, class_weight="balanced",
            random_state=SEED, n_jobs=-1),        # fixed random_state: identical for any n_jobs
    }
def main() -> None:
    mlflow.set_tracking_uri(os.getenv("MLFLOW_TRACKING_URI", f"sqlite:///{ROOT / 'mlflow.db'}"))
    mlflow.set_experiment("retail_banking_fraud_reference")
    raw = ROOT / "data" / "raw" / "synthetic_transactions.csv"
    if not raw.exists():
        raw.parent.mkdir(parents=True, exist_ok=True)
        make_synthetic_transactions().to_csv(raw, index=False)
    snapshot = publish_snapshot(raw, os.getenv("DATASET_VERSION", "synthetic-2026-10-v1"))
    train, validation, test = chronological_partitions(load_snapshot(snapshot))
    # Select model AND threshold on validation only; the test period is touched once.
    fitted, val_ap = {}, {}
    for name, model in candidates().items():
        fitted[name] = model.fit(train[FEATURES], train[TARGET])
        val_ap[name] = average_precision_score(
            validation[TARGET], model.predict_proba(validation[FEATURES])[:, 1])
    best = max(val_ap, key=val_ap.get)
    model = fitted[best]
    val_scores = model.predict_proba(validation[FEATURES])[:, 1]
    threshold = choose_threshold(validation[TARGET], val_scores)
    val_metrics = metrics_at_threshold(validation[TARGET], val_scores, threshold)
    test_metrics = metrics_at_threshold(
        test[TARGET], model.predict_proba(test[FEATURES])[:, 1], threshold)
    with mlflow.start_run(run_name=f"{best}_candidate") as run:
        mlflow.log_input(mlflow.data.from_pandas(train, name=snapshot["dataset_version"],
                         digest=snapshot["sha256"][:16], targets=TARGET), context="training")
        mlflow.log_params({
            "algorithm": best, **{f"validation_ap_{k}": round(v, 5) for k, v in val_ap.items()},
            "dataset_version": snapshot["dataset_version"], "dataset_sha256": snapshot["sha256"],
            **source_state(), "python": platform.python_version(),
            "sklearn": sklearn.__version__, "numpy": np.__version__, "pandas": pd.__version__,
            "train_end": str(train["event_time"].max()), "test_start": str(test["event_time"].min()),
            "embargo_hours": EMBARGO.total_seconds() / 3600, "seed": SEED,
            "threshold_selection": "max_recall_at_validation_fpr", "max_validation_fpr": MAX_VALIDATION_FPR,
        })
        mlflow.log_metrics({f"validation_{k}": v for k, v in val_metrics.items()}
                           | {f"test_{k}": v for k, v in test_metrics.items()})
        info = mlflow.sklearn.log_model(          # skops: no pickle on load
            model, name="model", input_example=train[FEATURES].head(5),
            serialization_format="skops", registered_model_name=REGISTERED_MODEL)
        MlflowClient().set_model_version_tag(REGISTERED_MODEL, info.registered_model_version,
                                             "approval_status", "candidate_not_approved")
        mlflow.log_dict({"run_id": run.info.run_id, "dataset": snapshot, "features": FEATURES,
                         "model_version": info.registered_model_version,
                         "validation": val_metrics, "test": test_metrics}, "evaluation.json")
    print(json.dumps({"model_version": info.registered_model_version, "algorithm": best,
                      "test": test_metrics}, indent=2, default=str))
if __name__ == "__main__":
    main()

Verified behaviour. Running the script twice produces two runs and two registered versions with identical dataset digests, thresholds and test metrics. Tampering with a snapshot file stops training, and re-publishing different data under an existing version is rejected. A fixed random_state makes the random forest identical for any n_jobs; bit-for-bit reproducibility across OS, library builds and CPU architectures is still not guaranteed, so record the environment and define an acceptable tolerance. mlflow.sklearn.log_model uses name, because artifact_path is deprecated in MLflow 3.

5. Production banking architecture

Separate the offline development plane from the online decision plane. Governed ingestion lands extracts and events with lineage, encryption, retention and contracts. An offline feature pipeline produces versioned, point-in-time-correct features consistent with online definitions. A training platform runs reviewed code in pinned images and tracks parameters, metrics, artifacts and lineage. Independent validation challenges methodology, data, limitations, outcomes, stability and intended use. The registry and release pipeline promote only approved immutable versions, with manifests pinning version and digest. The online scoring service validates authenticated requests and scores with a known version. A separately governed decision engine combines score, rules, limits, velocity controls, step-up authentication and review queues. Monitoring feeds SLOs, data and model quality, outcomes, cost, security and customer impact into runbooks. For high-risk payment flows, a scoring failure must never silently become an approval; the fallback is an approved rules path, step-up or manual review.

6. Example inference service boundary

The service refuses to start unless MODEL_URI pins a numeric registry version: @champion, latest and run URIs are rejected, so a running instance cannot silently follow a moved alias. It also refuses a version not tagged approved, so an unvalidated candidate cannot be served by mistake. The model loads in the lifespan, passes a smoke prediction before readiness succeeds (a hard-coded "ready" proves nothing), and every response carries model name, version, producing run ID and feature-contract version for audit. Caller-supplied request IDs are accepted only in a bounded format because they are logged, and failures return 503 for the decision service to route to its fallback.

"""Illustrative inference boundary: returns a risk score, never an authorisation decision.
pip install fastapi uvicorn "mlflow>=3.4,<4" skops pandas
Runs behind the bank's gateway/mesh, which enforces mTLS or OAuth2 workload identity,
network policy, quotas and request-size limits.
"""
from __future__ import annotations
import logging
import math
import os
import re
import time
from contextlib import asynccontextmanager
from typing import Annotated
from uuid import uuid4
import mlflow
import mlflow.sklearn
import pandas as pd
from fastapi import FastAPI, Header, HTTPException, Request
from pydantic import BaseModel, ConfigDict, Field
logger = logging.getLogger("fraud-inference")
FEATURES = ["transaction_amount", "hours_since_last_txn", "customer_tenure_months",
            "merchant_risk_score", "is_foreign_country"]
FEATURE_CONTRACT = "card-fraud-features-v1"
PINNED_URI = re.compile(r"^models:/(?P<name>[\w.-]+)/(?P<version>\d+)$")  # no @alias
REQUEST_ID = re.compile(r"^[A-Za-z0-9-]{8,64}$")
class TransactionFeatures(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)
    transaction_amount: float = Field(gt=0, le=1_000_000)
    hours_since_last_txn: float = Field(ge=0, le=100_000)
    customer_tenure_months: int = Field(ge=0, le=1_200)
    merchant_risk_score: float = Field(ge=0, le=1)
    is_foreign_country: int = Field(ge=0, le=1)
@asynccontextmanager
async def lifespan(app: FastAPI):
    uri = os.environ["MODEL_URI"]
    if not (match := PINNED_URI.fullmatch(uri)):
        raise RuntimeError("MODEL_URI must pin a numeric version, e.g. models:/card_fraud/12")
    version = mlflow.MlflowClient().get_model_version(match["name"], match["version"])
    if version.tags.get("approval_status") != "approved":  # fail closed on unapproved artifacts
        raise RuntimeError(f"model version {version.version} is not approved for serving")
    model = mlflow.sklearn.load_model(uri)           # skops format: no pickle execution
    probe = pd.DataFrame([[100.0, 2.0, 24, 0.5, 0]], columns=FEATURES)
    if not 0.0 <= float(model.predict_proba(probe)[0, 1]) <= 1.0:
        raise RuntimeError("model failed startup smoke test")
    app.state.model, app.state.identity = model, {
        "model_name": version.name, "model_version": version.version,
        "run_id": version.run_id, "feature_contract": FEATURE_CONTRACT}
    yield
app = FastAPI(title="Fraud Risk Scoring", lifespan=lifespan)
@app.get("/health/live")
def liveness():
    return {"status": "alive"}
@app.get("/health/ready")
def readiness(request: Request):
    if getattr(request.app.state, "model", None) is None:
        raise HTTPException(503, "model_not_loaded")
    return {"status": "ready", **request.app.state.identity}
@app.post("/v1/fraud-score")
def score_transaction(body: TransactionFeatures, request: Request,
                      x_request_id: Annotated[str | None, Header()] = None):
    request_id = x_request_id if x_request_id and REQUEST_ID.fullmatch(x_request_id) else str(uuid4())
    started = time.perf_counter()
    try:
        row = pd.DataFrame([body.model_dump()], columns=FEATURES)
        score = float(request.app.state.model.predict_proba(row)[0, 1])
        if not (math.isfinite(score) and 0.0 <= score <= 1.0):
            raise ValueError("model returned a score outside [0, 1]")
    except Exception:
        logger.exception("scoring_failed request_id=%s", request_id)
        # The decision service routes this to the approved fallback (rules, step-up,
        # manual review). A technical failure must never become an approval.
        raise HTTPException(503, {"request_id": request_id, "message": "scoring_unavailable"})
    # No PAN, customer identifiers or raw features in logs.
    logger.info("score_completed request_id=%s latency_ms=%.2f", request_id,
                (time.perf_counter() - started) * 1000)
    return {"request_id": request_id, "fraud_score": score, **request.app.state.identity}

Integration requirements. Run behind the bank's gateway or mesh with mTLS or OAuth2 workload identity, segmentation, rate and size limits, secrets management and central audit. Test the model's output contract explicitly (predict may return labels; predict_proba returns probabilities). Add timeouts, concurrency limits, circuit breakers and graceful shutdown; keep idempotency and duplicate handling in the transaction orchestration layer; and test peak load, dependency outages, stale features, model-load failure and rollback. In production, set the approval tag only through the validated release workflow, never by hand.

7. Model risk, security, and governance controls

Independent validation and intended use

Validation should be proportionate to materiality, complexity and use, and should critically assess conceptual soundness, data and assumptions, implementation, outcomes, limitations and monitoring. Developer tests do not replace independent challenge where policy requires it. Record intended and prohibited uses, override procedures and revalidation triggers.

Security and privacy

Least-privilege roles for data scientists, operators, validators, approvers and deployers, with separation of duties.

Encryption in transit and at rest, managed keys and a secrets vault; no credentials in notebooks, code or MLflow parameters.

Private networking, gateway controls, service authentication and authorisation, rate limits and egress restrictions.

SBOMs, dependency and container scanning, signed build artifacts, provenance and patch management.

Data minimisation, tokenisation, retention and deletion, residency, purpose limitation and auditable access.

Tamper-evident audit events for dataset publication, registration, validation, approval, deployment, alias changes and rollback.

Monitoring

Service: availability, latency percentiles, throughput, timeouts, errors, saturation, dependency health.

Data: freshness, schema changes, nulls, ranges, feature distributions, missing-feature rates.

Model: score distribution, calibration, drift, stability, performance once delayed labels mature.

Business and customer: fraud loss, false declines and challenges, friction, analyst workload, segment impact.

Operations: model age, feature freshness, inference cost, alert quality, incident duration, rollback readiness.

Input drift is not concept drift, and neither proves deterioration. Treat drift alerts as investigation signals and confirm impact on matured outcomes before retraining or changing policy.

8. Controlled retraining, promotion, and rollback

Define retraining triggers in advance: schedule, label volume and maturity, monitored deterioration, or material data or product change.

Validate new data and labels; quarantine failed snapshots and preserve the approved version.

Train a candidate and compare it with the champion under the same temporal design, including operational and customer costs.

Obtain independent validation and the required business, risk, compliance, security and change approvals.

Promote through development, validation, staging, shadow or canary and production, recording each approver.

Pin model, feature contract, container, threshold or policy configuration and deployment manifest.

Roll back to the prior compatible release when guardrails are breached; rehearse it and verify feature compatibility.

Retire versions under retention and legal-hold rules without deleting evidence needed for investigation.

Putting it together

A reproducible release connects the chain: an approved, checksum-verified snapshot; a run recording exact code, environment, features and evaluation; a registered version with immutable lineage; attached validation and approvals; a deployment pinning exact model and policy versions; online scoring that reports model identity with every score; monitoring of operations and outcomes; and a rehearsed rollback to the prior approved release.

Takeaway. The decisive audit question is not "which model file was used?" but "for this transaction and decision, which data and feature values were available, which model and policy versions were active, what score was produced, what controls and overrides applied, and who authorised the release?"

Further reading: MLflow Model Registry (mlflow.org/docs/latest/ml/model-registry/); DVC remote storage (doc.dvc.org/user-guide/data-management/remote-storage); Federal Reserve SR 26-2, Revised Guidance on Model Risk Management (federalreserve.gov/supervisionreg/srletters/sr2602.htm); OCC Bulletin 2026-13; scikit-learn model evaluation; FastAPI security. This is an engineering reference, not legal, regulatory or model-validation advice.

Back to all blogs