From Notebook to Production: The Complete ML Project Checklist

From Notebook to Production: The Complete ML Project Checklist

A notebook proves that an idea can work. Production proves that it works reliably, repeatedly, securely, and at an acceptable cost.

Moving a model into production is not a matter of saving a file and exposing an endpoint. It needs a dependable system around the model: trustworthy data, reproducible pipelines, meaningful evaluation, secure deployment, monitoring, operational ownership and a controlled lifecycle. This checklist follows that journey from problem definition to retirement, for engineering, data science, platform, product, security and governance teams. Apply each control in proportion to the system's impact, risk and operating environment.

Scope. The checklist targets predictive ML. If the system uses LLMs or prompts, also version prompts, model identifiers, retrieval corpora and output schemas as release artifacts, and evaluate them on a fixed, adversarial golden set (including prompt injection) before each change. NIST's Generative AI Profile (NIST AI 600-1) complements the AI RMF for those risks.

1. Frame the problem and define success

Start with the decision the model will support, not with an algorithm. A model can score well offline and still fail to improve the business process.

Define the business problem, users, intended use, and the decision or action a prediction will influence.

Specify the task (classification, regression, ranking, forecasting, recommendation, anomaly detection), the prediction unit, frequency, decision horizon and when a prediction must be available.

Choose a primary metric that reflects the task, and business outcome measures such as revenue lift, cost avoided, time saved or loss prevented.

Define guardrails (false-positive and false-negative costs, latency, availability, fairness, safety, privacy, customer impact) and set acceptance thresholds before experimenting, including whether calibration and threshold selection matter.

Document the consequences of wrong predictions, who can override them, and the fallback process.

Compare ML with rules, heuristics, manual review or simple statistics, and confirm ML is justified.

Stage output: problem statement and intended use · metric, guardrails and acceptance thresholds · decision workflow and fallback · stakeholder agreement.

2. Understand, validate, and govern the data

Data quality and availability are production concerns, not one-time cleaning tasks. Training data must represent what is genuinely available at prediction time.

Inventory each source: owner, meaning, refresh cadence, retention, access method and dependency. Build a data dictionary with semantics, units, valid ranges, label definitions and the time each value becomes available.

Profile missing values, duplicates, invalid labels, outliers, inconsistent formats, unexpected categories and shifts in volume or distribution; check representativeness across segments, geographies and time, including sampling bias and class imbalance.

Prevent target and temporal leakage: a feature must exist at the actual prediction timestamp, so use point-in-time-correct joins for historical features.

Choose splits that mirror deployment: time-based or grouped where random splits would leak across time, users, devices or entities. Keep the final test set out of model selection and tuning. When cross-validating time-dependent data, use forward-chaining folds (train on the past, validate on the future); random k-fold reintroduces the leakage the split was designed to prevent.

Agree data contracts or schema checks with producers that catch breaking changes, unexpected nulls, invalid ranges and stale or incomplete feeds before training or inference.

Record lineage, transformations, source versions, approvals and snapshots or reproducible extraction criteria; apply privacy, consent, minimisation, retention, residency, licensing and regulatory requirements, restricting sensitive data and de-identifying where appropriate.

Stage output: source inventory and data dictionary · quality and representativeness report · leakage and split review · contracts, lineage and privacy review.

3. Establish a meaningful baseline

A baseline shows whether the model adds value. Compare against the existing business process as well as simple technical alternatives.

Build a rule-based, heuristic, naive or simple statistical baseline, and a simple interpretable model (logistic regression, a decision tree, a straightforward forecaster) where useful.

Evaluate every candidate with the same split, metric definitions and protocol, and quantify operational or business improvement over the current workflow, not only score improvement.

Record training and inference cost, latency, complexity, maintainability and failure modes; define the minimum improvement that justifies added complexity. If the baseline is sufficient, do not deploy a more complex model without a clear reason.

Stage output: baseline implementation · comparable evaluation report · business-value estimate and go/no-go threshold.

4. Make experiments reproducible and traceable

Another engineer should be able to reconstruct how a model was produced and why that version was approved. A notebook is for exploration; it must not be the only record of the system.

Keep code in version control with reviewed changes and tagged releases; separate exploratory notebooks from reusable modules and executable training and evaluation pipelines.

Move paths, thresholds and feature lists into validated configuration; never hard-code credentials.

Lock dependencies and record runtime, OS, hardware, library versions and container image; version datasets or record immutable snapshots, extraction queries, feature definitions and transformation code.

Track each run's parameters, metrics, artifacts, code commit (and whether the working tree had uncommitted changes), data and feature versions and environment.

Set seeds where useful, documenting that some hardware and distributed operations remain nondeterministic; store reports, plots, artifacts, logs and configuration as traceable outputs.

Make a clean run from a fresh environment part of the reproducibility check.

Stage output: version-controlled repository · reproducible environment and run instructions · experiment and artifact lineage · successful clean rerun.

5. Build modular, reliable ML pipelines

Replace hidden notebook state with explicit, testable stages, and make training and inference agree on feature definitions and transformations.

Separate ingestion, validation, transformation, feature engineering, training, evaluation, registration and deployment into maintainable components with typed or schema-validated inputs and outputs (names, types, units, nullability, ranges).

Separate code from configuration and make stages idempotent where practical, so safe retries do not create duplicate or inconsistent results.

Reuse one transformation definition for training and inference where feasible; where implementations differ, add parity tests to detect training-serving skew.

Adopt a feature store when shared, point-in-time-correct features and online/offline consistency justify its operational cost; it is not mandatory.

Orchestrate workflows with retries, timeouts, dependency management, backfills and failure notifications; make runs traceable with run IDs, timestamps, input versions, output artifact IDs and structured logs.

Define freshness expectations and stop stale, partial or failed upstream loads from silently feeding production predictions.

Stage output: modular training and inference code · validated feature schema and transformations · orchestrated pipeline with retry and failure behaviour · traceable artifacts.

6. Test data, model behaviour, and the surrounding system

ML testing covers more than whether a model returns a prediction: test the data, transformations, statistical behaviour, serving interface, infrastructure and downstream decision workflow.

Unit-test feature engineering, preprocessing, post-processing and business rules; add data tests for schema, types, ranges, null rates, categories, duplicates, freshness and distribution change.

Test pipeline integration, idempotency, retries, backfills and recovery from partial failure, and run a small training smoke test to catch broken dependencies and configuration.

Keep fixed evaluation datasets and regression tests, and compare candidates with both acceptance thresholds and the production model using task-appropriate metrics (precision, recall, F1, ROC-AUC or PR-AUC, RMSE/MAE, ranking metrics) and segment results.

For imbalanced classification, never rely on accuracy: evaluate precision-recall trade-offs, choose the threshold from business costs and capacity, and check calibration when scores are read as probabilities.

Test serialisation, loading, dependency compatibility, input validation, output schema and numerical stability; test API contracts, authentication, authorisation, rate limits, timeouts and error responses.

Load-test expected and peak traffic (concurrency, latency percentiles, memory, CPU/GPU, autoscaling) and exercise failures: malformed or missing features, stale data, unavailable dependencies, timeouts, model-load failure and resource exhaustion, verifying safe fallback.

Perform risk-appropriate fairness, robustness, explainability, privacy and security assessments, documenting groups or conditions where performance is weaker.

Stage output: unit, data, integration and pipeline tests · model-quality and regression report · API, load and failure-mode tests · responsible-AI and security review.

7. Package, document, and register the model

A model is not production-ready while it exists only as a file on a developer's laptop. It needs a deployable, identifiable artifact and enough metadata to reproduce, review and operate it.

Package the model with its preprocessing and post-processing, runtime dependencies and a defined input/output signature.

Store artifacts in controlled, durable storage and register approved versions in a model registry, recording training data reference, features, commit, parameters, metrics, evaluation report, environment, owner and approval status.

Document intended and out-of-scope uses, assumptions, limitations, data characteristics and known risks in a model card.

Treat deserialisation as code execution: prefer non-executable formats (ONNX, safetensors, skops for scikit-learn) over pickle or joblib, verify digests or signatures before loading, scan packages and dependencies, and restrict who can publish or promote.

Use immutable version identifiers for traceability; registry aliases or tags are mutable lifecycle labels and must not be the only audit record. Define states (experimental, candidate, validated, approved, production, retired) with explicit promotion criteria.

Stage output: deployable package · registry entry and immutable version · model card and evaluation evidence · integrity and approval record.

8. Design the production serving architecture

Choose the serving pattern by how predictions are consumed, how fresh they must be, and what happens when they are unavailable.

Batch scoring for scheduled bulk predictions; online APIs for low-latency decisions; streaming for event-driven processing; edge deployment when inference must run near the device or offline.

Define SLOs for availability, latency percentiles, throughput, freshness and recovery time, with capacity and scaling assumptions.

Design API contracts, validation, response schemas, versioning, authentication, authorisation, rate limiting and backward compatibility.

Use bounded timeouts, circuit breakers, bulkheads and graceful degradation. Retry only idempotent operations, with exponential backoff, jitter and a retry budget, to avoid retry storms; never use unbounded queues.

Plan concurrency, autoscaling, resource limits, warm-up, model loading, readiness checks and graceful shutdown; cache predictions only when reuse is safe and keys include every relevant input and the model version.

Define behaviour when the model or a dependency is unavailable (safe default, rules-based fallback, deferred processing or human review), separate training from serving, and document data flow, trust boundaries, dependencies and failure domains.

Stage output: serving pattern and architecture diagram · latency, throughput and availability targets · API contract and resilience design · fallback and capacity plan.

9. Deploy through controlled automation

Automated delivery reduces manual error, but an ML release must validate both software correctness and model quality; a green build is not evidence that a model is safe to promote.

Automate linting, unit tests, dependency checks, container builds, integration tests and artifact publishing; run data validation, model evaluation, threshold checks and required approvals before promotion.

Deploy to a production-like staging environment that exercises the real serving image, artifact, configuration, permissions and dependencies together, using infrastructure as code and controlled configuration promotion.

Keep secrets in a managed store; use least-privilege service identities, key rotation and encryption in transit and at rest; scan code, containers, dependencies and infrastructure configuration.

Roll out progressively with shadow traffic, canary, blue-green or A/B tests, stating what each is meant to validate. Halt or roll back automatically on leading indicators (errors, latency, invalid outputs, severe safety signals); delayed-label metrics cannot drive automatic rollback until labels mature.

Verify that the deployed artifact digest and model version match the approved release record, and ensure more than one authorised person can execute or recover a release.

Stage output: CI/CD pipeline with quality gates · staging evidence · infrastructure-as-code and security controls · progressive rollout and tested rollback.

10. Monitor production: service, data, and model quality

Deployment is the start of operations. A model can stay online while its inputs, predictions or real-world usefulness deteriorate, so monitor the whole service and separate signals that need different responses.

Use structured logs, metrics, traces, correlation IDs and model and data version tags so a single prediction can be traced without logging sensitive payloads. Every alert needs a severity, owner, runbook and escalation path; avoid alerting on every statistical fluctuation by using thresholds, windows and context suited to the use case.

Stage output: service and infrastructure dashboards · data-quality, drift and model-quality monitoring · business outcome and cost metrics · alerts, runbooks and traceability.

11. Retrain, improve, and retire safely

Retraining is an evidence-based release, not an automatic response to drift. New data can be stale, biased, mislabelled or less representative than the current training set.

Define triggers: schedule, sufficient new labelled data, verified degradation, meaningful drift or a documented business change. Investigate pipelines, instrumentation and population changes before deciding to retrain.

Validate incoming labels and feedback for correctness, delay, sampling bias and representativeness; rebuild from versioned code, data, features and configuration with complete lineage.

Compare the candidate with the champion on a fixed evaluation suite and recent data, including segment, fairness, calibration, robustness and cost checks; use champion/challenger, shadow or controlled experiments when they answer the deployment question.

Require risk-appropriate approval; never promote a model just because a scheduled job finished. Keep the previous stable model and a tested rollback, and define when rollback, fallback or human review applies.

Archive retired versions, data, reports and approvals per retention and audit policy, delete data when rules require it, and decommission deliberately, including consumers, credentials, scheduled jobs and monitoring.

Stage output: retraining policy and validated feedback process · candidate-versus-champion report · approval and rollback evidence · archival and retirement plan.

12. Establish ownership, security, and governance

Production ML is a socio-technical system. Accountability covers the data, model, software, operational process and the people affected by its outputs.

Name owners for the business outcome, data sources, feature pipeline, model, serving service, infrastructure, security and on-call; define who can change data, approve a model, promote a release, accept residual risk, disable the system and communicate an incident.

Maintain architecture and data-flow diagrams, dependency inventories, model documentation, known limitations and runbooks.

Apply least privilege, role-based access, network controls, encryption, secret rotation, audit logging and separation of duties where warranted.

Assess ML-specific threats (data poisoning, training-data exposure, model theft, adversarial inputs, insecure serialisation, endpoint abuse) and manage third-party data, pretrained models, libraries, hosted services, licences and supply-chain changes.

Define privacy, legal, regulatory, retention and records obligations for the actual jurisdiction and industry, and set risk-based requirements for explainability, fairness, human oversight, appeal or override and communication of limitations.

Keep an inventory of deployed models with periodic and change-triggered reviews; define incident procedures (detection, triage, containment, rollback or shutdown, recovery, communication, learning) and backup, disaster-recovery and continuity expectations for data, artifacts, registries, configuration and serving.

Stage output: ownership and decision-rights map · security and privacy assessment · governance and model inventory · incident, continuity and review procedures.

A practical production-readiness gate

Before promotion, require evidence, not verbal confirmation, for each applicable control. The gate is risk-based: a low-impact internal ranking tool needs a lighter process than a model used in credit, healthcare, employment or other consequential decisions.

Are intended use, affected users, business owner and decision workflow documented? (§1)

Were the success metric, business outcome, guardrails and acceptance thresholds agreed in advance? (§1)

Are data sources, permissions, lineage, quality, freshness, representativeness and point-in-time correctness validated, with leakage assessed and splits that reflect deployment? (§2)

Does the candidate meet acceptance criteria and add meaningful value over the baseline and current production model? (§3, §6)

Can training and evaluation be reproduced from versioned code, data references, configuration and environment? (§4, §5)

Have data, feature, model, calibration, API, integration, load, security and failure-mode tests passed? (§6)

Is the model packaged in a safe format, versioned, registered, documented and traceable to its run and evidence? (§7)

Are capacity, latency, availability, resilience, fallback and recovery defined and tested? (§8)

Are CI/CD gates, staged rollout, approval, halt criteria and rollback tested? (§9)

Are service health, data quality, drift, delayed-label performance, business outcomes and cost observable? (§10)

Is there a controlled process for feedback, retraining, review, rollback and retirement? (§11)

Are owners, on-call, runbooks, incident response, privacy and security controls and risk approvals in place? (§12)

Gate rule. If a critical answer is "no", do not promote the model until the gap is closed or an explicitly authorised, documented risk decision is made.

The operating principle

A production ML system is more than a trained model: it is a versioned, monitored decision system whose data, code, model, infrastructure, outcomes and ownership stay understandable over time. The goal is not to eliminate every failure but to detect failures early, limit their impact, recover safely and keep demonstrating that the system remains fit for its intended use. That depends on disciplined validation, reproducibility, controlled change, observability and clear accountability working together.

Further reading: Google for Developers, Rules of Machine Learning (developers.google.com/machine-learning/guides/rules-of-ml) and Productionization (developers.google.com/machine-learning/managing-ml-projects/production); Google Cloud Architecture Center, MLOps: Continuous delivery and automation pipelines in machine learning (docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning); MLflow Model Registry (mlflow.org/docs/latest/ml/model-registry/); NIST AI Risk Management Framework (nist.gov/itl/ai-risk-management-framework) and Generative AI Profile, NIST AI 600-1.

Back to all blogs