ML Pipelines

You have a notebook that loads data, fills missing values with the median, scales features, trains a model, and prints an accuracy number. It works, so you ship it.

A month later someone retrains the model and gets different numbers. The median was computed on the full dataset, test rows included: that's leakage. The scaling parameters were never saved, so inference runs on different statistics than training did. The feature engineering got copy-pasted between the training script and the serving code, and the two copies drifted apart. Then a categorical column shows up in production with a value the encoder has never seen, and the whole thing falls over.

None of this is exotic; it's the ordinary way ML systems break in production. Every one of these failures traces back to the same root cause: the transformations weren't packaged with the model. A pipeline fixes that by fusing every transformation step and the model into one ordered, reproducible object, fit once and applied identically everywhere.

What a Pipeline Is

A pipeline is an ordered sequence of data transformations that ends in a model, each step feeding its output straight into the next. Fit the whole thing once on training data, and at inference time that same fitted pipeline transforms new data and produces predictions — no separate code path, no re-deriving statistics.

The pipeline guarantees:

• Transformations are fitted only on training data (no leakage)
• The same transformations are applied at inference time
• The entire object can be serialized and deployed as one artifact
• Cross-validation applies the pipeline per fold, preventing subtle leakage

Data Leakage: The Silent Killer

Data leakage is any information from the test set, or from the future, sneaking into training. It inflates whatever accuracy number you report, and pipelines block the most common forms of it by construction.

Leaky (wrong):

X = df.drop("target", axis=1)
y = df["target"]
 
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
 
X_train, X_test = X_scaled[:800], X_scaled[800:]
y_train, y_test = y[:800], y[800:]

The scaler saw test data before the model ever did. Its mean and standard deviation already include test samples, so the accuracy you measure afterward is inflated.

Correct:

X_train, X_test = X[:800], X[800:]
 
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

With a pipeline, this stops being something you have to remember to get right. It's handled by construction, every time.

sklearn Pipeline

sklearn's Pipeline chains transformers and a final estimator into one object, exposing .fit(), .predict(), and .score() that run every step in order.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
 
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression()),
])
 
pipe.fit(X_train, y_train)
predictions = pipe.predict(X_test)

When you call pipe.fit(X_train, y_train):

1. Scaler calls fit_transform on X_train
2. Model calls fit on the scaled X_train

When you call pipe.predict(X_test):

1. Scaler calls transform (not fit_transform) on X_test
2. Model calls predict on the scaled X_test

The scaler never sees test data during fitting; that's the entire point of wrapping it in a Pipeline.

ColumnTransformer: Different Pipelines for Different Columns

ColumnTransformer routes different subsets of columns to different preprocessing pipelines, then combines the results into one feature matrix. Real datasets need this: numeric columns want imputation and scaling, categorical columns want a different imputation strategy and encoding, and hand-rolling that split every time is how column-order bugs get in.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
 
numeric_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
 
categorical_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore")),
])
 
preprocessor = ColumnTransformer([
    ("num", numeric_pipe, ["age", "income", "score"]),
    ("cat", categorical_pipe, ["city", "gender", "plan"]),
])
 
full_pipeline = Pipeline([
    ("preprocess", preprocessor),
    ("model", GradientBoostingClassifier()),
])

handle_unknown="ignore" on OneHotEncoder is the one setting that actually matters for production. When a new category shows up (a city the model never saw during training), it returns a zero vector instead of raising and taking the request down with it.

Experiment Tracking

Experiment tracking is the record of what happened across every training run: which hyperparameters, which dataset version, what the metrics came out to, which code was actually running. A pipeline makes any single run reproducible, but it doesn't by itself tell you what you tried last week or why run 47 beat run 46.

MLflow is the most common open-source solution:

import mlflow
 
with mlflow.start_run():
    mlflow.log_param("max_depth", 5)
    mlflow.log_param("n_estimators", 100)
    mlflow.log_param("learning_rate", 0.1)
 
    pipe.fit(X_train, y_train)
    accuracy = pipe.score(X_test, y_test)
 
    mlflow.log_metric("accuracy", accuracy)
    mlflow.sklearn.log_model(pipe, "model")

Every run gets recorded with its parameters, metrics, artifacts, and the full model, so you can compare runs, reproduce any of them, and deploy whichever version actually won.

Weights & Biases (wandb) provides the same functionality with a hosted dashboard:

import wandb
 
wandb.init(project="my-pipeline")
wandb.config.update({"max_depth": 5, "n_estimators": 100})
 
pipe.fit(X_train, y_train)
accuracy = pipe.score(X_test, y_test)
 
wandb.log({"accuracy": accuracy})

Model Versioning

Model versioning is knowing, at any moment, exactly which model is in production, which is staging, and which one you retired last week. Experiment tracking tells you what you tried; versioning tells you what's actually live.

MLflow's Model Registry provides:

• Version tracking: Every saved model gets a version number
• Stage transitions: "Staging", "Production", "Archived"
• Approval workflow: Models must be explicitly promoted to production
• Rollback: Switch back to a previous version instantly

Data Versioning with DVC

DVC (Data Version Control) versions large data files the way git versions code: a content hash lives in git, the actual bytes live in remote storage. Git alone can't do this, since it chokes on large binary files and was never built to track them.

dvc init
dvc add data/training.csv
git add data/training.csv.dvc data/.gitignore
git commit -m "Track training data"
dvc push

In practice that remote storage is S3, GCS, or Azure, and the .dvc file left behind in git is tiny: just the hash. Switch git commits and run dvc checkout, and it restores the exact data that commit was trained on.

That's the real payoff: every git commit now pins both code and data, so reproducing last month's experiment stops being an archaeology project.

Reproducible Experiments

A reproducible experiment requires four things:

1. Fixed random seeds: Set seeds for numpy, random, and the framework (torch, sklearn)
2. Pinned dependencies: requirements.txt or poetry.lock with exact versions
3. Versioned data: DVC or similar
4. Config files: All hyperparameters in a config, not hardcoded

import numpy as np
import random
 
def set_seed(seed=42):
    random.seed(seed)
    np.random.seed(seed)
    try:
        import torch
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
    except ImportError:
        pass

From Notebook to Production Pipeline

The typical progression:

1. Notebook exploration: Quick experiments, visualizations, feature ideas
2. Extract functions: Move preprocessing, feature engineering, evaluation into modules
3. Build Pipeline: Chain transformations into a sklearn Pipeline or custom class
4. Config management: Move all hyperparameters into a YAML/JSON config
5. Experiment tracking: Add MLflow or wandb logging
6. Data validation: Check schema, distributions, and missing value patterns before training
7. Tests: Unit tests for transformers, integration tests for the full pipeline
8. Deployment: Serialize the pipeline, wrap in an API (FastAPI, Flask), containerize

Common Pipeline Mistakes

The code in code/pipeline.py builds a complete ML pipeline from scratch, in four steps:

Step 1: Custom Transformer

class CustomTransformer:
    def __init__(self):
        self.means = None
        self.stds = None
 
    def fit(self, X):
        self.means = np.mean(X, axis=0)
        self.stds = np.std(X, axis=0)
        self.stds[self.stds == 0] = 1.0
        return self
 
    def transform(self, X):
        return (X - self.means) / self.stds
 
    def fit_transform(self, X):
        return self.fit(X).transform(X)

Step 2: Pipeline from Scratch

class PipelineFromScratch:
    def __init__(self, steps):
        self.steps = steps
 
    def fit(self, X, y=None):
        X_current = X.copy()
        for name, step in self.steps[:-1]:
            X_current = step.fit_transform(X_current)
        name, model = self.steps[-1]
        model.fit(X_current, y)
        return self
 
    def predict(self, X):
        X_current = X.copy()
        for name, step in self.steps[:-1]:
            X_current = step.transform(X_current)
        name, model = self.steps[-1]
        return model.predict(X_current)

Step 3: Cross-Validation with Pipeline

The code demonstrates how cross-validation with a pipeline prevents data leakage: the scaler gets fit separately on each fold's training data, never on the fold it's about to score.

Step 4: Full Production Pipeline with sklearn

A full production pipeline: ColumnTransformer routing to multiple preprocessing paths, a model at the end, trained with proper cross-validation and experiment logging.

Step 5: Serialize and Verify

A fitted pipeline is a Python object like any other, so joblib can freeze it to disk and hand it to a process that never saw the training data:

import joblib
 
joblib.dump(pipe, "pipe.joblib")
 
loaded = joblib.load("pipe.joblib")
predictions_original = pipe.predict(X_test)
predictions_loaded = loaded.predict(X_test)
 
assert (predictions_original == predictions_loaded).all()

That assertion is the entire point. The loaded object carries the same fitted imputer, the same fitted encoder, the same trained model as the pipeline you trained. Nothing refits, nothing recomputes a statistic from a different sample, so training and production see exactly the same transformations. This is what "reproducible" cashes out to in practice: not a promise, a passing assert.

This lesson produces:

• outputs/prompt-ml-pipeline.md -- a skill for building and debugging ML pipelines
• code/pipeline.py -- a complete pipeline from scratch through sklearn

1. Build a pipeline that handles a dataset with 3 numeric columns and 2 categorical columns. Use ColumnTransformer to apply median imputation + scaling to numerics and most-frequent imputation + one-hot encoding to categoricals. Train with 5-fold cross-validation.

2. Deliberately introduce data leakage: fit the scaler on the full dataset before splitting. Compare the cross-validation score (leaky) to the pipeline cross-validation score (clean). How large is the difference?

3. Serialize your pipeline with joblib.dump. Load it in a separate script and run predictions. Verify the predictions are identical.

4. Add a custom transformer to the pipeline that creates polynomial features (degree 2) for the two most important numeric columns. Where should it go in the pipeline?

5. Set up MLflow tracking for the pipeline. Run 5 experiments with different hyperparameters. Use the MLflow UI (mlflow ui) to compare runs and pick the best model.

• scikit-learn Pipeline docs -- the official pipeline reference
• MLflow documentation -- experiment tracking and model registry
• DVC documentation -- data versioning
• Sculley et al., Hidden Technical Debt in Machine Learning Systems (2015) -- the seminal paper on ML systems complexity
• Google ML Best Practices: Rules of ML -- practical production ML advice