Capstone 07 — End-to-End Fine-Tuning Pipeline (Data to SFT to DPO to Serve)

A fine-tuning pipeline is the chain of steps that turns a raw corpus and an open base model into a served, evaluated, documented model: data prep, supervised fine-tuning (SFT), preference tuning, quantization, serving, and eval. Most serious AI teams in 2026 keep one on tap. They aren't shipping a frontier base model. The measurable wins sit downstream: domain SFT, DPO against labeled preferences, distilled drafts for speculative decoding, and serving with EAGLE-3.

The tooling is mature. Axolotl v0.8 handles multi-GPU SFT configs, TRL 0.15 handles DPO and GRPO, and Unsloth gives you fast single-GPU iteration. vLLM 0.7 with EAGLE-3 pushes decode throughput 2-3x without quality loss. What's left is craft: the YAMLs, the data hygiene, and the eval discipline.

You will take an 8B base (Llama 3.3, Qwen3, or Gemma 3) through SFT and then DPO on task-specific data, quantize it for serving, and measure the gains against lm-evaluation-harness, RewardBench-2, MT-Bench-v2, and MMLU-Pro. You will also write a model card under the Model Openness Framework (MOF). Reproducibility is the point: one command reruns the whole pipeline.

The pipeline has five stages, and each one's output is the next one's input.

• Data: dedup (MinHash / Datatrove), a quality filter (Nemotron-CC style classifier), a PII scrub, and a split-hygiene check for contamination from public benchmarks.
• SFT: an Axolotl YAML, ZeRO-3 on 8xH100, a cosine schedule, packed sequences, 2-3 epochs.
• DPO or GRPO: a TRL config, 1 epoch, preference pairs that are either human-labeled or model-judged, and a beta to tune.
• Quantize: GPTQ, AWQ, and GGUF, so you can deploy to whatever runtime the target needs.
• Serve: vLLM 0.7 with EAGLE-3 speculative heads (or SGLang with SpecForge), a K8s deployment, and an HPA on queue-wait.

The ablations are the deliverable: SFT-only vs SFT+DPO vs SFT+GRPO on three task-specific benchmarks. On the serving side, report tokens/s at batch 1 / 8 / 32, the EAGLE-3 acceptance rate, and $/1M tokens. Add a safety eval (Llama Guard 4 pass rate), and a model card covering bias evaluations, reproducibility seeds, and data licensing.

• Data: Datatrove for dedup, Nemotron-CC classifier for quality, Presidio for PII
• Base: Llama 3.3 8B, Qwen3 14B, or Gemma 3 12B
• SFT: Axolotl v0.8 with ZeRO-3, Flash Attention 3, packed sequences
• Preference tuning: TRL 0.15 for DPO or GRPO; Unsloth for single-GPU iteration
• Quantization: GPTQ (Marlin), AWQ, GGUF via llama.cpp
• Serving: vLLM 0.7 with EAGLE-3 speculative decoding (or SGLang 0.4 + SpecForge)
• Eval: lm-evaluation-harness, RewardBench-2, MT-Bench-v2, MMLU-Pro
• Safety eval: Llama Guard 4, ShieldGemma-2
• Infrastructure: Kubernetes + NVIDIA device plugin, HPA on queue-wait metric
• Observability: W&B for training, Langfuse for inference

1. Data pipeline. Run Datatrove dedup on the raw corpus, apply a Nemotron-CC-style quality classifier, and scrub PII with Presidio. Write train/val splits with an explicit seed.

2. Contamination check. For every validation split, compute MinHash against the MMLU-Pro, MT-Bench-v2, and RewardBench-2 test sets. Reject any overlap.

3. Axolotl SFT. Write a YAML with ZeRO-3, FA3, and sequence packing. Train 2-3 epochs on 8xH100 and log to W&B.

4. TRL DPO / GRPO. Start from the SFT checkpoint and run one epoch of DPO on preference pairs (or GRPO with a verifiable reward on math/code). Sweep beta.

5. Quantize. Produce three quants: GPTQ-INT4-Marlin, AWQ-INT4, and GGUF-Q4_K_M for llama.cpp. Record size and nominal throughput for each.

6. Serve with speculative decoding. Configure vLLM 0.7 with EAGLE-3 draft heads trained via Red Hat Speculators. Measure acceptance rate and tail latency at batch 1 / 8 / 32, then report $/1M tokens against Anthropic / OpenAI on the same eval.

7. Eval matrix. Run lm-eval-harness, RewardBench-2, MT-Bench-v2, and MMLU-Pro on base, SFT-only, SFT+DPO, and SFT+GRPO. Produce a table.

8. Safety eval. Measure the Llama Guard 4 pass rate on the dev set and add a ShieldGemma-2 output filter.

9. Model card. Fill in the MOF template: data, training, eval, safety, license, and a reproducibility section with the YAMLs and commit SHAs.

main.py in this lesson is a runnable scaffold of the same DAG: each stage is a function that consumes the previous stage's artifact by content hash, and python main.py prints the manifest, eval report, and endpoint summary. The numbers below are what a full run looks like.

$ ./pipeline.sh config/llama3.3-8b-domainX.yaml
[data]    300k raw, 282k deduped, 256k quality-filtered, 255k after PII scrub (seed=7)
[SFT]     3 epochs, 8xH100, 6h12m, val loss 1.42 -> 1.03
[DPO]     1 epoch, beta=0.08, 4xH100, 1h42m
[quant]   GPTQ-INT4 4.6 GB, AWQ-INT4 4.8 GB, GGUF-Q4_K_M 5.1 GB
[serve]   vLLM 0.7, EAGLE-3 acceptance 0.74, p99 126ms @ bs=8
[eval]    MMLU-Pro +3.2, MT-Bench-v2 +0.41, RewardBench-2 +0.08
[card]    model-card.md generated under 2026 MOF

outputs/skill-finetuning-pipeline.md describes the deliverable. One command runs data through SFT, DPO, quantization, serving, and eval, and emits a model card plus the served endpoint.

1. Run SFT-only, SFT+DPO, and SFT+GRPO on the same task-specific benchmark. Report which preference method wins and by how much.

2. Swap Llama 3.3 8B for Qwen3 14B. Measure the $/1M tokens at matched quality.

3. Measure the EAGLE-3 acceptance rate on domain data vs generic ShareGPT. Report the delta and what it means for latency budgets.

4. Inject 1% contamination (leak MMLU-Pro answers into the training data) and rerun eval. Watch MMLU-Pro accuracy jump unrealistically, then build a contamination-check CI gate that catches it.

5. Add LoRA SFT as an alternative to full fine-tuning. Measure the quality gap at 10x lower memory.

• Axolotl documentation: the reference SFT / DPO trainer
• TRL documentation: DPO and GRPO reference implementations
• Unsloth: single-GPU iteration reference
• DeepSeek R1 paper (arXiv:2501.12948): GRPO methodology
• vLLM + EAGLE-3 documentation: reference serving stack
• SGLang SpecForge: alternate speculative-decoding trainer
• The Model Openness Framework (arXiv:2403.13784): the LF AI & Data Generative AI Commons paper that defines the open-release grading classes
• lm-evaluation-harness: canonical eval runner