Building a Complete LLM Pipeline

Each previous lesson works on its own. The tokenizer trains, the tiny GPT pre-trains, the SFT dataset gets assembled, the reward model learns, DPO runs, evals score, the weights quantize, and an inference server comes up. Each of those is a notebook with its own conventions, output paths, and seed.

A frontier training run is not a notebook. Llama 3 405B took 30 million H100 hours over roughly 54 days. DeepSeek-V3 used around 2.8 million H800 hours. Over a span like that, one corrupted checkpoint, one data contamination, or one eval regression can cost a team a week of wall-clock and a month of GPU budget. Teams survive through pipeline hygiene: each stage has a deterministic input, a deterministic output, a manifest, a hash, and a gate.

This is the capstone, and you will not run it end-to-end on a laptop. You will write the orchestrator that coordinates the stages, the manifest that describes the run, the verifier that gates ship decisions, and the replay plan that lets a third party re-run your work from a single file. The code is small. The discipline is large.

The pattern holds from 100M to 1T parameters. The same four components (manifest, orchestrator, eval gate, artifact store) run Llama 3 and a hobby GPT alike. What changes is the size of the numbers inside each stage's config, not the shape of the pipeline.

The Twelve Stages

Each Phase 10 lesson is one stage. Here is the dependency graph.

Stages 07 and 08 can run in parallel, and so can 10 and 11, since both only read the stage 09 policy. The rest is a chain. A change in stage 02 (tokenizer) invalidates every downstream artifact. A change in stage 10 (eval) invalidates only the ship decision.

The Manifest

A manifest is a single file that describes a run well enough to replay it. Nothing the pipeline produces should depend on state outside it. The fields are boring and mandatory.

pipeline_version: 1.2.3
seed: 42
git_commit: a1b2c3d4
stages:
  01_tokenizer:
    recipe: bpe_32k
    input_hash: sha256:...
    output_hash: sha256:...
    wall_clock_sec: 3600
    cost_usd: 12

The output hash of stage N is the input hash of stage N+1, and a mismatch halts the pipeline. That is how you catch data corruption early. It is also how a teammate on a different continent checks that their replay produced the same artifact as yours.

In practice teams use a small YAML schema plus a manifest checker that diffs against the previous successful run. A delta outside the expected fields (cost, wall clock) is a red flag.

Artifact Typing

Each stage's output is a typed artifact: a named type with a known schema, not a directory blob or a pickle.

Typing prevents the most common failure, which is using a stage 08 output as a stage 06 input and shipping a DPO-trained model through the SFT path. With typed artifacts and typed stage signatures, that mistake becomes a compile-time failure instead of a day-five one.

The Eval Gate

Shipping does not mean training finished. It means training finished and the eval gate passed. You define the gate before the run starts.

gates:
  mmlu:      >= baseline + 0.5   # no regression
  humaneval: >= baseline + 1.0
  truthfulqa: >= baseline         # no drop
  safety_refusal_rate: <= 0.05
  kl_from_reference: <= 25.0
  cost_total_usd: <= 50000

Each gate is a numeric threshold. There are no "looks good" gates and no subjective sign-offs. If the gates pass, the artifact is marked shippable. If one fails, the run is held until a named reviewer overrides it, and that override is logged in the manifest.

Two gates catch most disasters. A regression gate (the new model must match or beat the previous one on core benchmarks) catches training bugs. A KL budget gate (the aligned policy must stay within X of its reference) catches alignment overcooking. Production pipelines tend to carry both.

The Orchestrator

A small piece of code that reads the manifest, dispatches stages, tracks artifacts, and halts on a contract violation. It is not Airflow and it is not Kubeflow. For pipeline hygiene you want something boring that you wrote yourself.

The job is narrow:

1. Resolve the DAG from the manifest.
2. For each stage, check whether the expected output already exists at the correct hash (skip if so).
3. Run the stage, capture stdout/stderr, and measure wall clock and cost.
4. Verify the output hash against the downstream stage's expected input hash.
5. On failure, write a partial manifest with the exact failing stage and exit nonzero.

That is a couple hundred lines of Python, and code/main.py in this lesson is a simulated version of it. In a real pipeline, torchrun or ray executes the individual stages on clusters, while the orchestrator itself runs on a single box.

Experiment Tracking and Artifact Storage

Two external systems anchor the pipeline.

Experiment tracker (wandb, neptune, mlflow). Logs loss curves, eval metrics, and system telemetry per stage. This is where you go to compare run A against run B three weeks later. Teams mostly use a hosted tracker, because writing your own eats time that belongs to training.

Artifact store (S3, R2, GCS). Immutable object store for checkpoints, datasets, tokenizers, and eval reports. Artifacts are addressed by hash, not by filename. A filename like latest.pt is a foot-gun, while ckpt-7b-step-20000-sha256:abc123.safetensors is a contract.

The orchestrator writes to both. The tracker is for humans looking at charts, and the artifact store is for the next stage looking up its inputs.

Costing

A frontier run has a dollar number attached, and budget discipline happens in two places.

Pre-run estimate. From the manifest, compute expected FLOPs (for pre-training: 6 x params x tokens), expected GPU hours (FLOPs / peak throughput / utilization), and dollar cost at the current rental rate. If the estimate exceeds the budget gate, the pipeline refuses to start.

In-run tracking. Log wall clock and cost to the manifest stage by stage, and check the remaining budget after each one. If a stage overran, the next stage's gate is evaluated against the new remaining budget. You do not want to find out you are out of money when the VC calls.

Llama 3's reported cost was $61M. DeepSeek-V3 reported $5.6M for the main pre-training run. The gap is mostly hardware efficiency plus mixture-of-experts, and the specific cost is visible because both teams tracked it per stage, not per run.

Reproducibility vs Determinism

These are different things. Reproducible means the same manifest, code, and infrastructure produce a checkpoint with equivalent downstream metrics. Deterministic means bit-identical output.

Modern LLM training is reproducible but not deterministic. Distributed reduce-order, GPU kernel non-determinism (cuBLAS, flash-attn), and mixed precision rounding combine to give floats that differ at the 1e-5 level between runs. The final metrics do not move, so this is fine. It hurts only if you try to debug with bit-level diffs. The fix is to log each stage's input hash, output hash, and headline metrics. If those match, the run is "reproduced" even when the weights are not bit-identical.

Rollback Plan

Before the run starts, write down what happens when each stage fails. There are three categories.

• Cheap to re-run (hours): tokenizer, eval, quantization, inference server. Just re-run.
• Medium (days): SFT, DPO, CAI. Keep the base model and re-run only the alignment stages.
• Expensive (weeks and millions of dollars): pre-training. Here the plan is not "re-run." It is "use the last good checkpoint and re-run the cheaper downstream stages with revised data."

Because stage dependencies are typed and hashed, the orchestrator can compute the rollback set automatically: invalidate the failed stage plus its descendants. A failure at stage 06 (SFT) invalidates 06, 07, 08, 09, 10, 11, 12. A failure at stage 11 (quantization) invalidates only 11 and 12. Settle this up front so nobody improvises while the team is exhausted at 4am.

Production Recipes Observed in 2026

Most frontier teams converged on the same skeleton.

• Tokenizer: 128k BPE with byte fallback. Trained on a small, balanced multilingual slice.
• Pre-training: 10-20T tokens, mostly web plus code plus synthetic. Muon or AdamW optimizer. FSDP2 or DeepSpeed ZeRO-3. Gradient checkpointing. BF16 weights, FP32 master.
• SFT: 500k-2M instruction pairs, mixed human and synthetic, with strict dedup against the eval set.
• Alignment: DPO or CAI + GRPO. RLHF only where the preference signal is too multidimensional for DPO.
• Eval: MMLU-Pro, MATH, HumanEval+, GPQA, SWE-Bench Verified, LiveBench, plus a private held-out set the public never sees.
• Quantization: 4-bit GPTQ or AWQ for serving, 8-bit for safety evals where accuracy deltas matter.
• Serving: vLLM, TensorRT-LLM, or in-house. Continuous batching. Speculative decoding. KV cache eviction.

The numbers change every six months. The skeleton does not.

The code here is an orchestrator and a manifest checker, not twelve training scripts. Each stage is simulated by a placeholder that emits an artifact with the right shape and a real SHA-256 hash. Running the orchestrator end-to-end proves the plumbing works before you spend GPU money on the real stages.

See code/main.py for the full implementation. The key pieces:

• Manifest dataclass: pipeline version, seed, git commit, stage records, gates, eval metrics, budget, and total cost.
• StageRecord dataclass: name, type, input hashes, output hash, wall clock, cost, and status.
• run(): walks the stage list in dependency order, checks that each input hash is in the store, hashes each output, and halts with halted_over_budget when cumulative cost passes the budget.
• gate(): compares each threshold in the manifest against eval_metrics and returns pass/fail with a reason per gate.
• ArtifactStore: an in-memory stub with put/get/has by hash, standing in for S3.
• simulate_stage(): the placeholder. It returns a deterministic blob plus a fixed wall clock and cost per stage type.

demo runs the twelve placeholder stages, prints the manifest summary, then runs the gate twice, once with passing metrics and once with a failing eval, to show what a held run looks like. Swap each placeholder for the real training script from the matching lesson and you have the skeleton of a real pipeline.

The canonical workflow has three commands.

python code/main.py plan    # validate manifest, compute cost estimate, print DAG
python code/main.py run     # execute stages, writing to manifest.out.yaml
python code/main.py gate    # read manifest.out.yaml, apply eval gates, ship-or-hold

Run plan first, every time. Most pipeline bugs show up at plan time: missing gate thresholds, stale hashes, budget overruns. plan is free and run is expensive, so catch bugs on the cheap side.

A note on the toy: the comments above describe the full workflow. In main.py, plan prints the DAG, the gates, and the budget (the cost estimate is Exercise 3). run prints the manifest as JSON to stdout, and gate reruns the simulated pipeline before applying the thresholds, so there is no manifest.out.yaml.

The output of gate ends in SHIP or HOLD, with a HOLD: <reason> line for each failed gate. A held run is not a failure. It is a decision point, where a named reviewer either overrides (and the override is logged) or approves the rollback.

This lesson produces outputs/skill-llm-pipeline-reviewer.md. Feed it a proposed pipeline manifest and it checks the contracts: stage typing, hash chain, gates, rollback plan, and cost estimate. It refuses to approve a manifest with a missing eval gate, an unbounded KL budget, or a run that mixes eval and training data.

1. Extend the orchestrator to support parallel execution of stages 07 and 08. Use the stdlib concurrent.futures module. Confirm the final manifest records both stages' outputs and that stage 09's input hash is a deterministic combination of both.

2. Add a "contamination check" gate. Given the eval dataset hash and the training dataset shards, compute the overlap (exact string match or 13-gram match). The gate fails if overlap exceeds 0.1%. Feed it a contaminated training set and confirm the gate holds the run.

3. Implement a cost estimator from first principles. For stage 04 (pre-training), estimate FLOPs as 6 x params x tokens, assume 40% MFU (model FLOPs utilization) on H100 at 989 TFLOPs BF16, at $2.50/GPU-hour. Report the estimate for a 7B model trained on 2T tokens. Compare to published Llama 2 numbers.

4. Build a partial rollback. Simulate a failure at stage 09 (CAI), then re-run stages 09 through 12 while leaving 01-08 cached. The orchestrator should detect the cached artifacts by hash and skip them. Measure wall-clock saved versus full re-run.

5. Add observability. Emit OpenTelemetry spans for each stage, with attributes for params, tokens seen, loss, and cost. Pipe the spans to a local collector. The point is not dashboards; the point is that every stage's health is traceable from a single trace ID.

• Dubey et al. (2024). The Llama 3 Herd of Models: the most detailed public description of a frontier pipeline, including data, training, alignment, and eval.
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report: an efficiency-first pipeline at roughly 1/10th the cost of Llama 3 class training.
• Kaplan et al. (2020). Scaling Laws for Neural Language Models: the original compute-data-params scaling relationship.
• Hoffmann et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla): the correction to Kaplan that recalibrated modern data budgets.
• PyTorch FSDP2 documentation: the distributed training primitive replacing FSDP1 in PyTorch 2.4+.
• Weights & Biases LLM Reports: real manifests and experiment tracker output for open-source LLM runs, useful as plagiarizable templates.