Large Corpus Downloader

A corpus downloader pulls a training corpus onto disk shard by shard and makes two guarantees: a dropped connection resumes from verified bytes, and near-duplicate documents get labelled before anyone tokenizes them. Skip either one and you find out at 200 GB.

The first time you train on a 200 GB corpus the network drops at percent 41 and the script exits with a urllib exception. The second time it drops at percent 78. By percent 99 you have rewritten the loop three times. Both failures have well-known solutions, and both get skipped because the pipeline starts as a one-line requests.get call that grew teeth.

Resume is an HTTP problem. The server has to honour Range, the client has to track the verified offset against an on-disk record, and that offset has to survive process death. If the offset and the file diverge by even one byte, the resumed download writes garbage, and the corruption only shows up during tokenization.

Deduplication is a signature problem. Exact-hash dedup misses near-duplicates: the same Wikipedia article with three different boilerplate footers, the same code file with a different license header, the same blog post with a tracking parameter on every link. MinHash plus LSH catches these at sub-linear cost, paying one signature per document and one bucket lookup per signature.

Streaming with urllib

The standard-library urllib.request.urlopen returns a file-like object. StreamingDownloader.download reads it in 64 KiB chunks and appends each chunk to <cache_dir>/<shard_id>.zst, so the compressed shard lands on disk in full before anything is decompressed. It then re-reads that file through zstandard.ZstdDecompressor().stream_reader, and the document iterator pulls decompressed lines from the reader. Only decompression is streamed, and it streams from disk rather than from the network. Neither the compressed nor the decompressed shard is ever held whole in memory: the memory cost is one chunk buffer, the line buffer, the MinHash signature for the current document, and the LSH index.

Resume with Range

Resume works because the downloader writes two files per shard: the shard itself and a .partial.json checkpoint. The checkpoint records verified_bytes, expected_size, sha256_prefix_hex (computed over the first verified_bytes bytes), and the source URL. On startup the downloader reads the checkpoint checks that the file size equals verified_bytes, and recomputes the sha256 over the on-disk bytes. It resumes only if both checks pass. If the hash is wrong, the partial is discarded and the download restarts from byte zero. A corrupted partial can't slip through, because the verified bytes are checked, not assumed.

MinHash plus LSH

MinHash estimates the Jaccard similarity of two sets in fixed space. For a document, the set is the shingles (overlapping n-grams) of its text. The signature is k minimum hash values, one per independent hash function. Two documents with Jaccard similarity s agree on any single component of the signature with probability s.

LSH then groups the k components into b bands of r rows each, where k = b * r. Two documents collide in at least one band with probability 1 - (1 - s^r)^b, which is a sharp threshold around the value of s you tune (b, r) to. The threshold sits near s = (1/b)^(1/r). For the demo parameters k = 128, b = 32, r = 4 that is about 0.42, a loose setting that flags documents sharing well under half their shingles. A stricter threshold near 0.7 needs fewer, longer bands, for example b = 16, r = 8.

Shard manifest as a contract

The manifest is the output downstream stages read. (Shards and checkpoints also persist in cache_dir, but nothing downstream should trust them directly.) For each shard it holds the URL, the decompressed byte count, the document count, the unique document count after dedup, and the sha256 of the final shard file. Downstream tokenization reads the manifest, not the directory listing. If a shard is missing or its sha256 is wrong, the manifest tells the next stage to refuse to start. That is the difference between "the data is downloaded" and "the data is downloaded and verifiable".

code/main.py implements seven pieces:

• ShardPlanner - reads a list of shard URLs and produces planned manifest entries.
• StreamingDownloader - opens a urllib stream with optional Range, appends chunks to <shard_id>.zst in the cache directory, rewrites the .partial.json checkpoint on every chunk, and verifies the sha256 prefix on resume.
• ZstdDocIterator - wraps the file-like stream in zstandard.ZstdDecompressor and yields one document per line.
• MinHasher - produces a k-component signature for a string using a fixed family of hash seeds.
• LSHIndex - buckets signatures by band and reports collisions.
• Dedup - combines hasher and index to label each document keep or near_duplicate along with the matching shard id.
• ManifestWriter - collects per-shard stats and writes manifest.json plus a manifest.json.lock holding the manifest's sha256.

The demo at the bottom of the file builds a small synthetic corpus on disk, compresses it with zstandard, downloads it through a file:// URL, deduplicates, and prints the manifest. The demo builds its MinHasher with shingle_width=3 because its sentences are short, while the class default is 5.

Run it:

python3 code/main.py

The script exits zero and prints a manifest summary.

Four patterns carry this lesson to real corpora.

Checkpoint before write. .partial.json is rewritten before each chunk is appended to the shard. The checkpoint goes to a .json.tmp file and is swapped in with os.replace, so it is never half-written. The order matters because the opposite one lets bytes land in the shard that the checkpoint doesn't cover, and the next resume would append from the wrong offset. The cost shows up on a crash between the two steps: verified_bytes is then larger than the file, _verify_partial sees the size mismatch, and the shard restarts from zero. That wastes one shard's progress and never corrupts it. It's the same discipline as a write-ahead log. The code never calls fsync, so a power loss can still reorder the writes on disk. A production version should fsync the checkpoint file and the shard before trusting either.

Sharded LSH index. A single LSH index over the whole corpus does not fit in RAM at the 200 GB scale. Partition the index by the first band hash, store the partitions on disk, and consult only the partition a new signature would land in. That costs one extra disk read per document, and in exchange the LSH index stops being a hard memory ceiling.

Tombstone, not delete. Record dropped duplicates in the manifest with verdict near_duplicate and the shard id of the document they collided with. Deleting them loses the link between the duplicate and its keeper. A tombstone preserves the audit trail and lets a downstream pass change its mind about the threshold.

Per-shard sha256 in the manifest, plus a manifest sha256. The manifest gets its own content hash (stored in manifest.json.lock), and downstream stages verify that hash before they trust the per-shard entries. Without it the manifest is a silent attack surface: someone who can edit a single file can corrupt the whole pipeline.

Three habits that hold up in a real pipeline:

• Resume on every CI run. CI runners are ephemeral, so the downloader has to assume a fresh disk on each run and recover from cache or remote. cache_dir is a first-class constructor parameter of StreamingDownloader, so point it at a persistent CI cache. The code has no CLI flags.
• Dedup before tokenization. Tokenization is expensive, and running it twice on the same document doubles the cost for the same loss curve. Dedup sits upstream of tokenization, not downstream.
• Manifest as merge gate. The training run reads the manifest sha256 from a pinned commit, and a new dataset version requires a new manifest commit. The link between code and data is git, not folklore.

On a real project, outputs/skill-corpus-downloader.md would describe which URLs feed the downloader, how the checkpoint directory is laid out, what shingle width and (k, b, r) triple the dedup uses, and where the manifest lives in version control. This lesson ships the engine.

1. Add a --shingle-width flag and measure how the dedup verdict changes at widths 3, 5, and 9. Defend the default you pick.
2. Add gzip support next to zstd by sniffing the magic bytes. The caller should not have to specify the codec.
3. Add a --resume-only mode that refuses to start a fresh download when no checkpoint exists. In CI this keeps one run from accidentally re-pulling 200 GB.
4. Move the LSH index to a shelf or sqlite file and compare throughput against the in-memory variant.
5. Add a manifest sha256 check on startup. The downloader should fail closed if the manifest on disk disagrees with the manifest hash in manifest.json.lock.

• RFC 7233 - HTTP Range requests, the resume protocol
• Zstandard format specification - frame format that makes streaming decompression safe
• MinHash - the signature family this lesson uses
• Locality-sensitive hashing - the banding scheme behind the dedup threshold
• Phase 19 · 43 - the HDF5 tokenized corpus the downloader feeds
• Phase 19 · 44 - the cosine schedule that trains on the corpus
• Phase 19 · 45 - the AMP loop that consumes the schedule