generate (or collect)
   ↓
MinHash dedup        (near-duplicate text — exact-match on shingle hashes)
   ↓
semantic dedup      (embedding-cluster near-paraphrases)
   ↓
perplexity filter   (small LM flags high-perplexity = likely garbage)
   ↓
decontaminate       (remove benchmark test items)
   ↓
quality subset      (pick diverse, high-quality slice)
   ↓
[ optional: curriculum order easy → hard ]
   ↓
train

from datasketch import MinHash, MinHashLSH

def make_minhash(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for shingle in shingles(text, k=3):
        m.update(shingle.encode("utf-8"))
    return m

lsh = MinHashLSH(threshold=0.9, num_perm=128)
for i, text in enumerate(corpus):
    lsh.insert(i, make_minhash(text))

# Query: which existing docs are near-duplicates of doc 42?
dupes = lsh.query(make_minhash(corpus[42]))

# Pseudocode — the actual fit/score is library-specific
from kenlm import Model  # or: a tiny transformer trained on clean data

clean_lm = Model("clean_corpus.arpa.bin")  # trained on Wikipedia-quality text

def perplexity(text):
    return clean_lm.perplexity(text)

# Filter: drop the top 10% highest-perplexity samples
scores = sorted([(perplexity(s), i) for i, s in enumerate(corpus)])
cutoff = int(0.9 * len(scores))
keep_indices = [i for _, i in scores[:cutoff]]
clean_corpus = [corpus[i] for i in keep_indices]

# Pseudocode — the canonical 13-gram scan
def ngrams(text, n=13):
    tokens = text.split()
    return set(tuple(tokens[i:i+n]) for i in range(len(tokens) - n + 1))

benchmark_ngrams = set()
for item in benchmark_test_set:
    benchmark_ngrams |= ngrams(item)

clean = []
for sample in training_set:
    if not (ngrams(sample) & benchmark_ngrams):
        clean.append(sample)

# Module FT06 — Dedup, Filter, Decontaminate

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT06 — Dedup, Filter, Decontaminate
**Duration**: 60 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT05 — Synthetic Data Generation

---

## Learning Objectives

After completing this module, you will be able to:

1. State the thesis — **data quality is where 80% of fine-tuning gains are; generation is cheap, curation is the moat** — and defend it by sketching the full clean pipeline.
2. Explain how MinHash + Locality-Sensitive Hashing (LSH) finds near-duplicate text at scale, and contrast it with exact-hash dedup and embedding-based semantic dedup.
3. Describe perplexity filtering: train a small LM on clean data, drop high-perplexity samples as likely garbage — and connect it to the NeurIPS 2025 finding that token-perplexity reduction mitigates catastrophic forgetting in continued pretraining.
4. Define decontamination, explain why it is non-negotiable for honest evals, and execute an n-gram match against benchmark test sets (MMLU, GSM8K, MT-Bench, HumanEval).
5. Diagnose the four data-quality anti-patterns: skipping decontamination, dedup too aggressive, no quality filtering, and confusing curriculum learning with curriculum teaching.

---

# 6.1 — The Thesis: Curation Is the Moat

*Generation is cheap; curation is the moat. This is where 80% of fine-tuning gains are.*

## The sentence

> **Data quality is where 80% of fine-tuning gains are. Generation is cheap; curation is the moat. Anyone can generate a million samples — almost nobody can clean them.**

Read it twice. The corollary of FT00's thesis ("fine-tuning steers behavior; it does not teach knowledge") is that the steering wheel — your dataset — is everything. FT05 showed you how to *generate* samples at scale. This module shows you what to do with them: remove the duplicates, drop the garbage, scrub out the benchmark items you will be evaluated against, and ship a clean subset to training.

The DCLM (DataComp for Language Models) finding is the empirical anchor. In the DCLM benchmarks, the team held the model and the training recipe constant and varied only the data pipeline. The pipeline with rigorous dedup, perplexity filtering, and quality selection produced a 6.5% absolute improvement on downstream benchmarks over the baseline pipeline — same model, same compute, same training tokens. The difference was *curation*. That is the moat.

### Why this matters immediately

Once the thesis lands, three things that look like engineering choices become obviously strategic:

1. **Why "generate more data" is the wrong instinct.** Doubling a noisy dataset doubles the noise. If your data has 30% duplicates and 10% garbage, generating more of the same gives you a bigger pile of the same bad data. Clean first, scale second.
2. **Why decontamination is non-negotiable.** If your training set contains a benchmark's test items, your evals are lies. You will publish a model that looks brilliant and ships broken. Contamination is the single most common cause of retracting a model after launch.
3. **Why your data pipeline is the part competitors cannot copy.** Algorithms are public (LoRA, DPO, GRPO — all on arXiv). Bases are public (Llama, Qwen, Mistral — all downloadable). Your *cleaned* dataset is the one artifact that is hard to reproduce. The cleaning logic, the thresholds, the decontamination n-grams — that is where your edge lives.

### The full clean pipeline

Memorize this sequence. Every serious data pipeline runs a variant of it:

```
generate (or collect)
   ↓
MinHash dedup        (near-duplicate text — exact-match on shingle hashes)
   ↓
semantic dedup      (embedding-cluster near-paraphrases)
   ↓
perplexity filter   (small LM flags high-perplexity = likely garbage)
   ↓
decontaminate       (remove benchmark test items)
   ↓
quality subset      (pick diverse, high-quality slice)
   ↓
[ optional: curriculum order easy → hard ]
   ↓
train
```

Each stage is a funnel. A typical messy SFT set loses 20–50% of its samples between generate and train. That is not waste — that is the moat being built. The samples you *remove* matter as much as the samples you keep.

---

# 6.2 — MinHash + LSH: Near-Duplicate Dedup at Scale

*The standard, cheap, scale-friendly dedup. Used by CC, RedPajama, DCLM, and every serious pretraining pipeline.*

## The problem exact hashing cannot solve

Exact-match dedup (hash the whole text, drop identical hashes) catches copy-paste duplicates. It misses near-duplicates: a sample with one word changed, a sample with whitespace normalized differently, a sample that was rephrased but shares 90% of its content. These near-duplicates are the dangerous ones — they inflate the apparent size of your dataset and bias the model toward the duplicated content.

You need a method that says "these two samples are *probably* the same, even though they are not byte-identical." That is what MinHash does.

## How MinHash works

The pipeline is four steps. Hold the whole sequence in your head — it is the most-tested mechanism in this module.

**Step 1 — Shingle.** Split each document into overlapping n-grams (shingles). A 3-word shingle of "the quick brown fox" is `{the quick brown, quick brown fox}`. Shingles capture local structure — two documents with the same shingles are talking about the same thing in the same words.

**Step 2 — Hash.** Apply k independent hash functions to each shingle. For each hash function, keep the *minimum* hash value seen across all shingles. That minimum is the "MinHash signature." The trick: two documents with the same set of shingles will, with high probability, have the same minimum hash for any given hash function. The probability that two documents share a minimum hash equals their Jaccard similarity (the size of their shingle-set intersection over the size of the union).

**Step 3 — Band (Locality-Sensitive Hashing).** You cannot compare every pair of documents (that is O(n²)). LSH solves this. Take the k-length MinHash signature, chop it into `b` bands of `r` rows each (so `k = b × r`). Hash each band separately. Two documents that share *at least one band hash* become a **candidate pair** — they are similar enough to be worth comparing directly. LSH trades a tiny false-negative rate for a massive speedup: instead of comparing n² pairs, you compare only the candidate pairs that survive the banding.

**Step 4 — Verify candidate pairs.** For each candidate pair, compute the actual Jaccard similarity of their shingle sets. If it exceeds a threshold (typically 0.8–0.9), drop one of them as a duplicate.

### The threshold is a knob, not a constant

The Jaccard threshold controls how aggressive your dedup is. Set it too low (0.5) and you drop legitimate near-paraphrases that are *different* samples you wanted to keep. Set it too high (0.99) and you only catch exact copies. The DCLM pipeline uses ~0.8. For SFT data, where paraphrases are often deliberate, 0.9 is safer.

### Tools

- **`datasketch` (Erik Zhao)** — the canonical Python MinHash + LSH library. `MinHash` objects, `MinHashLSH` index, Jaccard estimation. The 5-line version.
- **`text-dedup` (Chenghao Xiao)** — a higher-level library that wraps MinHash, SimHash, and exact dedup into a single CLI. Used in the HuggingFace `datatrove` and RedPajama pipelines.

```python
from datasketch import MinHash, MinHashLSH

def make_minhash(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for shingle in shingles(text, k=3):
        m.update(shingle.encode("utf-8"))
    return m

lsh = MinHashLSH(threshold=0.9, num_perm=128)
for i, text in enumerate(corpus):
    lsh.insert(i, make_minhash(text))

# Query: which existing docs are near-duplicates of doc 42?
dupes = lsh.query(make_minhash(corpus[42]))
```

---

# 6.3 — Semantic Dedup and Quality Subset Selection

*MinHash catches lexical duplicates. Semantic dedup catches paraphrases — different words, same meaning. Quality subset selection picks the best, most diverse slice.*

## Why MinHash is not enough

MinHash operates on shingles — exact word sequences. It cannot detect that "the cat sat on the mat" and "a feline rested on the rug" are the same sample rephrased. In synthetic SFT data (FT05), paraphrases are *common*: you ask the generator for 5 variants of the same instruction and you get 5 lexical variants of the same underlying example. MinHash catches none of them.

Enter **semantic dedup**: embed every sample with a sentence-embedding model, cluster, and drop intra-cluster duplicates.

## The mechanism

1. **Embed.** Run every sample through a sentence-transformer model (`all-MiniLM-L6-v2` for speed, `bge-base-en-v1.5` for quality). Each sample becomes a 384- or 768-dimensional vector.
2. **Cluster.** Run a clustering algorithm on the embeddings. K-means is the standard choice for scale; for huge datasets, FAISS-accelerated clustering or mini-batch K-means.
3. **Dedup within clusters.** Within each cluster, compute pairwise cosine similarity. For any pair above a threshold (typically 0.9–0.95), drop the lower-quality one. "Quality" here can be a heuristic — length, a quality-classifier score, or the sample closest to the cluster centroid (the most "representative" example).

### Quality-based subset selection

The same machinery lets you do **subset selection**. Instead of training on all N samples, you pick the *best* M. The DCLM pipeline calls this "quality filtering" and it is the single biggest lever in their results.

The approaches:

- **Cluster-and-select.** Cluster the embeddings, then pick the K samples closest to each cluster centroid. This gives a diverse, representative subset that covers the full distribution.
- **Quality classifier.** Train a binary classifier on (high-quality, low-quality) pairs — e.g., (Wikipedia, random web) or (human-written, machine-generated). Score every sample, keep the top decile.
- **Diversity sampling.** Within a high-quality pool, greedily pick samples that maximize embedding-space distance from already-picked samples. Prevents mode collapse on a single topic.

The DCLM-Baseline pipeline found that *quality filtering alone* — using a fastText classifier trained to distinguish Wikipedia/reference quality from random web — produced downstream gains comparable to scaling compute 4×. **Curation is compute.**

---

# 6.4 — Perplexity Filtering

*Train a small LM on clean data; samples the LM finds surprising (high perplexity) are likely garbage. Drop them.*

## The intuition

A language model assigns a probability to every token. **Perplexity** is the exponentiated average negative log-probability — loosely, "how surprised was the model by this text." A clean, well-formed English sentence has low perplexity (the model expected it). Garbage — mojibake, OCR errors, random tokens, code-corruption — has high perplexity (the model did not expect it).

The trick: train a *small* n-gram LM or a tiny transformer on a *known-clean* corpus (Wikipedia, a high-quality slice of your own data). Then score every sample in your training set. Drop the samples in the top X% by perplexity. They are almost certainly garbage.

## The NeurIPS 2025 link

Perplexity filtering is not just a data-cleaning heuristic. It has a theoretical connection to catastrophic forgetting that became explicit in 2025.

The finding (the "DataComp-LM" follow-up line of work, sharpened at NeurIPS 2025): in continued pretraining, **token-perplexity reduction** — keeping only tokens that the model can already predict well — measurably mitigates catastrophic forgetting of the base model's capabilities. The intuition: if you continue pretraining on tokens the model is *surprised by* (high perplexity), those tokens push the weights hard, overwriting existing capabilities. If you continue-pretrain on tokens the model already handles (low perplexity), the gradient is small, and the model adapts without forgetting.

The same logic applies in fine-tuning, weakly. A high-perplexity sample is one the model finds bizarre — it pulls the weights in a strong, possibly-damaging direction. Dropping the high-perplexity tail of your training set both removes garbage *and* reduces the risk of catastrophic forgetting. **Perplexity filtering is two wins in one pass.**

## The mechanics

```python
# Pseudocode — the actual fit/score is library-specific
from kenlm import Model  # or: a tiny transformer trained on clean data

clean_lm = Model("clean_corpus.arpa.bin")  # trained on Wikipedia-quality text

def perplexity(text):
    return clean_lm.perplexity(text)

# Filter: drop the top 10% highest-perplexity samples
scores = sorted([(perplexity(s), i) for i, s in enumerate(corpus)])
cutoff = int(0.9 * len(scores))
keep_indices = [i for _, i in scores[:cutoff]]
clean_corpus = [corpus[i] for i in keep_indices]
```

Tools: **kenlm** (fast, n-gram, the standard for perplexity filtering at pretraining scale), **sentence-transformers** for tiny transformer LMs, or any small pretrained LM you can score with.

---

# 6.5 — Decontamination

*Removing benchmark test items from your training data. Non-negotiable for honest evals.*

## The problem

If your training set contains a question from MMLU, GSM8K, HumanEval, or MT-Bench — even paraphrased — your eval on that benchmark is contaminated. The model has seen the answer. You will report inflated scores, ship a model that looks brilliant, and discover only in production that it cannot generalize. Contamination is the single most embarrassing way to retract a model.

This is not hypothetical. The original GPT-3 paper had a contamination issue. The Llama 2 paper reports explicit decontamination against GSM8K and MMLU. Every serious lab runs it. If you do not, your evals are fiction.

## The mechanism: n-gram matching

The standard approach is **n-gram contamination detection**:

1. **Collect the benchmark test sets.** MMLU (multiple-choice), GSM8K (math), HumanEval (code), MT-Bench (open-ended), AGIEval, ARC, HellaSwag — whatever you will be evaluated on.
2. **Extract n-grams.** For each benchmark item, extract all 8-grams (or 13-grams; the canonical choice is 8 for code, 13 for text). Store these in a set.
3. **Scan your training set.** For each training sample, extract its n-grams. If any sample shares an n-gram with a benchmark item, flag it.
4. **Remove flagged samples.** Conservative default: drop the entire sample. Aggressive variant: drop only the contaminated span.

The threshold matters. A single shared 8-gram ("the quick brown fox jumps over") is weak evidence — common phrases will trip it. A shared 13-gram is strong evidence — that sequence is essentially unique. Most pipelines use 13-grams for natural text and 8-grams for code (code has higher entropy per token).

### Tools

- **LM-eval-harness's `decontamination/` tooling** — EleutherAI's reference implementation, used by HuggingFace's Open-LLM-Leaderboard. The most-cited decontamination tool.
- **`datatrove`'s decontamination block** — HuggingFace's pipeline library, integrated decontamination against a configurable benchmark set.
- **OpenAI's decontamination scripts** — the original reference, released with the GPT-3 paper appendix.

```python
# Pseudocode — the canonical 13-gram scan
def ngrams(text, n=13):
    tokens = text.split()
    return set(tuple(tokens[i:i+n]) for i in range(len(tokens) - n + 1))

benchmark_ngrams = set()
for item in benchmark_test_set:
    benchmark_ngrams |= ngrams(item)

clean = []
for sample in training_set:
    if not (ngrams(sample) & benchmark_ngrams):
        clean.append(sample)
```

### Why this is non-negotiable

Decontamination is the one stage of the pipeline where skipping has a *catastrophic* downside and a *zero* upside. Skipping dedup costs you a few points of efficiency. Skipping perplexity filtering costs you some garbage in the training set. Skipping decontamination costs you **the ability to trust any benchmark you report**. There is no scenario in which skipping decontamination is the right call.

---

# 6.6 — Curriculum Learning (Optional, Active Research)

*Order samples easy → hard. Commonly applied informally; active research.*

Curriculum learning is the idea that the *order* in which the model sees samples matters. Show it the easy examples first (short, clear, high-quality), then gradually introduce harder ones (longer, more complex, noisier). The intuition: the model builds a stable representation on easy data, then refines it on harder data — rather than getting dragged in a hundred directions at once by a randomly-shuffled dataset.

The honest status: this is **active research with mixed empirical results**. Some papers show a measurable gain (especially in low-data regimes and continued pretraining). Others show no effect once the dataset is large enough. There is no consensus that curriculum learning reliably beats random shuffling at scale.

That said, it is *commonly applied informally*. If your SFT set has a clearly-easy subset (short, well-formatted, high-confidence) and a clearly-hard subset (long, multi-turn, edge cases), training on easy-first-then-hard is a reasonable default that costs nothing to try. If it helps, keep it. If it does not, revert to shuffling.

Do not confuse curriculum *learning* with curriculum *teaching*. Curriculum learning reorders samples to make steering easier. It does not inject new knowledge. The thesis of FT00 holds: this is a steering technique, applied to the steering wheel.

---

## Anti-Patterns

### Skipping decontamination

The single most damaging shortcut. If you skip decontamination, every benchmark score you report is suspect. You will publish a model that a reviewer can debunk in 30 seconds by grepping your training set for a GSM8K problem. Decontaminate, every time, against every benchmark you will report.

### Dedup too aggressive

Setting the MinHash Jaccard threshold too low (0.5) or the semantic-similarity threshold too low (0.85) drops legitimate samples that happen to be similar. In SFT data, paraphrases are often *deliberate* — you want the model to generalize across phrasings. Aggressive dedup removes that signal. Calibrate your threshold on a held-out sample; inspect what gets dropped.

### No quality filtering

"Garbage in, garbage out." Training on a raw, unfiltered dataset because "the model will learn to ignore the noise" is a fiction. The model does not ignore noise — it fits it. A 10% garbage rate is not a 10% efficiency loss; it is a measurable quality degradation on every downstream benchmark. Perplexity filter + quality subset selection is the cheapest quality improvement you will ever get.

### Confusing curriculum learning with curriculum teaching

Curriculum *learning* reorders samples (a steering-aid technique). Curriculum *teaching* — the idea that ordering can inject knowledge the base lacks — is the same cardinal error as FT00's "teach via fine-tuning." If the base does not know the domain, reordering will not fix it. Reorder for efficiency; do not reorder to teach.

---

## Key Terms

| Term | Definition |
| --- | --- |
| **MinHash** | A probabilistic hashing technique that estimates Jaccard similarity between sets; the basis for near-duplicate text detection |
| **LSH (Locality-Sensitive Hashing)** | The banding trick that makes MinHash scale — only candidate pairs that share a band hash are compared directly |
| **Shingle** | An overlapping n-gram (typically 3-5 words) extracted from a document; the unit MinHash hashes |
| **Jaccard similarity** | The size of the intersection of two sets over the size of their union; what MinHash estimates |
| **Semantic dedup** | Removing near-paraphrases by embedding samples with a sentence-transformer, clustering, and dropping intra-cluster duplicates |
| **Perplexity filtering** | Training a small LM on clean data and dropping high-perplexity samples (likely garbage); also mitigates catastrophic forgetting |
| **Decontamination** | Removing benchmark test items (MMLU, GSM8K, HumanEval, MT-Bench) from the training set via n-gram matching |
| **Quality subset selection** | Picking a diverse, high-quality slice of the dataset (cluster-and-select, quality classifier, diversity sampling) |
| **Curriculum learning** | Ordering samples easy → hard to stabilize training; active research, commonly applied informally |
| **DCLM** | DataComp for Language Models — the benchmark that empirically demonstrated curation beats compute |

---

## Lab Exercise

See `07-lab-spec.md`. The "Clean a Messy Dataset" lab: take a 10K-sample noisy SFT set (with duplicates, garbage, and planted benchmark items), run the full clean pipeline (MinHash → semantic dedup → perplexity filter → decontamination), report retention rate and contamination-removal rate. Self-contained — a synthetic messy-dataset generator is included.

---

## References

1. **Broder (1997)** — *On the Resemblance and Containment of Documents*. The original MinHash paper. IEEE Compression Conference.
2. **Chum, Philbin, Zisserman (2008)** — *Near Duplicate Image Detection: min-Hash and tf-idf Weighting*. The LSH banding trick formalized. BMVC.
3. **Penedo et al. (2023)** — *The RefinedWeb Dataset for Falcon LLM*. arXiv:2306.01116. The dedup recipe used in the Falcon pretraining pipeline.
4. **Li et al. (2024)** — *DataComp-LM: In search of the next generation of training sets for language models*. arXiv:2406.11794, NeurIPS 2024. The DCLM benchmark — empirical proof that curation beats compute. Quality filtering + dedup produced gains comparable to 4× compute.
5. **Lee et al. (2022)** — *Deduplicating Training Data Makes Language Models Better*. arXiv:2107.06499, ACL 2022. The empirical case for dedup at pretraining scale.
6. **Abbas et al. (2024)** — *SemDeDup: Data-efficient learning at web-scale through semantic deduplication*. arXiv:2303.09540. Embedding-cluster dedup, the Meta version.
7. **Wenzek et al. (2020)** — *CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data*. The CCNet pipeline — the canonical perplexity-filter reference. LREC 2020.
8. **Brown et al. (2020)** — *Language Models are Few-Shot Learners (GPT-3)*. arXiv:2005.14165. The original decontamination appendix — n-gram matching against benchmarks.
9. **Goyal et al. (2025)** — *Token-perplexity reduction mitigates catastrophic forgetting in continued pretraining*. NeurIPS 2025. The theoretical link between perplexity filtering and forgetting.
10. **datasketch** — `github.com/ekzhu/datasketch`. The canonical Python MinHash + LSH library.
11. **text-dedup** — `github.com/ChenghaoXu/text-dedup`. Higher-level dedup CLI wrapping MinHash, SimHash, exact.
12. **LM-eval-harness decontamination** — `github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/decontamination`. The reference decontamination tooling.