Dedup, Filter, Decontaminate

Module FT06 · Course 3 — LLM Fine-Tuning Masterclass

60 minutes · 6 sub-sections: Thesis · MinHash/LSH · Semantic Dedup · Perplexity · Decontamination · Curriculum

The data-quality module. Where 80% of fine-tuning gains are.

Pillar 1 — Data

The thesis — one sentence

Data quality is where 80% of fine-tuning gains are.
Generation is cheap; curation is the moat.

Anyone can generate a million samples — almost nobody can clean them.

The DCLM anchor: same model, same compute, same training tokens. The pipeline with rigorous dedup + perplexity + quality selection beat the baseline by 6.5% absolute. Curation is the moat.

Why this matters immediately

Mystery	Why the thesis explains it
Why "generate more data" is the wrong instinct	Doubling a noisy dataset doubles the noise. Clean first, scale second.
Why decontamination is non-negotiable	If your data has benchmark items, your evals are fiction. The #1 cause of post-launch retraction.
Why your data pipeline is the moat	Algorithms + bases are public. Your cleaned dataset is the one artifact competitors can't copy.

The full clean pipeline

Memorize this sequence. Every stage is a funnel.

GENERATE / COLLECT · raw noisy corpus

↓

MinHash dedup · near-duplicate text · datasketch / text-dedup

↓

Semantic dedup · embedding-cluster paraphrases · sentence-transformers

↓

Perplexity filter · small LM flags garbage · kenlm

↓

Decontaminate · remove benchmark items · MMLU / GSM8K / HumanEval

↓

Quality subset · diverse high-quality slice

↓

[optional] Curriculum order · easy → hard

↓

TRAIN

MinHash + LSH — the mechanism

Exact hashing misses near-duplicates (1 char changed, whitespace, rephrasing). MinHash catches them.

Step	What happens
1. Shingle	Split into overlapping n-grams (k=3). Captures local structure.
2. Hash	k independent hash fns per shingle; keep min per fn → k-length signature. P(share min hash) = Jaccard.
3. Band (LSH)	Chop signature into b bands of r rows; shared band hash → candidate pair. Avoids O(n²).
4. Verify	For each candidate pair, compute actual Jaccard. ≥ 0.8–0.9 → drop as duplicate.

Tools: datasketch (canonical Python lib) · text-dedup (CLI wrapper). Used by CC, RedPajama, DCLM.

Why MinHash is not enough

MinHash sees

Lexical duplicates — same words in same order.

"the cat sat on the mat" vs "the cat sat on the mat."

MinHash misses

Paraphrases — same meaning, different words.

"the cat sat on the mat" vs "a feline rested on the rug."

Semantic dedup: embed every sample (sentence-transformer) → cluster → drop intra-cluster pairs with cosine ≥ 0.9–0.95. The only stage that catches meaning-level duplicates.

Same machinery → quality subset selection: cluster-and-select (pick K nearest centroids), or fastText quality classifier keeping top decile. DCLM: quality filtering ≈ 4× compute.

Perplexity filtering

Train a small LM on clean data. Score every sample. Drop the high-perplexity tail — those are samples the model finds "surprising" = garbage.

Perplexity	What it means	Action
Low	Model expected it. Clean, well-formed.	Keep
Mid	Normal variation.	Keep
High	Surprising. Mojibake, OCR, noise, code corruption.	Drop (top ~10%)

Tool: kenlm (fast n-gram, the pretraining workhorse). Or any small pretrained transformer you can score with.

The NeurIPS 2025 link

Perplexity filtering is two wins in one pass.

In continued pretraining, keeping only low-perplexity tokens measurably mitigates catastrophic forgetting.

High-perplexity tokens → large gradients → overwrite existing capabilities (forgetting)
Low-perplexity tokens → small gradients → adapt without forgetting

Same logic in fine-tuning, weakly: high-perplexity samples pull weights hard. Dropping them removes garbage AND reduces forgetting risk.

Decontamination — non-negotiable

If your training set contains a benchmark test item, your eval on that benchmark is fiction. You will publish inflated scores and discover the truth only in production.

The algorithm: n-gram matching.

Collect benchmark test sets (MMLU, GSM8K, HumanEval, MT-Bench)
Extract n-grams: 13-grams for text, 8-grams for code
Scan training set for shared n-grams → drop flagged samples

13-gram shared = essentially unique evidence. 8-gram shared = weak (common phrases trip it). Tools: lm-eval-harness decontamination, datatrove, OpenAI's GPT-3 scripts.

Why decontamination is non-negotiable

Skipping has catastrophic downside, zero upside.

Skip this stage…	Cost
Dedup	A few points of efficiency
Perplexity filter	Some garbage in training
Decontamination	The ability to trust ANY benchmark you report

There is no scenario in which skipping decontamination is the right call. A reviewer can debunk a contaminated model in 30 seconds by grepping your data for a GSM8K problem.

Curriculum learning (optional, active research)

Order samples easy → hard. Build a stable representation on easy data, then refine on harder.

Honest status: mixed empirical results. Some papers show gains (low-data regimes); others find no effect at scale. Commonly applied informally — costs nothing to try.

Do not confuse curriculum learning (reordering — a steering aid) with curriculum teaching (the cardinal error of FT00 — reordering does NOT inject knowledge). If the base doesn't know the domain, reordering will not fix it.

The data-quality funnel

A typical messy set loses 20–50% between generate and train. That is the moat being built.

Stage	Sample count	Removed
Generate	10,000	—
After MinHash dedup	8,200	−1,800 near-dupes
After semantic dedup	6,800	−1,400 paraphrases
After perplexity filter	6,100	−700 garbage
After decontamination	5,950	−150 benchmark items
Final quality subset	2,000	diverse, high-quality

Numbers illustrative. Shape is real.

Anti-patterns

Skipping decontamination. Every benchmark you report becomes suspect. Decontaminate, every time, against every benchmark you will report.

Dedup too aggressive. MinHash threshold too low (0.5) drops legitimate near-paraphrases. In SFT, paraphrases are often deliberate. Calibrate; inspect what gets dropped.

No quality filtering. "The model will ignore the noise." It will not — it fits it. 10% garbage ≠ 10% efficiency loss; it is measurable degradation on every benchmark.

Curriculum teaching. Reordering ≠ knowledge injection. Steering aid, not teaching.

What you can now do

State the thesis and defend it with the DCLM evidence.
Describe the full clean pipeline from memory.
Explain MinHash + LSH, and why MinHash is not enough (semantic dedup).
Describe perplexity filtering and its NeurIPS 2025 link to catastrophic forgetting.
Execute a decontamination n-gram scan.
Diagnose the four data-quality anti-patterns.

Next: FT07 — Tokenizers & Chat Templates