Course: Course 3 — LLM Fine-Tuning Masterclass
Module: FT06 — Dedup, Filter, Decontaminate
Duration: ~40 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
Welcome to module FT06 — Dedup, Filter, Decontaminate. This is the second module of Pillar One, Data, and if you remember only one thing from this pillar, remember this: data quality is where eighty percent of fine-tuning gains are.
The previous module, FT05, taught you how to generate synthetic data at scale. This module teaches you what to do with that data before it ever sees a training loop. Because here is the truth nobody puts on the marketing page: a clean ten-thousand-sample dataset will outperform a noisy hundred-thousand-sample dataset, every time, on the same compute. Generation is cheap. Curation is the moat.
[SLIDE 2 — The thesis]
Here is the thesis. Read it twice. Data quality is where eighty percent of fine-tuning gains are. Generation is cheap; curation is the moat. Anyone can generate a million samples — almost nobody can clean them.
The empirical anchor is DCLM — DataComp for Language Models. The DCLM team held the model and the training recipe constant and varied only the data pipeline. The pipeline with rigorous dedup, perplexity filtering, and quality selection beat the baseline by six-point-five absolute points on downstream benchmarks. Same model. Same compute. Same training tokens. The difference was entirely curation. That is the moat.
[SLIDE 3 — Why this matters immediately]
Once the thesis lands, three things become obviously strategic.
First, "generate more data" is the wrong instinct. Doubling a noisy dataset doubles the noise. If your data has thirty percent duplicates and ten percent garbage, generating more of the same gives you a bigger pile of the same bad data. Clean first, scale second.
Second, decontamination is non-negotiable. If your training set contains a benchmark's test items, your evals are lies. You will publish a model that looks brilliant and ships broken. Contamination is the single most common cause of retracting a model after launch.
Third, your data pipeline is the part competitors cannot copy. Algorithms are public — LoRA, DPO, GRPO, all on arxiv. Bases are public — Llama, Qwen, Mistral, all downloadable. Your cleaned dataset, with its specific thresholds and cutoffs, is the one artifact that is hard to reproduce. That is where your edge lives.
[SLIDE 4 — The full clean pipeline]
Memorize this sequence. Every serious data pipeline runs a variant of it. Generate. Then MinHash dedup, for near-duplicate text. Then semantic dedup, for embedding-cluster paraphrases. Then perplexity filter, where a small language model flags garbage. Then decontaminate, removing benchmark items. Then quality subset selection, picking a diverse, high-quality slice. Optionally, curriculum order, easy to hard. Then train.
Each stage is a funnel. A typical messy SFT set loses twenty to fifty percent of its samples between generate and train. That is not waste — that is the moat being built. The samples you remove matter as much as the samples you keep.
[SLIDE 5 — MinHash, the problem]
Start with dedup. Exact-match dedup — hashing the whole text and dropping identical hashes — catches copy-paste. It misses near-duplicates: one word changed, whitespace normalized differently, a sample rephrased but sharing ninety percent of its content. Those near-duplicates are the dangerous ones. They inflate the apparent size of your dataset and bias the model toward the duplicated content.
You need a method that says "these two samples are probably the same, even though they are not byte-identical." That is what MinHash does.
[SLIDE 6 — MinHash mechanism]
MinHash is four steps. Hold the whole sequence in your head.
Step one, shingle. Split each document into overlapping n-grams. A three-word shingle of "the quick brown fox" is the set containing "the quick brown" and "quick brown fox." Shingles capture local structure.
Step two, hash. Apply k independent hash functions to each shingle. For each hash function, keep the minimum hash value seen across all shingles. That minimum is the MinHash signature. The trick: two documents with the same shingle set will, with high probability, share a minimum hash. The probability that they share one equals their Jaccard similarity.
Step three, band. This is locality-sensitive hashing, LSH. You cannot compare every pair — that is order n-squared. LSH chops the k-length signature into b bands of r rows. Hash each band separately. Two documents that share at least one band hash become a candidate pair. LSH trades a tiny false-negative rate for a massive speedup.
Step four, verify. For each candidate pair, compute the actual Jaccard similarity. If it exceeds your threshold — typically zero-point-eight to zero-point-nine — drop one as a duplicate.
The tools: datasketch is the canonical Python library. text-dedup is a higher-level CLI that wraps MinHash, SimHash, and exact dedup. Both are used in production pipelines from RedPajama to HuggingFace's datatrove.
[SLIDE 7 — MinHash is not enough]
MinHash catches lexical duplicates. It cannot detect that "the cat sat on the mat" and "a feline rested on the rug" are the same sample rephrased. In synthetic SFT data, paraphrases are common — you asked the generator for five variants, you got five lexical variants of the same underlying example. MinHash catches none of them.
Enter semantic dedup. Embed every sample with a sentence-transformer. Cluster the embeddings. Within each cluster, compute pairwise cosine similarity. For any pair above zero-point-nine to zero-point-nine-five, drop the lower-quality one.
The same machinery lets you do quality subset selection. Instead of training on all your samples, pick the best slice. Cluster, then pick the K samples closest to each centroid — that gives a diverse, representative subset. Or train a fastText quality classifier on labeled high-quality versus low-quality pairs, and keep the top decile. DCLM found that quality filtering alone produced gains comparable to four times compute. Curation is compute.
[SLIDE 8 — Perplexity filtering]
Next stage: perplexity filtering. The intuition is simple. A language model assigns a probability to every token. Perplexity is how surprised the model was by the text. Clean, well-formed English has low perplexity. Garbage — mojibake, OCR errors, random tokens, code corruption — has high perplexity.
The trick: train a small language model on known-clean data — Wikipedia, or a high-quality slice of your own corpus. Then score every sample in your training set. Drop the samples in the top ten percent by perplexity. They are almost certainly garbage.
The standard tool is kenlm — fast, n-gram, the workhorse of every pretraining pipeline. Or any small pretrained transformer you can score with.
[SLIDE 9 — The NeurIPS 2025 link]
Here is the part that elevated perplexity filtering from a cleaning heuristic to a theoretical result. NeurIPS twenty-twenty-five sharpened the connection to catastrophic forgetting.
The finding: in continued pretraining, keeping only low-perplexity tokens — tokens the model can already predict well — measurably mitigates catastrophic forgetting of the base model's capabilities. The intuition: if you continue-pretrain on tokens the model is surprised by, those tokens push the weights hard and overwrite existing capabilities. If you continue-pretrain on tokens the model already handles, the gradient is small, and the model adapts without forgetting.
The same logic applies in fine-tuning, weakly. A high-perplexity sample is one the model finds bizarre — it pulls the weights in a strong, possibly-damaging direction. Dropping the high-perplexity tail of your training set both removes garbage and reduces the risk of catastrophic forgetting. Two wins in one pass.
[SLIDE 10 — Decontamination]
Now the non-negotiable stage. Decontamination. Removing benchmark test items from your training data.
If your training set contains a question from MMLU, GSM8K, HumanEval, or MT-Bench — even paraphrased — your eval on that benchmark is contaminated. The model has seen the answer. You will report inflated scores, ship a model that looks brilliant, and discover only in production that it cannot generalize.
The standard approach is n-gram contamination detection. Collect the benchmark test sets. Extract n-grams — thirteen-grams for natural text, eight-grams for code. Scan your training set for any shared n-gram. Drop the offending samples.
A single shared eight-gram is weak evidence — common phrases will trip it. A shared thirteen-gram is strong evidence — that sequence is essentially unique. Most pipelines use thirteen-grams for text and eight-grams for code.
The tools: EleutherAI's lm-eval-harness ships reference decontamination tooling. HuggingFace's datatrove has an integrated decontamination block. The original GPT-3 paper appendix released OpenAI's decontamination scripts. Every serious lab runs one of these. If you do not, your evals are fiction.
[SLIDE 11 — Why decontamination is non-negotiable]
This is the one stage where skipping has catastrophic downside and zero upside. Skipping dedup costs you a few points of efficiency. Skipping perplexity filtering costs you some garbage. Skipping decontamination costs you the ability to trust any benchmark you report. There is no scenario in which skipping decontamination is the right call.
It is also the most embarrassing failure mode. A reviewer can debunk a contaminated model in thirty seconds by grepping your training set for a GSM8K problem. Do not be the team that ships a contaminated model.
[SLIDE 12 — Curriculum learning]
A brief word on curriculum learning. The idea: order your samples easy to hard. Show the model short, clear, high-quality examples first, then gradually introduce longer, more complex, noisier ones. The intuition: the model builds a stable representation on easy data, then refines it on harder data.
The honest status: this is active research with mixed empirical results. Some papers show gains, especially in low-data regimes. Others show no effect at scale. There is no consensus that curriculum learning reliably beats random shuffling.
But it is commonly applied informally. If your SFT set has a clearly-easy subset and a clearly-hard subset, training easy-first-then-hard is a reasonable default that costs nothing to try. If it helps, keep it. If not, revert to shuffling.
And do not confuse curriculum learning with curriculum teaching. Curriculum learning reorders samples to make steering easier. It does not inject new knowledge. The thesis of FT00 holds.
[SLIDE 13 — The anti-patterns]
Four anti-patterns to leave with.
First, skipping decontamination. The most damaging shortcut. Every benchmark you report becomes suspect. Decontaminate, every time, against every benchmark you will report.
Second, dedup too aggressive. Setting the MinHash threshold too low — zero-point-five — drops legitimate samples that happen to be similar. In SFT data, paraphrases are often deliberate. Aggressive dedup removes that signal. Calibrate your threshold on a held-out sample. Inspect what gets dropped.
Third, no quality filtering. "The model will learn to ignore the noise." It will not. It fits it. A ten-percent garbage rate is not a ten-percent efficiency loss — it is a measurable quality degradation on every downstream benchmark. Perplexity filter plus quality subset selection is the cheapest quality improvement you will ever get.
Fourth, confusing curriculum learning with curriculum teaching. Reordering is a steering aid. It does not inject knowledge. If the base does not know the domain, reordering will not fix it.
[SLIDE 14 — What you can now do]
You can now state the thesis and defend it with the DCLM evidence. You can describe the full clean pipeline from memory. You can explain MinHash and LSH, and why MinHash is not enough. You understand perplexity filtering and its NeurIPS twenty-twenty-five link to catastrophic forgetting. You can execute a decontamination scan. And you can diagnose the four data-quality anti-patterns.
The lab for this module is the capstone of Pillar One. You will take a ten-thousand-sample messy SFT set with planted duplicates, garbage, and benchmark items. You will run the full clean pipeline. You will report retention and contamination-removal rates. By the end, the moat will be something you have built with your own hands.
Next, module FT07: tokenizers and chat templates. The cleaned data has to be tokenized correctly before training, and a chat-template mistake will silently turn your SFT set into noise. Let's get the substrate right.
End of module FT06. Duration: approximately forty minutes at one-hundred-forty words per minute.
# Teaching Script — Module FT06: Dedup, Filter, Decontaminate **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT06 — Dedup, Filter, Decontaminate **Duration**: ~40 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] Welcome to module FT06 — Dedup, Filter, Decontaminate. This is the second module of Pillar One, Data, and if you remember only one thing from this pillar, remember this: data quality is where eighty percent of fine-tuning gains are. The previous module, FT05, taught you how to generate synthetic data at scale. This module teaches you what to do with that data before it ever sees a training loop. Because here is the truth nobody puts on the marketing page: a clean ten-thousand-sample dataset will outperform a noisy hundred-thousand-sample dataset, every time, on the same compute. Generation is cheap. Curation is the moat. [SLIDE 2 — The thesis] Here is the thesis. Read it twice. Data quality is where eighty percent of fine-tuning gains are. Generation is cheap; curation is the moat. Anyone can generate a million samples — almost nobody can clean them. The empirical anchor is DCLM — DataComp for Language Models. The DCLM team held the model and the training recipe constant and varied only the data pipeline. The pipeline with rigorous dedup, perplexity filtering, and quality selection beat the baseline by six-point-five absolute points on downstream benchmarks. Same model. Same compute. Same training tokens. The difference was entirely curation. That is the moat. [SLIDE 3 — Why this matters immediately] Once the thesis lands, three things become obviously strategic. First, "generate more data" is the wrong instinct. Doubling a noisy dataset doubles the noise. If your data has thirty percent duplicates and ten percent garbage, generating more of the same gives you a bigger pile of the same bad data. Clean first, scale second. Second, decontamination is non-negotiable. If your training set contains a benchmark's test items, your evals are lies. You will publish a model that looks brilliant and ships broken. Contamination is the single most common cause of retracting a model after launch. Third, your data pipeline is the part competitors cannot copy. Algorithms are public — LoRA, DPO, GRPO, all on arxiv. Bases are public — Llama, Qwen, Mistral, all downloadable. Your cleaned dataset, with its specific thresholds and cutoffs, is the one artifact that is hard to reproduce. That is where your edge lives. [SLIDE 4 — The full clean pipeline] Memorize this sequence. Every serious data pipeline runs a variant of it. Generate. Then MinHash dedup, for near-duplicate text. Then semantic dedup, for embedding-cluster paraphrases. Then perplexity filter, where a small language model flags garbage. Then decontaminate, removing benchmark items. Then quality subset selection, picking a diverse, high-quality slice. Optionally, curriculum order, easy to hard. Then train. Each stage is a funnel. A typical messy SFT set loses twenty to fifty percent of its samples between generate and train. That is not waste — that is the moat being built. The samples you remove matter as much as the samples you keep. [SLIDE 5 — MinHash, the problem] Start with dedup. Exact-match dedup — hashing the whole text and dropping identical hashes — catches copy-paste. It misses near-duplicates: one word changed, whitespace normalized differently, a sample rephrased but sharing ninety percent of its content. Those near-duplicates are the dangerous ones. They inflate the apparent size of your dataset and bias the model toward the duplicated content. You need a method that says "these two samples are probably the same, even though they are not byte-identical." That is what MinHash does. [SLIDE 6 — MinHash mechanism] MinHash is four steps. Hold the whole sequence in your head. Step one, shingle. Split each document into overlapping n-grams. A three-word shingle of "the quick brown fox" is the set containing "the quick brown" and "quick brown fox." Shingles capture local structure. Step two, hash. Apply k independent hash functions to each shingle. For each hash function, keep the minimum hash value seen across all shingles. That minimum is the MinHash signature. The trick: two documents with the same shingle set will, with high probability, share a minimum hash. The probability that they share one equals their Jaccard similarity. Step three, band. This is locality-sensitive hashing, LSH. You cannot compare every pair — that is order n-squared. LSH chops the k-length signature into b bands of r rows. Hash each band separately. Two documents that share at least one band hash become a candidate pair. LSH trades a tiny false-negative rate for a massive speedup. Step four, verify. For each candidate pair, compute the actual Jaccard similarity. If it exceeds your threshold — typically zero-point-eight to zero-point-nine — drop one as a duplicate. The tools: datasketch is the canonical Python library. text-dedup is a higher-level CLI that wraps MinHash, SimHash, and exact dedup. Both are used in production pipelines from RedPajama to HuggingFace's datatrove. [SLIDE 7 — MinHash is not enough] MinHash catches lexical duplicates. It cannot detect that "the cat sat on the mat" and "a feline rested on the rug" are the same sample rephrased. In synthetic SFT data, paraphrases are common — you asked the generator for five variants, you got five lexical variants of the same underlying example. MinHash catches none of them. Enter semantic dedup. Embed every sample with a sentence-transformer. Cluster the embeddings. Within each cluster, compute pairwise cosine similarity. For any pair above zero-point-nine to zero-point-nine-five, drop the lower-quality one. The same machinery lets you do quality subset selection. Instead of training on all your samples, pick the best slice. Cluster, then pick the K samples closest to each centroid — that gives a diverse, representative subset. Or train a fastText quality classifier on labeled high-quality versus low-quality pairs, and keep the top decile. DCLM found that quality filtering alone produced gains comparable to four times compute. Curation is compute. [SLIDE 8 — Perplexity filtering] Next stage: perplexity filtering. The intuition is simple. A language model assigns a probability to every token. Perplexity is how surprised the model was by the text. Clean, well-formed English has low perplexity. Garbage — mojibake, OCR errors, random tokens, code corruption — has high perplexity. The trick: train a small language model on known-clean data — Wikipedia, or a high-quality slice of your own corpus. Then score every sample in your training set. Drop the samples in the top ten percent by perplexity. They are almost certainly garbage. The standard tool is kenlm — fast, n-gram, the workhorse of every pretraining pipeline. Or any small pretrained transformer you can score with. [SLIDE 9 — The NeurIPS 2025 link] Here is the part that elevated perplexity filtering from a cleaning heuristic to a theoretical result. NeurIPS twenty-twenty-five sharpened the connection to catastrophic forgetting. The finding: in continued pretraining, keeping only low-perplexity tokens — tokens the model can already predict well — measurably mitigates catastrophic forgetting of the base model's capabilities. The intuition: if you continue-pretrain on tokens the model is surprised by, those tokens push the weights hard and overwrite existing capabilities. If you continue-pretrain on tokens the model already handles, the gradient is small, and the model adapts without forgetting. The same logic applies in fine-tuning, weakly. A high-perplexity sample is one the model finds bizarre — it pulls the weights in a strong, possibly-damaging direction. Dropping the high-perplexity tail of your training set both removes garbage and reduces the risk of catastrophic forgetting. Two wins in one pass. [SLIDE 10 — Decontamination] Now the non-negotiable stage. Decontamination. Removing benchmark test items from your training data. If your training set contains a question from MMLU, GSM8K, HumanEval, or MT-Bench — even paraphrased — your eval on that benchmark is contaminated. The model has seen the answer. You will report inflated scores, ship a model that looks brilliant, and discover only in production that it cannot generalize. The standard approach is n-gram contamination detection. Collect the benchmark test sets. Extract n-grams — thirteen-grams for natural text, eight-grams for code. Scan your training set for any shared n-gram. Drop the offending samples. A single shared eight-gram is weak evidence — common phrases will trip it. A shared thirteen-gram is strong evidence — that sequence is essentially unique. Most pipelines use thirteen-grams for text and eight-grams for code. The tools: EleutherAI's lm-eval-harness ships reference decontamination tooling. HuggingFace's datatrove has an integrated decontamination block. The original GPT-3 paper appendix released OpenAI's decontamination scripts. Every serious lab runs one of these. If you do not, your evals are fiction. [SLIDE 11 — Why decontamination is non-negotiable] This is the one stage where skipping has catastrophic downside and zero upside. Skipping dedup costs you a few points of efficiency. Skipping perplexity filtering costs you some garbage. Skipping decontamination costs you the ability to trust any benchmark you report. There is no scenario in which skipping decontamination is the right call. It is also the most embarrassing failure mode. A reviewer can debunk a contaminated model in thirty seconds by grepping your training set for a GSM8K problem. Do not be the team that ships a contaminated model. [SLIDE 12 — Curriculum learning] A brief word on curriculum learning. The idea: order your samples easy to hard. Show the model short, clear, high-quality examples first, then gradually introduce longer, more complex, noisier ones. The intuition: the model builds a stable representation on easy data, then refines it on harder data. The honest status: this is active research with mixed empirical results. Some papers show gains, especially in low-data regimes. Others show no effect at scale. There is no consensus that curriculum learning reliably beats random shuffling. But it is commonly applied informally. If your SFT set has a clearly-easy subset and a clearly-hard subset, training easy-first-then-hard is a reasonable default that costs nothing to try. If it helps, keep it. If not, revert to shuffling. And do not confuse curriculum learning with curriculum teaching. Curriculum learning reorders samples to make steering easier. It does not inject new knowledge. The thesis of FT00 holds. [SLIDE 13 — The anti-patterns] Four anti-patterns to leave with. First, skipping decontamination. The most damaging shortcut. Every benchmark you report becomes suspect. Decontaminate, every time, against every benchmark you will report. Second, dedup too aggressive. Setting the MinHash threshold too low — zero-point-five — drops legitimate samples that happen to be similar. In SFT data, paraphrases are often deliberate. Aggressive dedup removes that signal. Calibrate your threshold on a held-out sample. Inspect what gets dropped. Third, no quality filtering. "The model will learn to ignore the noise." It will not. It fits it. A ten-percent garbage rate is not a ten-percent efficiency loss — it is a measurable quality degradation on every downstream benchmark. Perplexity filter plus quality subset selection is the cheapest quality improvement you will ever get. Fourth, confusing curriculum learning with curriculum teaching. Reordering is a steering aid. It does not inject knowledge. If the base does not know the domain, reordering will not fix it. [SLIDE 14 — What you can now do] You can now state the thesis and defend it with the DCLM evidence. You can describe the full clean pipeline from memory. You can explain MinHash and LSH, and why MinHash is not enough. You understand perplexity filtering and its NeurIPS twenty-twenty-five link to catastrophic forgetting. You can execute a decontamination scan. And you can diagnose the four data-quality anti-patterns. The lab for this module is the capstone of Pillar One. You will take a ten-thousand-sample messy SFT set with planted duplicates, garbage, and benchmark items. You will run the full clean pipeline. You will report retention and contamination-removal rates. By the end, the moat will be something you have built with your own hands. Next, module FT07: tokenizers and chat templates. The cleaned data has to be tokenized correctly before training, and a chat-template mistake will silently turn your SFT set into noise. Let's get the substrate right. --- *End of module FT06. Duration: approximately forty minutes at one-hundred-forty words per minute.*