{ "module": "FT06 — Dedup, Filter, Decontaminate", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 40, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the central thesis of FT06?", "options": [ "More training data is always better — generate at maximum scale.", "Data quality is where 80% of fine-tuning gains are. Generation is cheap; curation is the moat.", "Algorithms matter more than data — pick the right optimizer and any data works.", "Dedup is optional; perplexity filtering is the only mandatory stage." ], "answer_index": 1, "rationale": "The thesis: data quality is where 80% of gains are. Generation is cheap; curation is the moat. DCLM empirically demonstrated this — same model, same compute, +6.5% absolute from curation alone." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the stages of the full clean pipeline in order.", "options": [ "Generate -> train -> dedup -> filter -> decontaminate", "Generate -> MinHash dedup -> semantic dedup -> perplexity filter -> decontaminate -> quality subset -> train", "Decontaminate -> dedup -> filter -> generate -> train", "Quality subset -> dedup -> generate -> perplexity filter -> train" ], "answer_index": 1, "rationale": "The canonical sequence: generate -> MinHash dedup (near-duplicate text) -> semantic dedup (embedding-cluster paraphrases) -> perplexity filter (small-LM garbage detection) -> decontaminate (benchmark items) -> quality subset (diverse high-quality slice) -> [optional curriculum] -> train. Memorize this." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What does decontamination remove, and why is it non-negotiable?", "options": [ "Low-quality samples — because they hurt training.", "Duplicates — because they waste compute.", "Benchmark test items (MMLU, GSM8K, HumanEval, MT-Bench) — because training on them makes every eval score you report fiction.", "Long samples — because they exceed the context window." ], "answer_index": 2, "rationale": "Decontamination removes benchmark test items. It is non-negotiable because contamination makes every benchmark score suspect — you ship a model that looks brilliant and breaks in production. The single most embarrassing way to retract a model." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "Your SFT set has many samples that are byte-identical copies of each other (copy-paste duplicates). What stage catches them, and what is the cheapest tool?", "options": [ "Semantic dedup with sentence-transformers — the only reliable approach.", "Exact-hash dedup (or MinHash at high threshold) — copy-paste duplicates share all shingles and all hashes. datasketch or text-dedup.", "Perplexity filtering — duplicates have similar perplexity.", "Decontamination — duplicates are benchmark items." ], "answer_index": 1, "rationale": "Byte-identical copies share all shingles and all hashes — exact-hash dedup catches them in O(n). MinHash also catches them (they have Jaccard = 1.0). You don't need embeddings for byte-identical duplicates; save semantic dedup for paraphrases." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You generated 5 paraphrased variants of every instruction with FT05's generator. MinHash catches none of them. Why, and what catches them?", "options": [ "MinHash is buggy — switch libraries.", "MinHash operates on shingles (exact word sequences); paraphrases use different words. Semantic dedup (embed + cluster + drop high-cosine pairs) catches them.", "The paraphrases are too short — increase shingle size.", "Perplexity filtering catches paraphrases because they have high perplexity." ], "answer_index": 1, "rationale": "MinHash is lexical — it sees exact word sequences. Paraphrases use different words with the same meaning, so their shingle sets do not overlap. Embed-based semantic dedup is the only stage that catches meaning-level duplicates." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You are building the decontamination scan for a math-reasoning model you will eval on GSM8K. What n-gram size, and against what set?", "options": [ "n=3 against the GSM8K *training* set — short n-grams catch more.", "n=13 against the GSM8K *test* set — long n-grams give strong evidence; scan against the set you will be evaluated on.", "n=1 (unigrams) against any math corpus — single words are sufficient.", "Skip the scan — GSM8K is small enough to not matter." ], "answer_index": 1, "rationale": "13-grams for natural text (8 for code). You scan against the TEST set of the benchmark you will be evaluated on — that is what would invalidate your eval. Short n-grams produce false positives on common phrases." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "Your decontamination scan finds that 0.5% of your 100K training set shares a 13-gram with MMLU. What do you do?", "options": [ "Keep them — 0.5% is negligible.", "Drop the entire 500 samples — conservative default. The downside of dropping a few legitimate samples is tiny; the downside of training on benchmark items is catastrophic.", "Drop only the contaminated span — surgical removal.", "Average the contaminated samples with non-contaminated ones." ], "answer_index": 1, "rationale": "Conservative default: drop the entire sample. Losing 500 of 100K (0.5%) costs essentially nothing. Training on benchmark items invalidates your eval. Span-level removal is an option but more error-prone." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You have a 50K-sample SFT set and need to reduce it to a 10K budget for training. What is the DCLM-endorsed approach?", "options": [ "Take the first 10K in file order.", "Random sample 10K.", "Quality subset selection: cluster embeddings, pick K samples closest to each centroid, or train a quality classifier and keep the top decile.", "Sort by length and keep the shortest 10K." ], "answer_index": 2, "rationale": "Quality subset selection. DCLM showed that quality filtering + cluster-and-select produces gains comparable to 4x compute scaling. Random sampling or file-order sampling leaves the full quality distribution in place; subset selection picks a diverse, high-quality slice." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You are doing continued pretraining (not SFT) and want to mitigate catastrophic forgetting. The NeurIPS 2025 finding says to apply which data-quality technique?", "options": [ "Aggressive MinHash dedup at threshold 0.7.", "Perplexity filtering — keep LOW-perplexity tokens; drop the high-perplexity tail. Tokens the model is surprised by push weights hard and cause forgetting.", "Decontamination against the base model's pretraining set.", "Curriculum learning from hard to easy." ], "answer_index": 1, "rationale": "Perplexity filtering, keeping the LOW-perplexity tail. The NeurIPS 2025 link: high-perplexity tokens produce large gradients and overwrite existing capabilities. Dropping the high-perplexity tail removes both garbage and the forgetting-causing signal." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "DCLM held the model and training recipe constant and varied only the data pipeline. The rigorous-dedup + perplexity + quality-selection pipeline beat the baseline by 6.5% absolute. What does this demonstrate about the steering thesis (FT00)?", "options": [ "It disproves the steering thesis — quality data must be injecting knowledge.", "It supports the thesis: the steering wheel (dataset) is everything. Same model, same compute — the only difference was the quality of the steering signal. Curation is the moat, and it is the cheapest gain you will ever get.", "It is irrelevant to the steering thesis — DCLM is about pretraining, not fine-tuning.", "It proves that more data beats better data." ], "answer_index": 1, "rationale": "The DCLM result is the empirical anchor for FT06's thesis and a direct application of FT00's 'your data matters more than your algorithm.' Same model, same compute, same training recipe — the only variable was data quality, and it produced a 6.5% absolute swing. The steering wheel is the moat." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team sets their MinHash Jaccard threshold to 0.5 'to catch more duplicates.' Their fine-tuned model's performance drops on every benchmark. What happened?", "options": [ "MinHash is fundamentally unreliable — switch to exact hashing.", "The threshold was too aggressive — it dropped legitimate near-paraphrases that the SFT set deliberately included to teach the model to generalize across phrasings. They removed the signal they wanted to train on.", "The model was undertrained.", "The base model was too small." ], "answer_index": 1, "rationale": "The 'dedup too aggressive' anti-pattern. A low Jaccard threshold (0.5) drops samples that are merely similar, not duplicates. In SFT data, paraphrases are often deliberate. Aggressive dedup removes the generalization signal. Calibrate on a held-out sample and inspect what gets dropped." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does perplexity filtering mitigate catastrophic forgetting in continued pretraining (NeurIPS 2025), and what is the practical implication?", "options": [ "It doesn't — the link is speculative.", "High-perplexity tokens are ones the model is 'surprised by' — they produce large gradients that overwrite existing capabilities. Dropping the high-perplexity tail keeps gradients small, so the model adapts without forgetting. Practical implication: a single filtering stage gives you two wins — garbage removal AND forgetting mitigation.", "Perplexity filtering injects new knowledge that prevents forgetting.", "Low-perplexity tokens are the dangerous ones — the model memorizes them." ], "answer_index": 1, "rationale": "Tokens the model is surprised by produce large gradients and push the weights hard — that is what causes forgetting. Keeping low-perplexity tokens keeps gradients small. The practical implication: perplexity filtering is two wins in one pass (garbage removal + forgetting mitigation)." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "A competitor copies your training algorithm (GRPO), your base model (Llama 3.x), and your hyperparameters. They cannot match your eval scores. Why, per the FT06 thesis?", "options": [ "They used the wrong random seed.", "Your cleaned dataset — with its specific dedup thresholds, perplexity cutoffs, decontamination n-grams, and quality-selection logic — is the one artifact that is hard to reproduce. Algorithms and bases are public; the data pipeline is the moat.", "Their GPU was slower.", "They forgot to quantize after training." ], "answer_index": 1, "rationale": "Algorithms are public (arXiv). Bases are public (HuggingFace). Your cleaned dataset is the one artifact competitors cannot easily copy — the specific thresholds, cutoffs, and selection logic are tuned to your domain and not visible in the released model. This is the strategic content of FT06's thesis." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Your team's pipeline runs MinHash dedup, then trains directly on the result — skipping perplexity filtering, decontamination, and quality selection. The model trains fine but underperforms a rival team's model on benchmarks. What is the most likely cause, and which stage was the cheapest to add?", "options": [ "The model is undertrained — add more epochs.", "Garbage in, garbage out — without perplexity filtering, garbage samples measurably degrade every benchmark. Perplexity filtering is the cheapest quality improvement you will ever get (a single pass with a small LM).", "MinHash is the wrong dedup algorithm.", "The base model is too small." ], "answer_index": 1, "rationale": "The 'no quality filtering' anti-pattern. The model does not ignore noise — it fits it. Perplexity filtering is a single pass with a small LM and produces measurable gains on every downstream benchmark. It is the cheapest quality improvement available." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team lead proposes 'curriculum teaching' — ordering the SFT set so the model 'learns' a new domain by seeing easy examples first, then hard ones. The model still fails the domain. Why, per the FT00 + FT06 synthesis?", "options": [ "The ordering was wrong — try hard-to-easy instead.", "Curriculum LEARNING (reordering for steering efficiency) is legitimate, but curriculum TEACHING (reordering to inject knowledge the base lacks) is the cardinal error of FT00. If the base does not know the domain, no ordering will fix it — you need a different base, CPT, or (usually) RAG.", "They need more samples in the curriculum.", "Curriculum learning only works for code, not text." ], "answer_index": 1, "rationale": "Curriculum learning reorders samples to make steering easier — a legitimate steering-aid technique. Curriculum teaching — the idea that ordering can inject knowledge — is the same cardinal error as FT00's 'teach via fine-tuning.' Reordering redirects probability mass; it does not add capability. If the base doesn't know the domain, you need a different intervention." } ] }