"What is the central thesis of FT06, in one sentence?"	"Data quality is where 80% of fine-tuning gains are. Generation is cheap; curation is the moat. Anyone can generate a million samples — almost nobody can clean them."	c3::ft06::recall
"Name the stages of the full clean pipeline, in order."	"Generate/collect -> MinHash dedup -> semantic dedup -> perplexity filter -> decontaminate -> quality subset -> [optional: curriculum order easy->hard] -> train. Each stage is a funnel that removes samples; the samples you remove matter as much as the ones you keep."	c3::ft06::recall
"What is MinHash and what does it estimate?"	"A probabilistic hashing technique for near-duplicate detection. For each document it produces a 'MinHash signature' (the min hash value per hash function across all shingles). The probability two docs share a min hash equals their Jaccard similarity (intersection over union of their shingle sets)."	c3::ft06::recall
"What are the four steps of the MinHash + LSH dedup pipeline?"	"(1) Shingle the document into overlapping n-grams. (2) Hash each shingle with k independent hash functions, keep the min per function -> k-length signature. (3) Band the signature into b bands of r rows, hash each band — shared band hash = candidate pair. (4) Verify candidate pairs by computing actual Jaccard; >= threshold -> drop."	c3::ft06::recall
"What problem does LSH (Locality-Sensitive Hashing) solve in MinHash dedup?"	"It makes MinHash scale. Without LSH you must compare every pair of documents (O(n^2)). LSH bands the signature so only documents that share at least one band hash become 'candidate pairs' worth comparing directly. Massive speedup for a tiny false-negative rate."	c3::ft06::application
"What is a shingle, and why use it instead of hashing the whole document?"	"A shingle is an overlapping n-gram (typically 3-5 words) extracted from a document. Hashing the whole document only catches byte-identical copies. Shingles capture LOCAL structure, so two documents with the same shingles are talking about the same thing in the same words — even if they differ by a few characters."	c3::ft06::application
"Why does MinHash miss paraphrases, and what catches them?"	"MinHash operates on shingles (exact word sequences). It cannot detect that 'the cat sat on the mat' and 'a feline rested on the rug' are the same meaning in different words. Semantic dedup catches them: embed every sample with a sentence-transformer, cluster, drop intra-cluster pairs above ~0.9-0.95 cosine similarity."	c3::ft06::analysis
"What does the Jaccard threshold control in MinHash dedup, and what are the failure modes if set wrong?"	"It controls how aggressive dedup is. Too low (0.5) drops legitimate near-paraphrases you wanted to keep. Too high (0.99) only catches exact copies. DCLM uses ~0.8; for SFT data where paraphrases are often deliberate, 0.9 is safer. Calibrate on a held-out sample and inspect what gets dropped."	c3::ft06::analysis
"What is perplexity filtering and how does it work?"	"Train a small LM (n-gram kenlm or a tiny transformer) on known-clean data, then score every sample in your training set. Drop the high-perplexity tail (typically top ~10%) — those are samples the clean model finds 'surprising', which is almost always garbage (mojibake, OCR errors, random tokens, code corruption)."	c3::ft06::recall
"What is the NeurIPS 2025 link between perplexity filtering and catastrophic forgetting?"	"In continued pretraining, keeping only LOW-perplexity tokens (ones the model already handles well) produces small gradients and lets the model adapt without overwriting existing capabilities. HIGH-perplexity tokens push the weights hard and cause forgetting. Perplexity filtering = two wins in one pass: removes garbage AND mitigates forgetting."	c3::ft06::analysis
"What is decontamination, and why is it non-negotiable?"	"Removing benchmark test items (MMLU, GSM8K, HumanEval, MT-Bench) from your training data via n-gram matching. Non-negotiable because skipping it means every benchmark score you report is fiction — the model has seen the answers. It is the single most embarrassing way to retract a model after launch."	c3::ft06::recall
"What n-gram size is standard for decontamination, and why?"	"13-grams for natural text, 8-grams for code (code has higher entropy per token). A single shared 8-gram ('the quick brown fox jumps over') is weak evidence — common phrases trip it. A shared 13-gram is essentially unique to that document. Use the longer n-gram for stronger evidence."	c3::ft06::application
"Describe the standard n-gram contamination-detection algorithm."	"(1) Collect benchmark test sets (MMLU, GSM8K, HumanEval, etc.). (2) Extract n-grams from each item, store in a set. (3) Extract n-grams from each training sample. (4) If any training sample shares an n-gram with a benchmark item, flag it. (5) Drop flagged samples (conservative) or just the contaminated span (aggressive)."	c3::ft06::application
"What tools implement MinHash dedup and decontamination respectively?"	"MinHash: 'datasketch' (the canonical Python library; MinHash + MinHashLSH) and 'text-dedup' (higher-level CLI wrapping MinHash, SimHash, exact). Decontamination: EleutherAI's LM-eval-harness decontamination tooling, HuggingFace datatrove's decontamination block, or OpenAI's original GPT-3 decontamination scripts."	c3::ft06::recall
"What is quality subset selection, and what are the three approaches?"	"Instead of training on all N samples, pick the best M. (1) Cluster-and-select: cluster embeddings, pick K samples closest to each centroid (diverse, representative). (2) Quality classifier: train fastText on (high-quality, low-quality) pairs, keep top decile. (3) Diversity sampling: greedily pick samples maximizing embedding-distance from already-picked ones."	c3::ft06::recall
"What did DCLM (DataComp-LM) demonstrate about curation vs compute?"	"Holding the model and training recipe constant and varying only the data pipeline, the rigorous-dedup + perplexity-filter + quality-selection pipeline produced a 6.5% absolute improvement on downstream benchmarks. Quality filtering ALONE produced gains comparable to scaling compute 4x. Curation is compute."	c3::ft06::analysis
"What is curriculum learning, and what is its honest status?"	"Ordering samples easy -> hard to stabilize training. ACTIVE RESEARCH with mixed empirical results — some papers show gains in low-data regimes, others find no effect at scale. Commonly applied informally (easy subset first, hard later) because it costs nothing to try. Do NOT confuse with curriculum teaching — reordering does not inject knowledge."	c3::ft06::analysis
"Why is 'generate more data' the wrong instinct when you have a noisy dataset?"	"Doubling a noisy dataset doubles the noise. If your data has 30% duplicates and 10% garbage, generating more of the same gives you a bigger pile of the same bad data. Clean first (dedup, perplexity filter, decontaminate), scale second. The moat is the cleaned subset, not the raw volume."	c3::ft06::analysis
"What is the difference between semantic dedup and quality subset selection?"	"Both use embeddings. Semantic dedup DROPS near-duplicates (pairs above ~0.9 cosine) to remove redundancy. Quality subset selection PICKS a diverse, high-quality slice (cluster-and-select, top-decile by classifier) to reduce a large clean pool to a fixed budget. Dedup is about removing; subset selection is about choosing."	c3::ft06::application
"Why is the data pipeline the part competitors cannot copy?"	"Algorithms are public (LoRA, DPO, GRPO — all on arXiv). Bases are public (Llama, Qwen, Mistral — all downloadable). Your cleaned dataset — with its specific dedup thresholds, perplexity cutoffs, decontamination n-grams, and quality-selection logic — is the one artifact that is hard to reproduce. That is where your edge lives."	c3::ft06::analysis
"What is the 'dedup too aggressive' anti-pattern?"	"Setting the MinHash Jaccard threshold too low (e.g., 0.5) or semantic-similarity threshold too low (e.g., 0.85) drops legitimate samples that happen to be similar. In SFT data, paraphrases are often DELIBERATE — you want the model to generalize across phrasings. Aggressive dedup removes that signal. Calibrate the threshold on a held-out sample and inspect what gets dropped."	c3::ft06::analysis
"What is the 'no quality filtering' anti-pattern, and why does it fail?"	"'The model will learn to ignore the noise.' It will not — it fits it. A 10% garbage rate is not a 10% efficiency loss; it is a measurable quality degradation on every downstream benchmark. Perplexity filter + quality subset selection is the cheapest quality improvement you will ever get. Garbage in, garbage out."	c3::ft06::analysis
"Sketch a typical retention shape through the clean pipeline."	"A 10K raw corpus typically loses 20-50% between generate and train. Example: MinHash removes ~15% (near-duplicates), semantic dedup ~10% (paraphrases), perplexity filter ~10% (garbage, fixed by cutoff), decontamination ~1-2% (benchmark items), quality subset selection reduces to a fixed budget (e.g., 2K of 6K). Final clean set: ~20-70% of raw, depending on whether you subset-select."	c3::ft06::application
"Why is the decontamination stage marked as 'danger / non-negotiable' in the diagrams?"	"It is the one stage of the pipeline where skipping has CATASTROPHIC downside and ZERO upside. Skipping dedup costs efficiency. Skipping perplexity filtering costs some garbage. Skipping decontamination costs you the ability to trust ANY benchmark you report. There is no scenario in which skipping decontamination is the right call."	c3::ft06::analysis
"What is the difference between curriculum learning and curriculum teaching?"	"Curriculum LEARNING reorders samples (easy -> hard) to make steering easier — a legitimate steering-aid technique, no knowledge injected. Curriculum TEACHING is the idea that ordering can inject knowledge the base lacks — the same cardinal error as FT00's 'teach via fine-tuning'. If the base doesn't know the domain, reordering will not fix it."	c3::ft06::analysis