Diagrams — Module FT06: Dedup, Filter, Decontaminate

Module: FT06 — Dedup, Filter, Decontaminate Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.

Diagram 1 — The Full Clean Pipeline

Type: Sequential funnel Purpose: The single diagram that anchors the module. Memorize this sequence. Reading the diagram: Top to bottom. Every stage is a filter that removes samples. The samples you remove matter as much as the ones you keep.

flowchart TD
  Gen["GENERATE / COLLECT\nraw noisy corpus"]
  Min["MinHash dedup\nnear-duplicate text\n(datasketch / text-dedup)"]
  Sem["Semantic dedup\nembedding-cluster paraphrases\n(sentence-transformers)"]
  Ppl["Perplexity filter\nsmall LM flags garbage\n(kenlm)"]
  Dec["Decontaminate\nremove benchmark items\n(MMLU / GSM8K / HumanEval)"]
  Qual["Quality subset\ndiverse high-quality slice\n(cluster-and-select)"]
  Cur["Curriculum order\neasy -> hard (optional)"]
  Train["TRAIN"]

  Gen --> Min --> Sem --> Ppl --> Dec --> Qual --> Cur --> Train

  style Gen fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Min fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Sem fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Ppl fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Dec fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8
  style Qual fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Cur fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style Train fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8

Diagram 2 — MinHash + LSH Mechanism

Type: Pipeline / mechanism Purpose: How near-duplicate detection works at scale. Four steps. Reading the diagram: Left to right. Shingle → Hash → Band → Verify. The LSH banding is the trick that makes this scale beyond O(n²).

flowchart LR
  Doc1["Document A\n'the quick brown fox'"]
  Doc2["Document B\n'the quick brown cat'"]

  subgraph S1["Step 1: SHINGLE"]
    SH["3-word overlapping n-grams\n{the quick brown, quick brown fox}"]
  end
  subgraph S2["Step 2: HASH (MinHash signature)"]
    MH["k independent hash fns\nkeep min hash per fn\n-> k-length signature"]
  end
  subgraph S3["Step 3: BAND (LSH)"]
    BAND["chop signature into b bands of r rows\nhash each band\nshared band hash -> candidate pair"]
  end
  subgraph S4["Step 4: VERIFY"]
    VER["compute actual Jaccard\n>= threshold (0.8-0.9)\n-> drop as duplicate"]
  end

  Doc1 --> SH
  Doc2 --> SH
  SH --> MH --> BAND --> VER

  style Doc1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Doc2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style S1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style S2 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style S3 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style S4 fill:#08080c,stroke:#5eead4,stroke-width:1.5px,color:#5eead4

Diagram 3 — Decontamination Flow

Type: Side-by-side scan Purpose: How benchmark contamination is detected and removed. Non-negotiable for honest evals. Reading the diagram: Two parallel inputs (your training set + the benchmark test sets). Both are reduced to n-gram sets. The intersection is the contaminated samples. Drop them.

flowchart LR
  Bench["Benchmark test sets\nMMLU · GSM8K · HumanEval · MT-Bench"]
  Train["Your training set\nraw or partially-cleaned"]

  subgraph NG["Extract n-grams (n=13 text, n=8 code)"]
    BN["benchmark n-gram set"]
    TN["training n-gram set"]
  end

  Isec["Intersection\n= contaminated samples"]
  Clean["Clean training set\n(eval is now honest)"]

  Bench --> NG
  Train --> NG
  BN --> Isec
  TN --> Isec
  Isec -.remove.-> Clean

  style Bench fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8
  style Train fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style NG fill:#08080c,stroke:rgba(240,128,128,0.4),color:#f08080
  style Isec fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
  style Clean fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8

Diagram 4 — The Data-Quality Funnel

Type: Quantitative funnel Purpose: How a raw corpus shrinks through the pipeline. A typical messy set loses 30-60% of samples between generate and train — and that is correct. Reading the diagram: Top to bottom. The numbers are illustrative (your retention will vary) but the shape is real: every stage removes samples, and the final clean set is materially smaller than the raw input.

flowchart TD
  A["10,000 raw samples\n(generate / collect)"]
  B["8,200 after MinHash dedup\n(-1,800 near-duplicates)"]
  C["6,800 after semantic dedup\n(-1,400 paraphrases)"]
  D["6,100 after perplexity filter\n(-700 garbage)"]
  E["5,950 after decontamination\n(-150 benchmark items)"]
  F["2,000 final quality subset\n(diverse, high-quality slice)"]
  T["TRAIN\nretention 20% of raw"]

  A --> B --> C --> D --> E --> F --> T

  style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style B fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style C fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style D fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style E fill:#14141f,stroke:#f08080,color:#e4e4e8
  style F fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style T fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4

Diagram 5 — Perplexity Filtering Concept

Type: Distribution / threshold Purpose: Why dropping high-perplexity samples removes garbage AND mitigates catastrophic forgetting. Reading the diagram: The x-axis is sample perplexity (low = expected by the small clean-trained LM; high = surprising). The right tail (high perplexity) is dropped. It overlaps with both "garbage" and "tokens that push weights hard" — the NeurIPS 2025 link.

flowchart LR
  subgraph Dist["Perplexity distribution of training samples"]
    direction LR
    Low["LOW perplexity\nmodel expected it\nclean, well-formed\n-> keep"]
    Mid["MID perplexity\nnormal variation\n-> keep"]
    High["HIGH perplexity\nmodel surprised\nGARBAGE / OCR / noise\n+ tokens that push weights hard\n-> DROP"]
  end

  Cut["cutoff: top ~10% dropped"]

  Low --> Mid --> High
  High -.-> Cut

  style Low fill:#14141f,stroke:#82e0aa,color:#e4e4e8
  style Mid fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style High fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8
  style Cut fill:#08080c,stroke:rgba(240,128,128,0.5),stroke-dasharray: 4 2,color:#f08080
  style Dist fill:#08080c,stroke:rgba(255,255,255,0.06),color:#9494a0

Validation notes

All five diagrams use the course design system colors: #14141f panel fill, #5eead4 accent for primary, #f08080 (danger) for the decontamination / garbage signal, rgba(255,255,255,0.12) for secondary borders, #e4e4e8 / #9494a0 for text.
Paste each into Mermaid Live Editor to render. All use stable Mermaid syntax (flowchart with subgraphs) supported in current Mermaid (v10.4+).
For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.
The decontamination diagram (Diagram 3) and the funnel (Diagram 4) deliberately use the #f08080 danger accent — these are the "non-negotiable" stages of the pipeline.

# Diagrams — Module FT06: Dedup, Filter, Decontaminate **Module**: FT06 — Dedup, Filter, Decontaminate **Diagram count**: 5 **Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live). --- ## Diagram 1 — The Full Clean Pipeline **Type**: Sequential funnel **Purpose**: The single diagram that anchors the module. Memorize this sequence. **Reading the diagram**: Top to bottom. Every stage is a filter that removes samples. The samples you *remove* matter as much as the ones you keep. ```mermaid flowchart TD Gen["GENERATE / COLLECT\nraw noisy corpus"] Min["MinHash dedup\nnear-duplicate text\n(datasketch / text-dedup)"] Sem["Semantic dedup\nembedding-cluster paraphrases\n(sentence-transformers)"] Ppl["Perplexity filter\nsmall LM flags garbage\n(kenlm)"] Dec["Decontaminate\nremove benchmark items\n(MMLU / GSM8K / HumanEval)"] Qual["Quality subset\ndiverse high-quality slice\n(cluster-and-select)"] Cur["Curriculum order\neasy -> hard (optional)"] Train["TRAIN"] Gen --> Min --> Sem --> Ppl --> Dec --> Qual --> Cur --> Train style Gen fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Min fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style Sem fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style Ppl fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style Dec fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8 style Qual fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style Cur fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0 style Train fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 ``` --- ## Diagram 2 — MinHash + LSH Mechanism **Type**: Pipeline / mechanism **Purpose**: How near-duplicate detection works at scale. Four steps. **Reading the diagram**: Left to right. Shingle → Hash → Band → Verify. The LSH banding is the trick that makes this scale beyond O(n²). ```mermaid flowchart LR Doc1["Document A\n'the quick brown fox'"] Doc2["Document B\n'the quick brown cat'"] subgraph S1["Step 1: SHINGLE"] SH["3-word overlapping n-grams\n{the quick brown, quick brown fox}"] end subgraph S2["Step 2: HASH (MinHash signature)"] MH["k independent hash fns\nkeep min hash per fn\n-> k-length signature"] end subgraph S3["Step 3: BAND (LSH)"] BAND["chop signature into b bands of r rows\nhash each band\nshared band hash -> candidate pair"] end subgraph S4["Step 4: VERIFY"] VER["compute actual Jaccard\n>= threshold (0.8-0.9)\n-> drop as duplicate"] end Doc1 --> SH Doc2 --> SH SH --> MH --> BAND --> VER style Doc1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style Doc2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style S1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4 style S2 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4 style S3 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4 style S4 fill:#08080c,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 ``` --- ## Diagram 3 — Decontamination Flow **Type**: Side-by-side scan **Purpose**: How benchmark contamination is detected and removed. Non-negotiable for honest evals. **Reading the diagram**: Two parallel inputs (your training set + the benchmark test sets). Both are reduced to n-gram sets. The intersection is the contaminated samples. Drop them. ```mermaid flowchart LR Bench["Benchmark test sets\nMMLU · GSM8K · HumanEval · MT-Bench"] Train["Your training set\nraw or partially-cleaned"] subgraph NG["Extract n-grams (n=13 text, n=8 code)"] BN["benchmark n-gram set"] TN["training n-gram set"] end Isec["Intersection\n= contaminated samples"] Clean["Clean training set\n(eval is now honest)"] Bench --> NG Train --> NG BN --> Isec TN --> Isec Isec -.remove.-> Clean style Bench fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8 style Train fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style NG fill:#08080c,stroke:rgba(240,128,128,0.4),color:#f08080 style Isec fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080 style Clean fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 ``` --- ## Diagram 4 — The Data-Quality Funnel **Type**: Quantitative funnel **Purpose**: How a raw corpus shrinks through the pipeline. A typical messy set loses 30-60% of samples between generate and train — and that is correct. **Reading the diagram**: Top to bottom. The numbers are illustrative (your retention will vary) but the shape is real: every stage removes samples, and the final clean set is materially smaller than the raw input. ```mermaid flowchart TD A["10,000 raw samples\n(generate / collect)"] B["8,200 after MinHash dedup\n(-1,800 near-duplicates)"] C["6,800 after semantic dedup\n(-1,400 paraphrases)"] D["6,100 after perplexity filter\n(-700 garbage)"] E["5,950 after decontamination\n(-150 benchmark items)"] F["2,000 final quality subset\n(diverse, high-quality slice)"] T["TRAIN\nretention 20% of raw"] A --> B --> C --> D --> E --> F --> T style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style B fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style C fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style D fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style E fill:#14141f,stroke:#f08080,color:#e4e4e8 style F fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style T fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4 ``` --- ## Diagram 5 — Perplexity Filtering Concept **Type**: Distribution / threshold **Purpose**: Why dropping high-perplexity samples removes garbage AND mitigates catastrophic forgetting. **Reading the diagram**: The x-axis is sample perplexity (low = expected by the small clean-trained LM; high = surprising). The right tail (high perplexity) is dropped. It overlaps with both "garbage" and "tokens that push weights hard" — the NeurIPS 2025 link. ```mermaid flowchart LR subgraph Dist["Perplexity distribution of training samples"] direction LR Low["LOW perplexity\nmodel expected it\nclean, well-formed\n-> keep"] Mid["MID perplexity\nnormal variation\n-> keep"] High["HIGH perplexity\nmodel surprised\nGARBAGE / OCR / noise\n+ tokens that push weights hard\n-> DROP"] end Cut["cutoff: top ~10% dropped"] Low --> Mid --> High High -.-> Cut style Low fill:#14141f,stroke:#82e0aa,color:#e4e4e8 style Mid fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style High fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8 style Cut fill:#08080c,stroke:rgba(240,128,128,0.5),stroke-dasharray: 4 2,color:#f08080 style Dist fill:#08080c,stroke:rgba(255,255,255,0.06),color:#9494a0 ``` --- ## Validation notes - All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `#f08080` (danger) for the decontamination / garbage signal, `rgba(255,255,255,0.12)` for secondary borders, `#e4e4e8` / `#9494a0` for text. - Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart` with subgraphs) supported in current Mermaid (v10.4+). - For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js. - The decontamination diagram (Diagram 3) and the funnel (Diagram 4) deliberately use the `#f08080` danger accent — these are the "non-negotiable" stages of the pipeline.