Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT06 — Dedup, Filter, Decontaminate Duration: 60–90 minutes (the data-quality lab — where most gains are) Environment: Python 3.11+. CPU-only is fine — sentence-transformers runs on CPU for 10K samples in a few minutes. ~2GB free disk for the embedding model.
By the end of this lab you will have:
This lab is the data-quality module felt end-to-end. You will not train a model — the point is that cleaning is upstream of training and yields disproportionate gains per hour invested.
python3.11 -m venv ft06-env && source ft06-env/bin/activate
# The data-quality stack — all CPU-friendly
pip install -q datasketch sentence-transformers numpy scikit-learn
# Optional, for faster embeddings if you have a GPU/MPS:
# pip install -q torch (already pulled in by sentence-transformers)
Verify:
import datasketch, sentence_transformers, numpy, sklearn
print(f"datasketch: {datasketch.__version__ if hasattr(datasketch,'__version__') else 'ok'}")
print(f"sentence-transformers: {sentence_transformers.__version__}")
print(f"numpy: {numpy.__version__}")
print(f"scikit-learn: {sklearn.__version__}")
This is a self-contained synthetic generator. It plants known pathologies so we can measure how well the pipeline catches them. Save as generate_messy.py.
"""generate_messy.py — synthetic 10K SFT dataset with planted pathologies."""
import json, random, string, hashlib
from pathlib import Path
random.seed(42)
OUT = Path("messy_sft.jsonl")
N_BASE = 6000 # unique "good" samples
N_NEAR_DUP = 1500 # near-duplicates (1-3 char edits)
N_PARAPHRASE = 1000 # semantic paraphrases (template rewording)
N_GARBAGE = 800 # high-perplexity noise
N_CONTAMINATED = 200 # planted benchmark items (GSM8K-style + MMLU-style)
# Total: 9500, plus ~500 more base to reach ~10K
N_TOTAL_TARGET = 10000
# A pool of clean, well-formed instruction/response pairs
CLEAN_TEMPLATES = [
("Explain what a hash function is.", "A hash function maps data of arbitrary size to a fixed-size value, typically for fast lookup or integrity checks."),
("What is a closure in JavaScript?", "A closure is a function that retains access to variables from its lexical scope, even when invoked outside that scope."),
("Summarize the CAP theorem.", "The CAP theorem says a distributed system can guarantee at most two of: consistency, availability, partition tolerance."),
("How does TCP establish a connection?", "TCP uses a three-way handshake: SYN, SYN-ACK, ACK, after which the connection is established."),
("Define eventual consistency.", "Eventual consistency guarantees that, given no new writes, all replicas converge to the same value over time."),
("What is idempotency?", "An operation is idempotent if calling it once has the same effect as calling it multiple times."),
("Explain the single responsibility principle.", "A class should have one, and only one, reason to change."),
("What does ACID stand for?", "Atomicity, Consistency, Isolation, Durability — the four guarantees of transactional systems."),
("Describe a bloom filter.", "A probabilistic data structure that tests set membership, with possible false positives but no false negatives."),
("What is a foreign key?", "A column referencing the primary key of another table, enforcing referential integrity."),
("Explain lazy evaluation.", "An evaluation strategy that delays computation until the result is actually needed."),
("What is the actor model?", "A concurrency model where actors communicate exclusively by exchanging messages, with no shared state."),
("Define immutability.", "An object whose state cannot be modified after creation."),
("What is a mutex?", "A synchronization primitive granting exclusive access to a resource."),
("Explain backpressure.", "A mechanism to handle a producer that is faster than its consumer, by signaling the producer to slow down."),
]
# Planted benchmark-style contamination items (you would NEVER train on these)
GSM8K_STYLE = [
("Solve: A store sells pencils at $0.50 each. If Maria buys 12 pencils, how much does she spend?",
"Maria spends 12 * $0.50 = $6.00."),
("Solve: A train travels 60 mph for 2.5 hours. How far does it go?",
"Distance = 60 * 2.5 = 150 miles."),
("Solve: A recipe needs 3/4 cup of sugar. If you make 4 batches, how much sugar?",
"4 * 3/4 = 3 cups of sugar."),
]
MMLU_STYLE = [
("Which sorting algorithm has O(n log n) average time complexity? A) Bubble B) Merge C) Selection D) Insertion",
"B) Merge sort has O(n log n) average time complexity."),
("What does the acronym SQL stand for? A) Simple Query Language B) Structured Query Language C) Standard Query Logic D) Sequential Query Language",
"B) Structured Query Language."),
("In Python, which keyword defines a function? A) func B) def C) function D) define",
"B) def."),
]
def random_garbage(min_len=20, max_len=120):
"""High-perplexity noise — random chars, mojibake, code-corruption."""
kind = random.choice(["rand_chars", "repeated", "mojibake", "mixed"])
if kind == "rand_chars":
return "".join(random.choices(string.printable + "é¥Ω£", k=random.randint(min_len, max_len)))
if kind == "repeated":
return ("".join(random.choices(string.ascii_letters, k=8)) + " ") * random.randint(5, 20)
if kind == "mojibake":
return "".join(random.choices("é¥Ω£ñ¥Ω", k=random.randint(min_len, max_len)))
return "asdf qwer " + "".join(random.choices("!@#$%^&*()1234567890", k=random.randint(min_len, max_len)))
def near_dup_of(text, n_edits=2):
"""1-3 char edits — invisible to exact hashing, caught by MinHash."""
chars = list(text)
for _ in range(n_edits):
if not chars: break
op = random.choice(["swap", "delete", "replace"])
i = random.randrange(len(chars))
if op == "swap" and i + 1 < len(chars):
chars[i], chars[i+1] = chars[i+1], chars[i]
elif op == "delete":
chars.pop(i)
else:
chars[i] = random.choice(string.ascii_lowercase)
return "".join(chars)
PARAPHRASE_PATTERNS = [
("Explain", "Describe in your own words"),
("What is", "Define"),
("How does", "Describe the way"),
("Define", "Briefly explain"),
("Describe", "Explain"),
]
def paraphrase_of(text):
"""Template rewording — invisible to MinHash, caught by semantic dedup."""
for a, b in PARAPHRASE_PATTERNS:
if text.startswith(a):
return b + text[len(a):]
return text + " (please.)"
def gen():
rows = []
# Base clean samples (deduplicated by template at generation; we still plant dupes below)
for _ in range(N_BASE):
q, a = random.choice(CLEAN_TEMPLATES)
# Light template variation so they aren't byte-identical
prefix = random.choice(["", "Q: ", "Question: ", ""])
rows.append({"prompt": prefix + q, "response": a, "label": "clean"})
# Near-duplicates of base samples
for i in range(N_NEAR_DUP):
q, a = random.choice(CLEAN_TEMPLATES)
rows.append({"prompt": near_dup_of(q, n_edits=random.choice([1,2,3])),
"response": near_dup_of(a, n_edits=1), "label": "near_dup"})
# Semantic paraphrases
for i in range(N_PARAPHRASE):
q, a = random.choice(CLEAN_TEMPLATES)
rows.append({"prompt": paraphrase_of(q), "response": a, "label": "paraphrase"})
# Garbage
for i in range(N_GARBAGE):
rows.append({"prompt": random_garbage(), "response": random_garbage(max_len=40),
"label": "garbage"})
# Contamination — benchmark items
for i in range(N_CONTAMINATED // 2):
q, a = random.choice(GSM8K_STYLE)
rows.append({"prompt": q, "response": a, "label": "contamination_gsm8k"})
for i in range(N_CONTAMINATED // 2):
q, a = random.choice(MMLU_STYLE)
rows.append({"prompt": q, "response": a, "label": "contamination_mmlu"})
# Pad to ~10K with extra clean base samples
while len(rows) < N_TOTAL_TARGET:
q, a = random.choice(CLEAN_TEMPLATES)
rows.append({"prompt": q, "response": a, "label": "clean"})
random.shuffle(rows)
with OUT.open("w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} samples to {OUT}")
# Print ground-truth counts (do NOT use these in the pipeline — they're for scoring only)
from collections import Counter
counts = Counter(r["label"] for r in rows)
print("Ground-truth label distribution:")
for k, v in counts.most_common():
print(f" {k}: {v}")
if __name__ == "__main__":
gen()
Run it:
python generate_messy.py
Record: the total count (should be ~10,000) and the ground-truth label distribution. You will use these to score the pipeline at the end.
What just happened (the teaching moment): You built a dataset with known pathologies — near-duplicates, paraphrases, garbage, and planted benchmark items. This lets you measure exactly what the clean pipeline catches and what it misses. In a real corpus you don't have ground-truth labels; here, you do.
"""phase2_minhash.py"""
import json
from datasketch import MinHash, MinHashLSH
from pathlib import Path
IN = Path("messy_sft.jsonl")
OUT = Path("after_minhash.jsonl")
def shingles(text, k=3):
tokens = text.split()
if len(tokens) < k:
return [" ".join(tokens)]
return [" ".join(tokens[i:i+k]) for i in range(len(tokens) - k + 1)]
def make_minhash(text, num_perm=128):
m = MinHash(num_perm=num_perm)
for sh in shingles(text):
m.update(sh.encode("utf-8"))
return m
rows = [json.loads(l) for l in IN.open()]
print(f"Loaded {len(rows)} samples")
# Build the LSH index over the concatenated prompt+response
THRESHOLD = 0.9 # Jaccard >= 0.9 -> near-duplicate
lsh = MinHashLSH(threshold=THRESHOLD, num_perm=128)
signatures = []
for i, r in enumerate(rows):
sig = make_minhash(r["prompt"] + " " + r["response"])
signatures.append(sig)
lsh.insert(i, sig)
# Identify duplicates: for each row, find existing near-duplicates
keep_idx = set(range(len(rows)))
dupes_found = 0
for i in range(len(rows)):
if i not in keep_idx:
continue
candidates = [c for c in lsh.query(signatures[i]) if c > i and c in keep_idx]
for c in candidates:
keep_idx.discard(c)
dupes_found += 1
clean = [rows[i] for i in sorted(keep_idx)]
with OUT.open("w") as f:
for r in clean:
f.write(json.dumps(r) + "\n")
print(f"MinHash dedup (threshold={THRESHOLD}): removed {dupes_found} near-duplicates")
print(f"Kept {len(clean)} / {len(rows)} = {100*len(clean)/len(rows):.1f}%")
Run it:
python phase2_minhash.py
Record: number of near-duplicates removed, retention percentage. Expect to catch a large chunk of the near_dup label (MinHash sees the 1–3 char edits).
What just happened: MinHash + LSH caught near-duplicates that exact hashing would have missed entirely. A single shared character change is invisible to exact hashing but shows up as a high-Jaccard signature here.
MinHash catches lexical duplicates. It misses paraphrases. Time for embeddings.
"""phase3_semantic.py"""
import json, numpy as np
from pathlib import Path
from sentence_transformers import SentenceTransformer
from sklearn.cluster import MiniBatchKMeans
IN = Path("after_minhash.jsonl")
OUT = Path("after_semantic.jsonl")
rows = [json.loads(l) for l in IN.open()]
texts = [r["prompt"] + " " + r["response"] for r in rows]
print(f"Loaded {len(rows)} samples — embedding (CPU; ~2 min for 10K)...")
model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dim, fast on CPU
emb = model.encode(texts, batch_size=64, show_progress_bar=True, normalize_embeddings=True)
print(f"Embeddings: {emb.shape}")
# Cluster, then drop near-duplicates within each cluster
N_CLUSTERS = min(200, len(rows) // 10)
km = MiniBatchKMeans(n_clusters=N_CLUSTERS, random_state=42, batch_size=256)
cluster_ids = km.fit_predict(emb)
SIM_THRESHOLD = 0.92 # cosine >= 0.92 -> semantic duplicate
keep = np.ones(len(rows), dtype=bool)
dropped = 0
for c in range(N_CLUSTERS):
idx = np.where(cluster_ids == c)[0]
if len(idx) < 2:
continue
sub = emb[idx]
# Pairwise cosine (already normalized -> dot product = cosine)
sim = sub @ sub.T
# Greedy: keep first, drop anyone >= threshold that hasn't been dropped
local_dropped = set()
for ii in range(len(idx)):
if idx[ii] in local_dropped:
continue
for jj in range(ii + 1, len(idx)):
if idx[jj] in local_dropped:
continue
if sim[ii, jj] >= SIM_THRESHOLD:
local_dropped.add(idx[jj])
keep[idx[jj]] = False
dropped += 1
clean = [rows[i] for i in range(len(rows)) if keep[i]]
with OUT.open("w") as f:
for r in clean:
f.write(json.dumps(r) + "\n")
print(f"Semantic dedup (cosine >= {SIM_THRESHOLD}): removed {dropped} paraphrases")
print(f"Kept {len(clean)} / {len(rows)} = {100*len(clean)/len(rows):.1f}%")
Run it:
python phase3_semantic.py
Record: paraphrases removed, retention percentage. Expect to catch a chunk of the paraphrase label — the ones MinHash missed.
What just happened: The sentence-transformer embedded every sample into a 384-dim semantic space. Clustering grouped paraphrases together; pairwise cosine within clusters caught the reworded versions. This is the stage that only semantic dedup can do.
Training a full kenlm model is out of scope for this lab. We approximate with a length-and-tokens heuristic that mimics perplexity filtering: well-formed English is smooth; garbage is jagged. (In production, use kenlm trained on Wikipedia-quality text.)
"""phase4_perplexity.py"""
import json, re, string
from pathlib import Path
import numpy as np
IN = Path("after_semantic.jsonl")
OUT = Path("after_perplexity.jsonl")
rows = [json.loads(l) for l in IN.open()]
print(f"Loaded {len(rows)} samples")
def approx_perplexity(text):
"""A cheap proxy for perplexity: ratio of 'weird' tokens to total tokens.
Real pipelines train a small LM on clean data and score with that.
Here we approximate: high weirdness score == high perplexity == likely garbage.
"""
text = text.strip()
if not text:
return 1e6
tokens = re.findall(r"\S+", text)
if not tokens:
return 1e6
weird = 0
for tok in tokens:
# Penalize tokens with lots of non-alphanumeric chars (mojibake, symbols, code-corruption)
non_alpha = sum(1 for c in tok if not c.isalpha() and c not in ".,;:'\"!?-()/$%")
if non_alpha / max(len(tok), 1) > 0.5:
weird += 1
# Penalize very long runs of a single char ("aaaaaaa")
if len(set(tok)) == 1 and len(tok) > 4:
weird += 1
# Penalize tokens with no vowels (likely noise)
if tok.isalpha() and not any(v in tok.lower() for v in "aeiou"):
weird += 0.5
return weird / len(tokens)
scores = [(approx_perplexity(r["prompt"] + " " + r["response"]), i) for i, r in enumerate(rows)]
scores.sort() # ascending — lowest weirdness first
# Keep the bottom 90% (drop the top 10% highest-perplexity)
cutoff = int(0.90 * len(scores))
keep_idx = sorted(i for _, i in scores[:cutoff])
clean = [rows[i] for i in keep_idx]
with OUT.open("w") as f:
for r in clean:
f.write(json.dumps(r) + "\n")
dropped = len(rows) - len(clean)
print(f"Perplexity filter (drop top 10%): removed {dropped} likely-garbage samples")
print(f"Kept {len(clean)} / {len(rows)} = {100*len(clean)/len(rows):.1f}%")
Run it:
python phase4_perplexity.py
Record: garbage removed. The drop rate is fixed (10%) by design; in production you'd tune this against a held-out quality sample.
The teaching moment (and the NeurIPS 2025 link): Perplexity filtering is two wins in one. It removes garbage and — when applied to continued pretraining — measurably mitigates catastrophic forgetting. High-perplexity tokens push the weights hard; dropping them lets the model adapt without overwriting existing capabilities.
We planted 200 benchmark-style items. The pipeline must catch them. Here is the n-gram scan against the planted benchmark items (in a real pipeline you would scan against the actual MMLU / GSM8K / HumanEval / MT-Bench test sets).
"""phase5_decontaminate.py"""
import json
from pathlib import Path
IN = Path("after_perplexity.jsonl")
OUT = Path("after_decontaminate.jsonl")
# In production, this is the union of n-gram sets from MMLU, GSM8K, HumanEval, MT-Bench test items.
# Here we reconstruct the planted benchmark items so we can scan against them.
BENCHMARK_ITEMS = [
"Solve: A store sells pencils at $0.50 each. If Maria buys 12 pencils, how much does she spend?",
"Solve: A train travels 60 mph for 2.5 hours. How far does it go?",
"Solve: A recipe needs 3/4 cup of sugar. If you make 4 batches, how much sugar?",
"Which sorting algorithm has O(n log n) average time complexity? A) Bubble B) Merge C) Selection D) Insertion",
"What does the acronym SQL stand for? A) Simple Query Language B) Structured Query Language C) Standard Query Logic D) Sequential Query Language",
"In Python, which keyword defines a function? A) func B) def C) function D) define",
]
def ngrams(text, n=8):
tokens = text.split()
if len(tokens) < n:
return [" ".join(tokens)]
return [" ".join(tokens[i:i+n]) for i in range(len(tokens) - n + 1)]
benchmark_ngrams = set()
for item in BENCHMARK_ITEMS:
benchmark_ngrams |= set(ngrams(item, n=8))
rows = [json.loads(l) for l in IN.open()]
print(f"Loaded {len(rows)} samples — scanning for benchmark contamination (n=8)...")
clean, flagged = [], 0
for r in rows:
text = r["prompt"]
if set(ngrams(text, n=8)) & benchmark_ngrams:
flagged += 1
continue
clean.append(r)
with OUT.open("w") as f:
for r in clean:
f.write(json.dumps(r) + "\n")
print(f"Decontamination: removed {flagged} benchmark-contaminated samples")
print(f"Kept {len(clean)} / {len(rows)} = {100*len(clean)/len(rows):.1f}%")
Run it:
python phase5_decontaminate.py
Record: contaminated samples removed. This should catch most (possibly all) of the surviving planted benchmark items.
The non-negotiable teaching moment: This is the one stage of the pipeline where skipping has catastrophic downside and zero upside. If you train on a benchmark's test items, your eval on that benchmark is fiction. You will publish inflated scores and discover the truth only in production.
"""phase6_score.py"""
import json
from pathlib import Path
from collections import Counter
raw = [json.loads(l) for l in Path("messy_sft.jsonl").open()]
clean = [json.loads(l) for l in Path("after_decontaminate.jsonl").open()]
raw_counts = Counter(r["label"] for r in raw)
clean_counts = Counter(r["label"] for r in clean)
n_raw = len(raw)
n_clean = len(clean)
print("=" * 60)
print("PIPELINE SCORECARD")
print("=" * 60)
print(f"Retention rate: {n_clean} / {n_raw} = {100*n_clean/n_raw:.1f}%")
print()
print(f"{'Label':<30} {'Raw':>8} {'Clean':>8} {'Removed':>10} {'%caught':>10}")
print("-" * 70)
total_contam_raw = sum(v for k, v in raw_counts.items() if k.startswith("contamination"))
total_contam_clean = sum(v for k, v in clean_counts.items() if k.startswith("contamination"))
for label in sorted(set(raw_counts) | set(clean_counts)):
r, c = raw_counts.get(label, 0), clean_counts.get(label, 0)
removed = r - c
pct = 100 * removed / r if r else 0
print(f"{label:<30} {r:>8} {c:>8} {removed:>10} {pct:>9.1f}%")
print("-" * 70)
if total_contam_raw:
contam_removed = total_contam_raw - total_contam_clean
print(f"Contamination-removal rate: {contam_removed} / {total_contam_raw} = "
f"{100*contam_removed/total_contam_raw:.1f}%")
else:
print("No planted contamination to score.")
print("=" * 60)
Run it:
python phase6_score.py
Record: retention rate and contamination-removal rate.
On the seed-42 generator, a correct pipeline produces:
n=8 instead of n=13, or scan the response too, not just the prompt.near_dup caught: ~70–95%. (A few may have so many edits they slip past MinHash at threshold 0.9.)paraphrase caught: ~40–70%. (Paraphrases that change only a leading verb are easy to catch; aggressive rewordings are harder.)garbage caught: ~80–95%. (The approximate perplexity proxy catches most mojibake; some repeated garbage may slip through if it happens to look alphabetic.)No code. Write 3–5 sentences answering:
Submit ft06-lab-report.md:
near_dup label.paraphrase and any remaining lexical duplicates that survived MinHash.garbage label and any near-duplicates that survived earlier stages.kenlm, train a 5-gram model on a clean subset of your data (or Wikipedia), and re-score Phase 4. Compare the garbage-caught rate against the cheap heuristic. (Sets up the DCLM pipeline.)# Lab Specification — Module FT06: Dedup, Filter, Decontaminate
**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT06 — Dedup, Filter, Decontaminate
**Duration**: 60–90 minutes (the data-quality lab — where most gains are)
**Environment**: Python 3.11+. **CPU-only is fine** — sentence-transformers runs on CPU for 10K samples in a few minutes. ~2GB free disk for the embedding model.
---
## Learning objectives
By the end of this lab you will have:
1. **Generated a synthetic messy SFT dataset** (10,000 samples) with planted pathologies: near-duplicates, semantic paraphrases, garbage, and benchmark-contamination items.
2. **Run the full clean pipeline** — MinHash dedup → semantic dedup → perplexity filter → decontamination — and watched each stage remove samples.
3. **Reported the retention rate** (what fraction of the original 10K survived) and the **contamination-removal rate** (what fraction of planted benchmark items were caught).
4. **Stated, in your own words, why this is the moat**: anyone can generate a million samples; the team that can clean them wins.
This lab is the data-quality module felt end-to-end. You will *not* train a model — the point is that cleaning is upstream of training and yields disproportionate gains per hour invested.
---
## Phase 0 — Environment setup (5 min)
```bash
python3.11 -m venv ft06-env && source ft06-env/bin/activate
# The data-quality stack — all CPU-friendly
pip install -q datasketch sentence-transformers numpy scikit-learn
# Optional, for faster embeddings if you have a GPU/MPS:
# pip install -q torch (already pulled in by sentence-transformers)
```
Verify:
```python
import datasketch, sentence_transformers, numpy, sklearn
print(f"datasketch: {datasketch.__version__ if hasattr(datasketch,'__version__') else 'ok'}")
print(f"sentence-transformers: {sentence_transformers.__version__}")
print(f"numpy: {numpy.__version__}")
print(f"scikit-learn: {sklearn.__version__}")
```
---
## Phase 1 — Generate the messy dataset (10 min)
This is a **self-contained synthetic generator**. It plants known pathologies so we can measure how well the pipeline catches them. Save as `generate_messy.py`.
```python
"""generate_messy.py — synthetic 10K SFT dataset with planted pathologies."""
import json, random, string, hashlib
from pathlib import Path
random.seed(42)
OUT = Path("messy_sft.jsonl")
N_BASE = 6000 # unique "good" samples
N_NEAR_DUP = 1500 # near-duplicates (1-3 char edits)
N_PARAPHRASE = 1000 # semantic paraphrases (template rewording)
N_GARBAGE = 800 # high-perplexity noise
N_CONTAMINATED = 200 # planted benchmark items (GSM8K-style + MMLU-style)
# Total: 9500, plus ~500 more base to reach ~10K
N_TOTAL_TARGET = 10000
# A pool of clean, well-formed instruction/response pairs
CLEAN_TEMPLATES = [
("Explain what a hash function is.", "A hash function maps data of arbitrary size to a fixed-size value, typically for fast lookup or integrity checks."),
("What is a closure in JavaScript?", "A closure is a function that retains access to variables from its lexical scope, even when invoked outside that scope."),
("Summarize the CAP theorem.", "The CAP theorem says a distributed system can guarantee at most two of: consistency, availability, partition tolerance."),
("How does TCP establish a connection?", "TCP uses a three-way handshake: SYN, SYN-ACK, ACK, after which the connection is established."),
("Define eventual consistency.", "Eventual consistency guarantees that, given no new writes, all replicas converge to the same value over time."),
("What is idempotency?", "An operation is idempotent if calling it once has the same effect as calling it multiple times."),
("Explain the single responsibility principle.", "A class should have one, and only one, reason to change."),
("What does ACID stand for?", "Atomicity, Consistency, Isolation, Durability — the four guarantees of transactional systems."),
("Describe a bloom filter.", "A probabilistic data structure that tests set membership, with possible false positives but no false negatives."),
("What is a foreign key?", "A column referencing the primary key of another table, enforcing referential integrity."),
("Explain lazy evaluation.", "An evaluation strategy that delays computation until the result is actually needed."),
("What is the actor model?", "A concurrency model where actors communicate exclusively by exchanging messages, with no shared state."),
("Define immutability.", "An object whose state cannot be modified after creation."),
("What is a mutex?", "A synchronization primitive granting exclusive access to a resource."),
("Explain backpressure.", "A mechanism to handle a producer that is faster than its consumer, by signaling the producer to slow down."),
]
# Planted benchmark-style contamination items (you would NEVER train on these)
GSM8K_STYLE = [
("Solve: A store sells pencils at $0.50 each. If Maria buys 12 pencils, how much does she spend?",
"Maria spends 12 * $0.50 = $6.00."),
("Solve: A train travels 60 mph for 2.5 hours. How far does it go?",
"Distance = 60 * 2.5 = 150 miles."),
("Solve: A recipe needs 3/4 cup of sugar. If you make 4 batches, how much sugar?",
"4 * 3/4 = 3 cups of sugar."),
]
MMLU_STYLE = [
("Which sorting algorithm has O(n log n) average time complexity? A) Bubble B) Merge C) Selection D) Insertion",
"B) Merge sort has O(n log n) average time complexity."),
("What does the acronym SQL stand for? A) Simple Query Language B) Structured Query Language C) Standard Query Logic D) Sequential Query Language",
"B) Structured Query Language."),
("In Python, which keyword defines a function? A) func B) def C) function D) define",
"B) def."),
]
def random_garbage(min_len=20, max_len=120):
"""High-perplexity noise — random chars, mojibake, code-corruption."""
kind = random.choice(["rand_chars", "repeated", "mojibake", "mixed"])
if kind == "rand_chars":
return "".join(random.choices(string.printable + "é¥Ω£", k=random.randint(min_len, max_len)))
if kind == "repeated":
return ("".join(random.choices(string.ascii_letters, k=8)) + " ") * random.randint(5, 20)
if kind == "mojibake":
return "".join(random.choices("é¥Ω£ñ¥Ω", k=random.randint(min_len, max_len)))
return "asdf qwer " + "".join(random.choices("!@#$%^&*()1234567890", k=random.randint(min_len, max_len)))
def near_dup_of(text, n_edits=2):
"""1-3 char edits — invisible to exact hashing, caught by MinHash."""
chars = list(text)
for _ in range(n_edits):
if not chars: break
op = random.choice(["swap", "delete", "replace"])
i = random.randrange(len(chars))
if op == "swap" and i + 1 < len(chars):
chars[i], chars[i+1] = chars[i+1], chars[i]
elif op == "delete":
chars.pop(i)
else:
chars[i] = random.choice(string.ascii_lowercase)
return "".join(chars)
PARAPHRASE_PATTERNS = [
("Explain", "Describe in your own words"),
("What is", "Define"),
("How does", "Describe the way"),
("Define", "Briefly explain"),
("Describe", "Explain"),
]
def paraphrase_of(text):
"""Template rewording — invisible to MinHash, caught by semantic dedup."""
for a, b in PARAPHRASE_PATTERNS:
if text.startswith(a):
return b + text[len(a):]
return text + " (please.)"
def gen():
rows = []
# Base clean samples (deduplicated by template at generation; we still plant dupes below)
for _ in range(N_BASE):
q, a = random.choice(CLEAN_TEMPLATES)
# Light template variation so they aren't byte-identical
prefix = random.choice(["", "Q: ", "Question: ", ""])
rows.append({"prompt": prefix + q, "response": a, "label": "clean"})
# Near-duplicates of base samples
for i in range(N_NEAR_DUP):
q, a = random.choice(CLEAN_TEMPLATES)
rows.append({"prompt": near_dup_of(q, n_edits=random.choice([1,2,3])),
"response": near_dup_of(a, n_edits=1), "label": "near_dup"})
# Semantic paraphrases
for i in range(N_PARAPHRASE):
q, a = random.choice(CLEAN_TEMPLATES)
rows.append({"prompt": paraphrase_of(q), "response": a, "label": "paraphrase"})
# Garbage
for i in range(N_GARBAGE):
rows.append({"prompt": random_garbage(), "response": random_garbage(max_len=40),
"label": "garbage"})
# Contamination — benchmark items
for i in range(N_CONTAMINATED // 2):
q, a = random.choice(GSM8K_STYLE)
rows.append({"prompt": q, "response": a, "label": "contamination_gsm8k"})
for i in range(N_CONTAMINATED // 2):
q, a = random.choice(MMLU_STYLE)
rows.append({"prompt": q, "response": a, "label": "contamination_mmlu"})
# Pad to ~10K with extra clean base samples
while len(rows) < N_TOTAL_TARGET:
q, a = random.choice(CLEAN_TEMPLATES)
rows.append({"prompt": q, "response": a, "label": "clean"})
random.shuffle(rows)
with OUT.open("w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} samples to {OUT}")
# Print ground-truth counts (do NOT use these in the pipeline — they're for scoring only)
from collections import Counter
counts = Counter(r["label"] for r in rows)
print("Ground-truth label distribution:")
for k, v in counts.most_common():
print(f" {k}: {v}")
if __name__ == "__main__":
gen()
```
Run it:
```bash
python generate_messy.py
```
**Record**: the total count (should be ~10,000) and the ground-truth label distribution. You will use these to score the pipeline at the end.
> **What just happened (the teaching moment):** You built a dataset with *known* pathologies — near-duplicates, paraphrases, garbage, and planted benchmark items. This lets you measure exactly what the clean pipeline catches and what it misses. In a real corpus you don't have ground-truth labels; here, you do.
---
## Phase 2 — MinHash dedup (10 min)
```python
"""phase2_minhash.py"""
import json
from datasketch import MinHash, MinHashLSH
from pathlib import Path
IN = Path("messy_sft.jsonl")
OUT = Path("after_minhash.jsonl")
def shingles(text, k=3):
tokens = text.split()
if len(tokens) < k:
return [" ".join(tokens)]
return [" ".join(tokens[i:i+k]) for i in range(len(tokens) - k + 1)]
def make_minhash(text, num_perm=128):
m = MinHash(num_perm=num_perm)
for sh in shingles(text):
m.update(sh.encode("utf-8"))
return m
rows = [json.loads(l) for l in IN.open()]
print(f"Loaded {len(rows)} samples")
# Build the LSH index over the concatenated prompt+response
THRESHOLD = 0.9 # Jaccard >= 0.9 -> near-duplicate
lsh = MinHashLSH(threshold=THRESHOLD, num_perm=128)
signatures = []
for i, r in enumerate(rows):
sig = make_minhash(r["prompt"] + " " + r["response"])
signatures.append(sig)
lsh.insert(i, sig)
# Identify duplicates: for each row, find existing near-duplicates
keep_idx = set(range(len(rows)))
dupes_found = 0
for i in range(len(rows)):
if i not in keep_idx:
continue
candidates = [c for c in lsh.query(signatures[i]) if c > i and c in keep_idx]
for c in candidates:
keep_idx.discard(c)
dupes_found += 1
clean = [rows[i] for i in sorted(keep_idx)]
with OUT.open("w") as f:
for r in clean:
f.write(json.dumps(r) + "\n")
print(f"MinHash dedup (threshold={THRESHOLD}): removed {dupes_found} near-duplicates")
print(f"Kept {len(clean)} / {len(rows)} = {100*len(clean)/len(rows):.1f}%")
```
Run it:
```bash
python phase2_minhash.py
```
**Record**: number of near-duplicates removed, retention percentage. Expect to catch a large chunk of the `near_dup` label (MinHash sees the 1–3 char edits).
> **What just happened:** MinHash + LSH caught near-duplicates that exact hashing would have missed entirely. A single shared character change is invisible to exact hashing but shows up as a high-Jaccard signature here.
---
## Phase 3 — Semantic dedup (15 min)
MinHash catches lexical duplicates. It misses paraphrases. Time for embeddings.
```python
"""phase3_semantic.py"""
import json, numpy as np
from pathlib import Path
from sentence_transformers import SentenceTransformer
from sklearn.cluster import MiniBatchKMeans
IN = Path("after_minhash.jsonl")
OUT = Path("after_semantic.jsonl")
rows = [json.loads(l) for l in IN.open()]
texts = [r["prompt"] + " " + r["response"] for r in rows]
print(f"Loaded {len(rows)} samples — embedding (CPU; ~2 min for 10K)...")
model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dim, fast on CPU
emb = model.encode(texts, batch_size=64, show_progress_bar=True, normalize_embeddings=True)
print(f"Embeddings: {emb.shape}")
# Cluster, then drop near-duplicates within each cluster
N_CLUSTERS = min(200, len(rows) // 10)
km = MiniBatchKMeans(n_clusters=N_CLUSTERS, random_state=42, batch_size=256)
cluster_ids = km.fit_predict(emb)
SIM_THRESHOLD = 0.92 # cosine >= 0.92 -> semantic duplicate
keep = np.ones(len(rows), dtype=bool)
dropped = 0
for c in range(N_CLUSTERS):
idx = np.where(cluster_ids == c)[0]
if len(idx) < 2:
continue
sub = emb[idx]
# Pairwise cosine (already normalized -> dot product = cosine)
sim = sub @ sub.T
# Greedy: keep first, drop anyone >= threshold that hasn't been dropped
local_dropped = set()
for ii in range(len(idx)):
if idx[ii] in local_dropped:
continue
for jj in range(ii + 1, len(idx)):
if idx[jj] in local_dropped:
continue
if sim[ii, jj] >= SIM_THRESHOLD:
local_dropped.add(idx[jj])
keep[idx[jj]] = False
dropped += 1
clean = [rows[i] for i in range(len(rows)) if keep[i]]
with OUT.open("w") as f:
for r in clean:
f.write(json.dumps(r) + "\n")
print(f"Semantic dedup (cosine >= {SIM_THRESHOLD}): removed {dropped} paraphrases")
print(f"Kept {len(clean)} / {len(rows)} = {100*len(clean)/len(rows):.1f}%")
```
Run it:
```bash
python phase3_semantic.py
```
**Record**: paraphrases removed, retention percentage. Expect to catch a chunk of the `paraphrase` label — the ones MinHash missed.
> **What just happened:** The sentence-transformer embedded every sample into a 384-dim semantic space. Clustering grouped paraphrases together; pairwise cosine within clusters caught the reworded versions. This is the stage that *only* semantic dedup can do.
---
## Phase 4 — Perplexity filter (10 min)
Training a full kenlm model is out of scope for this lab. We approximate with a length-and-tokens heuristic that mimics perplexity filtering: well-formed English is smooth; garbage is jagged. (In production, use kenlm trained on Wikipedia-quality text.)
```python
"""phase4_perplexity.py"""
import json, re, string
from pathlib import Path
import numpy as np
IN = Path("after_semantic.jsonl")
OUT = Path("after_perplexity.jsonl")
rows = [json.loads(l) for l in IN.open()]
print(f"Loaded {len(rows)} samples")
def approx_perplexity(text):
"""A cheap proxy for perplexity: ratio of 'weird' tokens to total tokens.
Real pipelines train a small LM on clean data and score with that.
Here we approximate: high weirdness score == high perplexity == likely garbage.
"""
text = text.strip()
if not text:
return 1e6
tokens = re.findall(r"\S+", text)
if not tokens:
return 1e6
weird = 0
for tok in tokens:
# Penalize tokens with lots of non-alphanumeric chars (mojibake, symbols, code-corruption)
non_alpha = sum(1 for c in tok if not c.isalpha() and c not in ".,;:'\"!?-()/$%")
if non_alpha / max(len(tok), 1) > 0.5:
weird += 1
# Penalize very long runs of a single char ("aaaaaaa")
if len(set(tok)) == 1 and len(tok) > 4:
weird += 1
# Penalize tokens with no vowels (likely noise)
if tok.isalpha() and not any(v in tok.lower() for v in "aeiou"):
weird += 0.5
return weird / len(tokens)
scores = [(approx_perplexity(r["prompt"] + " " + r["response"]), i) for i, r in enumerate(rows)]
scores.sort() # ascending — lowest weirdness first
# Keep the bottom 90% (drop the top 10% highest-perplexity)
cutoff = int(0.90 * len(scores))
keep_idx = sorted(i for _, i in scores[:cutoff])
clean = [rows[i] for i in keep_idx]
with OUT.open("w") as f:
for r in clean:
f.write(json.dumps(r) + "\n")
dropped = len(rows) - len(clean)
print(f"Perplexity filter (drop top 10%): removed {dropped} likely-garbage samples")
print(f"Kept {len(clean)} / {len(rows)} = {100*len(clean)/len(rows):.1f}%")
```
Run it:
```bash
python phase4_perplexity.py
```
**Record**: garbage removed. The drop rate is fixed (10%) by design; in production you'd tune this against a held-out quality sample.
> **The teaching moment (and the NeurIPS 2025 link):** Perplexity filtering is two wins in one. It removes garbage *and* — when applied to continued pretraining — measurably mitigates catastrophic forgetting. High-perplexity tokens push the weights hard; dropping them lets the model adapt without overwriting existing capabilities.
---
## Phase 5 — Decontamination (10 min)
We planted 200 benchmark-style items. The pipeline must catch them. Here is the n-gram scan against the planted benchmark items (in a real pipeline you would scan against the *actual* MMLU / GSM8K / HumanEval / MT-Bench test sets).
```python
"""phase5_decontaminate.py"""
import json
from pathlib import Path
IN = Path("after_perplexity.jsonl")
OUT = Path("after_decontaminate.jsonl")
# In production, this is the union of n-gram sets from MMLU, GSM8K, HumanEval, MT-Bench test items.
# Here we reconstruct the planted benchmark items so we can scan against them.
BENCHMARK_ITEMS = [
"Solve: A store sells pencils at $0.50 each. If Maria buys 12 pencils, how much does she spend?",
"Solve: A train travels 60 mph for 2.5 hours. How far does it go?",
"Solve: A recipe needs 3/4 cup of sugar. If you make 4 batches, how much sugar?",
"Which sorting algorithm has O(n log n) average time complexity? A) Bubble B) Merge C) Selection D) Insertion",
"What does the acronym SQL stand for? A) Simple Query Language B) Structured Query Language C) Standard Query Logic D) Sequential Query Language",
"In Python, which keyword defines a function? A) func B) def C) function D) define",
]
def ngrams(text, n=8):
tokens = text.split()
if len(tokens) < n:
return [" ".join(tokens)]
return [" ".join(tokens[i:i+n]) for i in range(len(tokens) - n + 1)]
benchmark_ngrams = set()
for item in BENCHMARK_ITEMS:
benchmark_ngrams |= set(ngrams(item, n=8))
rows = [json.loads(l) for l in IN.open()]
print(f"Loaded {len(rows)} samples — scanning for benchmark contamination (n=8)...")
clean, flagged = [], 0
for r in rows:
text = r["prompt"]
if set(ngrams(text, n=8)) & benchmark_ngrams:
flagged += 1
continue
clean.append(r)
with OUT.open("w") as f:
for r in clean:
f.write(json.dumps(r) + "\n")
print(f"Decontamination: removed {flagged} benchmark-contaminated samples")
print(f"Kept {len(clean)} / {len(rows)} = {100*len(clean)/len(rows):.1f}%")
```
Run it:
```bash
python phase5_decontaminate.py
```
**Record**: contaminated samples removed. This should catch most (possibly all) of the surviving planted benchmark items.
> **The non-negotiable teaching moment:** This is the one stage of the pipeline where skipping has catastrophic downside and zero upside. If you train on a benchmark's test items, your eval on that benchmark is fiction. You will publish inflated scores and discover the truth only in production.
---
## Phase 6 — Score the pipeline (5 min)
```python
"""phase6_score.py"""
import json
from pathlib import Path
from collections import Counter
raw = [json.loads(l) for l in Path("messy_sft.jsonl").open()]
clean = [json.loads(l) for l in Path("after_decontaminate.jsonl").open()]
raw_counts = Counter(r["label"] for r in raw)
clean_counts = Counter(r["label"] for r in clean)
n_raw = len(raw)
n_clean = len(clean)
print("=" * 60)
print("PIPELINE SCORECARD")
print("=" * 60)
print(f"Retention rate: {n_clean} / {n_raw} = {100*n_clean/n_raw:.1f}%")
print()
print(f"{'Label':<30} {'Raw':>8} {'Clean':>8} {'Removed':>10} {'%caught':>10}")
print("-" * 70)
total_contam_raw = sum(v for k, v in raw_counts.items() if k.startswith("contamination"))
total_contam_clean = sum(v for k, v in clean_counts.items() if k.startswith("contamination"))
for label in sorted(set(raw_counts) | set(clean_counts)):
r, c = raw_counts.get(label, 0), clean_counts.get(label, 0)
removed = r - c
pct = 100 * removed / r if r else 0
print(f"{label:<30} {r:>8} {c:>8} {removed:>10} {pct:>9.1f}%")
print("-" * 70)
if total_contam_raw:
contam_removed = total_contam_raw - total_contam_clean
print(f"Contamination-removal rate: {contam_removed} / {total_contam_raw} = "
f"{100*contam_removed/total_contam_raw:.1f}%")
else:
print("No planted contamination to score.")
print("=" * 60)
```
Run it:
```bash
python phase6_score.py
```
**Record**: retention rate and contamination-removal rate.
### Expected ballpark numbers
On the seed-42 generator, a correct pipeline produces:
- **Retention rate**: 65–80% (you removed 20–35% as duplicates, paraphrases, garbage, and contamination).
- **Contamination-removal rate**: ideally **100%**. If it is below 95%, your n-gram threshold is too high — try `n=8` instead of `n=13`, or scan the response too, not just the prompt.
- **`near_dup` caught**: ~70–95%. (A few may have so many edits they slip past MinHash at threshold 0.9.)
- **`paraphrase` caught**: ~40–70%. (Paraphrases that change only a leading verb are easy to catch; aggressive rewordings are harder.)
- **`garbage` caught**: ~80–95%. (The approximate perplexity proxy catches most mojibake; some `repeated` garbage may slip through if it happens to look alphabetic.)
---
## Phase 7 — The thesis, in your own words (5 min)
No code. Write 3–5 sentences answering:
1. What fraction of the original 10K samples survived to training? Where did the lost samples go — which stage removed the most?
2. What was the contamination-removal rate? Why does this matter specifically for the evals you will report on this model later?
3. If you doubled the size of the raw dataset (20K instead of 10K) with the *same* pathology rates, would your final clean dataset be higher quality? Why or why not?
---
## Deliverables
Submit `ft06-lab-report.md`:
- [ ] Phase 1: total sample count + ground-truth label distribution
- [ ] Phase 2: MinHash near-duplicates removed + retention %
- [ ] Phase 3: semantic paraphrases removed + retention %
- [ ] Phase 4: garbage removed + retention %
- [ ] Phase 5: contaminated samples removed + retention %
- [ ] Phase 6: full scorecard; retention rate; contamination-removal rate
- [ ] Phase 7: your 3–5 sentence thesis statement
---
## Solution key
- **Phase 2** (MinHash): on seed-42, expect ~1,200–1,500 near-duplicates removed. Mostly the `near_dup` label.
- **Phase 3** (semantic): expect ~600–900 paraphrases removed. A mix of `paraphrase` and any remaining lexical duplicates that survived MinHash.
- **Phase 4** (perplexity): exactly 10% by construction. Most of the removals should come from the `garbage` label and any near-duplicates that survived earlier stages.
- **Phase 5** (decontamination): expect to catch nearly all surviving planted benchmark items. If the contamination-removal rate is below 95%, check the n-gram size and ensure you are scanning the prompt (where the benchmark items live).
- **Phase 6** (scorecard): retention 65–80%; contamination-removal ~100%.
- **Phase 7** (thesis): a correct statement names (a) retention of ~70% with the bulk of losses in MinHash and semantic dedup; (b) contamination-removal ~100% matters because contaminated evals are fiction — you would publish a model that breaks in production; (c) doubling a noisy dataset does NOT improve quality — you get a bigger pile of the same bad data. Clean first, scale second. **Generation is cheap; curation is the moat.**
---
## Stretch goals
1. **Tune the MinHash threshold.** Re-run Phase 2 with thresholds 0.7, 0.8, 0.9, 0.95. Plot retention vs. near-dup-caught. Notice the trade-off: aggressive dedup drops legitimate near-paraphrases; conservative dedup lets near-duplicates through.
2. **Try a real perplexity model.** Install `kenlm`, train a 5-gram model on a clean subset of your data (or Wikipedia), and re-score Phase 4. Compare the garbage-caught rate against the cheap heuristic. (Sets up the DCLM pipeline.)
3. **Add quality subset selection.** After Phase 5, run cluster-and-select (cluster the embeddings, pick the K samples closest to each centroid) to reduce the clean set to a fixed budget of 2,000 samples. Report the diversity (number of distinct clusters represented) before and after. (Sets up DCLM-style subset selection.)
4. **Scan the real MMLU.** Download the MMLU test set from HuggingFace, build its 13-gram set, and re-run Phase 5 against it. Compare to the planted contamination. (Sets up the production decontamination pipeline.)