Özer Cem KELAHMETKELAHMET
AI Simulations · Training from scratch

Pre-training data: deduplicate, filter by quality and mix the sources

A model is mostly its data. Before the first training step, a pre-training corpus is deduplicated, filtered and mixed. This simulation runs that pipeline on 3,000 generated documents from five sources and shows what survives each stage.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The top bars follow the tokens through each stage: raw, after deduplication, after the quality threshold and after the language filter. The mixture bar shows each source's share once you apply the weights; the table shows sample documents and why each was kept or dropped. The side panel converts the mixture into epochs per source for a 15-trillion-token run.

Things to try

  • Turn deduplication off and on: web and forum data shrink the most, because copies concentrate there.
  • Lower the near-duplicate threshold to 0.5: more pages count as duplicates, including ones that only share boilerplate.
  • Raise the quality threshold to 0.7: forums and the web lose most of their tokens, books and papers keep theirs.
  • Weight papers at 4×: they become a large share of the mixture, but the side panel shows them repeated many times.

How it works in real systems

Real pipelines deduplicate with exact hashes and MinHash with locality-sensitive hashing for near duplicates, score quality with small classifiers trained to recognise reference-quality text, identify languages with fastText-style models, and remove personal data and toxic content. Datasets such as FineWeb document each filter and its effect. The mixture weights are then tuned with small proxy models, because they shape what the final model is good at.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Exact and near-duplicate removal with MinHash

# pip install datasketch
from datasketch import MinHash, MinHashLSH
import hashlib, re

def shingles(text, k=5):
    words = re.findall(r"\w+", text.lower())
    return {" ".join(words[i:i + k]) for i in range(max(1, len(words) - k + 1))}

def minhash(text, perm=128):
    m = MinHash(num_perm=perm)
    for s in shingles(text):
        m.update(s.encode())
    return m

docs = {"a": "the cat sat on the mat today", "b": "the cat sat on the mat today!", "c": "a completely different page"}
seen, lsh, keep = set(), MinHashLSH(threshold=0.8, num_perm=128), []
for k, t in docs.items():
    h = hashlib.sha1(t.strip().lower().encode()).hexdigest()
    if h in seen: continue                      # exact duplicate
    seen.add(h); m = minhash(t)
    if lsh.query(m): continue                   # near duplicate (Jaccard ≥ 0.8)
    lsh.insert(k, m); keep.append(k)
print(keep)

Step 2 · Quality and language filters

# pip install fasttext-wheel  (language id model: lid.176.bin from fasttext.cc)
import fasttext
lid = fasttext.load_model("lid.176.bin")

def lang(text):
    label, prob = lid.predict(text.replace("\n", " "))
    return label[0].replace("__label__", ""), float(prob[0])

def heuristics(text):
    words = text.split()
    if len(words) < 50: return False                          # too short
    if sum(w.isupper() for w in words) / len(words) > 0.3: return False
    lines = text.splitlines()
    if lines and sum(l.strip().endswith((".", "!", "?")) for l in lines) / len(lines) < 0.3: return False
    return True

def keep(text, quality_score, threshold=0.45):
    code, p = lang(text)
    return code == "en" and p > 0.65 and heuristics(text) and quality_score >= threshold

quality_score comes from a small classifier trained to separate reference-quality text from random web text, as in FineWeb-Edu.

Step 3 · Mix the sources and count epochs

tokens = {"web": 9.1e12, "code": 1.2e12, "books": 0.3e12, "papers": 0.25e12, "forums": 0.4e12}
weights = {"web": 1.0, "code": 1.2, "books": 3.0, "papers": 3.0, "forums": 0.6}
budget = 15e12

mixed = {k: tokens[k] * weights[k] for k in tokens}
total = sum(mixed.values())
for k in tokens:
    share = mixed[k] / total
    print(f"{k:7s} share {share:5.1%}  epochs {share * budget / tokens[k]:4.1f}")

Beyond about four epochs, repeating a source adds little; plan the mixture so small high-quality sources are not repeated dozens of times.

More simulations

← Anomaly detection · All AI Simulations · Tokenizer and vocabulary →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open