Skip to content

LLM-Sanitizer-Pro — Audit & SOTA Plan (2026-10-07)

Audited revision: b8b54c1 (v3.0.0). 25 modules, ~3.8k LOC source, 18 test files.

0. Redundancy check vs. Brainbrew

Verdict: not redundant, but there is duplicated code that runs the other way.

LLM-Sanitizer-Pro Brainbrew
Purpose Curate/clean existing datasets (any source, any size) Generate synthetic Q&A from PDFs/TXT with teacher LLMs
Interface CLI + Python library Streamlit no-code UI + headless CLI
Core loop read → clean/redact → filter → dedup → decontam → write chunk → generate → LLM-judge → filter → export → LoRA train → HF publish
Overlap — pipeline/sanitizer.py says it is "extracted from the standalone LLM Dataset Sanitizer PRO v2.8 script". With pii.py, dedup.py, decontam.py and quality.py, that is ~1.1k LOC, about 40% of Brainbrew's pipeline/

The overlap is a post-generation cleaning stage that was copied into Brainbrew. It does not make this repo redundant. The forks have already drifted. Brainbrew's PII is in some ways ahead: Luhn-checked cards, IBAN mod-97, IPv6, private-IP heuristics, a URL policy and a Presidio adapter. This repo is ahead on secrets, NER, pseudonymization, chat validation, resume, Hub input and reports.

Recommendation: port Brainbrew's validated PII detectors here (Phase 1). Then make Brainbrew depend on llm-sanitizer-pro through Sanitizer/SanitizerConfig and delete its vendored copy (Phase 2, separate Brainbrew PR).

1. Baseline measurements

Check Result
pytest 273 passed, 6 skipped (18.7 s)
ruff check (ruff 0.16.8, what CI installs today since it is unpinned) 555 errors. ruff 0.14.0 reports 0. CI's lint step will go red on the next run
mypy (configured strict = true, not run in CI) 85 errors in 17 files
Coverage (in-process) 58% overall. cli.py, config.py and worker.py show 0% because CLI tests use subprocess and coverage isn't collected there
Throughput ~3.9k rec/s single-process, ~14k rec/s with --jobs 4 (100k JSONL records, PII + secrets + exact dedup)
Python matrix 3.9–3.12. 3.9 is end-of-life (Oct 2025); 3.13/3.14 are untested

2. Findings (verified = reproduced in this audit)

P0: data loss, data corruption, privacy leaks

# Finding Where Evidence
1 The audit report leaks raw PII and secrets. "Before" samples and dropped samples are the unredacted input records, written to an HTML file meant to be "attached to a compliance ticket". cli.py:621,759, api.py:233,247, report.py:58-68 Verified: john.doe@example.com and AKIA… appear verbatim in --report output
2 --resume duplicates output after a hard crash. The checkpoint stores only records_read. Rows written after the last checkpoint (buffer flushes) are already on disk and get re-appended. cli.py:590-600, checkpoint.py Verified: SIGKILL at record 2500 with interval 1000 gave 3474 rows, of which 3000 unique (474 duplicates). SQLite dedup batches committed after the checkpoint also make stats wrong on resume
3 .json.gz output is not compressed. The atomic tmp path out.json.gz.tmp no longer ends in .gz, so smart_open writes plain text, which is then renamed to .gz. writers.py:205-206 Verified: gzip.BadGzipFile
4 CSV/Excel/Parquet silently drop columns. The header/schema comes from the first record and later keys are discarded (extrasaction='ignore', from_pylist first-row inference). Nested values are written as Python repr. writers.py:243-248,262,275 Verified: extra and meta fields lost in CSV
5 Non-English data is rejected by default. --min-ascii-ratio 0.85 is always on. cli.py:142, api.py:50 Verified: Chinese and Russian records gave 0/2 kept
6 HTML stripping corrupts text. a<b becomes a, x<y and y>z becomes x z, foo<b>bar</b>baz becomes foo bar baz (spaces injected), <script> bodies are kept, and close() is never called, so trailing buffered text is lost. It runs on every string, including code. pii.py:6-30 Verified
7 PII false positives destroy content. Card numbers have no Luhn check (order IDs), SSNs have no area/group validation, any 10-digit number counts as a phone (Unix timestamps), version strings count as IPs (1.2.3.4), and the URL regex eats trailing ).. pii.py:35-43 Verified on all five cases
8 PII false negatives. No IPv6, no IBAN, no non-US formats (no phonenumbers), English-only NER (en_core_web_sm). pii.py, ner.py Verified (IPv6, IBAN)
9 Secrets false negatives. Missing Hugging Face hf_…, GitLab glpat-, npm, Azure, AWS secret keys, and ENCRYPTED PRIVATE KEY PEM. The connection-string regex over-consumes trailing ",. secrets.py:21-58 Verified on all of these
10 Malformed input isn't counted. Bad JSONL lines and non-dict ijson items are skipped without touching stats.malformed, so the report says Malformed: 0. readers.py:91-103,73-76 Verified

P1: correctness and robustness

  • Decontamination only sees the first 8,192 chars. It checks the truncated quality_text, so contamination later in long records is missed (cli.py:635, quality.py:18).
  • Truncation destroys newlines. --max-tokens whitespace truncation re-joins with spaces, which contradicts "structure-preserving" (core.py:34-35).
  • Pseudonym IPs become invalid. The template 0.0.0.{n} is not a valid IP once n > 255 (pii.py:89).
  • Shared class state is mutated. _install_pseudo_templates writes into the class-level PseudoRegistry._TEMPLATES dict (secrets.py:82).
  • HF token check uses a substring. The token is sent when 'huggingface.co' in url; it should compare the parsed hostname (hub.py:40). Hub downloads are fully buffered in memory (resp.read()), with no retry, no revision pinning (the cache never invalidates) and no checksum.
  • API doesn't resolve benchmark names. SanitizerConfig.decontaminate skips resolve_benchmark_names, so 'all' or a typo raises KeyError (api.py:181).
  • Config files aren't validated. Unknown keys are silently ignored (typos) and values aren't type-checked, so min_chars: "50" crashes mid-run (config.py:35-39).
  • Splits and sampling aren't reproducible. --split and --sample use a global random stream, so the train/test assignment of a record changes whenever an upstream filter changes. The standard is a content-hash split. Random state also isn't restored on resume.
  • Semantic dedup has limited recall and doesn't scale. At t=0.85, 8×8-bit LSH bands give about 85% candidate recall, all vectors stay in a Python list in RAM, and buckets hold N/256 entries, which is roughly quadratic.
  • Chat validator rejects valid OpenAI tool-calling data: content: null with tool_calls, and multimodal content lists.
  • --jobs > 1 loses audit samples: no PII diffs or dropped samples.

P2: architecture and engineering

  • The pipeline is implemented twice. cli._handle and api._evaluate have already drifted: --sample is CLI-only and quality scripts are CLI-only. CLI main() is a 590-line function.
  • Config is duck-typed. argparse.Namespace is used as the config object everywhere (_ArgsView shim, getattr(args, …, default) with defaults repeated in 3 places). Worker state lives in module globals.
  • The version is hard-coded in 6 places ("3.0" strings in cli/stats/report).
  • CI: ruff is unpinned with no explicit select, mypy isn't run, there's no coverage gate, no 3.13/3.14, and actions aren't SHA-pinned. No benchmark/regression suite for detection quality (precision and recall are never measured).
  • Packaging: the all extra pulls transformers (and torch) into "all format extras". Development Status :: 5 - Production/Stable overstates the maturity.

3. SOTA score

Dimension Weight Score Notes
Feature breadth 15% 8.0 Covers most of the curation surface (PII, secrets, 3-tier dedup, decontam, chat lint, scoring, resume, reports)
Correctness / data integrity 20% 4.5 P0 #2–#7, #10
Privacy & security 15% 4.0 Report leak, regex-only PII, secret coverage gaps
Detection quality vs. 2026 SOTA 15% 4.5 langdetect, spaCy-sm, distilgpt2 perplexity, 8 dated benchmarks
Scale & performance 10% 4.5 Single node, in-RAM fuzzy/semantic indexes, buffered Parquet
Architecture & maintainability 10% 5.5 Duplicated pipeline, Namespace config
Engineering (tests, CI, typing) 10% 5.5 Good test count; lint will break, mypy unenforced
Docs & UX 5% 7.5 Clear README, profiles, config template
Weighted total 5.3 / 10

Reference points (2026): - NeMo Curator 26.02: Ray-based, GPU fuzzy/semantic dedup, model classifiers. - datatrove/FineWeb-2: GlotLID v3, Gopher/C4/FineWeb filters. - GLiNER2-PII (May 2026): multilingual, 42 PII types; outperforms OpenAI Privacy Filter and Presidio on span-level F1. - Gitleaks/TruffleHog: 150–800+ secret detectors with entropy and verification. - FineWeb-Edu, DCLM and Ultra-FineWeb fastText quality classifiers. - Rensa: Rust MinHash.

4. Execution plan to ≥ 8/10

Each item ships with a regression test that fails before the fix. Score estimates are cumulative.

Phase 0: CI hygiene (≈0.5 day) → 5.5

  1. Pin ruff==0.14.* and add an explicit [tool.ruff.lint] select = ["E","F","W","I","UP","B"]; fix the findings. Add mypy sanitizer_pro to CI with per-module ignore_errors overrides that are burned down in Phase 2.
  2. Matrix 3.10–3.14 and drop 3.9 (requires-python>=3.10). Collect subprocess coverage (COVERAGE_PROCESS_START) with a --cov-fail-under=80 gate.
  3. Pin GitHub Actions by SHA and add Dependabot for actions and pip.

Phase 1: P0/P1 bug fixes (≈4–5 days) → 6.8

  1. Report privacy: store dropped samples after redaction, and show PII diffs as redacted spans (kind, offset, length, masked preview) instead of raw text. Add an explicit --report-raw-samples opt-in with a loud warning.
  2. Resume integrity: checkpoint output_bytes (taken after flush() + os.fsync()) and truncate the output to it on resume. Commit SQLite dedup only at checkpoints (no autonomous batch commits under --resume). Persist the RNG state.
  3. Writers: gzip-aware tmp naming (out.json.tmp.gz). CSV: JSON-encode nested values and warn or error on new keys (--csv-strict), or do a schema pre-pass. Parquet: streaming ParquetWriter with a unified/promoted schema (fixes the OOM too). Excel: header union.
  4. Defaults: min_ascii_ratio default 0 (off). Add script-aware gates (--allowed-scripts). Move the length gates into profiles. Ship this as v4.0.0 with a CHANGELOG migration note, since defaults change.
  5. HTML: strip only when the text looks like markup. Drop script/style/noscript bodies, insert newlines only at block-level tags, call close(), and leave non-tag < alone. Optional trafilatura/resiliparse backend for full web pages.
  6. PII precision and recall:
  7. Port Brainbrew's Luhn, IBAN mod-97, IPv6 and URL-policy code.
  8. Add SSN area/group validation, ipaddress validation with version-context exclusion, and URL trailing-punctuation trimming.
  9. Make phone detection stricter by default, with an optional phonenumbers backend.
  10. Pseudonym IPs from TEST-NET/10.x space.
  11. Secrets: add HF, GitLab, npm, Azure, AWS-secret (context-gated), ENCRYPTED/unterminated PEM, Shannon-entropy gating for the generic rule, and connection-string terminators. Stop mutating class state (instance templates).
  12. Decontaminate the full concatenated text, not the 8 KB quality slice. Count malformed JSONL and non-dict items. Truncator preserves whitespace. Hub: hostname check, streamed download with retry/backoff, revision pinning (hf://…@rev), size check. API resolves benchmark names. Config files get unknown-key warnings and type coercion/validation.

Phase 2: one engine, typed config (≈3–4 days) → 7.3

  1. Introduce Pipeline = ordered Stage objects (Clean, Redact, Gate, Chat, Decontam, Score, Sample, Dedup) with a single evaluate() used by both the CLI and the Sanitizer API. The CLI becomes argument parsing plus I/O only (target: main() under 150 lines).
  2. Make SanitizerConfig the only config type, with validation, built from argparse, YAML or JSON. Delete _ArgsView and the getattr(args, …) defaults. Workers receive the config, not a Namespace.
  3. Deterministic content-hash --split/--sample (sha256(key) mod 2^64). Single-source version via importlib.metadata. Burn down mypy strict to 0.
  4. Brainbrew follow-up PR: replace pipeline/sanitizer.py/pii.py with llm-sanitizer-pro>=4. Needs push access to Brainbrew.

Phase 3: SOTA detection backends (≈6–8 days) → 8.2

  1. PII: a --pii-backend gliner backend (GLiNER2-PII or knowledgator/gliner-pii-base, multilingual, 40+ types) plus a Presidio adapter. Batched inference with nlp.pipe for spaCy. Add a PII eval harness (tests/eval/) reporting P/R/F1 per backend on a public benchmark (e.g. REDACT/SPY samples) and gate CI on regressions.
  2. Language ID: GlotLID v3 / OpenLID-v2 fastText backend (default when installed); keep langdetect as the fallback.
  3. Quality: pluggable classifier scorers: FineWeb-Edu (BERT), DCLM fastText, Ultra-FineWeb fastText. Add Gopher/C4/FineWeb rule sets (n-gram repetition ratios, line-level filters, bullet/ellipsis ratios). Batched GPU perplexity with a current small LM.
  4. Decontamination:
  5. Expand the registry: MMLU-Pro, GPQA, MATH-500, AIME 2024/2025, IFEval, BBH, MuSR, HumanEval+/MBPP+, LiveCodeBench.
  6. Index answers and choices too, using hashed n-grams.
  7. Add a per-benchmark contamination table in the report.
  8. Add an optional embedding stage for paraphrase contamination (the LLM-decontaminator approach).
  9. Chat: OpenAI tool-calling and multimodal schema, ShareGPT input, and token budgets via tokenizer.apply_chat_template.

Phase 4: scale (≈4–5 days) → 8.6

  1. A rensa (Rust MinHash) backend, disk-backed LSH buckets (SQLite/LMDB), and a usearch/FAISS index for semantic dedup (constant-memory bands, ≥95% recall at t=0.85, measured).
  2. Glob/multi-file input with per-file parallelism. Process-pool chunking that keeps PII counts, audit samples and pseudonyms correct (sharded registry with deterministic HMAC pseudonyms, --pseudo-key).
  3. A benchmarks/ suite in CI (nightly) with throughput targets: ≥15k rec/s single-core regex path, linear scaling to --jobs N.

Phase 4 outcome (measured, 100k synthetic records, 4-core x86):

Item Result
Fuzzy dedup Rensa signatures and verified LSH: 455 → 6.1k rec/s. Recall at thresholds 0.7–0.85 is 97–98%, against exact Jaccard. With the SQLite index, peak memory stays flat (~140 MB at 100k and 500k records).
Semantic dedup usearch HNSW matches exact search on 99.9% of decisions, at ~3k rec/s. LSH: 98% at 0.4k rec/s, and degrading as the index grows.
Multi-file input and chunked --jobs --jobs 4: 6.3k → 20.6k rec/s (3.3×). Results are identical to --jobs 1.
Single-core regex path 1.7k → 7.1k rec/s; no flags: 5.2k → 12.6k. The ≥15k target was not met. The remaining cost is per-record Python work: JSON in and out, NFKC normalization, gates, stats.
Benchmark suite benchmarks/ runs nightly. Its targets are regression guards set from these measurements. Throughput past one core comes from --jobs.

Phase 5: provenance and release quality (≈2 days) → 8.8

Run manifest (config hash, package and model versions, input digests, per-stage counts) and an auto-generated HF dataset card. sanitize diff for two stats files. Signed releases with SBOM. Docs site.

Minimum path to 8/10: Phases 0–3 (~14–18 engineering days). Phases 4–5 add margin.

5. Re-assessment after Phases 0–5 (2026-10-08)

Same rubric and weights as section 3.

Dimension Weight Before After Evidence Remaining gaps
Feature breadth 15% 8.0 9.0 Adds GlotLID/OpenLID language ID, Gopher/C4/FineWeb rules, FineWeb-Edu and DCLM scorers, GLiNER2 PII, tool-calling/ShareGPT chat, multi-file input, manifest, dataset card, sanitize diff No Presidio adapter, no Ultra-FineWeb scorer
Correctness / data integrity 20% 4.5 8.5 All P0/P1 findings fixed, each with a regression test. Output byte-identical across the performance work. --jobs N output, stats and pseudonyms identical to --jobs 1. Crash-safe resume. Much new code. Two bugs in this release cycle were caught only by the project's own tools (keyed pseudonyms re-redacted; a release-workflow version check)
Privacy & security 15% 4.0 7.5 Report redaction. GLiNER2 PII (micro F1 0.92 exact / 0.96 overlap). Keyed pseudonyms kept out of checkpoints, logs and manifests. Signed releases with SBOM and provenance. 21 secret detectors, against 150+ in Gitleaks/TruffleHog, and no live verification. PII accuracy measured on a synthetic set, not a public benchmark.
Detection quality vs. 2026 SOTA 15% 4.5 8.0 The FineWeb-2 LID model. Rule filters agree with datatrove on 98–100% of decisions. FineWeb-Edu/DCLM classifiers. 22 benchmarks, answers and choices included. No embedding-based paraphrase decontamination, no LiveCodeBench, no batched GPU scoring
Scale & performance 10% 4.5 7.0 2.4–4× single-core speedup. --jobs 4 20.6k rec/s. Fuzzy dedup 13× faster with flat-memory disk index. HNSW semantic index. Chunked multi-file input. Nightly benchmarks. Single node. Parent process caps parallel throughput at ~20k rec/s. 15k rec/s single-core target missed (7.1k). No Ray/GPU path (NeMo Curator's ground)
Architecture & maintainability 10% 5.5 8.0 One engine for CLI and API. Typed SanitizerConfig. main() is 97 lines. mypy --strict clean No Stage objects as planned. cli.py is still large (build_parser is 291 lines)
Engineering (tests, CI, typing) 10% 5.5 9.0 590 tests across 3 environments (3.9, 3.13, core-only), strict typing, ruff, actionlint, equivalence and accuracy tests, nightly benchmark gate —
Docs & UX 5% 7.5 9.0 Docs site, generated CLI reference (every option documented), changelog, release-verification guide —
Weighted total 5.3 8.2 / 10

The plan projected 8.8. The 0.6 shortfall is the gaps above, mostly: - secrets coverage; - paraphrase decontamination and the remaining scorers; - the single-node throughput ceiling.

Release status: 4.0.0 is built, signed (Sigstore), attested (SLSA) and has an SBOM. Publishing to PyPI waits on registering the trusted publisher on PyPI, which only the account owner can do.

Next steps by value: 1. Secrets: import the Gitleaks rule set (MIT) and add optional live verification. 2. Paraphrase-aware decontamination: embedding stage over the n-gram candidates. 3. Move dedup/stats/writing out of the single parent process (sharded dedup by key hash) to lift the ~20k rec/s ceiling. 4. A PII benchmark run on a public dataset alongside the synthetic set. 5. Brainbrew migration to llm-sanitizer-pro>=4.0.0 once it is on PyPI.

Sources

  • NeMo Curator — https://github.com/NVIDIA-NeMo/Curator
  • 2026 curation stack overview — https://www.spheron.network/blog/ai-pretraining-data-curation-nemo-curator-datatrove-fineweb-gpu-cloud/
  • GLiNER2-PII — https://arxiv.org/pdf/2605.09973 · https://fastino.ai/blog/gliner2-pii-open-source-privacy-filtering-with-pii-detection
  • OpenAI Privacy Filter evaluation — https://arxiv.org/pdf/2608.02616
  • knowledgator/gliner-pii-base — https://huggingface.co/knowledgator/gliner-pii-base-v1.0
  • Ultra-FineWeb — https://arxiv.org/pdf/2505.05427 · Nemotron-CC — https://arxiv.org/pdf/2412.02595
  • FineWeb2 (GlotLID v3) — https://arxiv.org/pdf/2506.20920 · OpenLID-v2 — https://huggingface.co/laurievb/OpenLID-v2
  • Rensa — https://pypi.org/project/rensa
  • LLM decontaminator — https://arena.ai/blog/llm-decontaminator/ · Rephrased-sample contamination — https://arxiv.org/abs/2311.04850v1
  • Secret scanners 2026 — https://appsecsanta.com/sca-tools/best-open-source-secret-scanning-tools
  • Benchmarks in 2026 model cards — https://futureagi.com/blog/llm-benchmarking-state-2026/