Skip to content

CLI reference

sanitize --input PATH --output PATH [options]
sanitize diff A B [--json] [--fail-on-change]

General

Option Default Description
-h, --help show this help message and exit
--version show program's version number and exit
--input Input file, directory (searched recursively), glob pattern (quote it: 'data/*.jsonl', 'data/**/*.parquet'), hf://… URI, or '-' for stdin. Multiple files are read in sorted order.
--output Output file or '-' for stdout.
--input-format FMT Override input format.
--output-format FMT Override output format.
--config PATH YAML or JSON config file.
--generate-config FMT Print config template and exit.
--profile NAME Apply a preset flag bundle: fine-tune, pretrain, rag (or 'list' to show them). Explicit flags and --config override it.

Quality Filters

Option Default Description
--min-chars 50 Drop records whose quality text is shorter than this many characters.
--max-chars 20000 Drop records whose quality text is longer than this many characters.
--min-words 8 Drop records with fewer words (each CJK/Thai character counts as a word).
--min-ascii-ratio Reject records whose ASCII-character share is below this (0 = off, the default; e.g. 0.85 keeps mostly-English text).
--min-unique-ratio 0.25 Drop records whose share of distinct words is lower (repetitive text).
--text-fields `` Comma-separated fields for quality scoring.
--text-fields-depth 20 How deep to descend into nested fields when collecting quality text.
--reject-allcaps Drop records that are almost entirely upper case.
--allcaps-min-len 50 Only apply --reject-allcaps to texts at least this long.
--allcaps-min-alpha 10 Only apply --reject-allcaps when the text has this many letters.
--require-fields `` Comma-separated fields every record must have (non-empty).
--quality-script PATH Python file defining keep(record) -> bool, applied after the built-in gates.
--max-depth 100 Maximum nesting depth cleaned and redacted inside a record.
--lang-filter `` Keep only these languages, e.g. en,zh or eng,cmn (ISO 639-1 or -3).
--lang-confidence Minimum language-ID confidence for --lang-filter to keep a record.
--lang-backend {auto,glotlid,openlid,langdetect} auto Language ID for --lang-filter: glotlid (fastText, 2000+ varieties; default when fastText is installed), openlid, or langdetect.
--lang-model PATH Local fastText language-ID model file (skips the Hub download).
--reject-code Reject records detected as code snippets.
--reject-profanity Reject records containing profanity.
--quality-scorer {heuristic,perplexity,fineweb-edu,dclm,fasttext} heuristic Scoring backend for --quality-min-score / --keep-top-percent / --quality-score-field: heuristic (default, no dependencies), perplexity, fineweb-edu (classifier, 0-5 mapped to 0-1), dclm (fastText, P(high quality)), or fasttext (your own model).
--quality-model NAME Model for the scorer: causal LM for perplexity (default distilgpt2), HF model for fineweb-edu, or a fastText .bin for fasttext.
--quality-label LABEL Positive label of a --quality-scorer fasttext model (e.g. __label__hq).
--quality-rules SETS `` Reject documents failing published pretraining rules: gopher, gopher-repetition, c4, fineweb (comma-separated, or all). English-tuned; thresholds as in datatrove.
--quality-min-score X Reject records with quality score < X (scores are in [0, 1]).
--keep-top-percent P Keep only the best P%% of surviving records by quality score. Buffers survivors in memory; output order is preserved.
--quality-score-field FIELD Annotate each output record with its quality score in FIELD.

Features

Option Default Description
--deduplicate Drop exact duplicates (SHA-256 of the cleaned record).
--fuzzy-dedup Near-duplicate detection: MinHash over word 3-shingles, LSH candidates verified by signature similarity. With --dedup-backend sqlite the index lives on disk (constant memory).
--fuzzy-threshold T 0.8 Jaccard similarity threshold for --fuzzy-dedup (0-1, default 0.8).
--fuzzy-backend {auto,rensa,datasketch} auto MinHash implementation: rensa (Rust, ~10x faster; pip install 'llm-sanitizer-pro[fuzzy]') or datasketch. auto prefers rensa.
--semantic-dedup Embedding-based near-dedup: drops paraphrases that share no n-grams (pip install model2vec; ~30MB model, no torch).
--semantic-threshold T 0.9 Cosine similarity threshold for --semantic-dedup (default 0.9).
--semantic-model NAME minishlab/potion-base-8M model2vec static embedding model for --semantic-dedup.
--semantic-index {auto,usearch,lsh} auto Nearest-neighbor index for --semantic-dedup: usearch (HNSW; stays fast as the index grows) or lsh (no extra dependency; slows down on large inputs). auto prefers usearch.
--dedup-fields `` Comma-separated fields to hash for exact dedup (default: the whole record).
--dedup-normalize Lower-case and collapse whitespace before hashing for exact dedup.
--dedup-backend {memory,sqlite} memory Where dedup state lives: memory, or sqlite on disk (constant memory; also used by --fuzzy-dedup).
--dedup-db-path PATH Keep the SQLite dedup database at PATH (needed for dedup across --resume).
--remove-pii Redact PII: emails, URLs, phones, payment cards, IBANs, SSNs, IP addresses.
--pii-mask Partially mask PII (j***n@example.com, ****-1111) instead of tokens.
--pii-pseudonymize Replace PII with stable pseudonyms (Person_0001, email_0002@…) instead of [PII_…] tokens.
--pseudo-key KEY Secret for keyed pseudonyms: the same value gets the same pseudonym in every run and worker that uses the key. Prefer the SANITIZE_PSEUDO_KEY environment variable (command lines are visible to other users). With --jobs > 1 and no key, a random per-run key is used.
--pseudo-map-file PATH Write the original -> pseudonym map (sensitive) to PATH.
--pii-patterns-file PATH JSON file of extra PII regexes: [{"pattern": …, "token": …, "kind": …}].
--pii-ner Also detect PII with a named-entity model (person names by default). Requires spacy (+en_core_web_sm) or transformers.
--pii-ner-backend {auto,spacy,transformers,gliner} auto NER engine for --pii-ner (auto picks what is installed).
--pii-ner-entities KINDS person Comma-separated entity kinds to redact (default: person): person, location, org; with --pii-ner-backend gliner also address, date_of_birth, id_number, financial, username, credential; or all.
--pii-ner-threshold T 0.5 GLiNER confidence threshold (default 0.5).
--pii-ner-model NAME Override the NER model (spaCy model name or HF model id).
--redact-secrets Detect and redact credentials — API keys, tokens, private keys, connection strings. Works with or without --remove-pii; honors --pii-mask / --pii-pseudonymize.
--clean-html Strip HTML markup from strings that look like HTML.
--paragraph-mode For .txt input: one record per blank-line-separated paragraph, not per line.
--txt-fallback-field FIELD For .txt output: the field to write when a record has no "text".
--field-config PATH JSON list of per-field actions: rename (with "to"), drop, pii_only, no_clean.
--max-tokens Truncate each text field to N tokens (counted with --tokenizer).
--tokenizer whitespace Token counter: whitespace, or a Hugging Face tokenizer name.
--sample Keep this fraction of records (0-1), chosen by content hash (see --seed).
--seed Salt for --sample/--split. Both are decided by a hash of each record's content, so results are reproducible across runs.
--split SPEC Write splits by content hash, e.g. train=0.9,val=0.05,test=0.05 (out.train.jsonl, …).
--quick Shortcut for --remove-pii --deduplicate --clean-html --dedup-normalize.
--format-chatml Format output as ChatML messages.
--format-instruct Format output as Alpaca/Instruct schema.

Benchmark Decontamination

Option Default Description
--decontaminate NAMES Comma-separated benchmark names to decontaminate against (e.g. mmlu,gsm8k), 'all', or 'list' to show available benchmarks. Test sets are downloaded from the Hugging Face Hub and cached.
--decontam-refs PATHS Comma-separated local reference files (any supported input format) whose text is treated as benchmark material.
--decontam-ngram N 8 Word n-gram size for overlap detection (default 8).
--decontam-min-hits N 1 Minimum colliding n-grams to flag a record (default 1).
--decontam-cache DIR Benchmark download cache dir (default ~/.cache/llm-sanitizer-pro/benchmarks).

Chat Dataset Validation

Option Default Description
--validate-chat Reject records whose conversation is invalid for chat fine-tuning (role alternation, empty turns, no assistant reply, malformed tool calls, unanswered or orphan tool results, …). Reads OpenAI "messages" (incl. content parts and tool_calls) and ShareGPT "conversations". Combine with --format-chatml to convert first.
--chat-lenient Only structural and tool-call checks (schema, known roles, non-empty content, assistant present, tool links); skip ordering/alternation rules.
--chat-max-tokens N Reject conversations longer than N tokens, counted with --tokenizer through its chat template when it has one (role markup included), else over message contents.
--chat-roles ROLES system,user,assistant Comma-separated allowed roles (default: system,user,assistant; add tool for tool-use data).

CSV / Excel Options

Option Default Description
--csv-delimiter CHAR CSV field delimiter (default: comma; tab for .tsv).
--csv-no-header The CSV has no header row (columns become col_0, col_1, …).
--csv-columns `` Comma-separated column names to use instead of the header row.
--excel-sheet-name NAME Excel sheet to read, by name.
--excel-sheet-index N Excel sheet to read, by position (0 = first).
--excel-warn-size 100 Warn when an Excel input is larger than this many MB (read in memory).

I/O Options

Option Default Description
--encoding utf-8 Text encoding of input and output files.
--shard-size N Write the output in shards of N records (out.00000.jsonl, …).
--json-path PATH item ijson path of the records inside a .json input (default: top-level array).
--hf-cache DIR Cache dir for hf:// dataset downloads (default ~/.cache/llm-sanitizer-pro/datasets).

Runtime

Option Default Description
--log-level {DEBUG,INFO,WARNING,ERROR} INFO Logging verbosity.
--log-format {text,json} text Log line format on stderr: human text or JSON lines (for structured ingestion). Default: text.
--quiet Only warnings and errors.
--no-progress Hide the progress bar.
--jobs 1 Worker processes for the per-record stage (output identical to --jobs 1; pseudonyms too only with --pseudo-key or SANITIZE_PSEUDO_KEY).
--chunk-size N 64 Records per task sent to a worker (record dispatch; JSONL input is read in 4 MB chunks).
--dry-run Process the first --dry-run-size records and write nothing.
--dry-run-size 10000 Records processed by --dry-run.
--stats-only Process everything but write no output (statistics only).
--debug-records Log why each record was dropped (sets --log-level DEBUG).
--stats-file PATH Write run statistics (JSON) to PATH.
--manifest PATH Write a run manifest (JSON): versions, models, full config and its hash, SHA-256 of every input and output file, per-stage counts.
--manifest-no-digest Skip the SHA-256 file digests in --manifest (sizes only).
--dataset-card PATH Write a Hugging Face dataset card (README.md) describing the processing, rendered from the run manifest.
--report PATH Write a self-contained HTML audit report (removal funnel, PII counts by type, sample diffs) to PATH. Samples are redacted.
--report-raw-samples Include verbatim (unredacted) record samples in --report. The report then contains raw PII/secrets; handle it like the input data.
--resume Checkpoint progress to .checkpoint.json and, when a checkpoint exists, continue the run from where it stopped (jsonl/txt/csv outputs, --jobs 1).
--checkpoint-interval N 10000 Write a checkpoint every N input records (default 10000).