Technical report: matra_paper_200k.pdf

LLM with Matra as tokenizer is still pending. If anyone wants to help me with compute, hmu @ e-mail

Live Demo

note: visit huggingface space link if the embed fails to load.

Matra: A Tokenizer Built for India’s Languages

200k, 128k vocab size. 23 languages. The lowest token counts and highest compression across the board.

In active development, more optimzation, tests, language specific tokenizers coming soon.

Most tokenizers treat Indian languages as an afterthought, fragmented syllables and half-formed characters scattered across a vocabulary designed for English. Matra was built from the ground up to fix that. The result, benchmarked on June 15 2026 against the IN22-Gen test split at 200k tokens per language, is unambiguous: Matra produces fewer tokens, denser representations, and dramatically lower fragmentation than GPT-5, Gemini 3.5 Flash, Gemma-4-31B, Qwen-3.6-MoE, Sarvam-105B, and Sutra-v2 across all 23 scheduled languages of India.

That is not a claim. It is a number: 1,103,089 total tokens to encode the entire benchmark corpus, versus 1,794,545 for the next-best competitor (Gemma-4-31B) and 3,225,298 for the worst (Qwen-3.6-MoE).

Architectural Innovations

Tokenization for Indic scripts is hard for a structural reason. A single syllable in Hindi, Tamil, or Kannada may be encoded as 2–4 UTF-8 bytes. A tokenizer trained predominantly on English text will decompose that syllable into individual bytes or Unicode code-points, producing a cascade of meaningless fragments. Every extra token is wasted context, wasted compute, and a weaker representation for the model.

Matra addresses this with seven architectural innovations.

Unicode-Aware Pre-Tokenization

The pre-tokenizer uses a hand-crafted regex pattern that respects Unicode letter, mark, and number categories (\p{L}, \p{M}, \p{N}). Diacritics are kept attached to their base letters. CJK characters are split individually (they are morphologically atomic). Right-to-left scripts and Arabic extensions are handled natively. Whitespace, four-space indents, tabs, newlines are preserved as explicit tokens for code-aware applications.

Language-Weighted Training (XLM-R Scaling)

A two-stage merge strategy with script-aware frequency scaling. During Stage 1, word frequencies are multiplied by a lang_weight factor when the word contains target-script characters - detected via is_target_script() across Devanagari, Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugu, Kannada, Malayalam, Sinhala, Arabic/Perso-Arabic (Urdu, Kashmiri, Sindhi), Ol Chiki (Santali), and Meetei Mayek (Manipuri).

The scaling follows the XLM-R alpha-smoothing formula:

$$w_w = w^{\frac{1-\alpha}{\alpha}}$$

With $\alpha = 0.75$, target-script words receive a ~1.33× frequency boost while English remains unscaled. Stage 2 uses a blended sentence-level multiplier: a sentence with 80% Hindi and 20% English gets 80% of the full weight, preventing code-switched “Hinglish” from distorting the merge budget.

Two-Stage Merge Strategy

Stage 1 learns intra-word subword merges (morphological prefixes, suffixes, character n-grams) constrained within word boundaries. Stage 2 extends merges across word boundaries within a sentence, learning multi-word “superword” compounds. The merge budget is split: transition_vocab_size tokens allocated to Stage 1, the remainder to Stage 2. This decouples morphological learning from syntactic collocation, and no single-character fragment is promoted until both stages complete.

Sentence Boundary Detection

Pre-tokenized word tokens are grouped into sentences using a zero-width lookbehind split on [.!?\n।] followed by a lookahead for whitespace. During Stage 2, any merge whose resulting token contains a sentence delimiter (. ! ? \n । 。 ？ ！) is permanently banned from the vocabulary. This prevents semantically invalid tokens like “delhi. the” from ever being learned, preserving clean sentence boundaries for downstream transformer models.

R2L Digit Grouping

Following Singh & Strouse (2024), digit sequences are chunked right-to-left in groups of three using a positive-lookahead regex:

(?:\p{N}{1,3}(?=(?:\p{N}{3})*(?!\p{N})))

This preserves base-10 alignment in right-to-left scripts and prevents the arithmetic hallucination that arises when digit strings are fragmented across arbitrary byte boundaries.

Hapax Legomena Pruning

The drop_singletons flag removes word entries with frequency ≤ 1.0 between Pass 1 and Stage 1. Hapax legomena - words appearing exactly once in the corpus - contribute noise to merge statistics without providing generalizable patterns. Pruning them frees the merge budget for higher-frequency morphological units. Sentences containing pruned words are also filtered before Stage 2.

This is mathematically defensible as a data-efficiency choice; for maximally pure results, drop_singletons defaults to False.

Resource Efficiency

2-Pass Streaming Architecture

A naive in-memory BPE trainer accumulates four large dicts simultaneously: word sequences, word frequencies, word-to-string maps, and sentence counters. On a 300 MB corpus (2.8 GB effective after alpha-smoothing), this peaks at 75+ GB RAM - well beyond consumer hardware.

Matra splits training into two streaming passes:

1. Pass 1 - Stream the corpus, populate word-level dicts, run Stage 1 BPE, then delete all three word dicts and call gc.collect().
2. Pass 2 - Re-stream the corpus (the iterator_factory is called a second time), populate sentence-level statistics, and run Stage 2 BPE.

This decouples word and sentence accumulation:

Training comfortably fits within a 16 GB machine budget.

RAM Shield

Stage 2 operates on a single concatenated 1D array of all corpus sentences - ~39M tokens for a 300 MB corpus. Each merge creates new pairs at the two affected boundaries, and the majority of those new pairs are singletons (freq=1). BPE requires freq >= 2 to execute a merge, but the dict still holds them - and with millions of merges, the dict grows unboundedly until the OOM killer fires.

The RAM Shield hard-caps memory: when len(pair_counts) > max_pairs (default 3,000,000), all singleton pairs are purged, the heap is rebuilt from live entries, and gc.collect() releases the freed memory. Purging is mathematically lossless - no pair with freq <= 1 can ever be selected for merging.

Early stopping terminates the merge loop when the top heap entry’s frequency drops below 2, preventing the trainer from grinding through dead heap pops. On typical corpora, Stage 2 halts at ~5K–15K merges (well before the budget) once no more repeat pairs exist.

Combined effect: Stage 2 peak RSS is hard-capped at ~2.5 GB regardless of corpus size, and OOM is impossible.

Checkpointing & Crash Recovery

The trainer auto-saves a .ckpt file after each phase boundary, keyed by a 16-char SHA-256 hash of the training config {vocab_size, alpha, drop_singletons, lang_weight, passes}. This gives two superpowers:

1. Crash recovery - kill mid-Stage 1 with Ctrl-C, restart with --resume-from pass1, the 20-minute Pass 1 is loaded from cache.
2. Hyperparameter sweep - train once with --vocab 200000 --alpha 0.5, then re-run with --vocab 256000 --alpha 0.5 --reuse-pass1 to skip Pass 1 entirely.

Checkpoints are written atomically (.tmp + os.replace()), surviving mid-write SIGKILL. On load, __version__, __stage__, and config_hash are verified against the current run - all mismatches raise ValueError with actionable messages.

Cython-Accelerated Hot Paths

The inner merge loop (_bpe_merge.pyx) and pair-position builder (_bpe_passb.pyx) are optionally compiled with Cython via pyximport. When available, they replace the equivalent Python loops at the two points that dominate training time. On a 200k-token corpus the difference is roughly 4–5× wall-clock time. The pure-Python fallback remains correct and usable without a build step, making Matra deployable in environments where compiling C extensions is impractical.

O(1) Encode Path

At inference time, the byte-to-token mapping is pre-computed as a 256-entry tuple indexed by raw byte value, eliminating per-byte dictionary lookups. The merge pass uses a lazy min-heap with integer-keyed _merge_ranks_int maps - pairs are resolved via (id0, id1) → rank lookups instead of string hashing. A bounded LRU cache (default 2,000,000 entries) means repeated subwords pay zero merge cost after the first occurrence. The end-to-end complexity is $O(n \log n)$ per sentence in the number of pre-tokenization tokens.

Benchmark Results: IN22-Gen, 23 Languages

Benchmark date: 15 June 2026. Test split: IN22-Gen. Corpus size: approximately 200,000 tokens per language. Tokenizers evaluated: Matra, GPT-5, Gemini 3.5 Flash, Gemma-4-31B, MUTANT (Krutrim-2), Qwen-3.6-MoE, Sarvam-105B, Sutra-v2.

Metrics:

• SeqRed (Sequence Reduction): percentage reduction in token count versus the raw byte sequence. Higher is better.
• NSL (Normalised Sequence Length): token count divided by the character count of the source text. Lower is better.
• BPT (Bytes Per Token): average bytes encoded per token. Higher is better.
• Fert (Fertility): average tokens produced per word. Lower is better.
• 1ch%: percentage of tokens that encode a single character. Lower is better.

Aggregate across all 23 languages

Matra’s total token count is 42% lower than Gemma-4-31B and 66% lower than Qwen-3.6-MoE on the same corpus. The fertility score of 2.06 means the average Indic word is encoded in just over two tokens; Qwen requires three times as many.

Selected per-language highlights

Hindi: Matra achieves 79.9% sequence reduction, a fertility of 1.09, and 32,218 total tokens. GPT-5 requires 51,905 and a fertility of 1.76.

Tamil: Matra achieves 77.3% sequence reduction and a BPT of 12.03, the highest encoding density of any tokenizer on this language. The next best is Gemma-4-31B at 9.67.

Bengali: Matra achieves 77.4% sequence reduction and 34,179 total tokens. Gemma-4-31B and Sarvam-105B are tied at 40,950.

Manipuri (Meitei script): Where GPT-5, Qwen, and MUTANT collapse entirely (negative sequence reduction, fertility above 16), Matra encodes Manipuri at 53.6% sequence reduction with a fertility of 3.05. Sarvam-105B and Sutra-v2 are the only other tokenizers with positive sequence reduction on this script.

Odia: GPT-5 achieves near-zero sequence reduction (−0.1%) on Odia. MUTANT and Qwen are worse. Matra achieves 71.5% with 49,755 tokens, the second-best after Sarvam-105B (54,656).

Matra 200K vs. 128K: Vocab Size Scaling

Matra was evaluated at two vocabulary scales (128K and 200K) to measure the impact of vocabulary size on compression quality. Both benchmarks use the same IN22-Gen test corpus, though the set of competing tokenizers differs - the 128K run compares against GPT-5, Gemini 2.5 Flash, Qwen-3.6-MoE, Sarvam-105B, and Sutra-v2, while the 200K run adds Gemma-4-31B and MUTANT (Krutrim-2).

Scaling vocabulary size from 128K to 200K delivers consistent across-the-board compression gains. Total token count drops by over a fifth, fertility approaches the ideal 2.0 threshold, and bytes-per-token rises to 8.90 - the highest encoding density among all tokenizers at either scale. The 128K variant remains a compact alternative for deployment environments where embedding table size is at a premium.

Architecture Notes

The core class is Matra, initialised with a target vocabulary size and a set of special tokens. The public interface is intentionally narrow.

from tokenizer import Matra

tok = Matra(target_vocab_size=300)
tok.train_from_iterator(texts)          # texts: Iterable[str]

ids = tok.encode("नमस्ते दुनिया")
text = tok.decode(ids)

tok.save("matra.json")
tok.load("matra.json")

The saved format is a JSON file containing the merge list (as space-separated string pairs) and the vocabulary (as a list of [id, token_str] pairs). The format is intentionally compatible with the HuggingFace tokenizer interface for downstream use.

Key parameters:

• target_vocab_size: final vocabulary size including the 256 byte-level base tokens and special tokens. Default 300.
• transition_vocab_size: intermediate vocabulary size used in Stage 1 training. Default 275.
• lang_weight: upsampling multiplier for target-script text during pair counting. Default 1.0; increase for corpus mixes that are predominantly English.
• gc_threshold: ratio of merge count to token count at which an intermediate garbage-collection pass is triggered during training. Default 3.0.
• max_cache_size: maximum number of entries in the encode cache. Default 2,000,000.

Limitations and Known Behaviour

The benchmark results on this page use a vocabulary of 200,000 tokens, trained on a ~300 MB multilingual corpus via the 2-pass streaming pipeline. A compact 300-token variant is also available for transfer-learning and low-resource scenarios - smaller vocabulary sizes increase cross-lingual transferability at the cost of raw compression.

Cython compilation is optional. The pure-Python pipeline is identical in output; only training speed differs (4-5× slower without the compiled extensions).

Reproducing the Benchmark

The benchmark was run on the IN22-Gen test split using the script included in the repository. Tokenizer weights for GPT-5, Gemini 3.5 Flash, Gemma-4-31B, Qwen-3.6-MoE, Sarvam-105B, Sutra-v2, and MUTANT (Krutrim-2) were loaded from their respective public releases. Matra was trained on a held-out training split; the benchmark corpus was not seen during training.

All metrics are computed at the token level against the original UTF-8 byte stream.