Technical report (arXiv preprint): matra-arxiv-pre.pdf | Previous report: matra_paper_200k.pdf

LLM with Matra as tokenizer is still pending. If anyone wants to help me with compute, hmu @ e-mail

Live Demo

note: visit huggingface space link if the embed fails to load.

Matra: Sentence-Constrained Superword Tokenization for Indic Scripts

200k vocab size. 24 languages. Aggregate fertility 1.78, 72.5% sequence reduction, 9.10 bytes per token.

In active development; more optimization, tests, and language-specific tokenizers coming soon.

Standard byte-pair encoding tokenizers trained on English-centric corpora fragment Indic scripts into single-character tokens, inflating sequence lengths by 3–8× relative to English. Matra addresses this with two modifications to the BPE training pipeline: a sentence-boundary constraint that prohibits cross-sentence merges, and word-level script-frequency weighting that applies XLM-R alpha-smoothing to individual token frequencies. On the MUTANT evaluation set across 22 Indic languages, English, and code, Matra produces fewer tokens and lower fertility than GPT-5, Gemini 3.5 Flash, Gemma-4-31B, Qwen-3.6-MoE, Sarvam-105B, and Sutra-v2.

2,125,481 total tokens to encode the entire MUTANT evaluation corpus: 31% fewer than Gemini 3.5 Flash (3,079,018), 62% fewer than Qwen-3.6-MoE (5,560,104), and 22% fewer than Sarvam-105B (2,714,325).

Architectural Innovations

Tokenization for Indic scripts is hard for a structural reason. A single syllable in Hindi, Tamil, or Kannada may be encoded as 2–4 UTF-8 bytes. A tokenizer trained predominantly on English text will decompose that syllable into individual bytes or Unicode code-points, producing a cascade of single-character tokens. Each extra token increases inference cost and reduces effective context length.

Matra addresses this with the following modifications.

Unicode-Aware Pre-Tokenization

The pre-tokenizer uses a hand-crafted regex pattern that respects Unicode letter, mark, and number categories (\p{L}, \p{M}, \p{N}). Diacritics are kept attached to their base letters. CJK characters are split individually (they are morphologically atomic). Right-to-left scripts and Arabic extensions are handled natively. Whitespace, four-space indents, tabs, newlines are preserved as explicit tokens for code-aware applications.

Language-Weighted Training (XLM-R Scaling)

A two-stage merge strategy with script-aware frequency scaling. During Stage 1, word frequencies are multiplied by a lang_weight factor when the word contains target-script characters - detected via is_target_script() across Devanagari, Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugu, Kannada, Malayalam, Sinhala, Arabic/Perso-Arabic (Urdu, Kashmiri, Sindhi), Ol Chiki (Santali), and Meetei Mayek (Manipuri).

The scaling follows the XLM-R alpha-smoothing formula:

$$w_w = w^{\frac{1-\alpha}{\alpha}}$$

With $\alpha = 0.75$, target-script words receive a ~1.33× frequency boost while English remains unscaled. Stage 2 uses a blended sentence-level multiplier: a sentence with 80% Hindi and 20% English gets 80% of the full weight, preventing code-switched “Hinglish” from distorting the merge budget.

Two-Stage Merge Strategy

Stage 1 learns intra-word subword merges (morphological prefixes, suffixes, character n-grams) constrained within word boundaries. Stage 2 extends merges across word boundaries within a sentence, learning multi-word “superword” compounds. The merge budget is split: transition_vocab_size tokens allocated to Stage 1, the remainder to Stage 2. This decouples morphological learning from syntactic collocation, and no single-character fragment is promoted until both stages complete.

Sentence-Boundary Constraint

Pre-tokenized word tokens are grouped into sentences using a zero-width lookbehind split on [.!?\n।] followed by a lookahead for whitespace. During Stage 2, any merge whose resulting token contains a sentence-terminal character (. ! ? \n । 。 ？ ！) is permanently prohibited. This prevents semantically invalid tokens like "city.The" or "delhi|mumbai" from ever being learned, preserving clean sentence boundaries for downstream transformer models.

This constraint is absent from both SuperBPE and MUTANT; removing it produces 23 delimiter-spanning tokens in the MUTANT vocabulary and 17 in Sutra-v2. Matra contains zero by construction.

R2L Digit Grouping

Following Singh & Strouse (2024), digit sequences are chunked right-to-left in groups of three using a positive-lookahead regex:

(?:\p{N}{1,3}(?=(?:\p{N}{3})*(?!\p{N})))

This preserves base-10 alignment in right-to-left scripts and prevents the arithmetic hallucination that arises when digit strings are fragmented across arbitrary byte boundaries.

Hapax Legomena Pruning

The drop_singletons flag removes word entries with frequency ≤ 1.0 between Pass 1 and Stage 1. Hapax legomena - words appearing exactly once in the corpus - contribute noise to merge statistics without providing generalizable patterns. Pruning them frees the merge budget for higher-frequency morphological units. Sentences containing pruned words are also filtered before Stage 2.

This is mathematically defensible as a data-efficiency choice; for maximally pure results, drop_singletons defaults to False.

Benchmark Results: MUTANT Evaluation Set (24 Languages)

Benchmark date: July 2026. Evaluation set: MUTANT (22 Indic languages, English, and code). Tokenizers evaluated: Matra, GPT-5, Gemma-4-31B, Gemini 3.5 Flash, Qwen-3.6-MoE, Sarvam-105B, Sutra-v2.

Metrics:

• SeqRed (Sequence Reduction): percentage reduction in token count versus the raw byte sequence. Higher is better.
• NSL (Normalised Sequence Length): token count divided by the character count of the source text. Lower is better.
• BPT (Bytes Per Token): average bytes encoded per token. Higher is better.
• Fert (Fertility): average tokens produced per word. Lower is better.
• 1ch%: percentage of tokens that encode a single character. Lower is better.

Aggregate values are language-averaged (mean of per-language values) to give equal weight to each language regardless of corpus size.

Aggregate across all 24 languages

Matra encodes the full MUTANT evaluation set in 2,125,481 tokens: 31% fewer than Gemini 3.5 Flash (3,079,018), 62% fewer than Qwen-3.6-MoE (5,560,104), and 22% fewer than Sarvam-105B (2,714,325). The aggregate fertility of 1.78 means the average word across all 24 languages is encoded in under two tokens; the next-best competitor (Sarvam-105B) requires 2.23. At fixed context length, a 22% reduction in token count translates directly to inference cost and latency savings.

Selected per-language highlights

Hindi: Matra achieves a fertility of 1.02, the lowest in the entire benchmark. The next-best is Sarvam-105B at 1.47.

Bengali: Matra achieves fertility 1.30, a 25.3% improvement over MUTANT-Indic (1.74) and well ahead of Sarvam-105B (1.69).

Tamil: Matra achieves fertility 1.91, outperforming Sarvam-105B (2.60) and Sutra-v2 (2.63).

Malayalam: Matra achieves fertility 2.44, the highest among Matra’s own results, reflecting the script’s heavy agglutination. Still ahead of Sarvam-105B (3.45) and Sutra-v2 (3.39).

Manipuri (Meetei Mayek): Where five of seven baselines collapse entirely (GPT-5 fertility 2.21, but with negative sequence reduction in prior benchmarks), Matra achieves fertility 1.91, matching Sarvam-105B and outperforming Sutra-v2 (2.23).

Odia: Where GPT-5 collapses to near-zero sequence reduction and fertility of 6.97, Matra achieves fertility 1.70.

Santali (Ol Chiki): Matra achieves fertility 2.72; Sarvam-105B leads at 2.01 and Sutra-v2 at 2.09. This is a known limitation of the current training corpus on Ol Chiki specifically.

English: Matra achieves fertility 1.15, outperforming all baselines including Sutra-v2 (1.19) and GPT-5 (1.34). Indic gains do not come at the cost of English regression.

Per-language fertility (MUTANT evaluation set)

The gains are not concentrated in a single script family. Devanagari languages achieve fertility between 1.02 and 2.12; Dravidian languages between 1.91 and 2.44; Perso-Arabic scripts between 1.17 and 1.46. The method targets structural properties of all Indic orthographies rather than handcrafted rules for specific languages.

Head-to-head with MUTANT-Indic

On 18 overlapping languages from the MUTANT evaluation set, Matra achieves lower fertility on 15 of 18 languages versus MUTANT-Indic, with a language-averaged fertility of 1.66 versus 1.89 (12.2% improvement).

Delimiter-spanning token analysis

A preliminary analysis of baseline vocabularies reveals semantically invalid tokens produced by cross-boundary merges:

The sentence-boundary constraint eliminates an entire class of alignment artifacts without any cost to compression ratio. When a high-frequency merge is prohibited, the merge budget is simply filled by the next-most-frequent valid pair.

Cross-lingual parity

Under GPT-5, English achieves fertility 1.34 while the average Indic fertility is 3.17 a ratio of 2.4×. Under Matra, English achieves 1.15 and the average Indic fertility is 1.78 a ratio of 1.5×. The script-frequency weighting closes roughly half of the parity gap without any degradation to English compression.

Vocab size scaling: 128K vs. 200K

Scaling from 128K to 200K follows a logarithmic law consistent with SuperBPE’s English results. The 128K variant remains a compact alternative for deployment environments where embedding table size is at a premium.

Previous Benchmark: IN22-Gen (23 Languages)

The following results are from the June 2026 IN22-Gen benchmark at 200K tokens per language. They are preserved for reference and comparison with the newer MUTANT evaluation above.

Limitations and Known Behaviour

Matra underperforms Sarvam-105B and Sutra-v2 on Santali in absolute fertility (2.72 vs. 2.01 and 2.09 respectively). These systems appear to have greater Ol Chiki representation in their training corpora. On Telugu, MUTANT-Indic achieves 1.88 versus Matra’s 2.22, and on Malayalam, MUTANT-Indic achieves 2.30 versus Matra’s 2.44. These are training data limitations, not algorithmic ones.

Downstream task evaluation is absent from the current paper. Fertility improvements are necessary but not sufficient evidence that a tokenizer improves model quality. Perplexity and task accuracy experiments require training language models on comparable data with comparable compute budgets, which is compute-intensive. This is an explicit limitation and future work direction.

The sentence-boundary constraint assumes that sentence boundaries are unambiguous and correctly detected by the pre-tokenizer. For highly informal text with non-standard punctuation, this assumption may fail. Evaluating on social media or conversational data is not done in this work.

Cython compilation is optional. The pure-Python pipeline is identical in output; only training speed differs (4-5× slower without the compiled extensions).

Reproducing the Benchmark

The primary benchmark was run on the MUTANT evaluation set using the script included in the repository. Tokenizer weights for GPT-5, Gemma-4-31B, Gemini 3.5 Flash, Qwen-3.6-MoE, Sarvam-105B, and Sutra-v2 were loaded from their respective public releases. Matra was trained on 10 GB of curated multilingual text; the benchmark corpus was not seen during training.

All metrics are computed at the token level against the original UTF-8 byte stream. The tokenizer serializes to a standard GPT-2-compatible JSON format (vocab + merges) that loads directly into HuggingFace tokenizers. Round-trip fidelity is guaranteed: decode(encode(x)) = x for any valid UTF-8 input x.

System Implementation

2-Pass Streaming Architecture

A naive in-memory BPE trainer accumulates four large dicts simultaneously: word sequences, word frequencies, word-to-string maps, and sentence counters. On a 10 GB corpus, this peaks at ~75 GB RAM.

Matra splits training into two streaming passes:

1. Pass 1: Stream the corpus, populate word-level dicts, run Stage 1 BPE, then delete all word dicts and call gc.collect().
2. Pass 2: Re-stream the corpus, populate sentence-level statistics, and run Stage 2 BPE.

On a 16 GB machine, training completes in ~41 minutes from 10 GB of streaming data. No GPU or external compiler is required.

Memory Cap

Stage 2 operates on a single concatenated 1D array. Each merge creates new pairs at affected boundaries, most of which are singletons (freq=1). BPE requires freq >= 2 to merge, but the dict still holds singletons. With millions of merges, the dict grows unboundedly.

When len(pair_counts) > max_pairs (default 3,000,000), all singleton pairs are purged and the heap is rebuilt. No pair with freq <= 1 can ever be selected for merging, so this is lossless. Early stopping terminates the loop when the top heap entry drops below frequency 2. Combined effect: Stage 2 peak RSS stays below ~2.5 GB.

Checkpointing

The trainer auto-saves a .ckpt file after each phase boundary, keyed by a SHA-256 hash of the training config. This enables crash recovery and hyperparameter sweeps: a killed run can be resumed from any phase boundary, and re-running with a different vocab_size but identical earlier config skips completed passes from cache.

Checkpoints are written atomically (.tmp + os.replace()). On load, __version__, __stage__, and config_hash are verified against the current run; mismatches raise ValueError.

Cython Compilation

The inner merge loop and pair-position builder are optionally compiled with Cython via pyximport. When available, they replace the equivalent Python loops at the two points that dominate training time. On a 200k-token corpus the difference is roughly 4–5× wall-clock time. The pure-Python fallback remains correct and usable without a build step.

Inference Optimizations

At inference time, the byte-to-token mapping is pre-computed as a 256-entry tuple indexed by raw byte value, eliminating per-byte dictionary lookups. The merge pass uses a lazy min-heap with integer-keyed _merge_ranks_int maps pairs are resolved via (id0, id1) → rank lookups instead of string hashing. A bounded LRU cache (default 2,000,000 entries) means repeated subwords pay zero merge cost after the first occurrence. End-to-end complexity is $O(n \log n)$ per sentence in the number of pre-tokenization tokens.