{"repo":"jserv/cjk-token-reducer","free":true,"listed":false,"github":"https://github.com/jserv/cjk-token-reducer","clone":"git clone https://github.com/jserv/cjk-token-reducer.git","description":"Reduce Claude Code token usage by 35-50% when using CJK (Chinese, Japanese, and Korean)","language":"Rust","stars":52,"topics":["claude-code","llm-inference","cjk-tokenizer"],"license":"MIT","category":"mcp-servers","readme_excerpt":"cjk-token-reducer Reduce Claude Code token usage by 35-50% when using CJK languages. The Problem CJK (Chinese, Japanese, Korean) languages consume 2-4x more tokens than English for the same semantic content. This discrepancy leads to higher costs, faster context exhaustion, and reduced context windows for RAG/agent workflows. Language Avg Token Ratio Typical Range Notes ---------- ----------------- --------------- ------- Chinese 2.0-3.0x 1.5-4.0x Rare characters may split into 3-4 tokens Japanese 2.12x 1.5-8.0x Mixed Kanji/Kana creates segmentation challenges Korean 2.36x 2.0-3.0x Agglutinative nature compounds inefficiency Token ratios based on BPE tokenizer analysis. Actual savings depend on text complexity and technical term density. Why Does This Happen? The inefficiency stems from the mechanics of Byte-Pair Encoding (BPE) and training data distribution: 1. Vocabulary Bias: Modern tokenizers train primarily on English corpora. Common English words merge into single tokens. CJK characters, occurring less frequently in training data, often fail to merge into \"words\" and split into individual character tokens or raw bytes. 2. UTF-8 Byte Fallback: A common cause of token expansion. - Many LLM tokenizers process text as UTF-8 bytes. - An English character is 1 byte. - A CJK character is typically 3 bytes in UTF-8. - If a CJK character is absent from the tokenizer's vocabulary, byte-level tokenizers may expand it into multiple tokens. The exact expansion depends on the tokeniz","default_branch":null,"files":null,"tree":[],"storefront":"/r/jserv","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jserv/cjk-token-reducer/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}