{"repo":"twardoch/split-markdown4gpt","free":true,"listed":false,"github":"https://github.com/twardoch/split-markdown4gpt","clone":"git clone https://github.com/twardoch/split-markdown4gpt.git","description":"A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.","language":"Python","stars":29,"topics":["data-preprocessing","gpt","gpt-3","gpt-35-turbo","gpt-35-turbo-16k","gpt-4","markdown","natural-language-processing","nlp","openai"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"split-markdown4gpt: split large Markdown for LLM context windows LLMs have finite context windows. A 50,000-word documentation site won't fit in one API call. split-markdown4gpt breaks Markdown files into token-bounded chunks that respect document structure — so each chunk stays coherent rather than cutting mid-sentence or mid-section. Why splitting matters Every LLM API call has a hard token limit (e.g. 4,096 tokens for GPT-3.5, 8,192 for GPT-4). Exceeding it raises an error. Naive character-based chunking cuts through sentences, code blocks, and headings — destroying context. This tool: 1. Parses the Markdown AST (via mistletoe ) 2. Counts tokens using the same tokenizer the model uses (via tiktoken ) 3. Packs as many complete sections as fit within the limit 4. When a single section exceeds the limit, falls back to sentence-level splitting (via syntok ) The result: chunks that respect heading hierarchy, never split a sentence across chunks, and are as large as possible without exceeding the token budget. Install CLI Outputs all sections joined by the separator. Redirect to a file or pipe to another tool. Flag Default Description ------ --------- ------------- --model gpt-3.5-turbo Tokenizer model (determines token counting) --limit model max Max tokens per chunk --separator === SPLIT === String inserted between chunks in output Python API Also accepts a string or file-like object: How it works MarkdownLLMSplitter class Methods: load md() , load md path() , load md str() , ","default_branch":null,"files":null,"tree":[],"storefront":"/r/twardoch","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/twardoch/split-markdown4gpt/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}