{"repo":"moeki0/baran","free":true,"listed":false,"github":"https://github.com/moeki0/baran","clone":"git clone https://github.com/moeki0/baran.git","description":"Text Splitter for Large Language Model (LLM) datasets.","language":"Ruby","stars":19,"topics":["ai","llm","markdown","ruby","gem"],"license":"MIT","category":"ai-agents","readme_excerpt":"Baran Text Splitter for Large Language Model datasets. To avoid token constraints and improve the accuracy of vector search in the Large Language Model, it is necessary to divide the document. This gem supports splitting the text in the specified manner. Features Baran provides efficient text splitting capabilities with the following key features: - Chunk Size Control : Split text into specified sizes - Overlap Management : Maintain continuity between chunks with configurable overlap - Context Preservation : Respect semantic boundaries in text - Metadata Support : Attach metadata to each chunk - Multiple Splitting Strategies : Character-based, recursive, sentence-based, and Markdown-aware splitting Installation Using Bundler Add this line to your application's Gemfile: And then execute: $ bundle install Direct Installation $ gem install baran Quick Start Usage Default Parameters - chunk size : 1024 (characters) - chunk overlap : 64 (characters) Character Text Splitter Splitting by the specified character. Recursive Character Text Splitter Splitting by the specified characters recursively, using the first separator found in the text. Sentence Text Splitter Splitting text by sentence boundaries (periods, exclamation marks, question marks). Markdown Text Splitter Splitting by Markdown structure with awareness of headers, code blocks, and other elements. Split with the following priority: \\n\\n\", # code block \"\\n\\n \\n\\n\", # horizontal rule \"\\n\\n---\\n\\n\", # horizontal rule \"\\n\\n \\n","default_branch":null,"files":null,"tree":[],"storefront":"/r/moeki0","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/moeki0/baran/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}