{"repo":"cwt/fts5-icu-tokenizer","free":true,"listed":false,"github":"https://github.com/cwt/fts5-icu-tokenizer","clone":"git clone https://github.com/cwt/fts5-icu-tokenizer.git","description":"FTS5 ICU Tokenizer for SQLite (mirror)","language":"Zig","stars":27,"topics":["fts5","sqlite","tokenizer"],"license":"MIT","category":"databases-storage","readme_excerpt":"FTS5 ICU Tokenizer for SQLite (Zig 0.16.0 Edition) Version 7.1.0 This project provides custom FTS5 tokenizers for SQLite implemented in Zig 0.16.0 using the International Components for Unicode (ICU) library to provide robust word segmentation and text normalization across multiple languages. --- Why Migrate to Zig 0.16.0? This project was originally written in C with CMake. The rewrite to Zig 0.16.0 resolves several fundamental issues inherent to C extension development: 1. Guaranteed Memory Safety & Zero Leaks - Old C Problem : Error handling across temporary text buffers and ICU handles required fragile goto cleanup; branches. Missing a free() or ubrk close() call on an error path caused memory leaks during high-throughput SQLite FTS indexing sessions. - Zig Solution : First-class defer and errdefer semantics guarantee that every heap allocation ( allocator.alloc , allocator.create ) and ICU handle ( ubrk close , utrans close ) is deterministically cleaned up, even when errors occur mid-tokenization. 2. Elimination of Build System & Toolchain Fragility - Old C Problem : Building cross-platform C extensions required complex CMake files, platform-specific Homebrew path workarounds ( brew --prefix icu4c ), Windows MSVC compiler flags ( /utf-8 ), and external generators ( make , ninja , Visual Studio). - Zig Solution : zig build replaces CMake and custom shell build scripts entirely. Zig operates as a unified compiler and build driver ( zig cc / b.addTranslateC ), handling C h","default_branch":null,"files":null,"tree":[],"storefront":"/r/cwt","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/cwt/fts5-icu-tokenizer/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}