{"repo":"pkoukk/tiktoken-go","free":true,"listed":false,"github":"https://github.com/pkoukk/tiktoken-go","clone":"git clone https://github.com/pkoukk/tiktoken-go.git","description":"go version of tiktoken","language":"Go","stars":946,"topics":["gpt-35-turbo","openai","tiktoken","chatgpt","go","golang","gpt-4"],"license":"MIT","category":"ai-agents","readme_excerpt":"tiktoken-go 简体中文 OpenAI's tiktoken in Go. Tiktoken is a fast BPE tokeniser for use with OpenAI's models. This is a port of the original tiktoken. Usage Install Cache Tiktoken-go has the same cache mechanism as the original Tiktoken library. You can set the cache directory by using the environment variable TIKTOKEN CACHE DIR. Once this variable is set, tiktoken-go will use this directory to cache the token dictionary. If you don't set this environment variable, tiktoken-go will download the dictionary each time you initialize an encoding for the first time. Alternative BPE loaders If you don't want to use cache or download the dictionary each time, you can use alternative BPE loader. Just call tiktoken.SetBpeLoader before calling tiktoken.GetEncoding or tiktoken.EncodingForModel . BpeLoader is an interface, you can implement your own BPE loader by implementing this interface. Offline BPE loader The offline BPE loader loads the BPE dictionary from embed files, it helps if you don't want to download the dictionary at runtime. Due to the size of the BPE dictionary, this loader is in other project. Include if you require this loader: tiktoken loader Examples Get Token By Encoding Get Token By Model Counting Tokens For Chat API Calls Below is an example function for counting tokens for messages passed to gpt-3.5-turbo or gpt-4. The following code was written based on openai-cookbook examples at Wednesday, 28 June 2023 . Please note that the token calculation method for the message ","default_branch":null,"files":null,"tree":[],"storefront":"/r/pkoukk","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/pkoukk/tiktoken-go/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}