{"repo":"philpher0x/vectrain","free":true,"listed":false,"github":"https://github.com/philpher0x/vectrain","clone":"git clone https://github.com/philpher0x/vectrain.git","description":"Vectrain is a high-performance, modular, plug-and-play RAG pipeline that ingests data, generates vector embeddings, and stores them in vector databases for semantic search, recommendations, and analytics.","language":"Go","stars":14,"topics":["golang","kafka","qdrant","vector-embeddings","embedding-vectors","embeddings","go","rag","vector-database","ai"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Vectrain — Vector Embeddings Pipeline Overview Vectrain is a high-performance vector embedding service built in Go. It provides a flexible and modular pipeline architecture for sourcing data, generating embeddings, and storing them in vector databases. Designed for scalability, Vectrain enables seamless integration of embeddings into semantic search, analytics, and recommendation systems. Features - Pipeline Architecture : Modular system with source, embedder, and storage components - Sources - Kafka Integration : Stream processing from Kafka topics - REST Integration : Accept data from external services via REST endpoints - Embedding - Vector Embeddings : Generate embeddings using Ollama models - Storage - Qdrant Storage : Store and query vector embeddings in Qdrant database - Configurable Components : Easily adjust batch sizes and worker counts - HTTP API : RESTful API for controlling and interacting with the pipeline Pipeline Description Vectrain’s pipeline manages the entire process of transforming raw data into vector embeddings. Architecture Vectrain implements a three-stage pipeline architecture: 1. Source - Reads data from configured sources (Kafka) 2. Embedder - Generates vector embeddings from the source data 3. Storage - Persists embeddings to vector databases (Qdrant) Key points - Configurable Adapters: When starting the pipeline, you specify the types of source , storage , and embedder . These adapters interface with external services. - Supported Sources: Curren","default_branch":null,"files":null,"tree":[],"storefront":"/r/philpher0x","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/philpher0x/vectrain/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}