{"repo":"Softlandia-Ltd/vision-is-all-you-need","free":true,"listed":false,"github":"https://github.com/Softlandia-Ltd/vision-is-all-you-need","clone":"git clone https://github.com/Softlandia-Ltd/vision-is-all-you-need.git","description":"Serverless Modal + FastAPI + React + ColPali + Qdrant + GPT4o Vision RAG (V-RAG) Demo","language":"TypeScript","stars":403,"topics":["colpali","fastapi","modal","qdrant","rag","react","v-rag","vision-rag"],"license":"MIT","category":"ai-agents","readme_excerpt":"Vision is All You Need: V-RAG (Vision RAG) Demo This is a demo of the Vision RAG (V-RAG) architecture. The V-RAG architecture utilizes a vision language model (VLM) to embed pages of PDF files (or any other document) as vectors directly, without the tedious chunking process. https://github.com/user-attachments/assets/44d72558-80e8-498d-903a-88672ec6ebd7 Check out the background blog post: https://softlandia.fi/en/blog/building-a-rag-tired-of-chunking-maybe-vision-is-all-you-need How does V-RAG work? 1. The pages of a PDF file are converted to images. - In theory these images can be anything, but the current demo uses PDF files since the underlying model has been trained on PDF files - pypdfium is used to convert the PDF pages to images 2. The images are passed through a VLM to get the embeddings. - ColPali is used as the VLM in this demo 3. The embeddings are stored in a database - QDrant is used as the vector database in this demo 4. The user passes a query to the V-RAG system 5. The query is passed through the VLM to get the query embedding 6. The query embedding is used to search the vector database for similar embeddings 7. The user query and images of the best matches from the search are passed again to a model that can understand images - we use GPT4o or GPT4o-mini in this demo 8. The model generates a response based on the query and the images How to run the demo? Make sure tou have an account in Hugging Face. Make sure you are logged into Hugging Face using transforme","default_branch":null,"files":null,"tree":[],"storefront":"/r/Softlandia-Ltd","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Softlandia-Ltd/vision-is-all-you-need/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}