{"repo":"AnkitNayak-dev/CrawlAI-RAG","free":true,"listed":false,"github":"https://github.com/AnkitNayak-dev/CrawlAI-RAG","clone":"git clone https://github.com/AnkitNayak-dev/CrawlAI-RAG.git","description":"CrawlAI RAG is an AI-powered website intelligence platform that allows users to crawl entire websites, index their content, and ask natural-language questions using Retrieval-Augmented Generation (RAG). It transforms static websites into queryable knowledge bases.","language":"Python","stars":153,"topics":["agent","crawler","rag","rag-chatbot","rag-pipeline"],"license":"MIT","category":"ai-agents","readme_excerpt":"CrawlAI RAG CrawlAI RAG is an AI-powered website intelligence platform that allows users to crawl entire websites, index their content, and ask natural-language questions using Retrieval-Augmented Generation (RAG) . It transforms static websites into queryable knowledge bases . --- Key Features Website Crawling - Crawls all internal pages of a website - Extracts clean, readable text RAG-Based Question Answering - Uses vector embeddings + LLM - Answers are grounded in website content - Minimizes hallucinations Multi-Website Indexing - Index multiple websites - All content stored in a shared vector database Fast & Scalable Backend - Built with FastAPI - ChromaDB for vector storage Simple Frontend - Built with Streamlit - Clean, single-query interface Secure Configuration - Environment variables via .env - API keys are never committed to GitHub --- Tech Stack Layer Technology ------ ----------- Backend FastAPI Frontend Streamlit AI / RAG LangChain Vector Database ChromaDB Embeddings Sentence-Transformers LLM Groq (LLaMA 3.3 70B) Web Scraping BeautifulSoup4 & Playwright Configuration python-dotenv --- Usage Guide 1. Index a Website 1. Enter a website URL 2. Click Index Website 3. Website content is crawled, chunked, and embedded 2. Ask Questions Ask natural-language questions such as: - What is this website about? - List all services mentioned - Who is the author? The system returns accurate, grounded answers based only on the indexed website content. --- How It Works 1. Website ","default_branch":null,"files":null,"tree":[],"storefront":"/r/AnkitNayak-dev","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/AnkitNayak-dev/CrawlAI-RAG/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}