{"repo":"jroakes/CrawlnChat","free":true,"listed":false,"github":"https://github.com/jroakes/CrawlnChat","clone":"git clone https://github.com/jroakes/CrawlnChat.git","description":"A modular web crawling and chat system that allows for ingesting website content through XML sitemaps, converting to vector embeddings, and providing AI-powered chat interfaces through multiple frontend options.","language":"Python","stars":28,"topics":["crawler","langgraph-python","openai","pinecone","rag"],"license":"MIT","category":"ai-agents","readme_excerpt":"Crawl n Chat A modular web crawling and chat system that allows for ingesting website content through XML sitemaps, converting to vector embeddings, and providing AI-powered chat interfaces through multiple frontend options. Features - Web Crawling : Asynchronously crawl websites from XML sitemaps, with built-in HTML-to-markdown conversion - Vector Storage : Efficient text chunking and embedding storage in Pinecone for high-performance similarity search - Agentic Routing : LangGraph-powered intelligent question routing that treats each website as a specialized knowledge tool - Multiple Interfaces : FastAPI (Swagger UI) and MCP (Anthropic Claude) frontends - Brand Compliance : Automated review of responses against brand guidelines - Configuration Flexibility : Support for both JSON and YAML configuration formats - Source Attribution : Automatic inclusion of source URLs in responses for attribution and verification - Enhanced Logging : Detailed logging of agent interactions, tool usage, and response generation System Architecture Installation Configuration Create a website configuration file in either JSON or YAML format: JSON Configuration Example YAML Configuration Example Environment Variables Set up your environment variables in the .env file. Here's a sample configuration: Usage Start the Service CLI Usage The CLI interface allows for quick testing and interaction with the service: API Endpoints FastAPI Interface The FastAPI server provides a comprehensive REST API with Sw","default_branch":null,"files":null,"tree":[],"storefront":"/r/jroakes","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jroakes/CrawlnChat/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}