{"repo":"Azzedde/clever_searcher","free":true,"listed":false,"github":"https://github.com/Azzedde/clever_searcher","clone":"git clone https://github.com/Azzedde/clever_searcher.git","description":"Intelligent web discovery agent with LLM-powered planning, multi-source search, smart deduplication, and GRPO preference dataset collection. Autonomously searches, analyzes, and summarizes web content while building training data for model fine-tuning.","language":"Python","stars":13,"topics":["ai-agents","automation","crawling","llm","openai","python","webscraping"],"license":"MIT","category":"ai-agents","readme_excerpt":"Clever Searcher An intelligent web discovery and content analysis agent that autonomously searches, fetches, analyzes, and summarizes web content based on your queries. Built with LLM-powered planning, multi-source search, smart deduplication, and personalized scoring. What It Actually Does Clever Searcher takes a natural language query and: 1. 🧠 Plans - Uses LLM to generate diverse search queries and crawl strategies 2. 🔍 Searches - Searches DuckDuckGo and/or Tavily for relevant content 3. 📥 Fetches - Downloads content using httpx with Playwright fallback for JS-heavy sites 4. 🔄 Deduplicates - Removes duplicate content using URL canonicalization and content hashing 5. 📝 Summarizes - Generates structured summaries with key points, tags, and entities 6. ⭐ Scores - Ranks content using embeddings and personalization (with simple fallback) 7. 📊 Outputs - Creates markdown digests, JSON exports, and database storage 8. 🎯 Learns - Collects preference data for GRPO training and personalization Key Features - 🆓 Free to Use : DuckDuckGo search (no API keys) + Ollama support (local LLM) - 🤖 LLM-Powered Planning : Intelligent query generation and crawl strategy - 🔄 Smart Deduplication : URL canonicalization + content fingerprinting - 📊 Structured Summaries : Key points, tags, entities, and read time estimates - 🎯 Personalization : Embedding-based scoring with user feedback learning - 📈 Preference Learning : GRPO dataset collection for model improvement - 🔧 Flexible Architec","default_branch":null,"files":null,"tree":[],"storefront":"/r/Azzedde","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Azzedde/clever_searcher/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}