{"repo":"MohaElbadry/SearchEngine_And_WebCrawler","free":true,"listed":false,"github":"https://github.com/MohaElbadry/SearchEngine_And_WebCrawler","clone":"git clone https://github.com/MohaElbadry/SearchEngine_And_WebCrawler.git","description":"SearchEngine and Web Crawler","language":"Java","stars":13,"topics":["crawler","search-engine","webcrawler"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"SearchEngine And WebCrawler: Built From Scratch An educational project to understand how modern search engines work, focusing on semantic search, web crawling, and vector embeddings. Key Features - Intelligent Web Crawling : Configurable depth and domain filtering - Semantic Understanding : Uses AI embeddings that capture meaning, not just keywords - Natural Language Queries : Search using conversational language - Vector-Based Retrieval : Returns results ranked by semantic similarity Architecture The system consists of four main components: %20(1).png) 1. Web Crawler : Extracts content from websites starting from seed URLs 2. Embedding Generator : - Converts text to vector embeddings using Ollama models - Supports different embedding models (default: nomic-embed-text ) 3. Elasticsearch Storage : Stores documents with their vector representations 4. Semantic Search Engine : Performs vector similarity search for natural language queries - Takes natural language queries as input - Converts queries to vector embeddings - Returns ranked results based on semantic similarity How It Works Crawling Flow 1. Start with a seed URL (default or user-provided) 2. Extract text content using JSoup HTML parsing 3. Generate vector embeddings for the text 4. Store document with embeddings in Elasticsearch 5. Extract all links from the page 6. Queue new discovered URLs for processing 7. Continue until depth limit is reached or queue is empty Search Flow 1. User enters a natural language query 2.","default_branch":null,"files":null,"tree":[],"storefront":"/r/MohaElbadry","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/MohaElbadry/SearchEngine_And_WebCrawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}