{"repo":"mihaimoga/WebSearchEngine","free":true,"listed":false,"github":"https://github.com/mihaimoga/WebSearchEngine","clone":"git clone https://github.com/mihaimoga/WebSearchEngine.git","description":"https://www.text-mining.ro/ demonstrates how to design & implement a Web Search Engine","language":"C++","stars":19,"topics":["crawler","search","search-engine","spider","mfc","mysql-database","visual-cpp","win32"],"license":"GPL-3.0","category":"scrapers-browser-automation","readme_excerpt":"WebSearchEngine This project demonstrates the design and implementation of a basic Web Search Engine , showcasing key components such as web crawling, indexing, and query processing. Overview The search engine operates by crawling websites, parsing their content, and indexing relevant information for efficient retrieval. It respects the robots.txt protocol and extracts metadata such as: - Page titles - HTML content - JavaScript and CSS - Headings - Meta tags Features - Web Crawler : Navigates through websites, obeying robots.txt directives. - Indexer : Processes and stores page content for fast lookup. - Query Processor : Handles user queries and retrieves relevant results. - Metadata Extraction : Captures structured data from HTML for improved relevance. How It Works 1. Web Crawling The engine starts by crawling web pages: - Downloads HTML content from URLs. - Follows links to discover new pages. - Respects robots.txt rules to avoid restricted areas. It extracts: - Page titles - Headings ( , , etc.) - Meta tags - JavaScript and CSS references - Raw HTML content 2. Indexing After crawling, the engine processes and stores content in an index: - Tokenization : Breaks text into searchable words. - Normalization : Lowercases text, removes punctuation, etc. - Metadata Storage : Links each word to the page(s) it appears in, with frequency and position. 3. Query Processing When a user enters a search query: - Parses the query. - Looks up relevant terms in the index. - Ranks pages ba","default_branch":null,"files":null,"tree":[],"storefront":"/r/mihaimoga","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mihaimoga/WebSearchEngine/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}