{"repo":"pc8544/Website-Crawler","free":true,"listed":false,"github":"https://github.com/pc8544/Website-Crawler","clone":"git clone https://github.com/pc8544/Website-Crawler.git","description":"Extract data from websites in LLM ready JSON or CSV format. Crawl or Scrape entire website with Website Crawler","language":"Java","stars":75,"topics":["crawler","data","json","scraper","scraping","llm","rag"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Website Crawler 🕷️ Website Crawler API The Website Crawler API allows developers to programmatically crawl websites and access structured metadata via six simple endpoints. This API gives you clean JSON responses and real-time crawl updates. The JSON/structured response generated by the /crawl/cwdata endpoint can be used for a variety of purpose. For example, as the data is JSON/LLM ready format, you can use it to train an AI model, use it for creating chatbots, audit websites, etc. --- 🔐 Authentication To use the API, you'll need an API Key . How to get one: 1. Visit websitecrawler.org 2. Create an account or log in 3. Go to the Settings page to generate your API key --- 🌐 Base URL https://www.websitecrawler.org/api --- 📡 Endpoints 1. POST /crawl/authenticate Obtain an access token through the API. This token must be included in all subsequent requests. - Key required in the JSON payload : - key (string): Your API Key - Sample Request to get the token : - - Sample Response : --- 2. POST /crawl/start Initiate a new crawl for a given domain. - Keys required in the JSON payload : - url (string, required): Target website (e.g. example.com ) i.e. a non redirecting main URL of the website. - limit (integer, required): Max pages to crawl (free tier is resticted to 100) - Sample Request to initiate crawling : - - Sample Response 1 : - - Sample Response 2 : --- 3. POST /crawl/cwdata Retrieve the structured crawl output once crawling has completed. - Required key in JSON payload :","default_branch":null,"files":null,"tree":[],"storefront":"/r/pc8544","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/pc8544/Website-Crawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}