{"repo":"oxylabs/ai-crawler-py","free":true,"listed":false,"github":"https://github.com/oxylabs/ai-crawler-py","clone":"git clone https://github.com/oxylabs/ai-crawler-py.git","description":"Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.","language":null,"stars":3198,"topics":["ai","ai-agents","ai-crawler","ai-studio","web-crawler","ai-web-crawler","crawl-agent","ai-scraping"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"AI-Crawler The AI-Crawler is an experimental data extraction app by Oxylabs AI Studio that uses advanced AI algorithms to crawl a given domain. It identifies relevant pages based on a natural language prompt and extracts structured JSON or Markdown output data. This low-code tool is designed to simplify complex data acquisition tasks, allowing developers and data scientists to focus on analysis rather than building and maintaining custom web scrapers. The AI web crawler offers advanced filtering, schema-based parsing, and seamless integration with various automation pipelines. Key features - Start a crawl from any given URL: Begin your data extraction from any valid web address using the AI Crawler as a starting point. - Natural language prompt: Define your data needs in plain English, and the crawl agent will interpret the prompt to find relevant content. - AI-assisted URL selection: The AI web crawler intelligently explores the site, identifying and prioritizing pages most aligned with your prompt. - Multiple output formats: Choose between structured JSON or Markdown output for seamless integration into automation or AI workflows. - Schema-based parsing: For JSON output, you can define a parsing schema in natural language to ensure the extracted data is structured to fit your application. How it works To get started with the AI Crawler, follow this four-step process: 1. Provide a starting URL of the website you want the web crawler to explore. 2. Describe the content you wa","default_branch":null,"files":null,"tree":[],"storefront":"/r/oxylabs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/oxylabs/ai-crawler-py/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}