{"repo":"Blacksuan19/scrapy-ai","free":true,"listed":false,"github":"https://github.com/Blacksuan19/scrapy-ai","clone":"git clone https://github.com/Blacksuan19/scrapy-ai.git","description":"Fully automated AI based web scraping.","language":"Python","stars":44,"topics":["openai-api","scrapy","webscraping","llm"],"license":"MIT","category":"ai-agents","readme_excerpt":"Scrapy-LLM LLM integration for Scrapy as a middleware. Extract any data from the web using your own predefined schema with your own preferred language model.     Features - Extract data from web page text using a language model. - Define a schema for the extracted data using pydantic models. - Validate the extracted data against the defined schema. - Seamlessly integrate with any API compatible with the OpenAI API specification. - Use any language model deployed on an API compatible with the OpenAI API specification. Installation Usage The guide below assumes that a Scrapy project has already been set up. If not, follow the official Scrapy tutorial to create a new project. Setup the middleware in the settings.py file and define the response model to use for extracting data from the web page text. then access extracted data from the response object. Creating a response model Response models are used to define the schema for the extracted data. The schema is used to validate the extracted data and ensure that it conforms to the desired structure. The response model should be a pydantic model with the desired fields and types. pydantic has support for many types including custom types like regex based types, emails, enums and more, for a full list of supported types check the pydantic documentation. In addition to the officially supported types there are other third-party libraries that add support for more types such as pydantic-extra-types which adds support for","default_branch":null,"files":null,"tree":[],"storefront":"/r/Blacksuan19","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Blacksuan19/scrapy-ai/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}