{"repo":"alexfazio/devdocs-to-llm","free":true,"listed":false,"github":"https://github.com/alexfazio/devdocs-to-llm","clone":"git clone https://github.com/alexfazio/devdocs-to-llm.git","description":"Turn any developer documentation into a GPT","language":"Jupyter Notebook","stars":104,"topics":["firecrawl","crawler","crawling","scraper","scraping"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Turn any developer documentation into a specialized GPT. Overview DevDocs to LLM is a tool that allows you to crawl developer documentation, extract content, and process it into a format suitable for use with large language models (LLMs) like ChatGPT. This enables you to create specialized assistants tailored to specific documentation sets. Features - Web crawling with customizable options - Content extraction in Markdown format - Rate limiting to respect server constraints - Retry mechanism for failed scrapes - Export options: - Rentry.co for quick sharing - Google Docs for larger documents Usage 1. Set up the Firecrawl environment 2. Crawl a website and generate a sitemap 3. Extract content from crawled pages 4. Export the processed content Requirements - Firecrawl API key - Google Docs API credentials (optional, for Google Docs export) Installation This project is designed to run in a Jupyter notebook environment, particularly Google Colab. No local installation is required. Configuration Before running the notebook, you'll need to set a few parameters: - sub url : The URL of the documentation you want to crawl - limit : Maximum number of pages to crawl - scrape option : Choose to scrape all pages or a specific number - num pages : Number of pages to scrape if not scraping all - pages per minute : Rate limiting parameter - wait time between chunks : Delay between scraping chunks - retry attempts : Number of retries for failed scrapes Contributing Contributions are welcome!","default_branch":null,"files":null,"tree":[],"storefront":"/r/alexfazio","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/alexfazio/devdocs-to-llm/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}