{"repo":"apify/crawlee-python","free":true,"listed":false,"github":"https://github.com/apify/crawlee-python","clone":"git clone https://github.com/apify/crawlee-python.git","description":"Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.","language":"Python","stars":9430,"topics":["apify","automation","beautifulsoup","crawler","crawling","headless","headless-chrome","parsel","pip","playwright","python","scraper","scraping","selenium","web-crawler","web-crawling","web-scraping"],"license":"Apache-2.0","category":"data_pipeline","readme_excerpt":"
\n \n \n \n \n \n \n \n A web scraping and browser automation library\n
\n\n
\n \n
\n\n
\n \n \n \n \n \n \n \n
\n\nCrawlee covers your crawling and scraping end-to-end and **helps you build reliable scrapers. Fast.**\n\nYour crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev/python/)** 👈\n\nWe also have a TypeScript implementation of the Crawlee, which you can explore and utilize for your projects. Visit our GitHub repository for more information [Crawlee for JS/TS on GitHub](https://github.com/apify/crawlee).\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/python/docs/introduction) in Crawlee documentation for more information.\n\nCrawlee is available as [`crawlee`](https://pypi.org/project/crawlee/) package on PyPI. This package includes the core functionality, while additional features are available as optional extras to keep dependencies and package size minimal.\n\nTo install Crawlee with all features, run the following command:\n\n```sh\npython -m pip install 'crawlee[all]'\n```\n\nThen, install the [Playwright](https://playwright.dev/) dependencies:\n\n```sh\nplaywright install\n```\n\nVerify that Crawlee is successfully installed:\n\n```sh\npython -c 'import crawlee; print(crawlee.__version__)'\n```\n\nFor detailed installation instructions see the [Setting up](https://crawlee.dev/python/docs/introduction/setting-up) documentation page.\n\n### With Crawlee CLI\n\nThe quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have [uv](https://pypi.org/project/uv/) installed:\n\n```sh\nuv --help\n```\n\nIf [uv](https://pypi.org/project/uv/) is not installed, follow the official [installation guide](https://docs.astral.sh/uv/getting-started/installation/).\n\nThen, run the CLI and choose from the available templates:\n\n```sh\nuvx 'crawlee[cli]' create my-crawler\n```\n\nIf you already have `crawlee` installed, you can spin it up by running:\n\n```sh\ncrawlee create my-crawler\n```\n\n## Examples\n\nHere are some practical examples to help you get started with different types of crawlers in Crawlee. Each example demonstrates how to set up and run a crawler for specific use cases, whether you need to handle simple HTML pages or interact with JavaScript-heavy sites. A crawler run will create a `storage/` directory in your current working directory.\n\n### BeautifulSoupCrawler\n\nThe [`BeautifulSoupCrawler`](https://crawlee.dev/python/api/class/BeautifulSoupCrawler) downloads web pages using an HTTP library and provides HTML-parsed content to the user. By default it uses [`ImpitHttpClient`](https://crawlee.dev/python/api/class/ImpitHttpClient) for HTTP communication and [BeautifulSoup](https://pypi.org/project/beautifulsoup4/) for parsing HTML. It is ideal for projects that require efficient extraction of data from HTML content. This crawler has very good performance since it does not use a browser. However, if you need to execute client-side JavaScript, to get your content, this is not going to be enough and you will need to use [`PlaywrightCrawler`](https://crawlee.dev/python/api/class/PlaywrightCrawler). Also if you want to use this crawler, make sure you install `crawlee` with `beautifulsoup` extra.\n\n```python\nimport asyncio\n\nfrom crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext\n\n\nasync def main() -> None:\n crawler = BeautifulSoupCrawler(\n # Limit the crawl to max requests. Remove or increase it for crawling all links.\n max_requests_per_crawl=10,\n )\n\n # Define the default request handler, which will be called for every request.\n @crawler.router.default_handler\n async def request_handler(context: BeautifulSoupCrawlingContext) -> None:\n context.log.info(f'Processing {context.request.url} ...')\n\n # Extract data from the page.\n data = {\n 'url': context.request.url,\n 'title': context.soup.title.string if context.soup.title else None,\n }\n\n # Push the extracted data to the default dataset.\n await context.push_data(data)\n\n # Enqueue all links found on the page.\n await context.enqueue_links()\n\n # Run the crawler with the initial list of URLs.\n await crawler.run(['https://crawlee.dev'])\n\n\nif __name__ == '__main__':\n asyncio.run(main())\n```\n\n### PlaywrightCrawler\n\nThe [`PlaywrightCrawler`](https://crawlee.dev/python/api/class/PlaywrightCrawler) uses a headless browser to download web pages and provides an API for data extraction. It is built on [Playwright](https://playwright.dev/), an automation library designed for managing headless browsers. It excels at retrieving web pages that rely on client-side JavaScript for content generation, or tasks requiring interaction with JavaScript-driven content. For scenarios where JavaScript execution is unnecessary or higher performance is required, consider using the [`BeautifulSoupCrawler`](https://crawlee.dev/python/api/class/BeautifulSoupCrawler). Also if you want to use this crawler, make sure you install `crawlee` with `playwright` extra.\n\n```python\nimport asyncio\n\nfrom crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext\n\n\nasync def main() -> None:\n crawler = PlaywrightCrawler(\n # Limit the crawl to max requests. Remove or increase it for crawling all links.\n max_requests_per_crawl=10,\n )\n\n # Define the default request handler, which will be called for every request.\n @crawler.router.default_handler\n async def request_handler(context: PlaywrightCrawlingContext) -> None:\n context.log.info(f'Processing {context.request.url} ...')\n\n # Extract data from the page.\n data = {\n 'url': context.request.url,\n 'title': await context.page.title(),\n }\n\n # Push the extracted data to the default dataset.\n await context.push_data(data)\n\n # Enqueue all links found on the page.\n await context.enqueue_links()\n\n # Run the crawler with the initial list of requests.\n await crawler.run(['https://crawlee.dev'])\n\n\nif __name__ == '__main__':\n asyncio.run(main())\n```\n\n### More examples\n\nExplore our [Examples](https://crawlee.dev/python/docs/examples) page in the Crawlee documentation for a wide range of additional use cases and demonstrations.\n\n## Features\n\nWhy Crawlee is the preferred choice for web scraping and crawling?\n\n### Why use Crawlee instead of just a random HTTP library with an HTML parser?\n\n- Unified interface for **HTTP & headless browser** crawling.\n- Automatic **parallel crawling** based on available system resources.\n- Written in Python with **type hints** - enhances DX (IDE autocompletion) and reduces bugs (static type checking).\n- Automatic **retries** on errors or when you’re getting blocked.\n- Integrated **proxy rotation** and session management.\n- Configurable **request routing** - direct URLs to the appropriate handlers.\n- Persistent **queue for URLs** to crawl.\n- Pluggable **storage** of both tabular data and files.\n- Robust **error handling**.\n\n### Why to use Crawlee rather than Scrapy?\n\n- **Asyncio-based** – Leveraging the standard [Asyncio](https://docs.python.org/3/library/asyncio.html) library, Crawlee delivers better performance and seamless compatibility with other modern asynchronous libraries.\n- **Type hints** – Newer project built with modern Python, and complete type hint coverage for a better developer experience.\n- **Simple integration** – Crawlee crawlers are regular Python scripts, requiring no additional launcher executor. This flexibility allows to integrate a crawler directly into other applications.\n- **State persistence** – Supports state persistence ","default_branch":"master","files":1304,"tree":[".editorconfig",".github/CODEOWNERS",".github/pull_request_template.md",".github/workflows/_checks.yaml",".github/workflows/manual_release_beta.yaml",".github/workflows/manual_release_docs.yaml",".github/workflows/manual_release_stable.yaml",".github/workflows/manual_version_docs.yaml",".github/workflows/on_issue.yaml",".github/workflows/on_master.yaml",".github/workflows/on_pull_request.yaml",".github/workflows/on_schedule_tests.yaml",".github/workflows/update_playwright_version.yaml",".gitignore",".markdownlint.yaml",".pre-commit-config.yaml",".rules.md","AGENTS.md","CHANGELOG.md","CLAUDE.md","CONTRIBUTING.md","GEMINI.md","LICENSE","README.md","codecov.yaml","docs/deployment/apify_platform.mdx","docs/deployment/aws_lambda.mdx","docs/deployment/code_examples/apify/crawler_as_actor_example.py","docs/deployment/code_examples/apify/get_public_url.py","docs/deployment/code_examples/apify/log_with_config_example.py","docs/deployment/code_examples/apify/proxy_advanced_example.py","docs/deployment/code_examples/apify/proxy_example.py","docs/deployment/code_examples/aws/beautifulsoup_crawler_lambda.py","docs/deployment/code_examples/aws/playwright_crawler_lambda.py","docs/deployment/code_examples/aws/playwright_dockerfile","docs/deployment/code_examples/google/cloud_run_example.py","docs/deployment/code_examples/google/google_example.py","docs/deployment/google_cloud.mdx","docs/deployment/google_cloud_run.mdx","docs/examples/add_data_to_dataset.mdx","docs/examples/beautifulsoup_crawler.mdx","docs/examples/capture_screenshot_using_playwright.mdx","docs/examples/capturing_page_snapshots_with_error_snapshotter.mdx","docs/examples/code_examples/adaptive_playwright_crawler.py","docs/examples/code_examples/add_data_to_dataset_bs.py","docs/examples/code_examples/add_data_to_dataset_dataset.py","docs/examples/code_examples/add_data_to_dataset_pw.py","docs/examples/code_examples/beautifulsoup_crawler.py","docs/examples/code_examples/beautifulsoup_crawler_keep_alive.py","docs/examples/code_examples/beautifulsoup_crawler_stop.py","docs/examples/code_examples/capture_screenshot_using_playwright.py","docs/examples/code_examples/configure_json_logging.py","docs/examples/code_examples/crawl_all_links_on_website_bs.py","docs/examples/code_examples/crawl_all_links_on_website_pw.py","docs/examples/code_examples/crawl_multiple_urls_bs.py","docs/examples/code_examples/crawl_multiple_urls_pw.py","docs/examples/code_examples/crawl_specific_links_on_website_bs.py","docs/examples/code_examples/crawl_specific_links_on_website_pw.py","docs/examples/code_examples/crawl_website_with_relative_links_all_links.py","docs/examples/code_examples/crawl_website_with_relative_links_same_domain.py","docs/examples/code_examples/crawl_website_with_relative_links_same_hostname.py","docs/examples/code_examples/crawl_website_with_relative_links_same_origin.py","docs/examples/code_examples/export_entire_dataset_to_file_csv.py","docs/examples/code_examples/export_entire_dataset_to_file_json.py","docs/examples/code_examples/extract_and_add_specific_links_on_website_bs.py","docs/examples/code_examples/extract_and_add_specific_links_on_website_pw.py","docs/examples/code_examples/file_download.py","docs/examples/code_examples/file_download_stream.py","docs/examples/code_examples/fill_and_submit_web_form_crawler.py","docs/examples/code_examples/fill_and_submit_web_form_request.py","docs/examples/code_examples/parsel_crawler.py","docs/examples/code_examples/parsel_crawler_with_error_snapshotter.py","docs/examples/code_examples/playwright_block_requests.py","docs/examples/code_examples/playwright_crawler.py","docs/examples/code_examples/playwright_crawler_with_camoufox.py","docs/examples/code_examples/playwright_crawler_with_error_snapshotter.py","docs/examples/code_examples/playwright_crawler_with_fingerprint_generator.py","docs/examples/code_examples/respect_robots_on_skipped_request.py","docs/examples/code_examples/respect_robots_txt_file.py","docs/examples/code_examples/resuming_paused_crawl.py","docs/examples/code_examples/run_parallel_crawlers.py","docs/examples/code_examples/using_browser_profiles_chrome.py","docs/examples/code_examples/using_browser_profiles_firefox.py","docs/examples/code_examples/using_sitemap_request_loader.py","docs/examples/crawl_all_links_on_website.mdx","docs/examples/crawl_multiple_urls.mdx","docs/examples/crawl_specific_links_on_website.mdx","docs/examples/crawl_website_with_relative_links.mdx","docs/examples/crawler_keep_alive.mdx","docs/examples/crawler_stop.mdx","docs/examples/export_entire_dataset_to_file.mdx","docs/examples/file_download.mdx","docs/examples/fill_and_submit_web_form.mdx","docs/examples/json_logging.mdx","docs/examples/parsel_crawler.mdx","docs/examples/playwright_crawler.mdx","docs/examples/playwright_crawler_adaptive.mdx","docs/examples/playwright_crawler_with_block_requests.mdx","docs/examples/playwright_crawler_with_camoufox.mdx","docs/examples/playwright_crawler_with_fingerprint_generator.mdx","docs/examples/respect_robots_txt_file.mdx","docs/examples/resuming_paused_crawl.mdx","docs/examples/run_parallel_crawlers.mdx","docs/examples/using_browser_profile.mdx","docs/examples/using_sitemap_request_loader.mdx","docs/guides/architecture_overview.mdx","docs/guides/avoid_blocking.mdx","docs/guides/code_examples/avoid_blocking/default_fingerprint_generator_with_args.py","docs/guides/code_examples/avoid_blocking/playwright_with_cloakbrowser.py","docs/guides/code_examples/avoid_blocking/playwright_with_fingerprint_generator.py","docs/guides/code_examples/cookie_management/initial_cookies.py","docs/guides/code_examples/cookie_management/persist_cookies.py","docs/guides/code_examples/cookie_management/playwright_cookies.py","docs/guides/code_examples/cookie_management/read_write_cookies.py","docs/guides/code_examples/cookie_management/retry_pinned_session.py","docs/guides/code_examples/cookie_management/retry_restore_cookies.py","docs/guides/code_examples/cookie_management/retry_single_session.py","docs/guides/code_examples/creating_web_archive/manual_archiving_parsel_crawler.py","docs/guides/code_examples/creating_web_archive/manual_archiving_playwright_crawler.py","docs/guides/code_examples/creating_web_archive/simple_pw_through_proxy_pywb_server.py","docs/guides/code_examples/error_handling/change_handle_error_status.py","docs/guides/code_examples/error_handling/disable_retry.py","docs/guides/code_examples/error_handling/handle_proxy_error.py","docs/guides/code_examples/http_clients/parsel_curl_impersonate_example.py","docs/guides/code_examples/http_clients/parsel_httpx_example.py","docs/guides/code_examples/http_clients/parsel_impit_example.py","docs/guides/code_examples/http_crawlers/__init__.py","docs/guides/code_examples/http_crawlers/beautifulsoup_example.py","docs/guides/code_examples/http_crawlers/custom_crawler_example.py","docs/guides/code_examples/http_crawlers/http_example.py","docs/guides/code_examples/http_crawlers/lexbor_parser.py","docs/guides/code_examples/http_crawlers/lxml_parser.py","docs/guides/code_examples/http_crawlers/lxml_saxonche_parser.py","docs/guides/code_examples/http_crawlers/parsel_example.py","docs/guides/code_examples/http_crawlers/pyquery_parser.py","docs/guides/code_examples/http_crawlers/scrapling_parser.py","docs/guides/code_examples/http_crawlers/selectolax_adaptive_run.py","docs/guides/code_examples/http_crawlers/selectolax_context.py","docs/guides/code_examples/http_crawlers/selectolax_crawler.py","docs/guides/code_examples/http_crawlers/selectolax_crawler_run.py","docs/guides/code_examples/http_crawlers/selectolax_parser.py","docs/guides/code_examples/http_headers/set_headers.py","docs/guides/code_examples/login_crawler/http_login.py","docs/guides/code_examples/login_crawler/playwright_login.py","docs/guides/code_examples/playwright_crawler/browser_configuration_example.py","docs/guides/code_examples/playwright_crawler/browser_pool_launch_hooks_example.py","docs/guides/code_examples/playwright_crawler/browser_pool_page_hooks_example.py","docs/guides/code_examples/playwright_crawler/multiple_launch_example.py","docs/guides/code_examples/playwright_crawler/navigation_hooks_example.py","docs/guides/code_examples/playwright_crawler/plugin_browser_configuration_example.py","docs/guides/code_examples/playwright_crawler_adaptive/handler.py","docs/guides/code_examples/playwright_crawler_adaptive/init_beautifulsoup.py","docs/guides/code_examples/playwright_crawler_adaptive/init_parsel.py","docs/guides/code_examples/playwright_crawler_adaptive/init_prediction.py","docs/guides/code_examples/playwright_crawler_adaptive/pre_nav_hooks.py","docs/guides/code_examples/proxy_management/inspecting_bs_example.py","docs/guides/code_examples/proxy_management/inspecting_pw_example.py","docs/guides/code_examples/proxy_management/integration_bs_example.py","docs/guides/code_examples/proxy_management/integration_pw_example.py","docs/guides/code_examples/proxy_management/quick_start_example.py","docs/guides/code_examples/proxy_management/session_bs_example.py","docs/guides/code_examples/proxy_management/session_pw_example.py","docs/guides/code_examples/proxy_management/tiers_bs_example.py","docs/guides/code_examples/proxy_management/tiers_pw_example.py","docs/guides/code_examples/pydantic_ai_crawler/additional_instructions_example.py","docs/guides/code_examples/pydantic_ai_crawler/basic_example.py","docs/guides/code_examples/pydantic_ai_crawler/custom_distiller_example.py","docs/guides/code_examples/pydantic_ai_crawler/debugging_example.py","docs/guides/code_examples/pydantic_ai_crawler/selector_extractor_example.py","docs/guides/code_examples/pydantic_ai_crawler/usage_limit_example.py","docs/guides/code_examples/request_loaders/rl_basic_example.py","docs/guides/code_examples/request_loaders/rl_basic_example_with_persist.py","docs/guides/code_examples/request_loaders/rl_tandem_example.py","docs/guides/code_examples/request_loaders/rl_tandem_example_explicit.py","docs/guides/code_examples/request_loaders/sitemap_basic_example.py","docs/guides/code_examples/request_loaders/sitemap_example_with_persist.py","docs/guides/code_examples/request_loaders/sitemap_tandem_example.py","docs/guides/code_examples/request_loaders/sitemap_tandem_example_explicit.py","docs/guides/code_examples/request_router/adaptive_crawler_handlers.py","docs/guides/code_examples/request_router/basic_request_handlers.py","docs/guides/code_examples/request_router/custom_router_default_only.py","docs/guides/code_examples/request_router/error_handler.py","docs/guides/code_examples/request_router/failed_request_handler.py","docs/guides/code_examples/request_router/http_pre_navigation.py","docs/guides/code_examples/request_router/playwright_pre_navigation.py","docs/guides/code_examples/request_router/router_middleware.py","docs/guides/code_examples/request_router/simple_default_handler.py","docs/guides/code_examples/request_throttling/throttling_example.py","docs/guides/code_examples/running_in_web_server/__init__.py","docs/guides/code_examples/running_in_web_server/crawler.py","docs/guides/code_examples/running_in_web_server/server.py","docs/guides/code_examples/scaling_crawlers/max_tasks_per_minute_example.py","docs/guides/code_examples/scaling_crawlers/min_and_max_concurrency_example.py","docs/guides/code_examples/scrapy_migration/crawlee_concurrency.py","docs/guides/code_examples/scrapy_migration/crawlee_crawlspider.py","docs/guides/code_examples/scrapy_migration/crawlee_error_handling.py","docs/guides/code_examples/scrapy_migration/crawlee_export.py","docs/guides/code_examples/scrapy_migration/crawlee_labels.py","docs/guides/code_examples/scrapy_migration/crawlee_playwright.py","docs/guides/code_examples/scrapy_migration/crawlee_post.py","docs/guides/code_examples/scrapy_migration/crawlee_proxy.py","docs/guides/code_examples/scrapy_migration/crawlee_quotes.py","docs/guides/code_examples/scrapy_migration/crawlee_throttling.py","docs/guides/code_examples/scrapy_migration/crawlee_user_data.py","docs/guides/code_examples/scrapy_migration/scrapy_authors.py","docs/guides/code_examples/scrapy_migration/scrapy_cb_kwargs.py","docs/guides/code_examples/scrapy_migration/scrapy_concurrency.py","docs/guides/code_examples/scrapy_migration/scrapy_crawlspider.py","docs/guides/code_examples/scrapy_migration/scrapy_errback.py","docs/guides/code_examples/scrapy_migration/scrapy_export.py","docs/guides/code_examples/scrapy_migration/scrapy_formrequest.py","docs/guides/code_examples/scrapy_migration/scrapy_playwright.py","docs/guides/code_examples/scrapy_migration/scrapy_playwright_settings.py","docs/guides/code_examples/scrapy_migration/scrapy_proxy.py","docs/guides/code_examples/scrapy_migration/scrapy_quotes.py","docs/guides/code_examples/scrapy_migration/scrapy_throttling.py","docs/guides/code_examples/service_locator/service_conflicts.py","docs/guides/code_examples/service_locator/service_crawler_configuration.py","docs/guides/code_examples/service_locator/service_crawler_event_manager.py","docs/guides/code_examples/service_locator/service_crawler_storage_client.py","docs/guides/code_examples/service_locator/service_locator_configuration.py","docs/guides/code_examples/service_locator/service_locator_event_manager.py","docs/guides/code_examples/service_locator/service_locator_storage_client.py","docs/guides/code_examples/service_locator/service_storage_configuration.py","docs/guides/code_examples/service_locator/service_storage_storage_client.py","docs/guides/code_examples/session_management/multi_sessions_http.py","docs/guides/code_examples/session_management/one_session_http.py","docs/guides/code_examples/session_management/sm_basic.py","docs/guides/code_examples/session_management/sm_beautifulsoup.py","docs/guides/code_examples/session_management/sm_http.py","docs/guides/code_examples/session_management/sm_parsel.py","docs/guides/code_examples/session_management/sm_playwright.py","docs/guides/code_examples/session_management/sm_standalone.py","docs/guides/code_examples/stagehand_crawler/basic_example.py","docs/guides/code_examples/stagehand_crawler/browserbase_example.py","docs/guides/code_examples/storage_clients/custom_storage_client_example.py","docs/guides/code_examples/storage_clients/file_system_storage_client_basic_example.py","docs/guides/code_examples/storage_clients/file_system_storage_client_configuration_example.py","docs/guides/code_examples/storage_clients/memory_storage_client_basic_example.py","docs/guides/code_examples/storage_clients/redis_storage_client_basic_example.py","docs/guides/code_examples/storage_clients/redis_storage_client_configuration_example.py","docs/guides/code_examples/storage_clients/registering_storage_clients_example.py","docs/guides/code_examples/storage_clients/sql_storage_client_basic_example.py","docs/guides/code_examples/storage_clients/sql_storage_client_configuration_example.py","docs/guides/code_examples/storages/cleaning_do_not_purge_example.py","docs/guides/code_examples/storages/cleaning_purge_explicitly_example.py","docs/guides/code_examples/storages/dataset_basic_example.py","docs/guides/code_examples/storages/dataset_with_crawler_example.py","docs/guides/code_examples/storages/dataset_with_crawler_explicit_example.py","docs/guides/code_examples/storages/helper_add_requests_example.py","docs/guides/code_examples/storages/helper_enqueue_links_example.py","docs/guides/code_examples/storages/kvs_basic_example.py","docs/guides/code_examples/storages/kvs_with_crawler_example.py","docs/guides/code_examples/storages/kvs_with_crawler_explicit_example.py","docs/guides/code_examples/storages/opening.py","docs/guides/code_examples/storages/rq_basic_example.py","docs/guides/code_examples/storages/rq_with_crawler_example.py","docs/guides/code_examples/storages/rq_with_crawler_explicit_example.py","docs/guides/code_examples/trace_and_monitor_crawlers/instrument_crawler.py","docs/guides/cookie_management.mdx","docs/guides/crawler_login.mdx","docs/guides/creating_web_archive.mdx","docs/guides/error_handling.mdx","docs/guides/http_clients.mdx","docs/guides/http_crawlers.mdx","docs/guides/http_headers.mdx","docs/guides/playwright_crawler.mdx","docs/guides/playwright_crawler_adaptive.mdx","docs/guides/proxy_management.mdx","docs/guides/pydantic_ai_crawler.mdx","docs/guides/request_loaders.mdx","docs/guides/request_router.mdx","docs/guides/request_throttling.mdx","docs/guides/running_in_web_server.mdx","docs/guides/scaling_crawlers.mdx","docs/guides/scrapy_migration.mdx","docs/guides/secure_scraping.mdx","docs/guides/service_locator.mdx","docs/guides/session_management.mdx","docs/guides/stagehand_crawler.mdx","docs/guides/storage_clients.mdx","docs/guides/storages.mdx","docs/guides/trace_and_monitor_crawlers.mdx","docs/introduction/01_setting_up.mdx","docs/introduction/02_first_crawler.mdx","docs/introduction/03_adding_more_urls.mdx","docs/introduction/04_real_world_project.mdx","docs/introduction/05_crawling.mdx","docs/introduction/06_scraping.mdx","docs/introduction/07_saving_data.mdx","docs/introduction/08_refactoring.mdx","docs/introduction/09_running_in_cloud.mdx","docs/introduction/code_examples/02_bs.py","docs/introduction/code_examples/02_bs_better.py","docs/introduction/code_examples/02_request_queue.py","docs/introduction/code_examples/03_enqueue_strategy.py","docs/introduction/code_examples/03_finding_new_links.py","docs/introduction/code_examples/03_globs.py","docs/introduction/code_examples/03_original_code.py","docs/introduction/code_examples/03_transform_request.py","docs/introduction/code_examples/04_sanity_check.py","docs/introduction/code_examples/05_crawling_detail.py","docs/introduction/code_examples/05_crawling_listing.py","docs/introduction/code_examples/06_scraping.py","docs/introduction/code_examples/07_final_code.py","docs/introduction/code_examples/07_first_code.py","docs/introduction/code_examples/08_main.py","docs/introduction/code_examples/08_routes.py","docs/introduction/code_examples/09_apify_sdk.py","docs/introduction/code_examples/__init__.py","docs/introduction/code_examples/routes.py","docs/introduction/index.mdx","docs/pyproject.toml","docs/quick-start/code_examples/beautifulsoup_crawler_example.py","docs/quick-start/code_examples/parsel_crawler_example.py","docs/quick-start/code_examples/playwright_crawler_example.py","docs/quick-start/code_examples/playwright_crawler_headful_example.py","docs/quick-start/index.mdx","docs/upgrading/upgrading_to_v0x.md","docs/upgrading/upgrading_to_v1.md","pyproject.toml","renovate.json","scripts/__init__.py","scripts/update_playwright_version.py","src/crawlee/__init__.py","src/crawlee/_autoscaling/__init__.py","src/crawlee/_autoscaling/_types.py","src/crawlee/_autoscaling/autoscaled_pool.py","src/crawlee/_autoscaling/py.typed","src/crawlee/_autoscaling/snapshotter.py","src/crawlee/_autoscaling/system_status.py","src/crawlee/_cli.py","src/crawlee/_consts.py","src/crawlee/_log_config.py","src/crawlee/_request.py","src/crawlee/_service_locator.py","src/crawlee/_types.py","src/crawlee/_utils/__init__.py","src/crawlee/_utils/blocked.py","src/crawlee/_utils/byte_size.py","src/crawlee/_utils/console.py","src/crawlee/_utils/context.py","src/crawlee/_utils/crypto.py","src/crawlee/_utils/docs.py","src/crawlee/_utils/file.py","src/crawlee/_utils/globs.py","src/crawlee/_utils/html_to_text.py","src/crawlee/_utils/http.py","src/crawlee/_utils/log.py","src/crawlee/_utils/models.py","src/crawlee/_utils/raise_if_too_many_kwargs.py","src/crawlee/_utils/recoverable_state.py","src/crawlee/_utils/recurring_task.py","src/crawlee/_utils/requests.py","src/crawlee/_utils/retry.py","src/crawlee/_utils/robots.py","src/crawlee/_utils/sitemap.py","src/crawlee/_utils/system.py","src/crawlee/_utils/time.py","src/crawlee/_utils/try_import.py","src/crawlee/_utils/urls.py","src/crawlee/_utils/wait.py","src/crawlee/_utils/web.py","src/crawlee/browsers/__init__.py","src/crawlee/browsers/_browser_controller.py","src/crawlee/browsers/_browser_plugin.py","src/crawlee/browsers/_browser_pool.py","src/crawlee/browsers/_playwright_browser.py","src/crawlee/browsers/_playwright_browser_controller.py","src/crawlee/browsers/_playwright_browser_plugin.py","src/crawlee/browsers/_stagehand_browser_controller.py","src/crawlee/browsers/_stagehand_browser_plugin.py","src/crawlee/browsers/_stagehand_types.py","src/crawlee/browsers/_types.py","src/crawlee/browsers/py.typed","src/crawlee/configuration.py","src/crawlee/crawlers/__init__.py","src/crawlee/crawlers/_abstract_http/__init__.py","src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py","src/crawlee/crawlers/_abstract_http/_abstract_http_parser.py","src/crawlee/crawlers/_abstract_http/_http_crawling_context.py","src/crawlee/crawlers/_abstract_http/py.typed","src/crawlee/crawlers/_adaptive_playwright/__init__.py","src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py","src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler_statistics.py","src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawling_context.py","src/crawlee/crawlers/_adaptive_playwright/_rendering_type_predictor.py","src/crawlee/crawlers/_adaptive_playwright/_result_comparator.py","src/crawlee/crawlers/_adaptive_playwright/_utils.py","src/crawlee/crawlers/_basic/__init__.py","src/crawlee/crawlers/_basic/_basic_crawler.py","src/crawlee/crawlers/_basic/_basic_crawling_context.py","src/crawlee/crawlers/_basic/_context_pipeline.py","src/crawlee/crawlers/_basic/_logging_utils.py","src/crawlee/crawlers/_basic/py.typed","src/crawlee/crawlers/_beautifulsoup/__init__.py","src/crawlee/crawlers/_beautifulsoup/_beautifulsoup_crawler.py","src/crawlee/crawlers/_beautifulsoup/_beautifulsoup_crawling_context.py","src/crawlee/crawlers/_beautifulsoup/_beautifulsoup_parser.py","src/crawlee/crawlers/_beautifulsoup/_utils.py"],"storefront":"/r/apify","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/apify/crawlee-python/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}
![\"Crawlee\"](\"https://raw.githubusercontent.com/apify/crawlee-python/master/website/static/img/crawlee-dark.svg?sanitize=true\")
\n ![\"PyPI](\"https://badge.fury.io/py/crawlee.svg\"){kind=icon}
\n ![\"PyPI](\"https://img.shields.io/pypi/dm/crawlee\")
![\"PyPI](\"https://img.shields.io/badge/python-3.10%2B-blue\"){kind=icon}
![\"Build](\"https://github.com/apify/crawlee-python/actions/workflows/on_master.yaml/badge.svg?branch=master\"){kind=icon}
\n ![\"Codecov](\"https://codecov.io/gh/apify/crawlee-python/graph/badge.svg?token=cCju61iPQG\"){kind=icon}
\n ![\"License\"](\"https://img.shields.io/pypi/l/crawlee\")
![\"Chat](\"https://img.shields.io/discord/801163717915574323?label=discord\")