{"repo":"appgoblin-dev/adscrawler","free":true,"listed":false,"github":"https://github.com/appgoblin-dev/adscrawler","clone":"git clone https://github.com/appgoblin-dev/adscrawler.git","description":"Crawler for Google, Apple Stores, app SDKs and App Ads.txt","language":"Python","stars":32,"topics":["android","apple","appstore","dashboard","itunes","scraper","app","google-play","store","appgoblin"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Crawl App Store Apps, App SDKs & App-Ads.txt Various tools used for collecting data for AppGoblin's free ASO app marketing tools and app SDK lists. These tools are used to build AppGoblin. This is a large monorepo combining other popular tools along with a database schema to store them in. Scrapers: - pull apps from app store and google play store top lists digitalmethodsinitiative/itunes-app-scraper & facundoolano/google-play-scraper - pull apps from some 3rd party stores - unzip/decompile Android APKs and iOS IPAs to look for 3rd party tracking/advertising tools, requires manual setup of iBotPeaches/apktool and majd/ipatool - App-ads.txt files are crawled based on the Interactive Advertising Bureau's Tech Lab specs. https://iabtechlab.com/ads-txt/ - Implementation of Waydroid emulator with MITM to capture HTTPs API traffic (requires manual steps for the headless implementation, please reach out if you need help) Setup and Installation Note: This project is not a one click setup but feel free to reach out for help. - Requires Object Storage / S3 - PostgreSQL: 17/18 - Setup your database. The files here use the database name 'madrone' - Add a password to your default db user if you dont have one yet ALTER USER postgres WITH PASSWORD 'xxx'; - Python environment: Python 3.13 - Setup python environment python3.12 -m venv .virtualenv & source .virtualenv/bin/activate - uv pip install -r pyproject.toml - cp example config.toml /config/adscrawler/config.toml and edit any needed val","default_branch":null,"files":null,"tree":[],"storefront":"/r/appgoblin-dev","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/appgoblin-dev/adscrawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}