{"repo":"proxidize/x-scraper","free":true,"listed":false,"github":"https://github.com/proxidize/x-scraper","clone":"git clone https://github.com/proxidize/x-scraper.git","description":"A Twitter/X scraper built with Playwright for browser automation and OpenAI GPT-4 for AI-powered tweet analysis. Features timeline scraping, historical search, keyword search, checkpoint/resume, proxy support, and comprehensive data extraction.","language":"Python","stars":33,"topics":["browser-automation","data-collection","gpt-4","openai","playwright","proxies","python","sentiment-analysis","social-media-scraping","twitter-scraper"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"X-Scraper Twitter/X scraper built with Playwright for browser automation and OpenAI for AI-powered tweet analysis. Features Core Scraping Capabilities - Timeline Scraping : Extract tweets from any user's timeline with full metadata - Historical Search : Scrape tweets from specific date ranges - Keyword Search : Search for tweets by keywords, hashtags, or phrases - Mixed Strategy : Combine timeline + historical search for comprehensive coverage Advanced Features - AI Analysis : Automatic sentiment analysis, topic extraction, and summaries using ChatGPT - Checkpoint/Resume : Resume interrupted scrapes from where they stopped - Progress Tracking : Real-time progress bars and detailed logging - Proxy Support : Built-in support for residential/mobile proxies - Session Persistence : Cookie-based authentication for reliable long-term scraping - Rate Limit Handling : Intelligent retry mechanisms with exponential backoff - Deduplication : Automatic removal of duplicate tweets Data Quality - Rich Metadata : Captures tweets, user info, engagement metrics, media, hashtags, URLs - Structured Output : Clean JSON format with proper typing and validation Prerequisites - Python 3.11+ - Chrome/Chromium browser (installed automatically by Playwright) - Twitter/X account credentials - Proxy service for IP rotation - (Optional) OpenAI API key for AI analysis Installation Standard Installation Docker Installation Configuration Edit config.ini with your settings: Required Fields - [TWITTER] : usern","default_branch":null,"files":null,"tree":[],"storefront":"/r/proxidize","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/proxidize/x-scraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}