{"repo":"MatthewWolff/TwitterScraper","free":true,"listed":false,"github":"https://github.com/MatthewWolff/TwitterScraper","clone":"git clone https://github.com/MatthewWolff/TwitterScraper.git","description":"Scrape a User's Twitter data! Bypass the 3,200 tweet API limit for a User!","language":"Python","stars":168,"topics":["twitter","twitter-api","twitter-scraper","scraper","tweepy","selenium","python3","twitter-bot","twitterscraper","data-science"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"TwitterScraper Description Twitter's API limits you to querying a user's most recent 3200 tweets. This is a pain in the ass. However, we can circumvent this limit using Selenium and doing some webscraping. We can query a user's entire time on twitter, finding the IDs for each of their tweets. From there, we can use the tweepy API to query the complete metadata associated with each tweet. You can adjust which metadata are collected by changing the variable METADATA LIST at the top of scrape.py . Personally, I was just collecting text to train a model, so I only cared about the full text field in addition to whether the tweet was a retweet. I've included a list of all available tweet attributes at the top of scrape.py so that you can adjust things as you wish. NOTE: This scraper will notice if a user has less than 3200 tweets. In this case, it will do a \"quickscrape\" to grab all available tweets at once (significantly faster). It will store them in the exact same manner as a manual scrape. Requirements (or rather, what I used) python3 Modules (via pip - see requirements.txt): selenium tweepy requests requests oauthlib beautifulsoup4 Chrome webdriver (you can use other drivers. Personally I use, brew install chromedriver because I work on a Mac) Twitter API developer credentials Example: I'll run the script two times on one of my advisors. By default, the scraper will start whenever the user created their twitter. I've chosen to look at a 1 year window, scraping at two week inte","default_branch":null,"files":null,"tree":[],"storefront":"/r/MatthewWolff","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/MatthewWolff/TwitterScraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}