{"repo":"lorae/roundup","free":true,"listed":false,"github":"https://github.com/lorae/roundup","clone":"git clone https://github.com/lorae/roundup.git","description":"Web scraper which aggregates pre-print academic economics papers from 20+ sources; presents titles, abstracts, authors and hyperlinks on an online dashboard. Auto-updates daily.","language":"Python","stars":10,"topics":["economics","streamlit","macroeconomics","microeconomics","api-scraping","html-scraping","selenium","streamlit-dashboard","streamlit-webapp","web-scraping"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"The website is active! View it here: https://roundup.streamlit.app/ About The purpose of this project is regularly track and present the most economics research on an interactive, sortable feed. We do so by web scraping research daily from academic organizations' websites. The data we collect are titles, authors, summaries, and links of working papers (also known as pre-print papers ), which present academic research that has not yet been vetted by the peer review process. The dashboard may be of use for those interested in active areas of economics research, such as economists, policy-oriented researchers, and students. Remotely run via GitHub Actions once daily, this project scrapes data from working paper publishers at 6:40 AM EST, compares newly collected data to a historic record of working papers, and presents the results on an interactive website. As of June 2024, the project incorporates data from 21 different sources. How it works The primary purpose of this repository is to maintain and improve the project dashboard. A detailed summary of the web scraping, GitHub Actions, and Streamlit components of this project follow. Web scraping The web scrapers in this project gather eight pieces of information on each economics working paper: - Title - Author(s) - Abstract : A one-paragraph long summary of the paper. - Date published : As reported by the website of origin. If the paper was posted and re-posted, the most recent date of publication is used. - estPubDate : Our be","default_branch":null,"files":null,"tree":[],"storefront":"/r/lorae","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/lorae/roundup/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}