{"repo":"Robokishan/Crunchy","free":true,"listed":false,"github":"https://github.com/Robokishan/Crunchy","clone":"git clone https://github.com/Robokishan/Crunchy.git","description":"Crunchy a Crunchbase crawler. Used MongoDB For storing data and used kafka for data streaming. rabbitmq for queuing purpose","language":"Python","stars":10,"topics":["kafka","playwright","rabbitmq","scrapy"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Crunchy This is Crunchy my experiment over some fancy technologies for learning purpose. Crunchy is scrapper for crunchbase.com. Background story I usually surf a lot on crunchbase for looking into which companies are doing what and follow progress. But it wasn't very convenient experience because of Bloatware ui. Web isn't that good any more.web should be Simple. Just like geo hotz's blog. so I got an idea to create clone frontend for crunchbase. but that comes with chaos. I wrap up my slives. and created a simple scrapper for crunchbase pushed data over kafka and made queue out of rabbitmq. and here it is crunchy a simple scrapper with rabbitmq Pending Work tor proxy servers I wasn't able to add proxy. since crunchbase uses cloudflare for security. Tried Couple of free proxy list and tor but cloudflare blocks all of these so i am left with isp's dhcp. lol i disconnect and reconnect to get new ip and using multiple isps this way it works and i get lots of new company list on my preference. If someone is interested please give good suggestion to use tor with crunchbase. since tor browser works without a problem with crunchbase. i say it is possible. but needs some digging. Structure Component Tech :-------------------- :-----------------------: Frontend NextJS Design System Tailwind ( All time fav ) Backend Django Database MongoDB Scrapper Scrapy with playwright Data Pipeline Kafka Queue Rabbitmq Deployment docker-compose Configurations Entire code base is configurable using ","default_branch":null,"files":null,"tree":[],"storefront":"/r/Robokishan","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Robokishan/Crunchy/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}