{"repo":"stevenhurwitt/reddit-streaming","free":true,"listed":false,"github":"https://github.com/stevenhurwitt/reddit-streaming","clone":"git clone https://github.com/stevenhurwitt/reddit-streaming.git","description":"streaming eight subreddits from reddit api using kafka producer & spark structured streaming.","language":"Jupyter Notebook","stars":19,"topics":["api","athena","glue","kafka","lambda","postgresql","pyspark","reddit","streaming"],"license":null,"category":"databases-storage","readme_excerpt":"reddit-streaming An attempt to stream multiple subreddits from the reddit api using kafka & spark, and store in s3 data lake as delta tables. nightly glue jobs processes raw data into clean data partitioned by year/month/day in athena.x Build dockerfiles Go to docker directory and run build script. ./build.sh Run docker-compose. docker-compose up -d --no-recreate Access jupyterlab shell. Can also attach VSCode to the jupyterlab container. docker exec -it jupyterlab bash Start streaming data Activate the virtual environment. source redditStreaming/reddit-env/bin/activate Go to reddit directory. cd redditStreaming/src/reddit Start pyspark streaming application. python3 -m reddit streaming.py Common errors: - java gateway exited before sending port number: make sure java home is set, java -version is 1.8 Start kafka producer Start the kafka producer. python3 -m reddit producer.py Common errors: - stored cluster id xyz does not match: go to /cluster config/kafka/logs/metadata.properties and change to correct cluster id Remove untagged docker images Remove untagged docker images. docker rmi $(docker images grep \"^ \" awk \"{print $3}\") Prune docker system volumes, containers & images. docker system prune && docker volume prune && docker container prune && docker image prune Note on versions When changing version of spark, hadoop, jupyterlab, etc, versions must be updated in build.sh , respective .Dockerfile , requirements.txt and reddit streaming.py . pyspark write stream to s3 erro","default_branch":null,"files":null,"tree":[],"storefront":"/r/stevenhurwitt","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/stevenhurwitt/reddit-streaming/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}