{"repo":"samvad-hq/samvad-news-harvester","free":true,"listed":false,"github":"https://github.com/samvad-hq/samvad-news-harvester","clone":"git clone https://github.com/samvad-hq/samvad-news-harvester.git","description":"Samvad News Harvester is a tiny Go service that fetches news links from provider feeds, enriches them with lightweight metadata, and fans structured events out to queues or webhooks. It is intentionally small, pluggable, and welcoming to new contributors.","language":"Go","stars":12,"topics":["golang","news","newsapi","scraper","crawler"],"license":"ISC","category":"scrapers-browser-automation","readme_excerpt":"samvad-news-harvester Samvad News Harvester is a tiny Go service that fetches news links from provider feeds, enriches them with lightweight metadata, and fans structured events out to queues or webhooks. It is intentionally small, pluggable, and welcoming to new contributors. The harvester only uses public sitemaps, or link lists, and never fetches or stores full article bodies. --- What it does Fetches links from providers defined in YAML/JSON config. Ships with a Google News sitemap fetcher tested across dozens of Indian news sources. (ndtv, thehindu, timesofindia, financialexpress, etc.) Enriches links with titles/descriptions/images using goquery (best effort; returns partial results on timeout or cancel). Publishes JSON events to multiple sinks (HTTP webhooks or queues: AWS SQS/SNS, GCP Pub/Sub) via a pluggable registry. Provides an optional dedupe layer ( bbolt file by default) to skip previously published article IDs. Quick architecture sketch --- Prereqs Go 1.24+ Access to whichever sinks you enable (webhook URL, SQS/SNS credentials, Pub/Sub topic, etc.) --- Quickstart (local) 1. Providers Edit configs/providers.yaml (or point PROVIDERS FILE to your own). Always set a valid user agent ; headers are not defaulted. 2. Publishers Copy: Then disable sinks you don’t own ( enabled: false ) or set the env vars for the ones you keep. The example enables GCP Pub/Sub — toggle it off if you don't have creds. 3. Run the harvester The process stays alive and runs every CRAWL INTE","default_branch":null,"files":null,"tree":[],"storefront":"/r/samvad-hq","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/samvad-hq/samvad-news-harvester/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}