{"repo":"Jason-nzd/countdown-scraper","free":true,"listed":false,"github":"https://github.com/Jason-nzd/countdown-scraper","clone":"git clone https://github.com/Jason-nzd/countdown-scraper.git","description":"Simple scraper for pulling supermarket product data. Powered by Node, Playwright, Cheerio, and Typescript.","language":"TypeScript","stars":16,"topics":["cheerio","playwright","typescript","nodejs"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Woolworths Scraper This project scrapes product info from Woolworths (Formerly Countdown) NZ website and optionally places the data into Azure CosmosDB. A history of price changes is stored within each product's database entry. Image files can also be scraped and placed into an Azure Storage Blob Container. Basic Setup With NodeJS installed, clone this repository, then run npm install to install dependencies. Playwright must also be installed when running for the first time with npx playwright install . The program can now be tested in dry run mode without any further setup using npm run dev . Optional Setup The .env file has variables that can be filled for more functionality. - The CosmosDB read-write connection string can be obtained from the Azure Portal CosmosDB Settings Keys . - A list of URLs to scrape can be put in file urls.txt , with one url per line. Usage npm run dev - will use dry-run mode, no azure connection is required and the results will log to console. npm run db - will scrape through the URLs and store the results into CosmosDB. npm run db https://sampleurl - a single url can be used as an argument. This will be scraped instead of the URLs text file. Other Command-Line Arguments images - will also upload images. headed - will run the browser in a window instead of a headless. Output Sample log output when running in dry run mode: This is a sample of a single product stored in CosmosDB. It was re-run at multiple dates to store changing prices:","default_branch":null,"files":null,"tree":[],"storefront":"/r/Jason-nzd","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Jason-nzd/countdown-scraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}