{"repo":"arnab2001/git-repo-parser","free":true,"listed":false,"github":"https://github.com/arnab2001/git-repo-parser","clone":"git clone https://github.com/arnab2001/git-repo-parser.git","description":"A tool to scrape all files from a GitHub repository and turn it into a JSON or TXT file, Useful for AI and LLM Projects","language":"TypeScript","stars":30,"topics":["ai","ai-tools","json","llm","llm-tools","repo","scraper","repo-scraper"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"git-repo-parser A powerful tool to scrape all files from a GitHub repository and convert them into JSON, Token-Oriented Object Notation (TOON), or RepoScript (an LLM-first transcript format). Installation Install the package globally using npm: Or add it to your project as a dependency: Usage Command Line Interface (CLI) This package provides three CLI commands: 1. git-repo-to-json : Scrapes a GitHub repository and saves the result as a JSON file. 2. git-repo-to-toon : Scrapes a GitHub repository and saves the result as a TOON file. 3. git-repo-to-text : Scrapes a GitHub repository and saves the result as a RepoScript transcript (formerly the “plain text” output). Example usage: The scraped data will be saved as files.json , files.toon , or files.txt in your current directory. When --tokens (or --token , --token-count , -t ) is supplied, the CLI also prints the token count using the CL100K vocabulary for any export format. Use --meta / --no-meta to toggle RepoScript metadata lines (default is no metadata). Benchmark Suite Run the bundled benchmark to evaluate scrape runtime and token usage across multiple public repositories: Results are saved under benchmark/ : - benchmark/results.json – machine-readable summary (durations, token counts, output sizes) - benchmark/results.md – markdown report per repository/format - benchmark/ .preview.txt – first 100 lines of each export for spot-checking Programmatic Usage You can also use the package in your Node.js projects: API scrapeRep","default_branch":null,"files":null,"tree":[],"storefront":"/r/arnab2001","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/arnab2001/git-repo-parser/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}