{"repo":"mishushakov/llm-scraper","free":true,"listed":false,"github":"https://github.com/mishushakov/llm-scraper","clone":"git clone https://github.com/mishushakov/llm-scraper.git","description":"Turn any webpage into structured data using LLMs","language":"TypeScript","stars":6911,"topics":["ai","artificial-intelligence","browser","browser-automation","gpt","gpt-4","langchain","llama","llm","openai","playwright","puppeteer","scraper"],"license":"MIT","category":"data_extraction","readme_excerpt":"# LLM Scraper\n\n<img width=\"1800\" alt=\"Screenshot 2024-04-20 at 23 11 16\" src=\"https://github.com/mishushakov/llm-scraper/assets/10400064/ab00e048-a9ff-43b6-81d5-2e58090e2e65\">\n\nLLM Scraper is a TypeScript library that allows you to extract structured data from **any** webpage using LLMs.\n\n> [!IMPORTANT]\n> **LLM Scraper was updated to version 2.0.**\n>\n> The new version comes with **Vercel AI SDK 6** support and updated examples.\n\n### Features\n\n- Supports GPT, Sonnet, Gemini, Llama, Qwen model series\n- Schemas defined with Zod or JSON Schema\n- Full type-safety with TypeScript\n- Based on Playwright framework\n- Streaming objects\n- [Code-generation](#code-generation)\n- Supports 6 formatting modes:\n  - `html` for loading pre-processed HTML\n  - `raw_html` for loading raw HTML (no processing)\n  - `markdown` for loading markdown\n  - `text` for loading extracted text (using [Readability.js](https://github.com/mozilla/readability))\n  - `image` for loading a screenshot (multi-modal only)\n  - `custom` for loading custom content (using a custom function)\n\n**Make sure to give it a star!**\n\n<img width=\"165\" alt=\"Screenshot 2024-04-20 at 22 13 32\" src=\"https://github.com/mishushakov/llm-scraper/assets/10400064/11e2a79f-a835-48c4-9f85-5c104ca7bb49\">\n\n## Getting started\n\n1. Install the required dependencies from npm:\n\n   ```\n   npm i zod playwright llm-scraper\n   ```\n\n2. Initialize your LLM:\n\n   **OpenAI**\n\n   ```\n   npm i @ai-sdk/openai\n   ```\n\n   ```js\n   import { openai } from '@ai-sdk/openai'\n\n   const llm = openai('gpt-4o')\n   ```\n\n   **Anthropic**\n\n   ```\n   npm i @ai-sdk/anthropic\n   ```\n\n   ```js\n   import { anthropic } from '@ai-sdk/anthropic'\n\n   const llm = anthropic('claude-3-5-sonnet-20240620')\n   ```\n\n   **Google**\n\n   ```\n   npm i @ai-sdk/google\n   ```\n\n   ```js\n   import { google } from '@ai-sdk/google'\n\n   const llm = google('gemini-1.5-flash')\n   ```\n\n   **Groq**\n\n   ```\n   npm i @ai-sdk/openai\n   ```\n\n   ```js\n   import { createOpenAI } from '@ai-sdk/openai'\n   const groq = createOpenAI({\n     baseURL: 'https://api.groq.com/openai/v1',\n     apiKey: process.env.GROQ_API_KEY,\n   })\n\n   const llm = groq('llama3-8b-8192')\n   ```\n\n   **Ollama**\n\n   ```\n   npm i ollama-ai-provider-v2\n   ```\n\n   ```js\n   import { ollama } from 'ollama-ai-provider-v2'\n\n   const llm = ollama('llama3')\n   ```\n\n3. Create a new scraper instance provided with the llm:\n\n   ```js\n   import LLMScraper from 'llm-scraper'\n\n   const scraper = new LLMScraper(llm)\n   ```\n\n## Example\n\nIn this example, we're extracting top stories from HackerNews:\n\n```ts\nimport { chromium } from 'playwright'\nimport { z } from 'zod'\nimport { Output } from 'ai'\nimport { openai } from '@ai-sdk/openai'\nimport LLMScraper from 'llm-scraper'\n\n// Launch a browser instance\nconst browser = await chromium.launch()\n\n// Initialize LLM provider\nconst llm = openai('gpt-4o')\n\n// Create a new LLMScraper\nconst scraper = new LLMScraper(llm)\n\n// Open new page\nconst page = await browser.newPage()\nawait page.goto('https://news.ycombinator.com')\n\n// Define schema to extract contents into\nconst schema = z.object({\n  top: z\n    .array(\n      z.object({\n        title: z.string(),\n        points: z.number(),\n        by: z.string(),\n        commentsURL: z.string(),\n      })\n    )\n    .length(5)\n    .describe('Top 5 stories on Hacker News'),\n})\n\n// Run the scraper\nconst { data } = await scraper.run(page, Output.object({ schema }), {\n  format: 'html',\n})\n\n// Show the result from LLM\nconsole.log(data.top)\n\nawait page.close()\nawait browser.close()\n```\n\nOutput\n\n```js\n[\n  {\n    title: \"Palette lighting tricks on the Nintendo 64\",\n    points: 105,\n    by: \"ibobev\",\n    commentsURL: \"https://news.ycombinator.com/item?id=44014587\",\n  },\n  {\n    title: \"Push Ifs Up and Fors Down\",\n    points: 187,\n    by: \"goranmoomin\",\n    commentsURL: \"https://news.ycombinator.com/item?id=44013157\",\n  },\n  {\n    title: \"JavaScript's New Superpower: Explicit Resource Management\",\n    points: 225,\n    by: \"olalonde\",\n    commentsURL: \"https://news.ycombinator.com/item?id=44012227\",\n  },\n  {\n    title: \"\\\"We would be less confidential than Google\\\" Proton threatens to quit Switzerland\",\n    points: 65,\n    by: \"taubek\",\n    commentsURL: \"https://news.ycombinator.com/item?id=44014808\",\n  },\n  {\n    title: \"OBNC – Oberon-07 Compiler\",\n    points: 37,\n    by: \"AlexeyBrin\",\n    commentsURL: \"https://news.ycombinator.com/item?id=44013671\",\n  }\n]\n```\n\nMore examples can be found in the [examples](./examples) folder.\n\n## Streaming\n\nReplace your `run` function with `stream` to get a partial object stream.\n\n```ts\n// Run the scraper in streaming mode\nconst { stream } = await scraper.stream(page, Output.object({ schema }))\n\n// Stream the result from LLM\nfor await (const data of stream) {\n  console.log(data.top)\n}\n```\n\n## Code-generation\n\nUsing the `generate` function you can generate re-usable playwright script that scrapes the contents according to a schema.\n\n```ts\n// Generate code and run it on the page\nconst { code } = await scraper.generate(page, Output.object({ schema }))\nconst result = await page.evaluate(code)\nconst data = schema.parse(result)\n\n// Show the parsed result\nconsole.log(data.top)\n```\n\n## Contributing\n\nAs an open-source project, we welcome contributions from the community. If you are experiencing any bugs or want to add some improvements, please feel free to open an issue or pull request.\n","default_branch":"main","files":19,"tree":[".gitignore",".prettierrc","LICENSE.md","README.md","examples/codegen.ts","examples/hn.ts","examples/ollama.ts","examples/streaming.ts","examples/toolUse.ts","package-lock.json","package.json","src/cleanup.ts","src/index.ts","src/models.ts","src/preprocess.ts","tests/index.ts","tests/scraper.test.ts","tsconfig.json","vitest.config.ts"],"storefront":"/r/mishushakov","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mishushakov/llm-scraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}