{"repo":"reidbarber/webmarker","free":true,"listed":false,"github":"https://github.com/reidbarber/webmarker","clone":"git clone https://github.com/reidbarber/webmarker.git","description":"Mark web pages for use with vision-language models","language":"TypeScript","stars":61,"topics":["prompt","prompt-engineering","som","vision-language-model","set-of-mark","claude","gemini","gpt4o","gpt4v","llms"],"license":"MIT","category":"ai-agents","readme_excerpt":"Mark web pages for use with vision-language models. Overview WebMarker adds visual markings with labels to elements on a web page. This can be used for Set-of-Mark prompting, which improves visual grounding abilities of vision-language models. How it works 1. Call the mark() function This marks the interactive elements on the page, and returns an object containing the marked elements, where each key is a mark label string, and each value is an object with the following properties: - element : The interactive element that was marked. - markElement : The label element that was added to the page. - boundingBoxElement : The bounding box element that was added to the page. You can use this information to build your prompt for the vision-language model. 2. Send a screenshot of the marked page to a vision-language model, along with your prompt Example prompt: 3. Programmatically interact with the marked elements. In a web browser (i.e. via Playwright), interact with elements as needed. For prompting or agent ideas, see the WebVoyager paper. Playwright example Options selector A custom CSS selector to specify which elements to mark. - Type: string - Default: 'a[href], button, input:not([type=\"hidden\"]), select, textarea, summary, [role=\"button\"], [tabindex]:not([tabindex=\"-1\"])' getLabel Provide a function for generating labels. By default, labels are generated as integers starting from 0. - Type: (element: Element, index: number) = string - Default: ( , index) = index.toString() mar","default_branch":null,"files":null,"tree":[],"storefront":"/r/reidbarber","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/reidbarber/webmarker/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}