{"repo":"rodricios/wxpath","free":true,"listed":false,"github":"https://github.com/rodricios/wxpath","clone":"git clone https://github.com/rodricios/wxpath.git","description":"wxpath - declarative web crawling with XPath; a Web Query Language (WQL)","language":"Python","stars":111,"topics":["crawler","scraper","web-graph-traversal","web-query-language","data-loader","rag","data-extraction","data-mining","extract-data"],"license":"AGPL-3.0","category":"scrapers-browser-automation","readme_excerpt":"wxpath - declarative web graph traversal with XPath NEW: TUI - Interactive terminal interface (powered by Textual) for testing wxpath expressions and exporting data. Install Requires Python 3.10+. --- What is wxpath? wxpath is a declarative web crawler where traversal is expressed directly in XPath. Instead of writing imperative crawl loops, wxpath lets you describe what to follow and what to extract in a single expression. wxpath executes that expression concurrently, breadth-first- ish , and streams results as they are discovered. This expression fetches a page, extracts links, and streams them concurrently - no crawl loop required: By introducing the url(...) operator and the /// syntax, wxpath's engine is able to perform recursive (or paginated) web crawling and extraction: Why wxpath? Most web scrapers force you to write crawl control flow first, and extraction second. wxpath converges those two steps into one: - You describe traversal declaratively - Extraction is expressed inline - The engine handles scheduling, concurrency, and deduplication RAG-Ready Output Extract clean, structured JSON hierarchies directly from the graph - feed your LLMs signal, not noise. Refer to LangChain Integration for more details. Deterministic wxpath is deterministic (read: not powered by LLMs). While we can't guarantee the network is stable, we can guarantee the traversal is. Documentation (WIP) Documentation is now available here. Contents - Example: Knowledge Graph - Language Design - ur","default_branch":null,"files":null,"tree":[],"storefront":"/r/rodricios","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/rodricios/wxpath/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}