{"repo":"adbar/htmldate","free":true,"listed":false,"github":"https://github.com/adbar/htmldate","clone":"git clone https://github.com/adbar/htmldate.git","description":"Fast and robust date extraction from web pages, with Python or on the command-line","language":"Python","stars":155,"topics":["metadata-extraction","date-parser","entity-extraction","natural-language-processing","nlp","web-scraping","webscraping","date","datetime","metadata"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Htmldate: Find the Publication Date of Web Pages Find original and updated publication dates of any web page. It is often not possible to do it using just the URL or the server response. On the command-line or with Python , all the steps needed from web page download to HTML parsing, scraping, and text analysis are included. The package is used in production on millions of documents and integrated into thousands of projects. In a nutshell With Python On the command-line Features - Flexible input: URLs, HTML files, or HTML trees can be used as input (including batch processing). - Customizable output: Any date format (defaults to ISO 8601 YMD). - Detection of both original and updated dates. - Multilingual. - Compatible with Python 3.10 and later. How it works Htmldate operates by sifting through HTML markup and if necessary text elements. It features the following heuristics: 1. Markup in header : Common patterns are used to identify relevant elements (e.g. link and meta elements) including Open Graph protocol attributes. 2. HTML code : The whole document is searched for structural markers like abbr or time elements and a series of attributes (e.g. postmetadata ). 3. Bare HTML content : Heuristics are run on text and markup: - In fast mode the HTML page is cleaned and precise patterns are targeted. - In extensive mode all potential dates are collected and a disambiguation algorithm determines the best one. Finally, the output is validated and converted to the chosen format. P","default_branch":null,"files":null,"tree":[],"storefront":"/r/adbar","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/adbar/htmldate/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}