{"repo":"tb0hdan/domains","free":true,"listed":false,"github":"https://github.com/tb0hdan/domains","clone":"git clone https://github.com/tb0hdan/domains.git","description":"World’s single largest Internet domains dataset","language":"JavaScript","stars":1155,"topics":["yacy","scrapy","dataset","search-engines","internet-domains","colly"],"license":"BSD-3-Clause","category":"auth-billing-email","readme_excerpt":"Domains Project: Processing petabytes of data so you don't have to ========== Products and services LLMSE is a LLM-based Search Engine that uses Ollama under the hood. World's single largest Internet domains dataset This public dataset contains freely available sorted list of Internet domains. Active Domains Graphical reports Github Dataset statistics vs Subscription dataset statistics Project news Support needed! You can support this project by doing any combination of the following: - Posting a link on your website to DomainsProject - Opening issue and attaching other domain datasets that are not here yet (be sure to scroll through this README first) - Publishing research work and linking to DomainsProject - Sponsoring this project. See Subscriptions - Running Passive DNS sensor. See Passive DNS sensor repository Milestones: Domains - [x] 10 Million - [x] 100 Million - [x] 1 Billion - [x] 1.7 Billion Github - [x] 2.8 Billion - Subscription - [x] 2.9 Billion - Subscription - [x] 3 Billion - Subscription (yes, it is true) (Wasted) Internet traffic: - [x] 500TB - [x] 925TB - [x] 1PB - [x] 1.3PB - [x] 1.5PB - [x] 5.7PB - [x] 8.1PB Random facts: - More than 1TB of Internet traffic is just 3 Mbytes of compressed data - 1 million domains is just 5 Mbytes compressed - More than 5.7PB of Internet traffic is necessary to crawl 1.7 billion domains (3.4TB / 1 million). - Only 4.6Gb of disk space is required to store 1.7 billion domains in compressed form - 1Gbit fully saturated link is","default_branch":null,"files":null,"tree":[],"storefront":"/r/tb0hdan","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/tb0hdan/domains/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}