{"repo":"thejordanprice/p2pspider","free":true,"listed":false,"github":"https://github.com/thejordanprice/p2pspider","clone":"git clone https://github.com/thejordanprice/p2pspider.git","description":"BitTorrent DHT crawler with real-time web UI. Indexes magnet links from DHT swarm into MongoDB/SQLite. Features responsive search, WebSocket updates, and torrent details.","language":"JavaScript","stars":35,"topics":["dht","nodejs","pm2","self-hosted","spider","torrent","dht-crawler","torrent-indexer","elasticsearch","redis"],"license":"MIT","category":"databases-storage","readme_excerpt":"p2pspider - DHT Spider A daemon that crawls the BitTorrent DHT network and an Express web application that provides a searchable database of magnet links with real-time updates through WebSockets. Intro DHT Spider can index over 1 million magnets per 24 hours on modest hardware (2GB of RAM and around 2MB/s connection). It's resource-intensive and will use available CPU and RAM, which can be controlled via the 'ecosystem.json' file. On 2GB RAM, it's recommended to use 8 instances of the daemon and 2 of the webserver, all limited at 175MB. Screenshots Getting Started Configuration You will need to have port 6881 (or your configured port) open to the internet for the DHT crawler to function properly. The application can be configured through the .env file: You can also fine-tune the crawler performance in the daemon.js file: It's not recommended to change the nodesMaxSize or maxConnections , but adjusting the timeout may increase indexing speed. Higher timeout values may require more RAM; the maximum recommended value is 5000ms. Component Control DHT Spider now allows you to run the daemon and webserver components independently: - RUN DAEMON : Set to \"true\" to run the P2P Spider daemon, or \"false\" to disable it - RUN WEBSERVER : Set to \"true\" to run the web server, or \"false\" to disable it This flexibility allows you to: - Run only the daemon for dedicated crawling - Run only the webserver for serving existing data - Run both components together (default behavior) Example usage:","default_branch":null,"files":null,"tree":[],"storefront":"/r/thejordanprice","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/thejordanprice/p2pspider/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}