{"repo":"marcelmay/hadoop-hdfs-fsimage-exporter","free":true,"listed":false,"github":"https://github.com/marcelmay/hadoop-hdfs-fsimage-exporter","clone":"git clone https://github.com/marcelmay/hadoop-hdfs-fsimage-exporter.git","description":"Exports Hadoop HDFS content statistics to Prometheus","language":"Java","stars":163,"topics":["prometheus-exporter","hadoop-fsimage","hadoop","hdfs","monitoring","hdfs-metrics"],"license":"Apache-2.0","category":"analytics","readme_excerpt":"Prometheus Hadoop HDFS FSImage Exporter ======= [ ][license] [ ][maven repo search] [ ][docker hub] Exports Hadoop HDFS statistics to Prometheus monitoring including total / per user / per group / per configured directory path / per set of paths number of directories number of files file size and optionally size distribution number of blocks file replication (overall / per user summary) The exporter parses the FSImage using the Hadoop FSImage Analysis library. This approach has the advantage of being fast (2.6 GB FSImage 50s) adding no heavy additional load to HDFS NameNode (no NameNode queries, you can run it on second NameNode) The disadvantage is no real time update, only when NameNode writes FSImage (interval of hours, see dfs.namenode.checkpoint.period). This should be sufficient for most cases (long-term trend, detecting HDFS small file abuses, user and group stats) parsing takes 2x-3x FSImage size in heap space The exporter parses fsimage data in background thread which checks every 60s for fsimage changes. This avoids blocking and long-running Prometheus scrapes and potentially stale metrics. Grafana dashboards Hadoop HDFS FSImage by kaszpir Requirements For building: JDK 17 Maven 3.9.x Docker (only required if building docker image) For running: JRE 17+ for running (JRE 21+ LTS recommended) Access to Hadoop FSImage file Docker (only required if building docker image) Downloading Available on and as docker image on [Docker Hub][docker hub] Building You can test the ex","default_branch":null,"files":null,"tree":[],"storefront":"/r/marcelmay","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/marcelmay/hadoop-hdfs-fsimage-exporter/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}