Hadoop Distributed File System
- Android app
- Not listed
- Free plan
- Yes
- Runs on
- api, Linux, Mac, self-hosted, Windows
Summary
Hadoop Distributed File System (HDFS) is the primary distributed storage used by Hadoop applications. A cluster centers on a NameNode that manages filesystem metadata and DataNodes that store data. It is designed for fault tolerance on low-cost commodity hardware and for high-throughput streaming access and batch processing, rather than low-latency work. Its design targets files from gigabytes to terabytes, clusters of hundreds of nodes and tens of millions of files in one instance. HDFS is written in Java and supported on major platforms, with self-hosted cluster deployment. Administration options include shell-like commands, NameNode and DataNode web servers, fsck diagnostics, safemode and a cluster balancer. Its permissions model shares much with POSIX and can optionally support POSIX ACLs. Hadoop offers simple operating-system identity and Kerberos authentication modes. Configured encryption protects data at rest and in transit through encryption zones and the Hadoop Key Management Server. WebHDFS provides REST access, and listed storage integrations include Amazon S3, Azure Blob Storage and Azure Data Lake Storage. HDFS is free and licensed under Apache License 2.0.
Who it is for
HDFS suits organizations building self-hosted Hadoop storage for high-throughput batch or streaming workloads. Its stated design fits large files and clusters, rather than low-latency access needs.
What is good
- Designed for fault tolerance on commodity hardware.
- Supports high-throughput batch and streaming workloads.
- Offers REST, Java and C API access.
- Configured encryption covers data at rest and in transit.
What to know first
- Designed for throughput rather than low latency.
- Support is through project mailing lists maintained by volunteers.
- Requires self-hosted cluster deployment.
Verdict
HDFS provides distributed storage and administration features for Hadoop workloads at large file and cluster scales. Its throughput focus makes workload fit an important consideration, and support is provided through project mailing lists.
Hadoop Distributed File System plans and pricing
All plansCompared on distributed file system software
- Deployment model
- self_hostedhadoop.apache.org
- Data replication
- Yeshadoop.apache.org
- Multi-site support
- Yeshadoop.apache.org
- File access protocols
- Java API, C API, REST API, HTTP browser, NFS gateway, FS shellhadoop.apache.org
- Encryption at rest
- Yeshadoop.apache.org
Facts
- Purpose
- HDFS is the primary distributed storage used by Hadoop applications.hadoop.apache.org · 30 Sept 2026
- Architecture
- An HDFS cluster primarily consists of a NameNode managing filesystem metadata and DataNodes storing data.hadoop.apache.org · 30 Sept 2026
- Fault tolerance
- HDFS is highly fault-tolerant and designed to run on low-cost commodity hardware.hadoop.apache.org · 30 Sept 2026
- Workload fit
- HDFS is designed for streaming access and batch processing with high throughput rather than low latency.hadoop.apache.org · 30 Sept 2026
- Scale
- HDFS is tuned for gigabyte-to-terabyte files, hundreds of nodes per cluster, and tens of millions of files in one instance.hadoop.apache.org · 30 Sept 2026
- Language and platforms
- Hadoop is written in Java and supported on all major platforms.hadoop.apache.org · 30 Sept 2026
- Administration
- HDFS provides shell-like commands, built-in NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer.hadoop.apache.org · 30 Sept 2026
- Permissions
- HDFS implements a permissions model sharing much of POSIX and optionally supports POSIX access control lists.hadoop.apache.org · 30 Sept 2026
- Authentication
- Hadoop supports simple operating-system identity mode and Kerberos authentication mode.hadoop.apache.org · 30 Sept 2026
- Encryption
- Configured HDFS encryption provides transparent end-to-end encryption for data at rest and in transit using encryption zones and the Hadoop Key Management Server.hadoop.apache.org · 30 Sept 2026
- REST integration
- The WebHDFS HTTP REST API supports the complete FileSystem and FileContext interfaces for HDFS.hadoop.apache.org · 30 Sept 2026
- Storage integrations
- The documentation lists Amazon S3, Azure Blob Storage, Azure Data Lake Storage, Tencent COS, and other Hadoop-compatible file systems.hadoop.apache.org · 30 Sept 2026
- Downloads and integrity
- Apache distributes source and binary tarballs and recommends verifying releases with GPG signatures or SHA-512 checksums.hadoop.apache.org · 30 Sept 2026
- Support
- Apache Hadoop support is provided through project mailing lists maintained by non-paid volunteers.hadoop.apache.org · 30 Sept 2026
- License
- Apache Hadoop is licensed under the Apache License 2.0.hadoop.apache.org · 30 Sept 2026
Best Hadoop Distributed File System alternatives
See all 12
Where it ranks on Everything Xiaomi
Is Hadoop Distributed File System yours?
Claim it for free: prove the domain, then correct facts, plans and screenshots. An editor reviews every change.
Sources
- hadoop.apache.org/docs/current/hadoop-project-dist/hadoop· checked 30 Sept 2026
- hadoop.apache.org/docs/current/hadoop-project-dist/hadoop· checked 30 Sept 2026
- hadoop.apache.org/docs/current/hadoop-project-dist/hadoop· checked 30 Sept 2026
- hadoop.apache.org/docs/current/hadoop-project-dist/hadoop· checked 30 Sept 2026
- hadoop.apache.org/docs/current/hadoop-project-dist/hadoop· checked 30 Sept 2026
- hadoop.apache.org/releases· checked 30 Sept 2026
- hadoop.apache.org/who.html· checked 30 Sept 2026
- apache.org/foundation/governance/corporation.html· checked 30 Sept 2026