Apache Hadoop HDFS
Apache Hadoop HDFS is a free, open-source distributed file system for high-throughput access to large application data sets on commodity hardware.
What Apache Hadoop HDFS is
Apache Hadoop HDFS is a free, open-source distributed file system for high-throughput access to large application data sets on commodity hardware. It is designed for clusters of hundreds of nodes and tens of millions of files in one instance. HDFS replicates file blocks across machines and detects failed DataNodes so it can begin re-replication. Its design favors batch processing and streaming throughput, not low-latency interactive workloads. Files use a write-once, read-many model: appends and truncates are supported, but arbitrary in-place updates are not. Applications can access HDFS through Java, a C wrapper, REST, a browser interface, or an NFS gateway. The documentation lists connectors for Google Cloud Storage, Amazon S3, Azure Blob Storage, and Azure Data Lake Storage. Transparent encryption is supported. Apache warns that without Kerberos caller authentication, people with network access to cluster servers can access data and run code, and recommends securing clusters before production. Apache also recommends checking release downloads using GPG signatures or SHA-512 checksums. The listed stable release is Hadoop 3.5.0, dated April 2, 2026.
Good
- Designed for clusters of hundreds of nodes.
- Replicates blocks and detects failed DataNodes.
- Provides Java, C, REST, browser, and NFS access.
- Supports transparent encryption.
- Connectors cover several cloud storage services.
Watch out
- Not tuned for low-latency interactive use.
- Arbitrary in-place file updates are unavailable.
- Clusters need Kerberos caller authentication for access control.
Who it suits
HDFS suits teams handling large application data sets that need distributed storage for batch or streaming workloads. It is less suited to interactive work requiring low latency or arbitrary in-place file updates.
Verdict
HDFS offers distributed, fault-tolerant storage for throughput-focused workloads, with several application interfaces and documented cloud connectors. Plan for its write-once model and secure cluster configuration before production.
Plans and prices
From Apache Hadoop HDFS's own pricing page, checked 4 Oct 2001.
| Plan | Price | What it gives you |
|---|---|---|
| Apache Hadoop HDFS | Free | Open-source software · Apache License 2.0 |
| Apache Hadoop / HDFS open-source distribution | Free | self-hosted · download from Apache mirrors · no hosted SLA stated |
The facts
Each one read from Apache Hadoop HDFS's own pages, with the date it was checked.
What it does
- Architecture
- A cluster uses a NameNode for namespace and access coordination and DataNodes for block storage and serving reads and writes.checked 15 Oct 2001
- Cloud integrations
- The current documentation lists compatible file systems for Amazon S3, Azure Blob, Azure Data Lake, Google Cloud Storage, Aliyun OSS, Tencent COS, Huawei OBS and VolcanoEngine TOS.checked 15 Oct 2001
- Current release
- Apache Hadoop 3.5.0 is the first stable release in the 3.5 line and adds full server-side Java 17 support.checked 15 Oct 2001
- Deployment modes
- The setup guide covers standalone, pseudo-distributed, fully distributed and Docker-container operation.checked 15 Oct 2001
- Documentation breadth
- The documentation set includes architecture, user guides, commands, Java API references, shell API, configuration references and security guides.checked 15 Oct 2001
- Encryption
- HDFS documentation includes transparent encryption as a supported feature.checked 4 Oct 2001
- Erasure coding
- The current documentation includes HDFS erasure-coding policies and operations.checked 15 Oct 2001
- Fault tolerance
- HDFS is designed to tolerate hardware failures and uses replicated blocks across DataNodes.checked 15 Oct 2001
- Hadoop scope
- Apache Hadoop provides open-source software for reliable, scalable distributed computing across clusters.checked 15 Oct 2001
- High availability
- NameNode high availability can use shared NFS storage or a distributed Journal edit log, with the Journal approach recommended.checked 15 Oct 2001
- Integrations
- Hadoop 3.5.0 includes a FileSystem implementation for Google Cloud Storage, and its documentation also lists connectors for Amazon S3, Azure Blob Storage, and Azure Data Lake Storage.checked 4 Oct 2001
- Interfaces
- Applications can use the Java FileSystem API, a C wrapper, WebHDFS REST, browser access or the NFS gateway.checked 15 Oct 2001
- Large-file model
- HDFS targets write-once-read-many workloads and commonly uses 128 MB blocks.checked 15 Oct 2001
- Legal and funding profile
- Crunchbase identifies the legal name as The Apache Software Foundation, a U.S. nonprofit established in 1999 and funded by individual donations and corporate sponsors.checked 15 Oct 2001
- License
- Apache packages are generally distributed under the OSI-approved Apache License 2.0 unless a package states otherwise.checked 15 Oct 2001
- Limit: learning and integration
- TrustRadius reviewers say the many modules take time to learn, components do not always integrate smoothly and optimization can be difficult.checked 15 Oct 2001
- Limit: operational complexity
- G2 reviewers complain that installation, NameNode administration, upgrades and security require substantial expertise and maintenance effort.checked 15 Oct 2001
- Limit: small files and cloud fit
- G2 reviewers report poor behavior with many small files and less flexibility or cost efficiency than cloud-native object storage.checked 15 Oct 2001
- Namespace
- HDFS offers hierarchical directories, permissions and quotas but does not support hard or soft links.checked 15 Oct 2001
- Operational tools
- The user guide documents rack awareness, safemode, fsck, balancing, upgrade rollback and checkpointing utilities.checked 15 Oct 2001
- Praise: large-scale data
- TrustRadius reviewers praise Hadoop for handling very large unstructured datasets, ETL pipelines, logging and broad data aggregation.checked 15 Oct 2001
- Praise: reliability
- G2 reviewers repeatedly praise HDFS for dependable large-file storage, fault tolerance and stable batch pipelines.checked 15 Oct 2001
The company
- Maker
- The Apache Hadoop project is developed as open-source software by the Apache Software Foundation.checked 4 Oct 2001
- Maker profile
- LinkedIn lists The Apache Software Foundation as a nonprofit software-development organization founded in 1999 with 5,001–10,000 employees and headquarters in Wilmington, Delaware.checked 15 Oct 2001
The full review of Apache Hadoop HDFS
Overview
Apache Hadoop HDFS is a self-hosted distributed file system for storing large application data sets across commodity hardware. It is best suited to teams building batch-processing or streaming systems that need cluster-scale storage. Its throughput and replication make a strong case for those workloads, but applications that depend on low-latency access or arbitrary file edits should look elsewhere.
Key features
HDFS is designed to scale to hundreds of nodes and tens of millions of files in one instance. It replicates file blocks across machines and detects failed DataNodes so it can initiate re-replication. That combination suits large workloads where continued access to data matters; it also makes HDFS a more specialized choice than storage built around interactive response times.
Files use a write-once, read-many model. Appends and truncates are supported, but arbitrary in-place updates are not, which fits batch pipelines better than applications that routinely revise existing files. Applications can use a Java API, C wrapper, REST API, browser interface, or NFS gateway.
Hadoop 3.5.0 includes a FileSystem implementation for Google Cloud Storage, and Hadoop documentation lists connectors for Amazon S3, Azure Blob Storage, and Azure Data Lake Storage. HDFS documentation also covers transparent encryption. Apache warns that without Kerberos caller authentication, anyone with network access to cluster servers can access data and run code, so securing a cluster is a prerequisite for production use.
Pricing
Apache Hadoop HDFS is free open-source software under the Apache License 2.0. The Apache Hadoop project is developed by the Apache Software Foundation, and HDFS uses a self-hosted deployment model. There is no free trial to weigh against a paid tier; the practical cost is operating the cluster and securing it for production.
Platforms
HDFS supports API access and is available for Linux and Windows in self-hosted deployments. Its Java, C, REST, browser, and NFS interfaces give application teams several ways to connect, while the deployment model makes cluster operation part of the adopter's responsibility.
Who it's for
HDFS is a good fit for organizations with large batch or streaming workloads, teams able to operate a cluster, and applications that benefit from replicated storage across machines. It is less suitable for low-latency interactive use or workloads that require arbitrary in-place file updates. Apache directs users seeking help to module mailing lists and asks that its non-paid volunteers not be approached for private support.
Pros and cons
- Pros: Designed to scale across hundreds of nodes and tens of millions of files, with block replication and failed-DataNode detection to support resilience.
- Pros: Multiple application interfaces and documented connectors for major cloud object-storage services broaden integration options.
- Pros: Free, open-source software with documented transparent encryption.
- Cons: Low-latency interactive workloads are not its target, and the write-once model rules out arbitrary in-place updates.
- Cons: Production clusters need deliberate security configuration; without Kerberos caller authentication, network access to cluster servers can expose data and code execution.
Alternatives
For a broader data lake software shortlist, compare tools by the storage and governance needs of your workload. HPCC Systems is another free, Apache 2.0-licensed self-hosted option, with API, web, and extension platforms in addition to Linux, macOS, and Windows. AWS Lake Formation may suit readers focused on permissions through integrated services: its permissions have no charge, while integrated services such as Amazon S3 use standard rates. Unilake is worth considering if you want a free platform the maker says can run fully isolated in an environment of your choice, under AGPL 3.0 and EUPL licenses.
Cloudera Data Lake Service is a paid option with custom pricing and no free plan. HPE Ezmeral Data Fabric offers a free plan and trial, with pricing that varies by reseller. lakeFS has a free plan and trial, with enterprise pricing by contacting sales. Tencent Cloud Application Performance Management has free-plan packages with stated trace-storage and Agent-hour allowances. Tencent Cloud GooseFS is a paid option with postpaid, hourly-settled management capacity.
Verdict
Choose Apache Hadoop HDFS if your team needs free, self-hosted storage for large batch or streaming workloads and can operate and secure a cluster. Its scale, replication, and multiple interfaces are compelling for that job. Look elsewhere when low-latency interaction or in-place file updates are central to the application.
Getting started
- Download the Hadoop source or binary tarball from the Apache Hadoop website.
- Verify the downloaded release with its GPG signature or SHA-512 checksum.
- Deploy HDFS in a self-hosted cluster.
- Configure Kerberos caller authentication and secure the cluster before production use.
- Connect applications through Java, C, REST, a browser interface, or an NFS gateway.
Limits worth knowing
HDFS is tuned for batch and streaming throughput rather than low-latency interactive use. Its write-once, read-many model permits appends and truncates, but not arbitrary in-place updates.
Questions
How much does Apache Hadoop HDFS cost?
HDFS is free and open source under the Apache License 2.0.
Who develops Apache Hadoop?
The Apache Hadoop project is developed as open-source software by the Apache Software Foundation.
Which platforms are listed?
API, Linux, self-hosted, and Windows.
How can applications access HDFS?
Through its Java API, a C wrapper, REST API, browser interface, or NFS gateway.
What should operators know about cluster security?
Apache warns that without Kerberos caller authentication, anyone with network access to cluster servers can access data and run code. It recommends securing clusters before production deployment.
Which cloud storage connectors are documented?
Hadoop 3.5.0 includes a FileSystem implementation for Google Cloud Storage, and the documentation lists connectors for Amazon S3, Azure Blob Storage, and Azure Data Lake Storage.
Alternatives to Apache Hadoop HDFS
- 1
HPCC Systems
Free planOpen sourceWebAPIExtensionLinux7.5Free plan - 2
Tencent Cloud Application Performance Management
WebAPILinuxSelf-hosted7.2 - 3
Parseable
Free planWebAPILinuxSelf-hosted6.9From $0.39 a month - 4
Oracle Big Data Service
WebAPILinux6.2 - 5
Qubole
WebAPI6.2 - 6
Apache Ozone
Free planWebAPILinuxSelf-hosted6.1Free plan