Hadoop and HDFS

Overview

Apache Hadoop is an open-source framework for storing and processing very large data sets on a cluster of ordinary servers. It has three core layers. HDFS, the Hadoop Distributed File System, stores the data. YARN shares the cluster's CPU and memory between jobs. MapReduce is the original batch processing model that runs on top. Hadoop is written mainly in Java and released under the Apache License 2.0.

Hadoop matters because it made a simple idea practical: instead of buying one very large and very expensive machine, you buy many cheap ones and let software handle the failures. Hard disks die, servers crash and network switches fail. Hadoop assumes all of this will happen and keeps working anyway. For about a decade it was the default platform for "big data", and many of today's tools, including Apache Hive and Apache Spark, grew up around it.

Think of a self-storage company. Rather than one giant warehouse, it rents out thousands of small units across several buildings. Every box you store is copied into three units, at least one of them in a different building. A central front desk keeps a ledger of which box is in which unit. If a unit floods, the desk simply points you to another copy. HDFS works the same way, with servers as units, racks as buildings and the NameNode as the front desk.

What it is

Hadoop is best understood as a set of cooperating services.

HDFS and YARN are separate. You can run Spark, Hive on Tez or other engines on YARN, reading data from HDFS, without ever writing a MapReduce job. In practice "Hadoop" often refers to the whole ecosystem that grew around these modules, not only the modules themselves.

Where it came from

Hadoop grew out of Apache Nutch, an open-source web search engine started in 2002 by Doug Cutting and Mike Cafarella. Nutch needed to store and process billions of web pages. In 2003 Google published a paper describing its distributed file system, GFS, and in 2004 a paper describing MapReduce. The Nutch developers built open-source versions of both ideas.

In early 2006 the storage and processing code was split out of Nutch into a new project called Hadoop, named after a toy elephant belonging to Cutting's son. Cutting joined Yahoo the same year, and Yahoo invested heavily in the project. In January 2008 Hadoop became a top-level Apache project. In April 2008 a Hadoop cluster set a record by sorting one terabyte in 209 seconds.

Later milestones include version 1.0 in December 2011, the Hadoop 2 line in 2013, which introduced YARN as a general resource manager and added NameNode high availability, and Hadoop 3.0 in December 2017, which added erasure coding to HDFS. The project remains active: the 3.4 line arrived in 2024 and the 3.5 line in 2026.

How it works

Storing a file

When a client writes a file, HDFS cuts it into blocks. The typical block size is 128 MB, and it is configurable. Each block is stored on several DataNodes; the default replication factor is three.

Placement is rack-aware. With a factor of three, the first replica goes on the writer's own machine if it is a DataNode, or otherwise on a node in the writer's rack. The second replica goes to a node in a different rack, and the third to another node in that same remote rack. This keeps one copy safe if a whole rack loses power, while limiting slow cross-rack traffic.

Writes are pipelined. The client streams data to the first DataNode, which stores it and forwards it to the second, which forwards it to the third. HDFS files follow a write-once, read-many model: once a file is closed you can append to it or truncate it, but you cannot edit bytes in the middle.

Keeping track of everything

The NameNode keeps the namespace in memory. It records every metadata change in an EditLog and periodically merges the log into a snapshot called the FsImage, a process called checkpointing. DataNodes send regular heartbeats to show they are alive and block reports listing what they hold. If a DataNode stops responding, the NameNode schedules new copies of its blocks elsewhere.

Each block also carries checksums. A client that reads a corrupt block detects the mismatch and fetches another replica.

Erasure coding

Three-way replication costs 200 per cent extra storage. Since Hadoop 3, HDFS can use erasure coding instead for colder data. With the default Reed-Solomon policy of six data and three parity units, a file needs 9 units of space rather than 18, an overhead of 50 per cent, at the cost of more CPU and network work on reads and repairs.

Processing the data

The key idea is to move computation to the data, not data to the computation. A MapReduce job runs map tasks on the machines that already hold the input blocks. Each map task emits key-value pairs. The framework then shuffles and sorts these pairs so that all values for the same key reach the same reduce task, which produces the final output. YARN decides where each task runs, preferring nodes or racks that hold the data locally.

Key concepts and terms

A worked example

This scenario is illustrative. A regional utility keeps meter readings on an on-premises cluster of 20 DataNodes spread over four racks, each node with 12 disks.

1. Every night it loads a 64 GB file of readings. HDFS splits it into 512 blocks of 128 MB.
2. With replication factor three, the file occupies 192 GB of raw disk. Each block has one copy in the loading node's rack and two in another rack.
3. A Spark job on YARN computes daily consumption per customer. YARN starts executors on nodes that hold the blocks, so most reads come from local disks.
4. During the job one DataNode fails. The NameNode notices the missing heartbeats and re-replicates that node's blocks to others. Tasks that were reading from the dead node retry against another replica. The job finishes, a little later than usual.
5. After 18 months the readings are rarely queried. The team moves them to a directory with the RS-6-3 erasure coding policy, cutting raw storage for that data from three times its size to one and a half times.

Strengths and limitations

Strengths:

• Scales out by adding commodity servers.
• Tolerates disk, node and rack failures by design.
• High aggregate throughput for large sequential reads.
• Data locality reduces network traffic for batch jobs.
• Mature, open source and supported by a large ecosystem.

Limitations:

• Operationally heavy: capacity planning, upgrades, security and tuning need a skilled team.
• Storage and compute are tied to the same machines, so you cannot scale one without the other.
• The NameNode keeps all metadata in memory, so very large numbers of small files strain it.
• MapReduce writes intermediate results to disk and is slow for interactive or iterative work.
• No in-place updates; transactional tables need a format layered on top.

How it compares with alternatives

Practical guidance and best practices

1. For new cloud projects, start with object storage and an open table format rather than a new HDFS cluster.
2. If you run HDFS, avoid millions of tiny files. Compact small files into larger ones, ideally close to the block size.
3. Store analytical data in columnar formats such as Apache Parquet rather than CSV or text.
4. Enable NameNode high availability so the master is not a single point of failure.
5. Configure rack awareness correctly; without it, replicas can end up in one rack.
6. Use erasure coding for cold, rarely read data and keep replication for hot data.
7. Use Spark or Hive on Tez for processing; reserve MapReduce for legacy jobs.
8. When migrating, move cold data first, run both systems side by side and compare query results before switching.

Common mistakes

• Treating HDFS like a normal file system and expecting in-place edits or low-latency random access.
• Setting replication to one on small clusters to save space, then losing data when a node dies.
• Letting small files pile up until the NameNode runs short of memory.
• Running a single NameNode in production with no standby.
• Assuming Hadoop is only MapReduce, and rejecting it for that reason, when Spark on YARN would work well.
• Lifting and shifting HDFS to the cloud without asking whether object storage would be cheaper and simpler.

Frequently asked questions

Is Hadoop still used?

Yes. Many organisations still run on-premises Hadoop clusters, and the Apache project continues to publish releases. New projects, especially in the cloud, usually choose object storage with engines such as Spark or Trino instead, but they often keep Hadoop's ideas and its client libraries.

What is the difference between Hadoop and HDFS?

HDFS is one module of Hadoop: the distributed file system. Hadoop as a whole also includes YARN for resource management, MapReduce for processing and common libraries. People sometimes use "Hadoop" to mean the wider ecosystem of tools that run on these modules.

Why is the default block size so large?

Large blocks keep the amount of metadata small and make reads mostly sequential. A 1 TB file in 128 MB blocks needs about 8,000 block entries; in 4 KB blocks it would need hundreds of millions. Large blocks also give each map task a sizeable, efficient piece of work.

What happens when a DataNode fails?

The NameNode stops receiving its heartbeats and marks it dead. Blocks that now have fewer replicas than required are copied from surviving replicas to other nodes. Clients reading from the failed node switch to another replica automatically.

Do I need MapReduce to use Hadoop?

No. YARN can run other engines, and most teams use Spark, Hive on Tez or similar tools. MapReduce is still available but is mainly kept for older jobs.

Can Spark read from HDFS and S3 at the same time?

Yes. Spark uses Hadoop's file system client libraries, which support HDFS and, through connectors such as S3A, object stores. A single job can read from one and write to the other, which is a common pattern during migrations.

Glossary

• Commodity hardware: standard, inexpensive servers rather than specialised machines.
• Hadoop Common: the shared libraries used by all Hadoop modules.
• Heartbeat: a regular signal from a DataNode to show it is alive.
• Block report: a DataNode's list of all blocks it stores.
• Checkpoint: merging the EditLog into a new FsImage.
• Standby NameNode: a second NameNode ready to take over if the active one fails.
• MapReduce: a batch model of parallel map tasks, a shuffle and reduce tasks.
• YARN: Yet Another Resource Negotiator, Hadoop's resource manager.
• Write-once, read-many: files are written, closed and then only read, appended or truncated.

Summary

Hadoop showed that a cluster of cheap machines, run by software that expects failure, could store and process data at a scale that once needed specialist systems. HDFS splits files into large replicated blocks, the NameNode tracks them, YARN shares the cluster and engines from MapReduce to Spark do the processing near the data. Its weaknesses, mainly operational effort and tightly coupled storage and compute, pushed many teams towards cloud object storage, warehouses and lakehouse designs. Even so, the ideas Hadoop popularised, such as replication, data locality and scale-out on commodity hardware, remain the foundations of modern big data systems.

Related

It belongs to Storage and formats in my stack, under Big data. Others in the same group: