4

According to everything I've read so far about the CAP Theorem, no distributed system can provide all three of: Availability, Consistency and Partition Tolerance.

Now, Hadoop 2.x introduced a new feature, which can be configured in order to remove the single point of failure that the hadoop clusters had (the single namenode). With that, the cluster becomes highly available, consistent and partition tolerant. Am I right? Or am I missing something? According to CAP, if the system tries to provide all three features, it should pay the price in latency, does the new feature add this latency to the cluster? Or has Hadoop cracked the CAP Theorem?

Deleteman
  • 8,500
  • 6
  • 25
  • 39

2 Answers2

5

HDFS does not provide Availability in case of multiple correlated failures (for instance, three failed data nodes with the same HDFS block).

From CAP Confusion: Problems with partition tolerance

Systems such as ZooKeeper are explicitly sequentially consistent because there are few enough nodes in a cluster that the cost of writing to quorum is relatively small. The Hadoop Distributed File System (HDFS) also chooses consistency – three failed datanodes can render a file’s blocks unavailable if you are unlucky. Both systems are designed to work in real networks, however, where partitions and failures will occur, and when they do both systems will become unavailable, having made their choice between consistency and availability. That choice remains the unavoidable reality for distributed data stores.

Andrey Sozykin
  • 926
  • 12
  • 13
2

HDFS High Availability makes HDFS more available, but not completely. If a network partition makes the client unable to communicate with either NameNode then the cluster is effectively unavailable.

Jeremy Beard
  • 2,727
  • 1
  • 20
  • 25