4

Assuming that the block size is 128MB, the cluster has 10GB (so ~80 available blocks). Suppose that I have created 10 small files which together take 128MB on disk (block files, checksums, replication...) and 10 HDFS blocks. If I want to add another small file to HDFS, then what does HDFS use, the used blocks or the actual disk usage, to calculate the available blocks?

80 blocks - 10 blocks = 70 available blocks or (10 GB - 128 MB)/128 MB = 79 available blocks?

Thanks.

Bao Bui
  • 185
  • 2
  • 6

1 Answers1

8

Block size is just an indication to HDFS how to split up and distribute the files across the cluster - there is not a physically reserved number of blocks in HDFS (you can change the block size for each individual file if you wish)

For your example, you need to also take into consideration the replication factor and checksum files, but essentially adding lots of small files (less than the block size) does not mean that you have wasted 'available blocks' - they take up as much room as they need (again you need to remember that replication will increase the physical data footprint required to store the file) and the number of 'available blocks' will be closer to your second calculation.

A final note - having lots to small files means that your name node will require more memory to track them (blocks sizes, locations etc), and its generally less efficient to process 128x1MB files than single 128MB file (although that depends on how you're processing it)

Chris White
  • 29,949
  • 4
  • 71
  • 93
  • Yes. Thanks Chris! So the block size is the (logical) allocation unit, right? Say the remaining disk space is 250MB (< 2 blocks), we cannot add a 200MB file (2 blocks are needed with replication = 1). – Bao Bui Mar 28 '13 at 12:14
  • 1
    with 250 MB remaining, you can add a 200 MB file with 128MB block size - you'll just get one block of 128MB, and another block of 72MB (replication factor 1) – Chris White Mar 28 '13 at 23:15