4

I understand that the block system in HDFS is a logical partition on top of underlying file system. But how is the file retrieved when I issue a cat command.

Let say I have a 1 GB file. My default HDFS block size is 64 MB.

I issue the following the command:

hadoop -fs copyFromLocal my1GBfile.db input/data/

The above command copies the file my1GBfile.db from my local machine to input/data directory in HDFS:

I have 16 blocks to be copied and replicated ( 1 GB / 64 MB ~ 16 ).

If I have 8 datanodes, a single datanode might not have all blocks to reconsitute the file.

when I issue the following command

hadoop -fs cat input/data/my1GBfile.db | head

what happens now?

How is the file reconstituted? Although blocks are just logical partitions, how is the 1 GB file physically stored. It is stored on HDFS. does each datanode get some physical portion of the file. so by breaking input 1GB file into 64 MB chunks, we might break something at record level (say in between the line). How is this handled?

I checked in my datanode and I do see a blk_1073741825, which when opened in editor actually displays contents of the file.

so is the chunks of files that is made is not logical but real partition of data happens?

kindly help clarify this

brain storm
  • 30,124
  • 69
  • 225
  • 393
  • possible duplicate of [How does Hadoop process records records split across block boundaries?](http://stackoverflow.com/questions/14291170/how-does-hadoop-process-records-records-split-across-block-boundaries) – Zeus Jul 23 '14 at 22:09
  • @Zeus: That is not a dupliate. It talks about input split which are conceptually very different than HDFS blocks – brain storm Jul 23 '14 at 22:14
  • oh shoot wrong thread. Please check the 1st answer on this thread : http://stackoverflow.com/questions/10857880/storage-format-in-hdfs – Zeus Jul 23 '14 at 22:19
  • @Zeus: Again, you have to read my question above. I ask about the physical location of files in HDFS. None of the posts you posted talk about them and that talks more on compression – brain storm Jul 23 '14 at 22:21

4 Answers4

7

So far I understand from your question, my answer goes like this as per my understanding...

First of all, you need to understand the difference b/w HDFS block size and inputSplit size.

Block size - Block size of HDFS (64/128/256 MB) actually contains the data of the original (1 GB) file. And internally/ultimately this data is stored in blocks (4/8 KB) on fileSystem (ext, etc). So, block size of HDFS is a physical partition of the original file.

InputSplit - A file is broken into input split, which is a logical partition of the file. Logical partition means -- it will just have the information of the blocks address/location. Hadoop uses this logical representation of the data (input split) stored in file blocks. When a MapReduce job client calculates the input splits, it figures out where the first whole record in a block begins and where the last record in the block ends.

In cases where the last record in a block is incomplete, the input split includes location information for the next block and the byte offset of the data needed to complete the record.

Hope, above clairfies the difference b/w block size and input split size.

Now coming to your question on working of 'hadoop fs -cat /'----->

All the information about the locations of blocks are stored in NameNode as metadata. If a node is getting split at record level, then DataNode sends the address/location information of the blocks to NameNode.

So, when client issues 'cat' command to Hadoop, then basically client sends a request to NameNode that -- "I want to read fileA.txt, please provide me the locations of all the blocks of this file stored at various locations". It's duty of NameNode to provide the locations of the blocks stored on various DataNodes.

Based on those locations, client directly contacts with DataNodes for those blocks. And finally client reads all those blocks in same order/manner those blocks were stored (here NameNode returns the addresses of all the blocks of a file to the client) in HDFS--resulting in complete file to the client.

Hope, I could clarify your doubt a bit.

anshuman sharma
  • 121
  • 1
  • 4
4

Blocks are literally just files on a datanode. When you cat a file in HDFS, your machine streams these blocks directly from their respective datanodes and reconstructs the entire file locally.

Mike Park
  • 10,845
  • 2
  • 34
  • 50
  • so a 1 GB file is physically chopped into 16 files (which are blocks). Hence blocks are not abstraction or logical layer correct? – brain storm Jul 23 '14 at 22:37
  • Yes it is physically chopped up. I'm not sure what you mean by them not being an abstraction or logical layer though. – Mike Park Jul 23 '14 at 22:40
  • This talks about blocks being abstraction layer: http://stackoverflow.com/questions/13012924/large-block-size-in-hdfs-how-is-the-unused-space-accounted-for – brain storm Jul 23 '14 at 22:54
  • Semantics... you could argue that HDFS blocks are an abstraction on top of OS level blocks. – Mike Park Jul 23 '14 at 23:03
2

Blocks

When you save a file to HDFS, the file is split into chunks called blocks (each of size 128MB by default, but this value can be configured). Blocks are replicated (3 times by default) and each copy is saved on a different (whenever possible) node in the Hadoop cluster. This is the reason why it's recommended to have a minimum of three datanodes in a cluster with replication 3.

Where blocks are physically placed is explained here:

For the common case, when the replication factor is three, HDFS’s placement policy is to put one replica on the local machine if the writer is on a datanode, otherwise on a random datanode in the same rack as that of the writer, another replica on a node in a different (remote) rack, and the last on a different node in the same remote rack.

To check the location of the blocks in your file $FILENAME use:

hdfs fsck $FILENAME -files -blocks -locations

This will show you the number of blocks, their status, as well as the IP addresses of all the nodes where blocks and their replicas are saved.

Now what happens when you process a file? (*) Hadoop will look for the blocks that are closest to the machine where the read command has been issued in order to minimize latency. Quote from the docs:

To minimize global bandwidth consumption and read latency, HDFS tries to satisfy a read request from a replica that is closest to the reader. If there exists a replica on the same rack as the reader node, then that replica is preferred to satisfy the read request. If HDFS cluster spans multiple data centers, then a replica that is resident in the local data center is preferred over any remote replica.

Partitions

Concerning partitions, these come into play in the context of MapReduce and Spark and they are logical divisions of data that constitute the basis of parallelism. Transformations are carried out in parallel on the data partitions.

Note that since "data" may consist of one or more files, a partition can comprise blocks from different files.

(*) you wrote: "when opened in editor actually displays contents of the file" ... as a side remark: it's not a good idea in general to open big files for editing! In this case OK to open just a block.

user2314737
  • 27,088
  • 20
  • 102
  • 114
0

In addition to previous answer, you could get info about blocks using Java API. E.g., you could use FilterFileSystem class and FilterFileSystem method in this class. So, you could see how you file splits into blocks and stored in cluster. Link to JavaAPI: http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FilterFileSystem.html

morsik
  • 1,250
  • 14
  • 17