2

I have one question regarding the HDFS read/write process:

Assuming that we have a client (for the sake of the example let's say that the client is a HADOOP map process) who requests to read a file from HDFS and or to write a file to HDFS, which is the process which actually does the read/write from/to the HDFS?

I know that there is a process for the Namenode and a process for each Datanode, what are their responsibilities to the system in general but I am confused in this scenario.

Is it the client's process by itself or is there another process in the HDFS, created and dedicated to the this specific client, in order to access and read/write from/to the HDFS?

Finally, if the second answer is true, is there any possibility that this process can be suspended for a while?

I have done some research and the most important solutions that I found were Oozie and JobControl class from hadoop API.

But, because I am not sure about the above workflow, I am not sure what process I am suspending and resuming with these tools.

Is it the client's process or a process which runs in HDFS in order to serve the request of the client?

Ravindra babu
  • 37,698
  • 11
  • 250
  • 211
dinosaur
  • 59
  • 1
  • 8

2 Answers2

1

Have a look at these SE posts to understand how HDFS writes work:

Hadoop 2.0 data write operation acknowledgement

Hadoop file write

Hadoop: HDFS File Writes & Reads

Apart from file/block writes, above question explain about datanode failure scenarios.

The current block on the good datanodes is given a new identity, which is communicated to the namenode, so that the partial block on the failed datanode will be deleted if the failed datanode recovers later on. The failed datanode is removed from the pipeline, and a new pipeline is constructed from the two good datanodes.

One failure in datanode triggers corrective actions by framework.

Regarding your second query :

You have two types of schedulers :

FairScheduler

CapacityScheduler

Have a look at this article on suspend and resume

In a multi-application cluster environment, jobs running inside Hadoop YARN may be of lower-priority than jobs running outside Hadoop YARN like HBase. To give way to other higher-priority jobs inside Hadoop, a user or some cluster-level resource scheduling service should be able to suspend and/or resume some particular jobs within Hadoop YARN.

When target jobs inside Hadoop are suspended, those already allocated and running task containers will continue to run until their completion or active preemption by other ways. But no more new containers would be allocated to the target jobs.

In contrast, when suspended jobs are put into resume mode, they will continue to run from the previous job progress and have new task containers allocated to complete the rest of the jobs.

Community
  • 1
  • 1
Ravindra babu
  • 37,698
  • 11
  • 250
  • 211
  • Thanks for your answer! I do not understand the last part. What about the example with the second client with higher priority? – dinosaur Mar 21 '16 at 16:27
  • Now that I am trying to reconsider my previous answer, could it work with the use of data queues based on these two posts? http://stackoverflow.com/questions/34464187/hadoop-file-write/34464676#34464676 http://stackoverflow.com/questions/32038000/hadoop-2-0-data-write-operation-acknowledgement – dinosaur Mar 21 '16 at 16:34
  • and concernig my initial post: the [JobControl](https://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapreduce/lib/jobcontrol/JobControl.html) class suspends the client's process and not the HDFS read/write right? Is there a way to pause the HDFS read/write process? – dinosaur Mar 21 '16 at 16:46
  • Very useful info, but what about suspending ***manually*** the read/write requests? The [YARN-2172] patch does not work according to the comments. – dinosaur Mar 21 '16 at 20:43
  • I don't see any issues with documentation. Running tasks of suspended jobs can't be stopped. They will run with limited containers. Once the job is resumed, tasks will get more containers. – Ravindra babu Mar 22 '16 at 13:58
0

So as far as I understand the process of a Datanode receives the data from the client's process (who requests to store some data in HDFS) and stores it. Then this Datanode forwards the exact same data to another Datanode (to achieve replication) and so on. When the replication will finish, an acknowledgement will go back to the Namenode who will finally inform the client about the completion of his write-request.

Based on the above flow, It is impossible to suspend an HDFS write operation in order to serve a second client's write-request (let's assume that the second client has higher priority) because if we suspend the Datanode by itself it will remain suspended for everyone who wants to write on it and as a result this part of the HDFS will be remained blocked. Finally, if I suspend a job from JobController class functions, I actually suspend the client's process (if I actually manage to catch it before his request will be done). Please correct me if I am wrong.

dinosaur
  • 59
  • 1
  • 8