2

How can I copy files from remote Unix and Windows servers into HDFS without intermediate staging from the command line?

Mogsdad
  • 44,709
  • 21
  • 151
  • 275
buckeyeosu
  • 45
  • 8
  • 1
    There is no standard command which achieves this. Good work arounds are given [here](http://one-line-it.blogspot.com/2013/05/hadoop-copy-directly-to-hdfs-from.html) and [here](http://stackoverflow.com/questions/11270509/putting-a-remote-file-into-hadoop-without-copying-it-to-local-disk). Hope this helps. – Ramzy May 28 '15 at 22:29
  • @buckeyeosu .. Could you please share some efficient way to achieve this. I got stuck with same problem. I will also refer to other options as mentioned here. – vikrant rana Jun 27 '19 at 12:53

2 Answers2

1

You can use following command:

hadoop fs -cp /user/myuser/copyTestFolder/* hdfs://remoteServer:8020/user/remoteuser/copyTestFolder/

or vice versa to copy from server to local machine.

You can also read the hadoop documentation.

Mauro Midolo
  • 1,841
  • 3
  • 14
  • 34
1

You can use WebHDFS and cURL to upload files. This will not require having any hadoop binaries on your client, just a cURL or cURL like client. The BigInsights Knowledge Center has information on how to administer the file system using the HttpFS REST APIs.

proksch_ibm
  • 278
  • 1
  • 9