scp remote file into hadoop without copying it to edge node

Question

Want to copy a file from a remote server to hadoop without copying to edge node.

1.Per below article we can do it in 2 step by first doing scp to local edge node and then perform hdfs fs command to move from edge node to hdfs https://community.cloudera.com/t5/Support-Questions/Import-data-from-remote-server-to-HDFS/td-p/233148

2.Per below article we can do ssh cat, but we have files like .gz which cannot be CAT putting a remote file into hadoop without copying it to local disk

But I am looking for a 3rd option where we can scp instead of ssh CAT and copy to hadoop without copying to Edge node.

score 1 · Accepted Answer · answered Feb 21 '20 at 10:07

1

Hadoop doesnt have an SCP upload feature.

If you want to get files in without an edge node or SSH, then that's what WebHDFS or the NFSGateway offer

answered Feb 21 '20 at 10:07

OneCricketeer

I dont see a copy option provided in WebHDFS. Can you please any link/ example command to achieve this. Thanks in advance!! – narayana kandukuri Mar 26 '20 at 00:09
You POST a file, not copy – OneCricketeer Mar 26 '20 at 01:55

score -1 · Answer 2 · answered Apr 23 '21 at 05:39

-1

Transfer using pipe

mkfifo - this creates pipe on local server (this doesn't store any data)

try mkfifo <pipename - some path on your server where ssh keys are present> | scp : | hdfs dfs -put | rm

answered Apr 23 '21 at 05:39

Prashant

Please improve your answer to be more readable. Write the code part in a code format – Khaled Apr 23 '21 at 07:30

2 Answers2