-1

I can ssh to our box and do a hadoop fs -ls /theFolder and browse in for the files, etc.. but that's all I know too :) My goal is to copy one of those files - they are Avro - on to my local home folder.

How can do this? I found also a get command but not sure how to sue that either.

  • If you use MapR, you can mount your entire HDFS with NFS and then easily copy files around easily. – kichik Apr 01 '13 at 15:45
  • can you explain it a little more? I am a hadoop noob :) –  Apr 01 '13 at 15:47
  • MapR is a distribution of Hadoop. You can download it as a VM to play with it and mount the HDFS using NFS. – kichik Apr 01 '13 at 17:13

3 Answers3

4

First, use hadoop fs -get /theFolder to copy it into the current directory you are ssh'ed into on your box.

Then you can use either scp or my preference of rsync to copy the files between your box and your local system like so. Here's how I'd use rsync after having used the -get, still in the same directory:

rsync -av ./theFolder username@yourlocalmachine:/home/username

This will copy theFolder from the local fs on your box into your home folder on your machine's fs. Be sure to replace username with your actual username in both cases, and yourlocalmachine with your machine's hostname or ip address.

Quetzalcoatl
  • 3,037
  • 2
  • 18
  • 27
2

Using hadoop's get you can copy the files from HDFS to your box's file system. Read more about using get here.

Then, using scp (this is similar to doing ssh) you may copy those files to your local system. Read more about using scp here.

Amar
  • 11,930
  • 5
  • 50
  • 73
0
hadoop fs -get theFolder

is great just like previous answer. For syncing with local machine, I think you can set up git. That's easy as well.

Zsolt
  • 1
  • 1