12

I don't know what's going on here but I am trying to copy a simple file from a directory in my local filesystem to the directory specified for hdfs.

In my hdfs-site.xml I have specified that the directory for hdfs will be /home/vaibhav/Hadoop/dataNodeHadoopData using the following properties - 

<name>dfs.data.dir</name>
<value>/home/vaibhav/Hadoop/dataNodeHadoopData/</value>

and 

<name>dfs.name.dir</name>
<value>/home/vaibhav/Hadoop/dataNodeHadoopData/</value>

I am using the following command - 

bin/hadoop dfs -copyFromLocal /home/vaibhav/ml-100k/u.data /home/vaibhav/Hadoop/dataNodeHadoopData

to copy the file u.data from it's local filesystem location to the directory that I specified as Hdfs directory. But when I do this, nothing happens - no error, nothing. And no file gets copied to the hdsf. Am I doing something wrong? Any permissions issue could be there?

Suggestions needed.

I am using pseudo distributed single node mode.

Also, on a related note, I want to ask that in my map reduce program I have set the configuration to point to the inputFilePath as /home/vaibhav/ml-100k/u.data. So would it not automatically copy the file from given location to hdfs ?

harpun
  • 4,022
  • 1
  • 36
  • 40
Kumar Vaibhav
  • 2,632
  • 8
  • 32
  • 54

1 Answers1

23

I believe dfs.data.dir and dfs.name.dir have to point to two different and existing directories. Furthermore make sure you have formatted the namenode FS after changing the directories in the configuration.

While copying to HDFS you're incorrectly specifying the target. The correct syntax for copying a local file to HDFS is:

bin/hadoop dfs -copyFromLocal <local_FS_filename> <target_on_HDFS>

Example:

bin/hadoop dfs -copyFromLocal /home/vaibhav/ml-100k/u.data my.data

This would create a file my.data in your user's home directory in HDFS. Before copying files to HDFS make sure, you master listing directory contents and directory creation first.

harpun
  • 4,022
  • 1
  • 36
  • 40
  • Cool. I tried it. I changed the config to point to two different dir locations, formatted the namenode, started all daemons and tried the copyFromLocal as you suggested. Now it says - org.apache.hadoop.Security.AccessControlException : Permission denied : user = root, access = write, inode = "":vaibhav:supergroup:rwxr-xr-x – Kumar Vaibhav Feb 06 '13 at 04:16
  • So I gave chmod 777 -R to hdfs directory(/home/vaibhav/Hadoop/dataNodeHadoopData/). But now there is some other error - it says File /user/vaibhav/u.data could only be replicated to 0 nodes, instead of 1. I have no idea why it's trying to write to this folder ! – Kumar Vaibhav Feb 06 '13 at 04:56
  • Per default files are copied to the user's home directory on HDFS. In your case `/user/vaibhav`. For the replication error, see [this](http://stackoverflow.com/questions/5293446/hdfs-error-could-only-be-replicated-to-0-nodes-instead-of-1) and the [runtime errors part](http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment) if hadoop wiki. Most likely the hadoop dfs daemons did not start correctly. – harpun Feb 07 '13 at 19:09