Hadoop DFS permission issue when running job

Question

I'm getting this following permission error, and am not sure why hadoop is trying to write to this particular folder:

hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar pi 2 100000
Number of Maps  = 2
Samples per Map = 100000
Wrote input for Map #0
Wrote input for Map #1
Starting Job
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=myuser, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

Any idea why it is trying to write to the root of my hdfs?

Update: After temporarily setting hdfs root (/) to be 777 permissions, I seen that a "/tmp" folder is being written. I suppose one option is to just create a "/tmp" folder with open permissions for all to write to, but it would be nice from a security standpoint if this is instead written to the user folder (i.e. /user/myuser/tmp)

No, I am using Cloudera, so hadoop is running as service, presumably as root — Dolan Antenucci, Sep 08 '11 at 18:20
Try setting the permissions to / to 777 with hadoop fs -chmod and see what it writes. — Donald Miner, Sep 08 '11 at 18:46
@orangeoctopus - it is writing /tmp/hadoop-mapred -- is there a setting to change this location? — Dolan Antenucci, Sep 08 '11 at 19:00
@spike - I'd prefer it if non-root users can run hadoop from their own accounts. I noticed it is writing /tmp to hdfs, so trying to figure out why that is occurring, so let me know if you have any ideas on that. thanks — Dolan Antenucci, Sep 08 '11 at 19:29
When we set the root directory, eg /app/hadoop/ , this is is located on master. right? So that means all data are saved to master?? — Paschalis, Feb 02 '13 at 12:14
@Paschalis - the `mapreduce.jobtracker.staging.root.dir` setting points to a location on HDFS. — Dolan Antenucci, Feb 03 '13 at 16:42

score 15 · Accepted Answer · answered Sep 09 '11 at 19:58

15

I was able to get this working with the following setting:

<configuration>
    <property>
        <name>mapreduce.jobtracker.staging.root.dir</name>
        <value>/user</value>
    </property>

    #...

</configuration>

Restart of jobtracker service required as well (special thanks to Jeff on Hadoop mailing list for helping me track down problem!)

answered Sep 09 '11 at 19:58

Dolan Antenucci

15,432
17
74
100

thanks, ran into the very same problem. it's a pity this isn't the default value. – stephen mulcahy Nov 08 '11 at 15:43
hhhm, it looks like if I set this, the permissions on mapred.system.dir (/hadoop/mapred/system) need to more permissive – stephen mulcahy Nov 09 '11 at 10:36
I have the /mapred/system directory locked down to just the mapred user (mapred.system.dir=/mapred/system). hdfs root (/) is set to only hdfs user too. If you still have problems, let me know what errors you are getting – Dolan Antenucci Nov 10 '11 at 01:30
@stephenmulcahy - BTW, I just ran into permission issue with Pig, which is due to it having its own temp directory setting. more info here: http://stackoverflow.com/questions/7194069/apache-pig-permissions-issue – Dolan Antenucci Nov 11 '11 at 14:57
What file did you add this too? Hadoop Config or Hive Config? – Dan Ciborowski - MSFT Aug 29 '12 at 18:10
@djc391 - It was added to mapred-site.xml (Hadoop config) – Dolan Antenucci Aug 29 '12 at 18:14
@dolan I am facing the same permission issue but I do not have control over the hdfs root directory. But looking at the /tmp directory, it has 777 permission. So I am wondering what causes this issue to me. `mapreduce.jobtracker.staging.root.dir = ${hadoop.tmp.dir}/mapred/staging` and `hadoop.tmp.dir = /tmp/hadoop-${user.name}`. May be it tries to write some other dir in the hdfs root other than '/tmp'? Also, I see this error only for large number of input paths; For smaller inputs, it goes smoothly. I appreciate your help. Thanks! – NPE Feb 01 '13 at 01:56
@sachin2182 -- Can you confirm you can create directories in hdfs's /tmp? If you can, create one for your user, then try setting the paths in the config so they are hard-coded to your new directories (create all the children ones too -- e.g. mapred/staging) -- give them all 777 permissions as a starting test. Restart Hadoop, and then see if it still errors. Did you check /var/log/hadoop to see if you can figure out where it is trying to write (not sure if it'll tell you, but worth a look)? – Dolan Antenucci Feb 01 '13 at 02:18
@dolan -- I have permission to create dirs under /tmp in hdfs. It turns out that while parsing my input, there is some unexpected character (which is used to determine the output location) and the map-reduce system tries to write in root dir. I fixed my script to take care of the unexpected character and now the jobs run without any issues. Sorry for the confusion and it took me a while to figure that something is wrong with my input/script. – NPE Feb 01 '13 at 03:49

score 2 · Answer 2 · answered Feb 16 '15 at 09:37

2

1) Create the {mapred.system.dir}/mapred directory in hdfs using the following command

sudo -u hdfs hadoop fs -mkdir /hadoop/mapred/

2) Give permission to mapred user

sudo -u hdfs hadoop fs -chown mapred:hadoop /hadoop/mapred/

answered Feb 16 '15 at 09:37

minhas23

9,291
3
58
40

score 0 · Answer 3 · answered Apr 09 '12 at 14:19

0

You need to set the permission for hadoop root directory (/) instead of setting the permission for the system's root directory. Even I was confused, but then realized that the directory mentioned was of hadoop's file system and not the system's.

answered Apr 09 '12 at 14:19

divinedragon

5,105
13
50
97

When you say "hadoop root directory", do you mean the root of HDFS? I chose not to make permissions in the HDFS root loose so that I could prevent users from writing to the root (and force them to one particular directory -- their "home": /user/). – Dolan Antenucci Apr 10 '12 at 03:17
Thats right. The configuration in the above question seems to be incorrect. It should point to the specific directory on HDFS. In my case it was pointing to /opt/hadoop/mapred/local/staging. I gave the write permission to the directory and it was working just fine. – divinedragon Apr 10 '12 at 06:27
Are you saying the setting mapreduce.jobtracker.staging.root.dir=/user is incorrect? Note that I'm also referring to an HDFS location in my setting for this. Also, I'm using Cloudera in case that makes a difference. – Dolan Antenucci Apr 10 '12 at 12:59

score 0 · Answer 4 · answered Jan 23 '12 at 15:11

0

You can also make a new user named "hdfs". Quite simple solution but not as clean probably.

Of course this is when you are using Hue with Cloudera Hadoop Manager (CDH3)

answered Jan 23 '12 at 15:11

Highmastdon

6,960
7
40
68

Hadoop DFS permission issue when running job

4 Answers4