Permission denied at hdfs

Question

I am new to hadoop distributed file system, I have done complete installation of hadoop single node on my machine.but after that when i am going to upload data to hdfs it give an error message Permission Denied.

Message from terminal with command:

hduser@ubuntu:/usr/local/hadoop$ hadoop fs -put /usr/local/input-data/ /input
put: /usr/local/input-data (Permission denied)

hduser@ubuntu:/usr/local/hadoop$

After using sudo and adding hduser to sudouser:

hduser@ubuntu:/usr/local/hadoop$ sudo bin/hadoop fs -put /usr/local/input-data/ /inwe
put: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="":hduser:supergroup:rwxr-xr-x

hduser@ubuntu:/usr/local/hadoop$

do you have access rights to the directory - are you using sudo? — ali haider, Jul 21 '12 at 16:00
Yes,after using sudo,,,,hduser@ubuntu:/usr/local/hadoop$ sudo bin/hadoop fs -put /usr/local/input-data/ /inwe put: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="":hduser:supergroup:rwxr-xr-x hduser@ubuntu:/usr/local/hadoop$ — Vignesh Prajapati, Jul 21 '12 at 17:17
In my case, it was because I was trying to download files in a location in my filesystem where I did not have permissions. — optimist, Jun 18 '15 at 14:08

score 77 · Accepted Answer · edited Jan 24 '16 at 20:04

77

I solved this problem temporary by disabling the dfs permission.By adding below property code to conf/hdfs-site.xml

<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

edited Jan 24 '16 at 20:04

gsamaras

71,951
46
188
305

answered Jul 22 '12 at 19:02

Vignesh Prajapati

2,320
3
28
38

4

This is the 1st thing I set in a new cluster ;-) – Thomas Decaux Oct 17 '16 at 20:42
where is the `conf` folder? – Klas Mellbourn May 11 '17 at 07:31
1

The conf folder is the configuration of the namenode. In my case using homebrew installed hadoop however it was in /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hdfs-site.xml – Anders Eriksson Aug 24 '17 at 13:41
7

How can this be the solution? It's ok for testing, not for production – Alex Pollan May 17 '18 at 14:53
@KlasMellbourn [HADOOP_ROOT]/etc/hadoop – SRF Aug 20 '18 at 06:52
1

This solution is like plug-in the ethernet cable when the wifi doesn't work :) – Daniel Argüelles Nov 21 '18 at 14:34

Roman Nikitchenko · Answer 2 · 2014-08-28T15:15:29.253

56

I had similar situation and here is my approach which is somewhat different:

 HADOOP_USER_NAME=hdfs hdfs dfs -put /root/MyHadoop/file1.txt /

What you actually do is you read local file in accordance to your local permissions but when placing file on HDFS you are authenticated like user hdfs. You can do this with other ID (beware of real auth schemes configuration but this is usually not a case).

Advantages:

Permissions are kept on HDFS.
You don't need sudo.
You don't need actually appropriate local user 'hdfs' at all.
You don't need to copy anything or change permissions because of previous points.

edited Aug 28 '14 at 15:15

answered Nov 15 '13 at 13:32

Roman Nikitchenko

12,800
7
74
110

you can replace the first hdfs by your user, example : HADOOP_USER_NAME=your_user hdfs dfs -put /source /destination – mourad m Apr 30 '20 at 09:16
THis is not a solution, it is a work arround. You basically say "My user do not have rights ? Lets use the superuser"... – Itération 122442 Dec 17 '21 at 08:33
For 2014 this was a perfect solution and even now it has its target auditory. This was clearly unlinking your HDFS users from system ones. Yes, the fact you have OS user does not give you anything on hdfs. As I said, 'other authentication methods' may be used if you want to authenticate HSDP users. Just that time nobody was able to do anything other than perimeter security. Think about it like service account for backend processes (with real users authenticated on API level or so) - in small/medium company works 100%. – Roman Nikitchenko Dec 19 '21 at 10:07

Donald Miner · Answer 3 · 2012-07-22T21:01:51.450

19

You are experiencing two separate problems here:

hduser@ubuntu:/usr/local/hadoop$ hadoop fs -put /usr/local/input-data/ /input put: /usr/local/input-data (Permission denied)

Here, the user hduser does not have access to the local directory /usr/local/input-data. That is, your local permissions are too restrictive. You should change it.

hduser@ubuntu:/usr/local/hadoop$ sudo bin/hadoop fs -put /usr/local/input-data/ /inwe put: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="":hduser:supergroup:rwxr-xr-x

Here, the user root (since you are using sudo) does not have access to the HDFS directory /input. As you can see: hduser:supergroup:rwxr-xr-x says only hduser has write access. Hadoop doesn't really respect root as a special user.

To fix this, I suggest you change the permissions on the local data:

sudo chmod -R og+rx /usr/local/input-data/

Then, try the put command again as hduser.

edited Jul 22 '12 at 21:01

answered Jul 21 '12 at 18:58

Donald Miner

38,889
8
95
118

copyFromLocal: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="":hduser:supergroup:rwxr-xr-x After changing permission above error promted. – Vignesh Prajapati Jul 22 '12 at 08:50
2

don't do `put` as `sudo`. Do it as `hduser`. – Donald Miner Jul 22 '12 at 21:01
It was group permission issue in my case, solved by chmod as suggested by @DonaldMiner. – novato Sep 14 '21 at 07:14

score 6 · Answer 4 · edited Oct 10 '14 at 07:05

6

I've solved this problem by using following steps

su hdfs
hadoop fs -put /usr/local/input-data/ /input
exit

edited Oct 10 '14 at 07:05

Andrey Korneyev

26,353
15
70
71

answered Oct 10 '14 at 06:24

java_dev

992
1
8
11

Martin Tapp · Answer 5 · 2014-05-15T12:39:48.373

5

Start a shell as hduser (from root) and run your command

sudo -u hduser bash
hadoop fs -put /usr/local/input-data/ /input

[update] Also note that the hdfs user is the super user and has all r/w privileges.

edited May 15 '14 at 12:39

answered May 06 '14 at 15:30

Martin Tapp

3,106
3
32
39

1

It is not required to start a shell. `sudo -u hdfs hadoop fs -chmod 777 /hbase` works as well. – 030 Jun 10 '14 at 09:39
Good point, I assumed you would want to run multiple commands as hduser. – Martin Tapp Jun 10 '14 at 12:21

score 4 · Answer 6 · answered Dec 01 '19 at 23:12

For Hadoop 3.x, if you try to create a file on HDFS when unauthenticated (e.g. user=dr.who) you will get this error.

It is not recommended for systems that need to be secure, however if you'd like to disable file permissions entirely in Hadoop 3 the hdfs-site.xml setting has changed to:

<property>
  <name>dfs.permissions.enabled</name>
  <value>false</value>
</property>

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Permission denied at hdfs

6 Answers6

Linked