Hadoop java.io.IOException: Mkdirs failed to create /some/path

Question

When I try to run my Job I am getting the following exception:

Exception in thread "main" java.io.IOException: Mkdirs failed to create /some/path
    at org.apache.hadoop.util.RunJar.ensureDirectory(RunJar.java:106)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:150)

Where the /some/path is hadoop.tmp.dir. However when I issue the dfs -ls cmd on /some/path I can see that it exists and the dataset file is present (was copied before lunching the job). Also the path is correctly defined in hadoop configs. Any suggestions will be appreciated. I am using hadoop 0.21.

score 95 · Answer 1 · edited Dec 24 '22 at 05:51

95

Just ran into this problem running mahout from CDH4 in standalone mode in my MacBook Air.

The issue is that a /tmp/hadoop-xxx/xxx/LICENSE file and a /tmp/hadoop-xxx/xxx/license directory are being created on a case-insensitive file system when unjarring the mahout jobs.

I was able to workaround this by deleting META-INF/LICENSE from the jar file like this:

zip -d mahout-examples-0.6-cdh4.0.0-job.jar META-INF/LICENSE

and then verified it with

jar tvf mahout-examples-0.6-cdh4.0.0-job.jar | grep -i license

edited Dec 24 '22 at 05:51

starball

20,030
7
43
238

answered Jul 08 '12 at 01:08

Todd Nemet

1,096
1
7
4

4

Woh, thank you very much, that was also my problem. Just to be a bit clearer : "mahout-examples-0.6-cdh4.0.0-job.jar" is the mapreduce job to perform. So the general case is : zip -d META-INF/LICENSE – JohnCastle Dec 04 '12 at 20:12
I keep finding this solutions online but it doesnt work for me!! After running the zip and jar commands above I still get:: Exception in thread "main" java.io.IOException: Mkdirs failed to create /var/folders/9y/4dzrwg8n45z7fbhmlqc7bsgc0000gn/T/hadoop-unjar5690365448328571882/license – alex9311 Jun 19 '15 at 15:52
2

@alex9311 I had the same issue as yours and I used this to solve it: `zip -d examples.jar LICENSE` – mbbce Oct 20 '15 at 13:13

ngrislain · Answer 2 · 2012-11-26T18:49:17.157

13

The problem is OSX specific it is due to the fact that by default the filesystem is set to case-insensitive on a Mac (case preserving but case insensitive, which to my opinion is very bad).

A hack to circumvent this is to create a .dmg disk image with disk utility which is case sensitive and mount this image where you need it (i.e. hadoop.tmp.dir or /tmp) with the following command (as a superuser):

sudo hdiutil attach -mountpoint /tmp <my_image>.dmg

I hope it helps.

edited Nov 26 '12 at 18:49

answered Nov 26 '12 at 18:14

ngrislain

944
12
19

How to create the dmg file? – hakunami Apr 21 '14 at 08:46

score 12 · Accepted Answer · answered May 09 '12 at 23:24

12

This is a file on the local disk that is being created (to unpack your job jar into), not in HDFS. Check you have permissions to mkdir this directory (try it from the command line)

answered May 09 '12 at 23:24

Chris White

29,949
4
71
93

9

yes, I checked and I have the permission to mkdir, but still have the problem. – hakunami Apr 21 '14 at 08:44
1

@NathanielFord - apparently the [much more highly upvoted answer didn't quite do the job](http://stackoverflow.com/a/11379938/1364007). Puzzling. – Wai Ha Lee Dec 14 '15 at 23:05
2

@WaiHaLee The highest voted answer is often not selected, but the OP goes so far as to explicitly state this answer doesn't work. Bizarre! – Nathaniel Ford Dec 14 '15 at 23:06
@NathanielFord - the highest voted answer is for a case-insensitive filesystem, something which the OP probably isn't working with (and certainly isn't the standard on a unix OS) – Chris White Jan 22 '16 at 00:02
I solved it by explicitly writing to hdfs: like this: `df.write.parquet("hdfs://localhost:8020/user/akavall/my_df_name"` – Akavall Jan 20 '18 at 01:03
what about non-nfs – mathtick Feb 23 '18 at 18:26
Another case might be there is no space left on device. – buxizhizhoum Jul 29 '19 at 13:18
@hakunami how did you resolve it then? – Priyanshu Sharma Nov 09 '22 at 15:02

RATabora · Answer 4 · 2012-11-01T16:27:33.370

I ran into this issues several times in the past, I believe it is a Mac specific issue. Since I use Maven to build my project, I was able to get around it by adding a line in my Maven pom.xml like this:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer">
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

But how would you use this with leiningen? – user1158559 Jul 02 '14 at 16:17 — user1158559, Jul 02 '14 at 16:17

Garry · Answer 5 · 2015-09-28T12:33:17.557

In my case below lines of code in pom.xml in Maven project worked on Mac.

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.0</version>
    <configuration>
      <shadedArtifactAttached>true</shadedArtifactAttached>
    </configuration>
    <executions>
      <execution>
        <phase>package</phase>
        <goals>
          <goal>shade</goal>
        </goals>
          <configuration>
            <filters>
              <filter>
                <artifact>*:*</artifact>
                <excludes>
                  <exclude>META-INF/*.SF</exclude>
                  <exclude>META-INF/*.DSA</exclude>
                  <exclude>META-INF/*.RSA</exclude>
                  <exclude>META-INF/LICENSE*</exclude>
                  <exclude>license/*</exclude>
                </excludes>
              </filter>
            </filters>
        </configuration>
      </execution>
    </executions>
  </plugin>

Make sure you are using the "maven-shade-plugin" plugin, assembly plugin doesn't have filters in configuration. May be there is another way to define filters in assembly plugin. But this works for me and checked some others as well. — Garry, Sep 28 '15 at 12:54

score 3 · Answer 6 · answered Jul 01 '15 at 13:07

3

Check the Required space is available or not. THis is problem mostly get because of the space issues.

answered Jul 01 '15 at 13:07

Kumar Basapuram

179
2
7

this is what i just met ! – kiwi Mar 02 '17 at 03:08

score 3 · Answer 7 · answered Feb 10 '17 at 17:09

I ran into this same issue while building MapReduce jobs on a Mac with MacOS Sierra. The same code runs without problems on Ubuntu Linux (14.04 LTS and 16.04 LTS). MapReduce distribution was 2.7.3, and was configured for Single Node, standalone operation. The problem appears to be related to copying license files into a META_INF directory. My problem was solved by adding a transformer into the Maven Shade plugin configuration, specifically: ApacheLicenseResourceTransformer.

Here is the relevant section of the POM.xml, which goes as part of the <build> section:

<plugin>                                                                                                             <groupId>org.apache.maven.plugins</groupId>                                                                      
   <artifactId>maven-shade-plugin</artifactId>                                                                      
   <version>3.0.0</version>                                                                                         
   <executions>                                                                                                     
     <execution>                                                                                                    
       <phase>package</phase>                                                                                       
       <goals>                                                                                                      
         <goal>shade</goal>                                                                                         
       </goals>                                                                                                     
       <configuration>                                                                                              
         <transformers>                                                                                             
           <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">       
             <mainClass>path.to.your.main.class.goes.here</mainClass>                                        
           </transformer>                                                                                           
           <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer">  
           </transformer>                                                                                           
         </transformers>                                                                                            
       </configuration>                                                                                             
     </execution>                                                                                                   
   </executions>                                                                                                    
 </plugin>

Notice that I also use the ManifestResourceTransformerto specify the main class for the MapReduce Job.

score 0 · Answer 8 · answered Oct 22 '16 at 13:02

0

In my case I just renamed the file "log_test.txt"

Because the OS (UBUNTU) was trying to generate a folder with the same name. "log_test.txt/__results.json"

answered Oct 22 '16 at 13:02

Ammar Bozorgvar

1,230
19
30

Hadoop java.io.IOException: Mkdirs failed to create /some/path

8 Answers8

Linked