java.io.FileNotFoundException and (No space left on device)

Question

I have this Java code, which should read a 0.5M files and write after removing some useless information(I’m using Enron E-mails Dataset)

public void getInboxFiles(File directory){
    File[] usersFolders;
    File[] userFolders;
    File[] inboxFiles;
    usersFolders = directory.listFiles();
    for(File temp:usersFolders){
        userFolders = temp.listFiles();         
        for(File temp2:userFolders){                                    
            inboxFiles = temp2.listFiles();
            for(File tmp3:inboxFiles){
                if(tmp3.isDirectory())
                    continue;                       
                readNPrase(tmp3, new File("/media/ADATA SH12/datasets/parsedEnron/"+temp.getName()+tmp3.getName()+".txt"));                         
            }
        }
    }   

}

the function readNParse is:

public void readNPrase(File in,File out){
    BufferedReader br=null;
    BufferedWriter bw =null;
    try{
        br = new BufferedReader(new FileReader(in));
        bw= new BufferedWriter(new FileWriter(out));
        boolean messageContent = false;
        String line = null;
        while((line = br.readLine()) != null){              
            if(line.trim().equals(""))
                messageContent = true;
            if(messageContent && !isHeader(line) && !line.trim().equals("")){
                bw.write(line);
                bw.newLine();
            }

        }
            bw.flush();
            br.close();
            bw.close();
    } catch (IOException e) {
        e.printStackTrace();
    }finally{
        try{
            bw.close();
            br.close();
        }catch(Exception e){
            e.printStackTrace();
        }
    }
}

after running this code for 10 minuets it stops working and gave me this error:

java.lang.NullPointerException java.io.FileNotFoundException: /media/ADATA SH12/datasets/parsedEnron/causholli-m98.txt (No space left on device)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
at java.io.FileWriter.<init>(FileWriter.java:90)
at EnronMailParser.readNPrase(EnronMailParser.java:16)
at EnronMailParser.getInboxFiles(EnronMailParser.java:71)
at EnronMailParser.main(EnronMailParser.java:84)

I checked the space on the drive but there is a too much of empty space some of people said it is related to file name, and some said it is related to inode which is an os issue, I have no idea what is it and how it solve it. I am on Ubuntu 12.04.

If its runing for 10mins must be a rather large amount of files in the tree you are processing, and you then write the parsed versions all to one directory (so flattening that tree) so could well just be running out of space or hitting a limit on number of files. Can you see how many files are in /media/ADATA SH12/datasets/parsedEnron after its failed? and the space on the /media filesystem (df -k) — dethorpe, Jul 01 '14 at 13:09

score 12 · Accepted Answer · answered Jul 01 '14 at 14:25

As you indicated, "No space left on device" could mean that you're out of inodes on the filesystem where the new file would be stored. Unix filesystems (including Ubuntu) commonly use a data structure called an inode for each file stored in the filesystem. The number of inodes is fixed when the filesystem is created, and it limits the number of files that can be created in the filesystem. If a filesystem is out of inodes, you can't create new files even when there is free space.

You can run df -i to see how many inodes each filesystem has and how many are free.

Filesystems are normally created with plenty of inodes, so it's a little unusual to run out. The filesystem is likely to have a large number of very small or empty files on it. It's not uncommon to find that some program has been quietly creating empty files in some obscure directory. You should inspect the filesystem to see if there are some files you can delete. Remember that you're looking for a large number of files to delete, because you want to free a large number of inodes. It doesn't matter how big each file is.

If the filesystem is out of inodes, there's not a lot you can do within your program to work around the issue. If your program is the one that created a bunch of empty files, then you should reconsider how your program works. Otherwise, your only choice is to create the files somewhere else where inodes aren't a problem.

score 4 · Answer 2 · edited May 23 '17 at 12:32

4

Most filesystems have a limit to the number of files which can be in a single directory. I would guess that you are hitting that limit. i would recommend adding some levels of directory nesting so that all the result files do not end up in the same directory. Some limit info here: https://stackoverflow.com/a/466596/552759 .

edited May 23 '17 at 12:32

Community

1
1

answered Jul 01 '14 at 13:02

jtahlborn

52,909
5
76
118

java.io.FileNotFoundException and (No space left on device)

2 Answers2