18

Im using Files.WalkFileTree() to navigate folder and counting audio files, but there is a problem when it encounters a tar file, it seems to be treating it as an actual folder I was expecting it to just skip over it.

I cannot see any options that let me control this behaviour

Code:

package com.jthink.songkong.fileloader;


import com.jthink.songkong.cmdline.SongKong;
import com.jthink.songkong.ui.MainWindow;

import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.concurrent.Callable;
import java.util.logging.Level;

/**
 * Count the number of files that can be loaded, for information purposes only
 */
public class CountFilesinFolder implements Callable<Boolean> {
    public static class CountFiles
            extends SimpleFileVisitor<Path> {
        private int fileCount = 0;
        private final PathMatcher matcher;

        CountFiles(String pattern) {
            matcher =
                    FileSystems.getDefault()
                            .getPathMatcher("regex:" + pattern);
        }

        /**
         * Find Music file
         *
         * @param file
         * @param attr
         * @return
         */
        @Override
        public FileVisitResult visitFile(Path file,
                                         BasicFileAttributes attr) {
            Path name = file.getFileName();
            if (name != null && matcher.matches(name)) {
                fileCount++;
            }
            return FileVisitResult.CONTINUE;
        }

        public int getFileCount() {
            return fileCount;
        }
    }


    private Path scanDir;
    public CountFilesinFolder(Path scanDir) {
        this.scanDir = scanDir;
    }

    public Boolean call() {
        CountFiles countFiles = null;
        try {
            countFiles = new CountFiles("^(?!._).*[.](?:mp3|mp4|m4p|m4b|m4a|ogg|flac|wma)$");
            Files.walkFileTree(scanDir, countFiles);
        }
        catch (Exception e) {
            MainWindow.logger.log(Level.SEVERE, "Unable to find file for deriving base folder", e);
        }
        MainWindow.logger.severe("Music File Count:"+countFiles.getFileCount());
        SongKong.setMaxProgress(countFiles.getFileCount());
        return true;
    }
}

gives this stacktrace

java.nio.file.NoSuchFileException: Z:\Scratch\fred.tar
    at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79)
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
    at sun.nio.fs.WindowsDirectoryStream.<init>(WindowsDirectoryStream.java:86)
    at sun.nio.fs.WindowsFileSystemProvider.newDirectoryStream(WindowsFileSystemProvider.java:526)
    at java.nio.file.Files.newDirectoryStream(Files.java:411)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:179)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
    at java.nio.file.Files.walkFileTree(Files.java:2591)
    at java.nio.file.Files.walkFileTree(Files.java:2624)
    at com.jthink.songkong.fileloader.CountFilesinFolder.call(CountFilesinFolder.java:68)
    at com.jthink.songkong.fileloader.CountFilesinFolder.call(CountFilesinFolder.java:15)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

but that is a a remote drive (nas drive), I get no such error on local drive

EDIT Implemented the following based o the answer below I thought worked

    @Override
            public FileVisitResult preVisitDirectory(Path dir, 
BasicFileAttributes attrs)
                    throws IOException {
                if(dir.endsWith(".tar"))
                {
                    return FileVisitResult.SKIP_SUBTREE;
                }
                return super.preVisitDirectory(dir, attrs);
            }

but my testing was amiss, it doesnt in fact work because the code in FileTreeWalker that fails is called before the previsit method

try {
            DirectoryStream<Path> stream = null;
            FileVisitResult result;

            // open the directory
            try {
                stream = Files.newDirectoryStream(file);
            } catch (IOException x) {
                return visitor.visitFileFailed(file, x);
            } catch (SecurityException x) {
                // ignore, as per spec
                return FileVisitResult.CONTINUE;
            }

            // the exception notified to the postVisitDirectory method
            IOException ioe = null;

            // invoke preVisitDirectory and then visit each entry
            try {
                result = visitor.preVisitDirectory(file, attrs);
                if (result != FileVisitResult.CONTINUE) {
                    return result;
                }
Paul Taylor
  • 13,411
  • 42
  • 184
  • 351
  • 3
    *error loading libastral.so*, show your code, please – Denis Tulskiy Jan 21 '13 at 09:53
  • 3
    Could you post your code? It seems like an interesting issue but without a running example it is hard to help. – Jagger Jan 21 '13 at 09:56
  • code added, just realized error only occurs if the files are on a remote drive – Paul Taylor Jan 21 '13 at 10:00
  • How do you mount the remote drive - SMB? And what FS on the remote drive? Could this be a permissions issue? – Anders R. Bystrup Jan 21 '13 at 10:05
  • Windows properties says its a NTFS fileystem, its mounted from a Netgear ReadyNASDuo using CIFS, Not permissions issue as I can see/open folder in WIndows Explorer. I Just realized the tar is actually empty,maybe not even a tar but that doesnt justify it throwing said exception. – Paul Taylor Jan 21 '13 at 10:18
  • and there is no such error remotely accessing said file from OSX – Paul Taylor Jan 21 '13 at 10:25
  • 1
    I tried to reproduce the issue by putting an empty `test.tar` file on a network drive and running your code but I was not able to do so. The exception is not thrown and the `call()` method ends without any problems. – Jagger Jan 21 '13 at 10:40
  • @Jagger same for me. I put two files with "mp3" extension and one empty fred.tar file into the directory, and the code properly returns a count of `2` with no exception. – Andreas Fester Jan 21 '13 at 10:42
  • I do not know whether this is any important but I ran my test on Windows 7 and JRE 1.7.0_11. – Jagger Jan 21 '13 at 10:46
  • Hm, Im using Windows 7 and Java 1.7.0_10 – Paul Taylor Jan 21 '13 at 10:48
  • @PaulTaylor can you quickly cross-check on _11? (You should upgrade anyways due to security issues with _10 ;) ) – Andreas Fester Jan 21 '13 at 10:55
  • Can you do a quick test to see if it actually consideres the .tar to be a directory: public static void main(String[] argv) { Path p = Paths.get("Z:/Scratch/fred.tar"); System.out.println(Files.isDirectory(p)); } – Aksel Willgert Jan 21 '13 at 11:01
  • Hi, yes tried and it considers to be a directory on Z:\drive, but not on c:\drive – Paul Taylor Jan 21 '13 at 11:08
  • 1
    I think it makes sense to limit the question to a smaller example then. Or does site-rules prevents doing this? – Aksel Willgert Jan 21 '13 at 11:11
  • @PaulTaylor Does it behave the same when you have a `.zip` file instead of `.tar`? – Andreas Fester Jan 21 '13 at 11:19
  • 1
    Implementation if something is a directory or not is depending on the state of the values returned by http://docs.oracle.com/javase/7/docs/api/java/nio/file/spi/FileSystemProvider.html#readAttributes(java.nio.file.Path, java.lang.Class, java.nio.file.LinkOption...) try to check which filesystem is in play: Path p1 = Paths.get("z:/"); Path p2 = Paths.get("c:/"); FileSystem fs = p1.getFileSystem(); FileSystem fs2 = p2.getFileSystem(); System.out.println(fs); System.out.println(fs2); – Aksel Willgert Jan 21 '13 at 11:48
  • In both cases it returns sun.nio.fs.WindowsFileSystem. I wonder if that is correct as I assume ReadyNAS is running Linux. – Paul Taylor Jan 21 '13 at 11:57
  • If the drive is seen as a letter, then I guess it is correct. You might have found a deeper problem here, I do not think the Java is working here wrongly, it is rather an underlying operating system API. – Jagger Jan 21 '13 at 13:21
  • 1
    A shot in the dark: I guess the TAR has a directory attribute on the native Linux system, but doesn't have it after copying it to your local drive. Best would be to check on the native file system if you can access it somehow. Maybe this is of any help: http://stackoverflow.com/q/4855373/44522 – MicSim Jan 21 '13 at 16:08
  • I think you are right, but I get no such error accessing the same faile from my OSX (unix based) system. So assume it also sees it as an archive, yet doesnt complain about it. – Paul Taylor Jan 21 '13 at 16:36

1 Answers1

2

Workaround for the problem at hand:

But implement a visitFileFailed and you should be ok.

public class MyFileVisitor extends SimpleFileVisitor<Path> {
    @Override
    public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {
        if (file.toString().endsWith(".tar")) {
            return FileVisitResult.CONTINUE;
        }
        return super.visitFileFailed(file, exc);
    }
}

Update: If we look closer we can see that walkFileTree uses the Files.readAttributes which turns to the current provider in play: WindowsFileSystemProvider.readAttributes to determine if a path is a directory.

As someone mentioned in the comments I also dont think the fault is in the Java-implementation but the OS-native-call that returns the wrong attribute

If you wanted to do a workaround for this, one option would be to implement your own FileSystem that wraps the WindowsFileSystem implementation transparently, except readAttributes returns .tar-paths as file instead of dir.

Aksel Willgert
  • 11,367
  • 5
  • 53
  • 74
  • Thankyou that works but ONLY if you change (file.endsWith(".tar")) to (file.toString().endsWith(".tar")) otherwise it never matches, i.e only works if you simple string match rather than file specific endsWith match. I think we have bug with Java somewhere in here. – Paul Taylor Jan 21 '13 at 21:31
  • But if the OS-native-call is returning incorrectly then don't you think WindowsFileSystemProvider shoud handle this issue rather than me (and eveyone else thatmay encounter this issue). Even if the error is not with java code I think its still something that needs to be fixed/workarounds in the system java code. – Paul Taylor Jan 22 '13 at 13:11
  • yes ofc :), but it was more as a sugestion for workaround, fixes to jre tends to take time. – Aksel Willgert Jan 22 '13 at 13:18
  • oh okay, well the visitFileFailed workaround is working for this quite obscure issue so I think I'll stick with that and move on. – Paul Taylor Jan 22 '13 at 17:14