Why does this code get stuck in the inner while loop?

Question

public static void Hash() throws IOException
{
    int i = 0;
    for (var k : allFiles.keySet())
    {
        for (var file : allFiles.get(k))
        {
            FileInputStream fis = new FileInputStream(file.getAbsolutePath());
            int b = fis.read();
            int xor = 0;
            while (b != -1)
            {
                xor ^= b;
                b = fis.read();
            }
            fis.close();
            System.out.println(i++);
            System.out.println("End elaboration file " + file.getAbsolutePath());
            long xored = file.length() ^ xor;
            if (allByHash.get(xored) != null)
            {
                allByHash.get(xored).add(file);
            } else
            {
                allByHash.put(xored, new LinkedList<File>());
                allByHash.get(xored).add(file);
            }
        }
    }
}

It stucks at i = 122; what is wrong? how can it keep looping in the while?

The aim is to read all files byte by byte and compute an hash of their size and content to compare them in search of duplicates.

The method blocks in the while and i can't understand why.

I can't debug the reading of the file byte by byte :p

Poor title. Rewrite to summarize your specific technical issue. — Basil Bourque, Oct 20 '21 at 21:51
Which file does it have a problem with? What happens if you just try to read that file by itself? — David Conrad, Oct 20 '21 at 22:00
I am guessing that you are trying to read from some file-like object that is endless, such as /dev/zero or /dev/urandom, or from a socket or named pipe that is blocking, waiting for data to be written to it. — David Conrad, Oct 20 '21 at 22:02
@DavidConrad I removed that file, it still blocks at the same point. — Donald Shimoda, Oct 20 '21 at 22:12
the one on which the program blocked @DavidConrad. anyway i solved, thank you :-) — Donald Shimoda, Oct 20 '21 at 22:27
Does this answer your question? [What is a debugger and how can it help me diagnose problems?](https://stackoverflow.com/questions/25385173/what-is-a-debugger-and-how-can-it-help-me-diagnose-problems) — EJoshuaS - Stand with Ukraine, Oct 20 '21 at 23:02

score 0 · Accepted Answer · answered Oct 20 '21 at 22:05

File 122 is either [A] quite large, or [B] literally endless (something like /dev/null is literally endless), or [C] a special file that can block (such as /dev/random, which can block and is endless).

Calling read() on a FileInputStream is literally 60,000 times slower than it needs to be. Modern disks are simply incapable of reading a single byte at a time; instead, they read blocks at a time. However, FileInputStream cannot buffer, so when ask for '1 byte' (which is what read() does), it fetches an entire block, tosses all but 1 byte in the bin, and returns it. Then you ask for another byte and we do this all over again. Thus, you're reading that file 'blockSize' times too many.

Either program this using a buffer (The .read(byteArray) variant), or wrap your FileInputStream in a BufferedInputStream first, which adds some memory to buffer those reads out.

The upshot is: If the 122nd file is quite large, it'll look like it's stuck, but it's just taking that long, as it's running 60k times slower than it needs to.

thank you @rzwitserloot for your clear explanation. that file was actually the first to be grater than a bounch of byte. I didn't realized that :-) — Donald Shimoda, Oct 20 '21 at 22:18
`/dev/null` is not endless, it is zero-length. In fact, it will immediately return EOF if you read from it. You may be thinking of `/dev/zero`. — David Conrad, Oct 20 '21 at 22:23

Why does this code get stuck in the inner while loop?

1 Answers1