Java FileInputStream limited to 65535 bytes

Question

I have a program that is opening a fileInputStream, reading it and writing it to an outputstream.

I'm using AIX.

When I run from terminal, I have no issues. However, when my third party application runs it, it encounters an issue where the FileInputStream only reads the first 65535 bytes from the file and then the next call to the .read() function returns -1, even though the file is slightly bigger than 65535 bytes (it's 68372 bytes). This results in a truncated output file.

My question is, what can cause this limitation? It doesn't appear to be an intrinsic limit of FileInputStream. I suspect there is a java option being set somewhere but I can't for the life of me determine where. Could this be an OS limitation somehow?

Here's my basic code:

OutputStream lOut = new FileOutputStream("/home/fileOut.txt");
FileInputStream fIn = new FileInputStream(new File("/home/fileIn.txt"));
int ch;
byte[] buf = new byte[65536];

while((ch = fIn.read(buf)) > 0)
{
    lOut.write(buf);
}

fIn.close();
lOut.close();

Hard to believe. NB Your write call is incorrect. It should be `lOut.write(buf, 0' ch);` Strange buffer size too. You should use a power of two. — user207421, Feb 27 '16 at 00:18
@EJP Outputstream documentation states that passing just a byte array is the same as the call you've specified. Link: https://docs.oracle.com/javase/7/docs/api/java/io/OutputStream.html. also, the buffer size was a typo. It was actually 1 << 16. — zeusalmighty, Feb 28 '16 at 02:37
No. It doesn't state that at all. It states that `write(buf)` is equivalent to `write(buf, 0, buf.length)`. Not the same thing at all. You are writing junk at the end of the file by using `buf.length` instead of `ch`. You can't assume either that `read()` fills the buffer or that the file length is a multiple of the buffer size. — user207421, Feb 28 '16 at 02:41
Not to labour the point but it can't possibly have said that `write(buf)` is equivalent to `write(buf, 0, ch)`, as `ch` is a local variable in your code, not part of the API. A very strange misreading. — user207421, Feb 28 '16 at 18:39

score 2 · Accepted Answer · edited Feb 28 '16 at 03:46

Leaving aside that your test code is broken; see @EJP's comments. (Yes. He >>is<< correct. Trust me / him.)

My question is, what can cause this limitation?

AFAIK, there is no such limitation.

It doesn't appear to be an intrinsic limit of FileInputStream.

There isn't one.

I suspect there is a java option being set somewhere but I can't for the life of me determine where.

There is no JVM option that would cause this behavior¹.

Could this be an OS limitation somehow?

In theory yes. Or it could be a bug / limitation in the file system drivers, a "resource limit" (see "man ulimit") or a storage media error. In practice, all of these explanations are unlikely. But you could confirm this the problem by trying to read ... or copy ... the same file using an OS utility that is known to work.

Actually, I suspect that the real problem is a bug in the third-party application² you are trying to using. It could be a similar bug to the bug in your test code, or something else.

^{1 - A slight overstatement. If the 3rd-party app did something really stupid (i.e. catching and squashing Throwable), then setting the heap size too small could in theory "cause" this behavior. If you ever encounter a problem like that, change vendors as soon as you can!}

^{2 - Just to be clear, a "third party application" is an application written & supplied by a third party. You / your organization are the first party, your JVM vendor is the second party ...}

The third party application is peoplesoft and all it is doing is instantiating the java object and calling my class. — zeusalmighty, Feb 28 '16 at 12:51
So, based on your assessment, I went back and double checked and you are correct. The third party application hadn't closed their file before passing it to me. When closing the file, their buffer was evidently flushed and I got all the data. — zeusalmighty, Feb 28 '16 at 14:05

Java FileInputStream limited to 65535 bytes

1 Answers1