2GB limitation Java Input Stream

Question

I was able to read files less than 2GB from http location. However if I try to read files greater then 2 GB I get exception. I am not reading fully instead reading in chunks. Below is my code snippet and exception message. Help me incase if you have a clue to proceed further

BufferedOutputStream bos = new BufferedOutputStream(os);
                    URL url = new URL(fileName);
                    LOGGER.debug(url.toString());
                    URLConnection connection = url.openConnection();

                    LOGGER.debug("Before getting input stream 100 MB, open input stream::" + downloadFileName);
                    // BufferedInputStream in = new BufferedInputStream(url.openStream());
                    InputStream in = url.openStream();
                  //  InputStreamReader inReader = new InputStreamReader(in);

                    // InputStream in = connection.getInputStream();
                    LOGGER.debug("Afters getting input stream, open inputstream::" + downloadFileName);
                    // in.

                    // LOGGER.debug("in1::" + in1);

                    final byte[] buffer = new byte[100 * 1024 * 1024]; // 100Mb
                    while (true)
                    {
                        LOGGER.debug("Reading..");

                        final int read = in.read(buffer);
                        LOGGER.debug("Read..");

                        if (read < 0)
                        {
                            break;
                        }
                        LOGGER.debug("Writing..");
                        bos.write(buffer, 0, read);

                        LOGGER.debug("Wrote..");

                        // bos.

                    }

are you able to / have you tried changing the max heap size for the JVM? — bcr, Jan 23 '14 at 09:59
Can you try reading without writing it (comment out bos) and check if it can read more than 2GB of data? — Sudhanshu Umalkar, Jan 23 '14 at 09:59
"Connection reset" means that the *remote peer* dropped the connection. Your problem may have nothing to do with the 2 GB size and if it does, it is on the *sender's* side. — Marko Topolnik, Jan 23 '14 at 10:00
possible duplicate of [Problems reading a huge file of 12 MB (java.lang.OutOfMemoryError)](http://stackoverflow.com/questions/8477669/problems-reading-a-huge-file-of-12-mb-java-lang-outofmemoryerror). Its an `InputStream` based problem, too. — jww, Jan 23 '14 at 10:57

score 1 · Answer 1 · answered Aug 15 '17 at 18:10

1

The BufferedInputStream interface uses signed integers for its indices and marklimit, meaning it is capped at 2gig maximum (Integer.MAX_VALUE bytes). If you read a file larger than that, it rolls over to negative and fails the marklimit comparison.

answered Aug 15 '17 at 18:10

jsears

4,511
2
31
36

score 0 · Answer 2 · answered Jan 23 '14 at 10:03

[SPACE] [ActiveMQ Session Task] [createreviseswpart_0:69:1:1:1] DEBUG 10:33:55 (CreateReviseSWPart.java:348) - Checkin Exception::java.net.SocketException: Connection reset

Looking at the javadoc for SocketException it states that

Thrown to indicate that there is an error creating or accessing a Socket.

This is most likely caused by a TCP error.

In your case it seems that the connection has been closed by the server end of the connection. This could be for many many reasons...

Try debugging the network traffic with a tool like Wireshark to view the actual packets. Hav you tried accessing the 2GB file using another tool (wget/curl/a webbrowser?).

score 0 · Answer 3 · edited May 23 '17 at 12:03

"connection reset" means the connection has gone in problem on your side activity on the socket, "connection reset by peer" means that on the other end point and there the other side (peer) has gone out the standard protocol of a goodbye handshake

This not help to identify where is the real problem (if it is in the code) but cold take you to the correct direction.

Wireshark could help to identify what is in "fly" on the net and if there is a close of the connection. Is it applicable to your enviorment?

Other interesting thing is to monitor is if there are some timeout.

2GB is a strange dimension so I prefer to advise you about:

If both are in JAVA 2GB could be a limit. Also memory (buffers) could be a limit. We assume that other side is ok.

2 GB is a JVM limit it's related to Integer.MAX_VALUE

It' not well known and bug in code are everywhere.

In some situations there are a workaround http://todayguesswhat.blogspot.it/2012/07/file-download-java-servlet-example-2gb.html

Your solution split in chunk using a custom packet or a correlation queue....

This could solve timeout caused by long activities but has the drawback of worst throughput.

What has this do to with the Connection reset error? Possibly the remote server has this limitation (if its running java at all), but the client code shown above should work just fine? — Peter Svensson, Jan 23 '14 at 10:55
@PeterLiljenberg "connection reset" means the connection times out on your end, "connection reset by peer" means it times out on the other end or is lost due to network conditions — user1594895, Jan 24 '14 at 09:59
For the record, "connection reset" and "connection reset by peer" are synonyms; some code paths on some platforms report one, some the other. A timeout on the local side results in "operation/connection timed out" and if the connection was closed at the initiative of the local side, you'll get "stream closed" or "socket closed". — Marko Topolnik, Jan 24 '14 at 11:59

2GB limitation Java Input Stream

3 Answers3