Looping over byte channels

Question

While reading from the good-old InputStream, I used the following code(with which I was never comfortable) :

    int read = 0;
    InputStream is = ....;

    while((i = is.read() != -1){

        ....
    }

Now I'm trying to read 10MB from an InputStream using NIO :

        protected void doPost(HttpServletRequest request,
            HttpServletResponse response) throws ServletException, IOException {
        // TODO Auto-generated method stub

        System.out.println("In Controller.doPost(...)");

        ByteBuffer chunk = ByteBuffer.allocateDirect(1000000);

        /* Source channel */
        int numRead = 0;
        ReadableByteChannel rbc = Channels.newChannel(request.getInputStream());

        /* Destination channel */
        File destFile = new File(
                "D:\\SegyDest.sgy");
        FileOutputStream destFileFos = new FileOutputStream(destFile);
        FileChannel destFileChannel = destFileFos.getChannel();

        /* Read-Write code */
        while (numRead >= 0) {
            chunk.rewind();
            numRead = rbc.read(chunk);

            System.out.println("numRead = " + numRead);

            chunk.rewind();
            destFileChannel.write(chunk);
        }

        /* clean-up */
        rbc.close();
        destFileChannel.close();
        destFileFos.close();

        request.setAttribute("ops", "File Upload");
        request.getRequestDispatcher("/jsp/Result.jsp").forward(request,
                response);
    }

My question is /* How to loop over the source channel to read all the bytes ? */

top page on google for ReadableByteChannel has what looks like a working example http://www.exampledepot.com/egs/java.nio/ReadChannel.html you need to reset() then read() then reset() then write(). — Darryl Miles, Sep 20 '12 at 06:50
Hi Darryl, I edited my original post to include the implementation as suggested at the link provided by you. But it doesn't seem to 'loop' - with whatever size I initialize the 'chunk', the destination file is of same size. The output is : numRead = 96 numRead = -1 Am I missing something here? — Kaliyug Antagonist, Sep 20 '12 at 09:08
FWIW why are you using non-blocking IO inside what looks like a Servlet ? you are aware the Servlet API is blocking ? You are trying to build a NIO scheme on top of a blocking IO handle. To do NIO you must use OS non-blocking IO handles as the basis to build on top of. You are better off using one of the new WebSocket Servlet APIs for a non-blocking IO model. — Darryl Miles, Sep 20 '12 at 09:34
Using InputStream as-is is perfectly fine in a Servlet since the IO model is one thread per active request.The only reason to consider NIO is if you had something useful to do from the same thread at the same time in between reading/writing IO. Your example does not demonstrate this need, it also does not check how many bytes written successfully nor does it have a strategy for handling backlog (where input IO is faster than output IO you are stuck holding data).I think your attempt to use NIO here is misguided possibly around some "uncomfortable" feeling you have, a shrink is best for that. — Darryl Miles, Sep 20 '12 at 09:45
I'm not gonna write the I/O code in the Servlet in the final code, here, I have written in the doPost(...) only for getting my approach clear. Actually, the question that I have asked is in the following context : http://stackoverflow.com/questions/12500185/upload-of-huge-file-using-a-web-application Please check the same and guide me. — Kaliyug Antagonist, Sep 20 '12 at 10:59

Darryl Miles · Accepted Answer · 2012-09-24T07:50:55.467

OR perform IO in chunks of more than 1 byte the API like so:

byte[] bA = new byte[4096];
int i;
InputStream is = ....;
OutputStream os = ....;

while((i = is.read(bA) != -1){
    os.write(bA, 0, i);
}

I've looked at your other question and my comments still stand. NIO is not the solution you are looking for. You have a low end machine with limits RAM acting as a proxy.

The best you can do is have your Servlet create a new thread, have this thread create and setup an outgoing connection using NIO sockets/HTTP-libraries. This new (and extra) thread is waiting on any of 3 things to happen and it pushes whatever APIs to try and make progress in these 3 areas.

The 3 things are:

Trying to write data to the remote server (if there is buffered in memory data to send)
Waiting for the main Servlet thread to indicate there is new data in the shared buffer. Or that End-of-stream was reached.
Waiting for the main Servlet thread to indicate the extra thread needs to shutdown (this is error recovery and cleanup).

You probably need a drainWithTimeout(long millis) function that the doPost() method calls on the extra thread to give it an amount of time to push the final data to the remote server. This gets called when an End-of-Stream if observed by the Servlet from the InputStream.

You MUST ensure your extra thread is 100% reliably reaped before the doPost() method returns. So controlling startup/shutdown of it is important, especially in the scenarios that the InputStream had an error because the sending client disconnected or was idle too long.

Then two threads (the normal Servlet thread in doPost() and the new thread you create) would setup and share some arbitrary memory buffer, maybe 16Mb or more that is shared.

If you can not have a 16Mb buffer due to limitations in clients/concurrent-users and 2Gb RAM then you really should stick with the example code at the top of this answer, since the network and the O/S kernels will already buffer some Mb's of data.

The point of using two threads is that you can not fix the issue that the Servlet API receiving the data is a blocking I/O API, you can not change that if you are writing the application to conform to Servlet specification/standards. If you know your specific Servlet container has a feature then that is outside the scope of this answer.

The two threads allow the main Servlet doPost thread to be in control and STILL use a blocking I/O API for InputStream.

There is no point using one thread and a blocking InputStream with a non-blocking OutputStream, you still have the problem that you can not service the output stream while the in.read() API call is blocked (waiting for more data or End-of-stream).

This code does not compile. OutputStream.write() does not return a byte count value or -1: it is void, and it blocks until all the data has been written. There is no such thing as a non-blocking OutputStream. — user207421, Sep 20 '12 at 23:06
Yes of course you are correct there, I'd hope the question asker could read between the lines with untested examples provided. I've amended to comment out those parts. The question asker did not provide any particular output API so I obliged with OutputStream but the NIO APIs will have a bytes written return count if the question asker is still sure they want to include NIO in their design. — Darryl Miles, Sep 21 '12 at 06:58
How about *removing* the junk from your code, instead of just commenting it out? It's never going back; it was always wrong: lose it. — user207421, Sep 21 '12 at 10:03
I need some time with all the inputs provided here - will be back ! Thanks everyone :) — Kaliyug Antagonist, Sep 24 '12 at 06:32

score 0 · Answer 2 · answered Sep 24 '12 at 10:59

The correct way to copy between NIO channels is as follows:

while (in.read(buffer) > 0 || buffer.position() > 0)
{
  buffer.flip();
  out.write(buffer);
  buffer.compact();
}

Note that this automatically takes care of EOS, partial reads, and partial writes.

Looping over byte channels

2 Answers2

Linked