Java NIO: transferFrom until end of stream

Question

I'm playing around with the NIO library. I'm attempting to listen for a connection on port 8888 and once a connection is accepted, dump everything from that channel to somefile.

I know how to do it with ByteBuffers, but I'd like to get it working with the allegedly super efficient FileChannel.transferFrom.

This is what I got:

ServerSocketChannel ssChannel = ServerSocketChannel.open();
ssChannel.socket().bind(new InetSocketAddress(8888));

SocketChannel sChannel = ssChannel.accept();
FileChannel out = new FileOutputStream("somefile").getChannel();

while (... sChannel has not reached the end of the stream ...)     <-- what to put here?
    out.transferFrom(sChannel, out.position(), BUF_SIZE);

out.close();

So, my question is: How do I express "transferFrom some channel until end-of-stream is reached"?

---

^{Edit: Changed 1024 to BUF_SIZE, since the size of the buffer used, is irrelevant for the question.}

Answer 1

There are few ways to handle the case. Some background info how trasnferTo/From is implemented internally and when it can be superior.

• 1st and foremost you should know how many bytes you have to xfer, i.e. use FileChannel.size() to determine the max available and sum the result. The case refers to FileChannel.trasnferTo(socketChanel)
• The method does not return -1
• The method is emulated on Windows. Windows doesn't have an API function to xfer from filedescriptor to socket, it does have one (two) to xfer from the file designated by name - but that's incompatible with java API.
• On Linux the standard sendfile (or sendfile64) is used, on Solaris it's called sendfilev64.

in short for (long xferBytes=0; startPos + xferBytes<fchannel.size();) doXfer() will work for transfer from file -> socket. There is no OS function that transfers from socket to file (which the OP is interested in). Since the socket data is not int he OS cache it can't be done so effectively, it's emulated. The best way to implement the copy is via standard loop using a polled direct ByteBuffer sized with the socket read buffer. Since I use only non-blocking IO that involves a selector as well.

That being said: I'd like to get it working with the allegedly super efficient "? - it is not efficient and it's emulated on all OSes, hence it will end up the transfer when the socket is closed gracefully or not. The function will not even throw the inherited IOException, provided there was ANY transfer (If the socket was readable and open).

I hope the answer is clear: the only interesting use of File.transferFrom happens when the source is a file. The most efficient (and interesting case) is file->socket and file->file is implemented via filechanel.map/unmap(!!).

Answer 2

I'm not sure, but the JavaDoc says:

An attempt is made to read up to count bytes from the source channel and write them to this channel's file starting at the given position. An invocation of this method may or may not transfer all of the requested bytes; whether or not it does so depends upon the natures and states of the channels. Fewer than the requested number of bytes will be transferred if the source channel has fewer than count bytes remaining, or if the source channel is non-blocking and has fewer than count bytes immediately available in its input buffer.

I think you may say that telling it to copy infinite bytes (of course not in a loop) will do the job:

out.transferFrom(sChannel, out.position(), Integer.MAX_VALUE);

So, I guess when the socket connection is closed, the state will get changed, which will stop the transferFrom method.

But as I already said: I'm not sure.

Answer 3

Answering your question directly:

while( (count = socketChannel.read(this.readBuffer) )  >= 0) {
   /// do something
}

But if this is what you do you do not use any benefits of non-blocking IO because you actually use it exactly as blocking IO. The point of non-blocking IO is that 1 network thread can serve several clients simultaneously: if there is nothing to read from one channel (i.e. count == 0) you can switch to other channel (that belongs to other client connection).

So, the loop should actually iterate different channels instead of reading from one channel until it is over.

Take a look on this tutorial: http://rox-xmlrpc.sourceforge.net/niotut/ I believe it will help you to understand the issue.

Answer 4

allegedly super efficient FileChannel.transferFrom.

If you want both the benefits of DMA access and nonblocking IO the best way is to memory-map the file and then just read from the socket into the memory mapped buffers.

But that requires that you preallocate the file.

Answer 5

This way:

URLConnection connection = new URL("target").openConnection();
File file = new File(connection.getURL().getPath().substring(1));
FileChannel download = new FileOutputStream(file).getChannel();

while(download.transferFrom(Channels.newChannel(connection.getInputStream()),
        file.length(), 1024) > 0) {
    //Some calculs to get current speed ;)
}

Answer 6

Building on top of what other people here have written, here's a simple helper method which accomplishes the goal:

public static void transferFully(FileChannel fileChannel, ReadableByteChannel sourceChannel, long totalSize) {
    for (long bytesWritten = 0; bytesWritten < totalSize;) {
        bytesWritten += fileChannel.transferFrom(sourceChannel, bytesWritten, totalSize - bytesWritten);
    }
}

Answer 7

transferFrom() returns a count. Just keep calling it, advancing the position/offset, until it returns zero. But start with a much larger count than 1024, more like a megabyte or two, otherwise you're not getting much benefit from this method.

EDIT To address all the commentary below, the documentation says that "Fewer than the requested number of bytes will be transferred if the source channel has fewer than count bytes remaining, or if the source channel is non-blocking and has fewer than count bytes immediately available in its input buffer." So provided you are in blocking mode it won't return zero until there is nothing left in the source. So looping until it returns zero is valid.

EDIT 2

The transfer methods are certainly mis-designed. They should have been designed to return -1 at end of stream, like all the read() methods.