Java TCP-Sockets transmit files larger than 4gb

Question

I am trying to transfer a file that is greater than 4gb using the Java SocketsAPI. I am already reading it via InputStreams and writing it via OutputStreams. However, analyzing the transmitted packets in Wireshark, I realise that the Sequence number of the TCP-packets is incremented by the byte-length of the packet, which seems to be 1440byte.

This leads to the behavior that when I try to send a file greater than 4gb, the total size of the Sequence-Number field of TCP is exceeded, leading to lots of error packages, but no error in Java.

My code for transmission currently looks like this:

 DataOutputStream fileTransmissionStream = new DataOutputStream(transmissionSocket.getOutputStream());
                    FileInputStream fis = new FileInputStream(toBeSent);
                    int totalFileSize = fis.available();
                    fileTransmissionStream.writeInt(totalFileSize);
                    while (totalFileSize >0){
                        if(totalFileSize >= FileTransmissionManagementService.splittedTransmissionSize){
                            sendBytes = new byte[FileTransmissionManagementService.splittedTransmissionSize];
                            fis.read(sendBytes);
                            totalFileSize -= FileTransmissionManagementService.splittedTransmissionSize;
                        } else {
                            sendBytes = new byte[totalFileSize];
                            fis.read(sendBytes);
                            totalFileSize = 0;
                        }
                        byte[] encryptedBytes = DataEncryptor.encrypt(sendBytes);
                        /*byte[] bytesx = ByteBuffer.allocate(4).putInt(encryptedBytes.length).array();
                        fileTransmissionStream.write(bytesx,0,4);*/
                        fileTransmissionStream.writeInt(encryptedBytes.length);
                        fileTransmissionStream.write(encryptedBytes, 0, encryptedBytes.length);

What exactly have I done wrong in this situation, or is it not possible to transmit files greater than 4gb via one Socket?

You realize that `FileInputStream.available()` is **not** the total length of the file, right? — RealSkeptic, May 27 '15 at 08:17
See also [Java multiple file transfer over socket](http://stackoverflow.com/questions/10367698/java-multiple-file-transfer-over-socket). — user207421, May 27 '15 at 08:17

user207421 · Accepted Answer · 2015-05-27T08:21:07.097

TCP can handle infinitely long data streams. There is no problem with the sequence number wrapping around. As it is initially random, that can happen almost immediately, regardless of the length of the stream. The problems are in your code:

DataOutputStream fileTransmissionStream = new DataOutputStream(transmissionSocket.getOutputStream());
FileInputStream fis = new FileInputStream(toBeSent);
int totalFileSize = fis.available();

Classic misuse of available(). Have a look at the Javadoc and see what it's really for. This is also where your basic problem lies, as values > 2G don't fit into an int, so there is a truncation. You should be using File.length(), and storing it into a long.

fileTransmissionStream.writeInt(totalFileSize);
while (totalFileSize >0){
    if(totalFileSize >= FileTransmissionManagementService.splittedTransmissionSize){
        sendBytes = new byte[FileTransmissionManagementService.splittedTransmissionSize];
        fis.read(sendBytes);

Here you are ignoring the result of read() here. It isn't guaranteed to fill the buffer: that's why it returns a value. See, again, the Javadoc.

        totalFileSize -= FileTransmissionManagementService.splittedTransmissionSize;
    } else {
        sendBytes = new byte[totalFileSize];

Here you are assuming the file size fits into an int, and assuming the bytes fit into memory.

        fis.read(sendBytes);

See above re read().

        totalFileSize = 0;
    }
    byte[] encryptedBytes = DataEncryptor.encrypt(sendBytes);
    /*byte[] bytesx = ByteBuffer.allocate(4).putInt(encryptedBytes.length).array();
    fileTransmissionStream.write(bytesx,0,4);*/

We're not interested in your commented-out code.

    fileTransmissionStream.writeInt(encryptedBytes.length);
    fileTransmissionStream.write(encryptedBytes, 0, encryptedBytes.length);

You don't need all this crud. Use a CipherOutputStream to take care of the encryption, or better still SSL, and use the following copy loop:

byte[] buffer = new byte[8192]; // or much more if you like, but there are diminishing returns
int count;
while ((count = in.read(buffer)) > 0)
{
    out.write(buffer, 0, count);
}

score 0 · Answer 2 · answered May 27 '15 at 08:28

It seems that your protocol for the transmission is:

Send total file length in an int.
For each bunch of bytes read,
1. Send number of encrypted bytes ahead in an int,
2. Send the entrypted bytes themselves.

The basic problem, beyond the misinterpretations of the documentation that were pointed out in @EJP's answer, is with this very protocol.

You assume that the file length can be sent oven in an int. This means the length it sends cannot be more than Integer.MAX_VALUE. Of course, this limits you to files of 2G length (remember Java integers are signed).

If you take a look at the Files.size() method, which is a method for getting the actual file size in bytes, you'll see that it returns long. A long will accommodate files larger than 2GB, and larger than 4GB. So in fact, your protocol should at the very least be defined to start with a long rather than an int field.

The size problem really has nothing at all to do with the TCP packets.

Java TCP-Sockets transmit files larger than 4gb

2 Answers2