Sending a large file over TCP channel with serialization

Question

I'm developing a client-server system in Java. I'm sending the messaeges over a TCP channel with serialized objects.

I will also need to send files. I could just define my own custom message class:

public class SendFile implements Serializable {
    private byte[] fileContents;

    ...
}

and send it. This poses a problem if we have a large file, as from what I understand it will try to have everything in memory before sending the file to the other side.

I've read about Externalizable but that won't be any better if the file is still being read to all my computer's memory before sending it over the TCP channel. I wanted some "lazy" way to do it (reading it from disk as I'm sending it over the network, and discarding from memory what was already read).

What's the cleanest way of handling this issue? In the worst case, I could just divide this into small chunks, but I'd like to avoid it if possible.

EDIT:

Of course I could just open a stream on the file and "send it". But that would imply treating sending a file and all the other messages in distinct ways, something I'm looking for to avoid.

Thanks

Couldn't you just have a tuneably-sized read ahead cache to keep only a few steps a head of the packets with the disk access? — Cris Stringfellow, Feb 28 '12 at 10:11
I'm not sure I've understood you. When we have a serializable object, the whole object is serialized before sending it over to the other side, right? — devoured elysium, Feb 28 '12 at 10:13
Yes if you serialize it automagically. But if you do it yourself...Why is tht your 'worst case'? — Cris Stringfellow, Feb 28 '12 at 10:18

score 2 · Answer 1 · answered Feb 28 '12 at 10:24

2

One way to get around it to use streaming. Streaming approach does not load the entire contents in memory. You can open two streams, input-stream from the object to be sent over the network and output-stream for the TCP socket connection.

In java, both the InputStream and OutputStream. support reading and writing of bytes.

You can define a buffer size (a byte array). Read (from file) the content into that buffer using InputStream.read(byte[] b) and write (to TCP socket) the same buffer to output-stream using OutputStream.write(byte[] b)

answered Feb 28 '12 at 10:24

Santosh

17,667
4
54
79

That is the obvious answer. But that would imply treating in a different way this message or the other kinds of messages, which is something I'm looking to avoid. – devoured elysium Feb 28 '12 at 10:30
why not just stream them all....you are writing a custom low level socket based app on the stack aren't you? Maybe the original TCP rfc woud be useful http://op.to/42Kb/plain/text/ietf+ – Cris Stringfellow Mar 01 '12 at 14:21

score 0 · Answer 2 · answered Mar 02 '12 at 19:13

0

Instead of using Java Serilization, you can directly read the file in Java in a buffer and send that buffer over the network. Why involve java serialization in b/w and complicate stuff. Using simple n/w program, you will be able to send your files.

answered Mar 02 '12 at 19:13

MoveFast

3,011
2
27
53

Sending a large file over TCP channel with serialization

EDIT:

2 Answers2