Large file transfer over java socket

Question

I have written a small client-server code for transferring small file. It uses Data output stream and readFully() method of data input stream. This code does not work for larger files for obvious reasons. I was thinking of fragmenting large files into smaller chunks of 1Kb each before sending them to client. But I can't think of any solution (like how to write multiple chunks on data output stream with correct offset and how to reassemble them at receiving end. Can anyone provide a workaround? It would be very helpful if you could modify my code:

Sender (Server):

public void sendFileDOS() throws FileNotFoundException {
    runOnUiThread( new Runnable() {
          @Override
          public void run() {
              registerLog("Sending. . . Please wait. . .");
          }
        });
    final long startTime = System.currentTimeMillis();
    final File myFile= new File(filePath); //sdcard/DCIM.JPG
    byte[] mybytearray = new byte[(int) myFile.length()];
    FileInputStream fis = new FileInputStream(myFile);  
    BufferedInputStream bis = new BufferedInputStream(fis);
    DataInputStream dis = new DataInputStream(bis);
    try {
        dis.readFully(mybytearray, 0, mybytearray.length);
        OutputStream os = socket.getOutputStream();
        //Sending file name and file size to the client  
        DataOutputStream dos = new DataOutputStream(os);     
        dos.writeUTF(myFile.getName());     
        dos.writeLong(mybytearray.length);     
        int i = 0;
        final ProgressBar myProgBar=(ProgressBar)findViewById(R.id.progress_bar);
        while (i<100) {
            dos.write(mybytearray, i*(mybytearray.length/100), mybytearray.length/100);
            final int c=i;
            runOnUiThread( new Runnable() {
                  @Override
                  public void run() {
                      myProgBar.setVisibility(View.VISIBLE);
                      registerLog("Completed: "+c+"%");
                      myProgBar.setProgress(c);
                      if (c==99)
                          myProgBar.setVisibility(View.INVISIBLE);
                  }
                });
            i++;
        }    
        dos.flush();

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    runOnUiThread( new Runnable() {
          @Override
          public void run() {
              long estimatedTime = (System.currentTimeMillis() - startTime)/1000;
              registerLog("File successfully sent");
              registerLog("File size: "+myFile.length()/1000+" KBytes");
              registerLog("Elapsed time: "+estimatedTime+" sec. (approx)");
              registerLog("Server stopped. Please restart for another session.");
              final Button startServerButton=(Button)findViewById(R.id.button1);
              startServerButton.setText("Restart file server");
          }
        });
}

Receiver (Client):

public class myFileClient {
final static String servAdd="10.141.21.145";
static String filename=null;
static Socket socket = null;
static Boolean flag=true;

/**
 * @param args
 */
public static void main(String[] args) throws IOException {
    // TODO Auto-generated method stub
    initializeClient();
    receiveDOS();      
}
public static void initializeClient () throws IOException {
    InetAddress serverIP=InetAddress.getByName(servAdd);
    socket=new Socket(serverIP, 4444);
}
public static void receiveDOS() {
    int bytesRead;
    InputStream in;
    int bufferSize=0;

    try {
        bufferSize=socket.getReceiveBufferSize();
        in=socket.getInputStream();
        DataInputStream clientData = new DataInputStream(in);
        String fileName = clientData.readUTF();
        System.out.println(fileName);
        OutputStream output = new FileOutputStream("//home//evinish//Documents//Android//Received files//"+ fileName);
        long size = clientData.readLong();
        byte[] buffer = new byte[bufferSize];
        while (size > 0
                && (bytesRead = clientData.read(buffer, 0,
                        (int) Math.min(buffer.length, size))) != -1) {
            output.write(buffer, 0, bytesRead);
            size -= bytesRead;
        }

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

Please help! Thanks in advance! :)

look here http://stackoverflow.com/questions/5113914/large-file-transfer-with-sockets?rq=1 — XXX, Jun 24 '13 at 22:06

user207421 · Accepted Answer · 2013-07-19T05:26:40.767

6

You're right, this is a poor way to do it. It wastes both memory and time; it assumes the file size is 32 bits; it assumes the entire file fits into memory; it assumes the entire file is read in one read; and it doesn't send anything until the entire file has been read.

The canonical way to copy a stream in Java is this:

while ((count = in.read(buffer)) > 0)
{
  out.write(buffer, 0, count);
}

It will work with any size buffer you like and therefore with any size file you can come up with. Use the same code at both ends, although you don't have to use the same size buffer at both ends. As you're copying over a network you might think that 1k or 1.5k is the best size, but that overlooks the presence of the socket send and receive buffers in the kernel. When you take them into account it is probably better to use 8k or more.

edited Jul 19 '13 at 05:26

answered Jun 24 '13 at 23:48

user207421

305,947
44
307
483

Thanx! Let me re-write my code! Will get back to you with results. :) – Vinit Shandilya Jun 25 '13 at 06:20
:: I tried the above method, however, the received file is incomplete. I'm not sure how to reassemble the packets at receiver. Can you please modify the sender and receiver code as per your suggestion? My mind is completely blocked and nothing seems to work! :( – Vinit Shandilya Jun 25 '13 at 08:42
You don't have to 'reassemble the packets at the receiver'. Just write them to your file. Same code at both ends. I've already said that. – user207421 Jun 25 '13 at 10:36
@MarquisofLorne when 50% transfer is done the server closes. I don't know why but on 50% the count value is -1 with every file. please help. – The Destroyer Nov 24 '20 at 04:37
I tried above method, but the received file is courepted (over sized then original file). please help me. I'm stuck from more then a month. sorry for bad English. – The Destroyer Dec 23 '20 at 11:05

Vinit Shandilya · Answer 2 · 2013-07-19T07:21:21.780

5

I finally solved the problem. Here is my modified source code for server and client. Hope this would help other people too! :) Server Side code snippet (sender):

final File myFile= new File(filePath); //sdcard/DCIM.JPG
    byte[] mybytearray = new byte[8192];
    FileInputStream fis = new FileInputStream(myFile);  
    BufferedInputStream bis = new BufferedInputStream(fis);
    DataInputStream dis = new DataInputStream(bis);
    OutputStream os;
    try {
        os = socket.getOutputStream();
        DataOutputStream dos = new DataOutputStream(os);
        dos.writeUTF(myFile.getName());     
        dos.writeLong(mybytearray.length);
        int read;
        while((read = dis.read(mybytearray)) != -1){
            dos.write(mybytearray, 0, read);
        }

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

Client side code snippet (Receiver):

int bytesRead;
    InputStream in;
    int bufferSize=0;

    try {
        bufferSize=socket.getReceiveBufferSize();
        in=socket.getInputStream();
        DataInputStream clientData = new DataInputStream(in);
        String fileName = clientData.readUTF();
        System.out.println(fileName);
        OutputStream output = new FileOutputStream("//home//evinish//Documents//Android//Received files//"+ fileName);
        byte[] buffer = new byte[bufferSize];
        int read;
        while((read = clientData.read(buffer)) != -1){
            output.write(buffer, 0, read);
        }

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

A bit faster way of writing to output stream:

long acc=0;
long N=myFile.length();
 while(acc<N){
            noofbytes=dis.read(mybytearray, 0, 16384);
            dos.write(mybytearray, 0, noofbytes);
            acc=acc+noofbytes; } dos.flush();

I saved around 7 seconds while transferring a video file of 72MB.

edited Jul 19 '13 at 07:21

answered Jun 26 '13 at 12:21

Vinit Shandilya

1,643
5
24
44

This code does not work. You are, pointlessly, writing the buffer size with writeLong() but never reading it, so you are writing the extra 8 bytes to the file instead. You don't need to double the /'s either. The rest of it is substantially the same as my answer, without acknowledgement. – user207421 Jul 19 '13 at 05:48
Well, you're right, I'm writing file size pointlessly, I had included that for testing purpose and forgot to remove it. But for the rest of the code, it's working absolutely fine. As a matter of fact, I have devised a new solution for writing to output stream which is a bit faster. You can refer to edit section. – Vinit Shandilya Jul 19 '13 at 07:22
You're dreaming. I gave you working code weeks ago. The only way you can make that faster is by using larger buffers. Your edit is invalid. Limiting the read to 16384 bytes won't make it any faster. You should let it fill the input buffer if it can. Not testing for -1 is positively unsafe: you are in danger of getting an `ArrayIndexOutOfBoundsException` with that code, and also of getting an incorrect value for 'acc', which will cause the `while` condition to malfuntion. You're just making it worse. You still haven't fixed the `//`s either. – user207421 Jul 19 '13 at 07:30
Can you please explain- "you are in danger of getting an ArrayIndexOutOfBoundsException with that code, and also of getting an incorrect value for 'acc', which will cause the while condition to malfuntion."? noofbytes will always return the actual amount of data read (no garbage whatsoever). At max, the buffer can read up to 16K, but in the end if left-over size is less thn 16K, the noofbytes would return the actual number of bytes read. And, yes! I did a comparison of different buffer sizes and found 16K is working best in my case (1 GHz dual core with 1 GB RAM) – Vinit Shandilya Jul 19 '13 at 07:50
You are in danger of getting an `ArrayIndexOutOfBoundsException` because it is possible to pass -1 as the third parameter to `write()`, which will throw that exception under that condition. You will also subtract 1 from `acc` instead of adding to it if `read()` returns -1. There is no advantage to taking the size of the file first and then looping until you've read exactly that many bytes. What if the file size changes? If your buffer is 16k you shouldn't specify it twice in your code: just use `read(buffer)`, which automatically adjusts itself to whatever the buffer size actually is. – user207421 Jul 19 '13 at 08:06
1

Use clientData.readLong(); for file size, your code works beautifully, thank you very much :) – Ben Jan 03 '14 at 16:25
@SB He isn't sending the file size. He is sending his own buffer size, which is pointless. – user207421 Aug 24 '20 at 07:15

Large file transfer over java socket

2 Answers2

Linked