How could I upload a large file in chunks using Java?

Question

I need to upload a large file in chunks using Java.

Is there any sample code I can refer to?

Just set chunked transfer mode on the `HttpURLConnection`. – user207421 Feb 26 '20 at 22:14 — user207421, Feb 26 '20 at 22:14

score 5 · Answer 1 · answered Apr 08 '15 at 19:12

*It can be done using plupload. Here is sample. My index.html is as under:-

    <!DOCTYPE html>
    <html>
    <head>
    <meta http-equiv="content-type" content="text/html; charset=UTF-8"/>
    <title>Upload</title>
    <!-- production -->
    <script type="text/javascript" src="js/plupload.full.min.js"></script>
    <!-- debug 
    <script type="text/javascript" src="../js/moxie.js"></script>
    <script type="text/javascript" src="../js/plupload.dev.js"></script>
    -->
    </head>
    <body style="font: 13px Verdana; background: #eee; color: #333">
    <div id="filelist"></div>
    <br />
        <button id="pickfiles" >Select file</button> 
        <button id="uploadfiles" >Upload</button>
        <div id="container">
    </div>
    <br />
    <pre id="console"></pre>
    <script type="text/javascript">
    // Custom example logic
    var uploader = new plupload.Uploader({
        runtimes : 'html5',
        browse_button : 'pickfiles', // you can pass an id...
        container: document.getElementById('container'), // ... or DOM Element itself
        url : 'UploadAction',//upload.php
        chunk_size : '1mb',
        method:'POST',
        flash_swf_url : 'js/Moxie.swf',
        silverlight_xap_url : 'js/Moxie.xap',

        filters : {
            max_file_size : '100gb',
            mime_types: [
                {title : "Image files", extensions : "jpg,gif,png"},
                {title : "Zip files", extensions : "zip,txt,vmdk"}
            ]
        },
        init: {
            PostInit: function() {
                document.getElementById('filelist').innerHTML = '';
                document.getElementById('uploadfiles').onclick = function() {
                    uploader.start();
                    return false;
                };
            },
            FilesAdded: function(up, files) {
                plupload.each(files, function(file) {
                    document.getElementById('filelist').innerHTML += '<div id="' + file.id + '">' + file.name + ' (' + plupload.formatSize(file.size) + ') <b></b></div>';
                });
            },
            UploadProgress: function(up, file) {
                document.getElementById(file.id).getElementsByTagName('b')[0].innerHTML = '<span>' + file.percent + "%</span>";
            },
            Error: function(up, err) {
                document.getElementById('console').appendChild(document.createTextNode("\nError #" + err.code + ": " + err.message));
            }
        }
    });
    uploader.init();
    </script>
    </body>
    </html>
<!-- end snippet -->

My java backend code(Servlet) is as under:-

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.commons.fileupload.FileItemIterator;
import org.apache.commons.fileupload.FileItemStream;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.fileupload.util.Streams;
public class UploadAction extends HttpServlet {
    private static final long serialVersionUID = 3447685998419256747L;
    private static final String RESP_SUCCESS = "{\"jsonrpc\" : \"2.0\", \"result\" : \"success\", \"id\" : \"id\"}";
    private static final String RESP_ERROR = "{\"jsonrpc\" : \"2.0\", \"error\" : {\"code\": 101, \"message\": \"Failed to open input stream.\"}, \"id\" : \"id\"}";
    public static final String JSON = "application/json";
    public static final int BUF_SIZE = 2 * 1024;
    public static final String FileDir = "/home/asjha/uploads/";

    private int chunk;
    private int chunks;
    private String name;
    private String user;
    private String time;
    protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        String responseString = RESP_SUCCESS;
        boolean isMultipart = ServletFileUpload.isMultipartContent(req);

        if(isMultipart){
            ServletFileUpload upload = new ServletFileUpload();
            try {
                FileItemIterator iter = upload.getItemIterator(req);
                while (iter.hasNext()) {
                    FileItemStream item = iter.next();
                    InputStream input = item.openStream();
                    // Handle a form field.
                    if(item.isFormField()){
                        String fileName = item.getFieldName();
                        String value = Streams.asString(input);
                        if("name".equals(fileName)){
                            this.name = value;
                        }else if("chunks".equals(fileName)){
                            this.chunks = Integer.parseInt(value);
                        }else if("chunk".equals(fileName)){
                            this.chunk = Integer.parseInt(value);
                        }else if("user".equals(fileName)){
                            this.user = value;
                        }else if("time".equals(fileName)){
                            this.time = value;
                        }
                    }

                    // Handle a multi-part MIME encoded file.
                    else {
                        File dstFile = new File(FileDir);
                        if (!dstFile.exists()){
                            dstFile.mkdirs();
                        }

                        File dst = new File(dstFile.getPath()+ "/" + this.name);

                        saveUploadFile(input, dst);
                    }
                }
            }
            catch (Exception e) {
                responseString = RESP_ERROR;
                e.printStackTrace();
            }
        }

        // Not a multi-part MIME request.
        else {
            responseString = RESP_ERROR;
        }

        if(this.chunk == this.chunks - 1){
            System.out.println("name"+this.name);
        }
        resp.setContentType(JSON);
        byte[] responseBytes = responseString.getBytes();
        resp.setContentLength(responseBytes.length);
        ServletOutputStream output = resp.getOutputStream();
        output.write(responseBytes);
        output.flush();
    }
    private void saveUploadFile(InputStream input, File dst) throws IOException {
        OutputStream out = null;
        try {
            if (dst.exists()) {
                out = new BufferedOutputStream(new FileOutputStream(dst, true),
                        BUF_SIZE);
            } else {
                out = new BufferedOutputStream(new FileOutputStream(dst),
                        BUF_SIZE);
            }
            byte[] buffer = new byte[BUF_SIZE];
            int len = 0;
            while ((len = input.read(buffer)) > 0) {
                out.write(buffer, 0, len);
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            if (null != input) {
                try {
                    input.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            if (null != out) {
                try {
                    out.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

Please refer to plupload for details and on github you can see sample projects by jakobadam and rocky.

Please let me know if multiple file upload is required. Using plupload we can upload any number of files of any sizes. This sample is for single file upload of very large size. dont forget to include plupload.full.min.js. Hope this helps*emphasized text**

Apache commons can also be used if we dont have any timeout issue like with nginx etc.. Plupload provides many ways to upload using flash,applet etc.. — Ashutosh Jha, Apr 08 '15 at 19:16

score 2 · Answer 2 · answered Aug 28 '18 at 13:26

Here is an example of native Java code that uploads a file using chunks:

final String       LF = "\r\n"; // Line separator required by multipart/form-data. Can be static class constant
final String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.

HttpURLConnection connection = (HttpURLConnection) new URL("http://some.com/upload").openConnection();
try {
    connection.setDoOutput(true);
    connection.setChunkedStreamingMode(4096);
    connection.setRequestMethod("POST");
    connection.addRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
    connection.addRequestProperty("Accept", "application/json");
    connection.addRequestProperty("Authorization", myToken);
    try (OutputStream os = connection.getOutputStream();
            Writer writer = new OutputStreamWriter(os, StandardCharsets.UTF_8)) {
        writer.append("--").append(boundary).append(LF);
        writer.append("Content-Disposition: form-data; name=\"dataFile\"; filename=\"file.zip\"").append(LF);
        writer.append("Content-Type: application/zip").append(LF);
        writer.append(LF);
        writer.flush();

        // Write body
        writeBinaryBody(os);
        writer.append(LF).append("--").append(boundary).append("--").append(LF);
        writer.flush();
        os.flush();
    }
    if (200 != connection.getResponseCode()) {
        try (Reader reader = new InputStreamReader(connection.getInputStream(), StandardCharsets.UTF_8)) {
            // Handle error here
        }
    }
} finally {
    connection.disconnect();
}

The code in this example is based on this answer about http file upload from java. The difference is the call to connection.setChunkedStreamingMode(4096); that defines that chunked streaming should be used.

@Tunelt I think that qualifies as an entirely different question. Please post a new question. I do not have the answer for that but I would be interested if there is a solution. — stenix, Feb 27 '20 at 06:13
I am trying to use this solution but getting network error. Probably reason is implementation of writeBinaryBody() method that is missing from this solution. Can you please edit and complete this solution to write a file in chunk ? — Rex, Feb 24 '21 at 14:07
I want to add request body example "api_key", What should I do? Thank you — Ben Jima, Apr 03 '23 at 18:02
@BenJima The binary body is the body and you cannot have more than one body in a request. You can however add http-header items for information such as "api_key". "Content-Type" is an example of a header. — stenix, Apr 26 '23 at 08:35

score 0 · Answer 3 · answered May 20 '11 at 16:16

0

You can simply break the file up yourself, send it using the Socket API, and re-assemble the file.

answered May 20 '11 at 16:16

Amir Afghani

37,814
16
84
124

score 0 · Answer 4 · answered May 20 '11 at 17:39

0

Try Apache Commons upload. It supports streaming, may be suitable for you.

answered May 20 '11 at 17:39

Jayan

18,003
15
89
143

score -1 · Answer 5 · edited May 23 '17 at 12:23

-1

Use a RandomAccessFile. This has already been covered on SO I believe.

java file input with rewind()/reset() capability

Basically you'd just seek to the starting point, write however many bytes you want to from there, and remember the point you stopped writing from.

edited May 23 '17 at 12:23

Community

1
1

answered May 20 '11 at 16:13

KyleM

4,445
9
46
78

Huh? This is totally unnecessary. You don't need to do any of that. I think the question is asking about the client side not the server side, by the way. – Robin Green May 20 '11 at 16:22
@Robin I guess the OP will decide that, but I may have misunderstood the question. – KyleM May 20 '11 at 16:30

How could I upload a large file in chunks using Java?

5 Answers5

Linked