Uploading chunked files directly to Amazon s3

Question

I'm looking for any straight forward examples on uploading directly to Amazon s3 in chunks without any server side processing (aside signing the request)

I've looked into many options and so far all examples either address just the chunking from the server or just sending to s3 from browser as a single put, or are so old they just don't work anymore.

Highest hope was Plupload, but I can't find any documentation for splitting large files into chunks, at least not in the amazon example.

Ultimately the goal is to send a 500mb file to s3 in under 5 seconds. Using the server in php I am able to get close to 10 seconds, but the client wants to avoid server processing entirely and go straight to s3, so those are the rules I'm bound by.

I find Amazon documentation highly convoluted and difficult to follow. Does anyone out there know of a complete example of browser chunking to s3?

Basic idea of Plupload settings:

$("#uploader").plupload({
    runtimes : 'html5,flash,silverlight',
    url : 'http://<?php echo $bucket; ?>.s3.amazonaws.com/',

    multipart: true,
    multipart_params: {
        'key': '${filename}', // use filename as a key
        'Filename': '${filename}', // adding this to keep consistency across the runtimes
        'acl': 'public-read',
        'Content-Type': 'Binary/Octet-Stream',
        'AWSAccessKeyId' : '<?php echo $accessKeyId; ?>',       
        'policy': '<?php echo $policy; ?>',
        'signature': '<?php echo $signature; ?>',
        'Multiple_queues': True,
    },
    file_data_name: 'file',
    filters : {
        // Maximum file size (apparently not per chunk)
        max_file_size : '5mb',
    },
    flash_swf_url : '../js/Moxie.swf',
    silverlight_xap_url : '../js/Moxie.xap'
});

It seems multipart is not multiple chunks, but maybe multiple files total. Not sure, I only need to send one.

Any input is appreciated.

Just to clarify... do you wish to send the file to S3 from the client (user's) computer, or from the server? If the latter, have you looked at the [`Transfer` class](https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.S3.Transfer.html) in the AWS SDK for PHP? It supports multipart upload. — John Rotenstein, Mar 12 '16 at 01:33
From the client. The standard AWS library handles server transfers straight out of the box, but inconveniently glosses over chunking from client. — Kai Qing, Mar 12 '16 at 15:41
Perhaps [Announcing the Amazon S3 Managed Uploader in the AWS SDK for JavaScript](https://blogs.aws.amazon.com/javascript/post/Tx3EQZP53BODXWF/Announcing-the-Amazon-S3-Managed-Uploader-in-the-AWS-SDK-for-JavaScript) or [HTML5 and Amazon S3 Multi-Part uploads](https://stackoverflow.com/questions/18075676/html5-and-amazon-s3-multi-part-uploads)? The latter uses [AWS S3 Multipart Upload from Browser](https://github.com/ienzam/s3-multipart-upload-browser). — John Rotenstein, Mar 12 '16 at 21:28
Yeah I've read over the amazon docs and the enzam link is the one I referred to in the post that doesn't work anymore. I'm currently looking into this that uses the jquery file library: https://github.com/eddturtle/direct-upload-s3-signaturev4 - however I don't see any evidence of concurrent transfer. It is chunked, just sending in single file and not in concurrent batches — Kai Qing, Mar 17 '16 at 19:30

score 1 · Accepted Answer · edited Jun 20 '19 at 11:00

1

Plupload does support chunked uploads so all you need to do is configure it properly:

var uploader = new plupload.Uploader({
    browse_button: 'browse', // this can be an id of a DOM element or the DOM element itself
    url: 'upload.php',
    chunk_size: '200kb',
    max_retries: 3
});

The last two lines of the above config will set chunking.

edited Jun 20 '19 at 11:00

TRiG

10,148
7
57
107

answered Feb 06 '18 at 00:09

JCL1178

1,155
1
10
31

This issue is old and no longer necessary. Client went with something else. However, if you want to update the answer with a good code snippet - in case that link goes dead - I can accept this as the answer – Kai Qing Feb 06 '18 at 00:34
Code snippet added. See above. – JCL1178 Feb 06 '18 at 01:47

Uploading chunked files directly to Amazon s3

1 Answers1