How to download a file from s3 using provided url?

Question

In my application I will get the url of s3 file like : https://s3.amazonaws.com/account-update/input.csv I have to download it and then process it. What I already done :

AmazonS3 s3 = new AmazonS3Client(credentials);
S3Object s3object = s3.getObject(new GetObjectRequest(
bucketName, key));

I am able to download the file by providing bucket name and key, but how can I download the file using the url(https://s3.amazonaws.com/account-update/input.csv) only?

score 22 · Answer 1 · answered May 22 '17 at 23:01

You could download the file via a standard curl/wget, the same as you would download any other file off the Internet.

The important part, however, is to enable access to the object from Amazon S3. A few options:

Make the object publicly-readable: This can be done via the console or CLI/API. However, anyone with that URL will be able to download it.
Create an Amazon S3 Bucket Policy that grants read access for the desired file/directory/bucket. But, again, anyone with the URL will be able to access those objects.
Keep the object private, but use a pre-signed URL that adds parameters to the URL to prove that you are permitted to download the object. This pre-signed URL is time-limited and can be generated with a few lines of code using current AWS credentials.

I saw examples using pre-signed URL for uploading objects. How can we generate pre-signed URL for downloading ? — Md. Arafat Al Mahmud, Jan 10 '19 at 10:40
@Md.ArafatAlMahmud Please create a new question rather than asking via a comment on an old question. — John Rotenstein, Jan 10 '19 at 12:28

Jawad · Answer 2 · 2018-04-06T21:44:07.200

21

You may consider using the AWS SDK class AmazonS3URI as shown below:

URI fileToBeDownloaded = new URI(" https://s3.amazonaws.com/account-update/input.csv"); 

AmazonS3URI s3URI = new AmazonS3URI(fileToBeDownloaded);

S3Object s3Object = s3Client.getObject(s3URI.getBucket(), s3URI.getKey());

From here on, you should be able to utilise the s3Object obtained in a similar fashion to the s3Object shown in your code.

For more Java related AWS SDK examples on using this class check here

edited Apr 06 '18 at 21:44

answered Apr 06 '18 at 21:05

Jawad

361
3
8

Is there something similar to AmazonS3URI in the newer SDK? I'm using software.amazon.awssdk:s3. – mrog Oct 23 '19 at 16:44
1

Not yet available on AWS SDK V2. You may follow this [here](https://github.com/aws/aws-sdk-java-v2/issues/272) – Jawad Oct 24 '19 at 17:35

score 3 · Answer 3 · answered May 22 '17 at 19:10

To enable access by HTTP, you must set the bucket up as a Static Website in the S3 console. Be warned, this will expose all of your data to the web unless you set up proper S3 access controls.

The method you are accessing via the Java SDK will not use this type of connection, though. It will connect via the built-in S3 protocol. You should inspect your IAM Role or Policy to ensure you have the correct permissions (s3:GetObject). You will also need s3:ListBucket to see a 404 if the object does not exist.

score 1 · Answer 4 · answered Apr 07 '18 at 08:21

The best way is using the pre-signed S3 URL to achieve your needs. You can add expiration time to your signed URLs and after that the URL not available.

For more information read the following page:

https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURLJavaSDK.html

score 1 · Answer 5 · edited Sep 27 '19 at 09:13

1

Just enter the url on the browser but make sure to replace \u0026 with & if you have downloaded the url via curl else you will get this error

<Error>
    <Code>AuthorizationQueryParametersError</Code>
    <Message>X-Amz-Algorithm only supports "AWS4-HMAC-SHA256"</Message>
</Error>

edited Sep 27 '19 at 09:13

RobC

22,977
20
73
80

answered Sep 27 '19 at 08:53

Riya John

474
4
14

score 0 · Answer 6 · answered May 28 '17 at 06:07

John Rotenstein is correct, you can download the file via the URL using standard curl/wget.

If you wanted to do this using Java, something like the following should do the trick; making use of the Apache HttpComponents package

private void downloadRequest(String url, String savedFile) {
    HttpClient client = HttpClients.createDefault();

    HttpGet httpGet = new HttpGet(url);
    HttpResponse response;
    try {
        response = client.execute(httpGet);
        HttpEntity entity = response.getEntity();
        if (entity != null) {
            try (FileOutputStream outstream = new FileOutputStream(savedFile)) {
                entity.writeTo(outstream);
            } catch (IOException e) {
                LOGGER.info(e.getMessage());
            }
        }

    } catch (IOException e) {
        e.printStackTrace();
    }
}

score 0 · Answer 7 · edited Aug 08 '19 at 19:13

You can't, but you can make the file attachment in the upload.

For example:

ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentType("application/csv;charset=utf-8");
objectMetadata.setContentDisposition("attachment");
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, arquivo, file, objectMetadata).withCannedAcl(CannedAccessControlList.PublicRead);
amazonS3.putObject(putObjectRequest);

I hope it helps.

score -4 · Answer 8 · answered Apr 07 '18 at 05:12

-4

using cli: aws s3 cp s3://bucket/prefix/key targetlocalfolder

answered Apr 07 '18 at 05:12

tooptoop4

234
3
15
45

This question is about using a url with http, which does not work with the CLI the same way a regular s3 path does. – szeitlin Nov 02 '18 at 23:38

How to download a file from s3 using provided url?

8 Answers8

Linked