Allowing users to upload content to s3

Question

I have an S3 bucket named BUCKET on region BUCKET_REGION. I'm trying to allow users of my web and mobile apps to upload image files to these bucket, provided that they meet certain restrictions based on Content-Type and Content-Length (namely, I want to only allow jpegs less than 3mbs to be uploaded). Once uploaded, the files should be publicly accessible.

Based on fairly extensive digging through AWS docs, I assume that the process should look something like this on my frontend apps:

const a = await axios.post('my-api.com/get_s3_id');

const b = await axios.put(`https://{BUCKET}.amazonaws.com/{a.id}`, {
   // ??
   headersForAuth: a.headersFromAuth,
   file: myFileFromSomewhere // i.e. HTML5 File() object
});

// now can do things like <img src={`https://{BUCKET}.amazonaws.com/{a.id}`} />
// UNLESS the file is over 3mb or not an image/jpeg, in which case I want it to be throwing errors

where on my backend API I'd be doing something like

import aws from 'aws-sdk';
import uuid from 'uuid';
app.post('/get_s3_id', (req, res, next) => {
  // do some validation of request (i.e. checking user Ids)
  const s3 = new aws.S3({region: BUCKET_REGION});
  const id = uuid.v4();
  // TODO do something with s3 to make it possible for anyone to upload pictures under 3mbs that have the s3 key === id
  res.json({id, additionalAWSHeaders});
});

What I'm not sure about is what exact S3 methods I should be looking at.

---

Here are some things that don't work:

• I've seen a lot of mentions of (a very old) API accessible with s3.getSignedUrl('putObject', ...). However, this doesn't seem to support reliably setting a ContentLength -- at least anymore. (See https://stackoverflow.com/a/28699269/251162.)

• I've also seen a closer-to-working example using an HTTP POST with form-data API that is also very old. I guess that this might get it done if there are no alternatives but I am concerned that it is no longer the "right" way to do things -- additionally, it seems to doing a lot of manual encrypting etc and not using the official node SDK. (See https://stackoverflow.com/a/28638155/251162.)

Answer 1

I think what might be better for this case in POSTing directly to S3, skipping your backend server.

What you can do is define a policy that explicitly specifies what can be uploaded to and to where, this policy is then signed using an AWS secret access key (using the AWS sig v4, can generate a policy using this).

An example usage of the policy and signature if viewable in the AWS docs

For your uses you can specify conditions like:

conditions: [
   ['content-length-range, 0, '3000000'],
   ['starts-with', '$Content-Type', 'image/']
]

This will limit uploads to 3Mb, and Content-Type to only items that begin with image/

Additionally, you only have to generate your signature for policy once (or whenever it changes), which means you don't need a request to your server to get a valid policy, you just hardcode it in your JS. When/if you need to update just regenerate the policy and signature and then update the JS file.

edit: There isn't a method through the SDK to do this as it's meant as way of directly POSTing from a form on a webpage, i.e. can work with no javascript.

edit 2: Full example of how to sign a policy using standard NodeJS packages:

import crypto from 'crypto';

const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;
const ISO_DATE = '20190728T000000Z';
const DATE = '20161201';
const REGION = process.env.AWS_DEFAULT_REGION || 'eu-west-1';
const SERVICE = 's3';
const BUCKET = 'your_bucket';

if (!AWS_ACCESS_KEY_ID || !AWS_SECRET_ACCESS_KEY) {
    throw new Error('AWS credentials are incorrect');
}

const hmac = (key, string, encoding) => {
    return crypto.createHmac("sha256", key).update(string, "utf8").digest(encoding);
};

const policy = {
    expiration: '2022-01-01T00:00:00Z',
    conditions: [
        {
            bucket: BUCKET,
        },
        ['starts-with', '$key', 'logs'],
        ['content-length-range', '0', '10485760'],
        {
            'x-amz-date': ISO_DATE,
        },
        {
            'x-amz-algorithm': 'AWS4-HMAC-SHA256'
        },
        {
            'x-amz-credential': `${AWS_ACCESS_KEY_ID}/${DATE}/${REGION}/${SERVICE}/aws4_request`
        },
        {
            'acl': 'private'
        }
    ]
};

function aws4_sign(secret, date, region, service, string_to_sign) {
    const date_key = hmac("AWS4" + secret, date);
    const region_key = hmac(date_key, region);
    const service_key = hmac(region_key, service);
    const signing_key = hmac(service_key, "aws4_request");
    const signature = hmac(signing_key, string_to_sign, "hex");

    return signature;
}

const b64 = new Buffer(JSON.stringify(policy)).toString('base64').toString();
console.log(`b64 policy: \n${b64}`);
const signature = aws4_sign(AWS_SECRET_ACCESS_KEY, DATE, REGION, SERVICE, b64);
console.log(`signature: \n${signature}\n`);

Answer 2

You need to get familiar with Amazon Cognito and especially with identity pool.

Using Amazon Cognito Sync, you can retrieve the data across client platforms, devices, and operating systems, so that if a user starts using your app on a phone and later switches to a tablet, the persisted app information is still available for that user.

Read more here: Cognito identity pools

Once you create new identify pool, you can reference it while using S3 JavaScript SDK which will allow you to upload content whit out exposing any credentials to the client.

Example here: Uploading to S3

Please read through all of it, especially the section "Configuring the SDK".

The second part of your puzzle - validations.

I would go about implementing a client-side validation (if possible) to avoid network latency before giving an error. If you would choose to implement validation on S3 or AWS Lambda you are looking for a wait-time until file reaches AWS - network latency.

Answer 3

This is something I know we have in our project, so I'll show you part of the codes:

you first need to post to your own server to get the creds for the upload, from that you will return the params from the client upload to S3.

these are params you send to the aws s3 service, you will need the bucket, upload path, and the file

 let params = {
        Bucket: s3_bucket,
        Key: upload_path,
        Body: file_itself
    };

this is the code I have for the actual upload to s3

config.credentials = new AWS.Credentials(credentials.accessKeyId, 
credentials.secretAccessKey, credentials.sessionToken);
    let s3 = new S3(config);
    return s3.upload(params, options).on("httpUploadProgress", handleProgress);

all of those credentials items you get from your backend of course.

Answer 4

On the backend you need to generate a timed, presigned URL and send that URL to the client for accessing the S3 object. Depending on your backend implementation technology you can use the AWS CLI or SDKs (e.g. for Java, .Net, Ruby or Go).

Please refer to CLI docs and SDK docs and more SDK

Content size restriction is not supported in link generation directly. But the link is just there to relay the access rights that the AWS user has.

For using a policy to restrict file size on upload you have to create a CORS policy on the bucket and use HTTP POST for the upload. Please see this link.

Answer 5

your servers acts like a proxy, also responsible for authorization, validation, etc. Some code snippet:

 upload(config, file, cb) {
  const fileType = // pass with request or generate or let empty
  const key = `${uuid.v4()}${fileType}`; // generate file name:
  const s3 = new AWS.S3();
  const s3Params = {
   Bucket: config.s3_bucket,
   Key: key,
   Body: file.buffer
 };

 s3.putObject(s3Params, cb);
}

and then you can send the key to the client and provide further access.