Intermittent 403 CORS errors when using S3 signed URL to PUT object

Question

I'm running into an issue where S3 will occasionally return a 403 on the OPTIONS request pre-flight when trying to upload an object from the client side with a PUT request to a signed URL. The frustrating part is that 90% of the time it works fine, and when it does fail, if I retry the same request a few times, then it will go through.

Has anyone experienced something similar, and if so, how did you solve it?

This seems related to this issue, but isn't exactly the same, because they're going through CloudFront, and I'm going directly to S3.

Generating the signed URL on the server side

const params = {
    Bucket: **my bucket**,
    Key: 'test/my-test-file.csv',
    Metadata: {
        'created-by': req.user.id,
        'job-id': job.id,
    },
};

const url = s3.getSignedUrl('putObject', params);

Angular.JS HTTP Request to the Signed URL

return $http({
    url: **signedUrlHere**,
    method: 'PUT',
    data: file,
    headers: {
        'Content-Type': '',
    },
    transformRequest: [],
});

Bucket CORS Settings

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

Request Headers

Host: s3.amazonaws.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: PUT
Access-Control-Request-Headers: content-type
Referer: https://**.***.com/*/*/*
Origin: https://**.***.com
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

Response Headers

HTTP/1.1 403 Forbidden
x-amz-request-id: *********
x-amz-id-2: *******
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Thu, 20 Jun 2019 02:54:39 GMT
Server: AmazonS3

I think the problem here is that why is it making option request to https://s3.amazonaws.com/ , can you try with virtual style path name in signed url. — James Dean, Jun 20 '19 at 16:32
Looks like there is a bug in the JS SDK that prevents me from using a virtual style path in the signed URL without a whole lot of extra effort. Hopefully they'll get this fixed soon. https://github.com/aws/aws-sdk-js/issues/2653 — sflogen, Jun 20 '19 at 18:29
*"If I retry the same request a few times, then it will go through."* Does this mean *exactly* the same request, and using exact same pre-signed URL? Or are you redoing the signing each time? — Michael - sqlbot, Jun 20 '19 at 21:32
Do these signed URLs include `AWSAccessKeyId` (Signature Version 2) or do they have `X-Amz-Credential` (Signature Version 4)? — Michael - sqlbot, Jun 20 '19 at 22:06
Signature Version 2. The signed URLs are `https://s3.amazonaws.com/mybucket/mypath/myfile.csv?AWSAccessKeyId=***********&Expires=1561127791&Signature=*******&x-amz-meta-created-by=2c8b358a-c717-4d2b-a1ff-f6c1f2415ac4&x-amz-meta-job-id=6d23e56d-bd53-409c-99e3-e2a812774135` — sflogen, Jun 21 '19 at 14:23
Did you ever figure this out, and if so what was the problem? — Rocket04, Oct 09 '20 at 15:48
@Rocket04, I still have not found a solution. We're just dealing with the intermittent errors by retrying in some cases and routing the files through our server in other cases. — sflogen, Oct 09 '20 at 21:18

score 2 · Answer 1 · answered Oct 19 '20 at 15:35

2

After looking into this for a while we found out what the problem was with these intermittent 403s, at least for us. Hopefully this helps you or anybody else with the same problem. In our case, it was caused by people having their computer setup to the wrong time. That explains the intermittent nature, because most people have their system set on automatic time and time-zone, the few who don't and have the wrong time keep getting errors.

answered Oct 19 '20 at 15:35

Rocket04

1,801
5
25
47

Glad to hear that! Next time I see it, I'll check the server time compared to the request time. I guess it's possible that the EC2 instance's clock is drifting. Although we don't usually see it limited to specific users. I've seen it a handful of times on my machine which is set on automatic time. – sflogen Oct 19 '20 at 20:59