Adding proper permissions to AWS S3 bucket to allow SEO

Question

I'm trying to verify my site for SEO purposes with Google using https://www.google.com/webmasters/tools/home?hl=en. I am using AWS S3 to host my content, and AWS Cloudfront to serve it through the CDN. I'm following this checklist: http://www.wikihow.com/Get-Your-Website-Indexed-by-Google and am on Step 4.

The steps Google lists to verify are:

1. Download this HTML verification file. [googlelongstringofcharacters.html]
2. Upload the file to https://www.dynamicdentaledu.com/
3. Confirm successful upload by visiting https://www.dynamicdentaledu.com/googlelongstringofcharacters.html in your browser.
4. Click Verify below. To stay verified, don't remove the HTML file, even after verification succeeds.

I've added the HTML file to my site's root. When I click confirm in step 3, I get:

So I skipped that and clicked Verify button in step 4. Google says:

Verification failed for https://www.dynamicdentaledu.com/ using the HTML file method (less than a minute ago). We were unable to connect to your server.

I think this is due to the permissions and bucket policies I have the S3 bucket. They are, respectively:

And

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::dynamicdentaledu.com/*"
        }
    ]
}

How can I enable Google to access what it needs?

---

EDIT: following AWS's bucket policies, I changed the policy to:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::dynamicdentaledu.com/*"
        }
    ]
}

Am now getting:

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>google*longstringofcharacters*.html</Key>
<RequestId>42DD1F1F0D5E06F7</RequestId>
<HostId>
zbmsLAEMz3ed2zKx3gKCHjrtHxeWmaLl16JJs6012zFcLQdnMH48mFJY1YOETD3WMS/8NwkU3SY=
</HostId>
</Error>

Answer 1

You have three issues.

• CloudFront will return errors to the browser for 5 minutes after you fix the problem, by default. When the origin server returns an error, usually there is no reason for CloudFront to continually retry. In a case like this, you may want to reconfigure the Error Caching TTL for 403 and 404 errors to 0 seconds in CloudFront. See my answer to Amazon CloudFront Latency for further explanation of this.

• You did not need to change your bucket policy. If your site was otherwise working and you uploaded this new object with the "make everything public" option selected (equivalent to setting x-amz-acl: public-read if using the API) then that should have been sufficient, though the 5 minute timer mentioned above could have complicated your troubleshooting process. Note also that in your bucket permissions, you are allowing Everyone to List the contents of your bucket. This is not actually causing the problem, here, but is a configuration that is potentially too permissive and needs to be mentioned. This option allows anyone to download a complete list of all your files, which seems like a bad idea in most cases.

• You didn't upload the file with the correct name. <Code>NoSuchKey</Code> is never returned for any reason other than, simply enough, there is no object with this key (path/filename.ext) in the bucket. It cannot be caused by policy, permissions, ACL, etc. Check in the S3 console: the file is not named as you intended, or is not in the right place, at the root of the bucket. The long string of characters is not, as far as I am aware, a secret value -- only an obscure/unpredictable value -- so if the information here doesn't help you resolve this, showing a screen shot of the console including this object and its properties should not pose any security issue for you. This may be necessary for further troubleshooting, should that be required.