1

I want to read and download data from an Amazon S3 Requester Pays bucket.

I've tried to enable a 'requester pays access' option in the .NET SDK but I can't find anything:

static async Task DownloadFile()
{
    TransferUtility fileTransferUtility =
        new TransferUtility(new AmazonS3Client(accessKey, secretAccessKey, Amazon.RegionEndpoint.USEast1));
    fileTransferUtility.Download("destination", "data-repository-client-1",
        "20221002-22%2709%2753_to_20221002-22%2723%2715_Trades_v5.csv.gz");
}

I get an access denied exception because of the Requester Pays access settings.

Ermiya Eskandary
  • 15,323
  • 3
  • 31
  • 44
pewocis495
  • 117
  • 1
  • 11
  • You probably need to refer to this: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestSyntax - specifically the x-amz-request-payer query parameter. I'm not 100% sure how to supply this in .NET though. – ProgrammingLlama Nov 11 '22 at 09:04
  • Specifically: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ObjectsinRequesterPaysBuckets.html - *"If x-amz-request-payer is not in the request, Amazon S3 returns a 403 error and charges the bucket owner for the request."*. – luk2302 Nov 11 '22 at 09:06
  • `AmazonS3Client`'s `DownloadToFilePathAsync` method has an "additionalProperties" parameter, which I'm guessing _might_ be involved in this, but I can't say for sure as I've never tried it. – ProgrammingLlama Nov 11 '22 at 09:09

1 Answers1

0

TransferUtility does not support downloading objects in Requester Pays buckets.

However, it is possible when using GetObject with a custom GetObjectRequest.

You will need to set the RequestPayer parameter to RequestPayer.Requester.

This should work:

using Amazon;
using Amazon.S3;
using Amazon.S3.Model;

var client = new AmazonS3Client(
    accessKey,
    secretAccessKey,
    RegionEndpoint.USEast1
);

var request = new GetObjectRequest()
{
    BucketName = "data-repository-client-1",
    Key = "20221002-22%2709%2753_to_20221002-22%2723%2715_Trades_v5.csv.gz",
    RequestPayer = RequestPayer.Requester
};

var filePath = "destination";
var appendToFile = false;

using (var response = await client.GetObjectAsync(request))
    await response.WriteResponseStreamToFileAsync(filePath, appendToFile, CancellationToken.None);
Ermiya Eskandary
  • 15,323
  • 3
  • 31
  • 44
  • Hi @Ermiya Thanks for the solution. I just got 1 question about the Key field, because I get the following error: "20221111T120928ZThe request signature we calculated does not match the signature you provided. Check your key and signing method.)'" Do you know where I can find the item key in the S3 Browser ? Or to list all items in the bucket ? – pewocis495 Nov 11 '22 at 12:10
  • @pewocis495 Are you sure your credentials are correct? The item key can be viewed within the console, listed via the SDK, CLI, API or should be viewable within S3 browser. Searching for 'list all items in S3 bucket' will yield plenty of solutions online – Ermiya Eskandary Nov 11 '22 at 13:17
  • @pewocis495 Ensure the file isn't within a 'folder' because if so, it will need to be mentioned in the key as well. Also I forgot - ask the bucket owner for the key as you probably won't have `s3:ListObjects` access to the bucket. – Ermiya Eskandary Nov 11 '22 at 15:11