How to use TransferUtility to upload multiple files

Question

I am trying to make sense of the documentation for:

• TransferUtility.UploadDirectory

The documentation does not describe the error condition of the upload. Typically I would guess something like System.Net.Http.HttpRequestException.

After reading multiple comments, it seems that S3 does not support TransactionScope. The only thing that seems to be supported is at file level:

• Are writes to Amazon S3 atomic (all-or-nothing)?

and

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel

So my questions are:

• Where can I find the error condition of UploadDirectory ?
• Does it makes sense to use UploadDirectory since atomic operations are at file (object) level ?

---

My question is about uploading multiples files (ie. s3 'object'), not about doing a multi-parts upload of single file.

Answer 1

AFAIK, the best you can do here is wrap it in a try catch:

try
{
    ...
}
catch (AmazonS3Exception e)
{
    // implement rollback operation
    ...
}
catch (Exception e)
{
    // no possible rollback operation, abort program ?
    ...
}

You can keep track of progress using an UploadDirectoryProgressEvent. In the event of an error, if you want to clean up, you'd have to compare the progress, note the diffs, and take action as appropriate (e.g. by removing objects if you don't want to keep them in S3 and you want the entire operation to be atomic).

Pay special attention to the fact that:

var request = new TransferUtilityUploadDirectoryRequest
{
  UploadFilesConcurrently = true,
};

Will have an impact on your rollback mechanism. Setting UploadFilesConcurrently to true imply that UploadDirectoryProgressArgs received in UploadDirectoryProgressEvent have a null value for CurrentFile:

• https://github.com/aws/aws-sdk-net/issues/317

In which case you can only implement rollback in the case where you can delete the full remote directory.

Note also the documentation on multi-part uploads:

If a multipart upload is interrupted, TransferUtility will attempt to abort the multipart upload. Under certain circumstances (network outage, power failure, etc.), TransferUtility will not be able to abort the multipart upload. In this case, in order to stop getting charged for the storage of uploaded parts, you should manually invoke TransferUtility.AbortMultipartUploads() to abort the incomplete multipart uploads.

The documentation has examples of both tracking and aborting muultipart uploads.

As for your other question:

Does it makes sense to use UploadDirectory since atomic operations are at file (object) level ?

I'd say that depends. The code to upload an entire directory of files might be somewhat cleaner, but since you still have to possible track and clean up, you might as well process the files one by one.