Amazon S3 client could not download file with spaces or hash?

Question

I am using AWSSDK.S3 version 3.3.17.2 and AWSSDK.Core version 3.3.21.16 to upload a file and then download the same file. The code below not able to download the file if the file name has spaces ( or #)

public class AmazonS3
{
    public async Task<string> UploadFileAsync(string sourceFile, string s3BucketUrl)
    {
        AmazonS3Uri s3Uri = new AmazonS3Uri(s3BucketUrl);
        using (var s3 = new AmazonS3Client(s3Uri.Region))
        {
            using (TransferUtility utility = new TransferUtility(s3))
            {
                TransferUtilityUploadRequest request = new TransferUtilityUploadRequest
                {
                    BucketName = s3Uri.Bucket,
                    ContentType = "application/pdf",
                    FilePath = sourceFile,
                    Key = s3Uri.Key + Path.GetFileName(sourceFile),
                };

                await utility.UploadAsync(request).ConfigureAwait(false);
            }
        }

        return Path.Combine(s3BucketUrl, Path.GetFileName(sourceFile));
    }  

    public async Task DownloadFileAsync(string destinationFilePath, string s3Url)
    {
        var s3Uri = new AmazonS3Uri(s3Url);
        var s3Client = new AmazonS3Client(s3Uri.Region);
        GetObjectRequest getObjectRequest = new GetObjectRequest
        {
            BucketName = s3Uri.Bucket,
            Key = s3Uri.Key
        };

        // dispose the underline stream when writing to local file system is done
        using (var getObjectResponse = await s3Client.GetObjectAsync(getObjectRequest).ConfigureAwait(false))
        {
            await getObjectResponse.WriteResponseStreamToFileAsync(destinationFilePath, false, default(System.Threading.CancellationToken)).ConfigureAwait(false);
        }
    }              
}

Then for testing purpose i am uploading the file and downloading the same file again

AmazonS3 s3 = new AmazonS3();

var uploadedFileS3Link = await s3.UploadFileAsync("C:\\temp\\my test file.pdf", @"https://mybucket.s3-us-west-2.amazonaws.com/development/test/");

// get exception at line below
await s3.DownloadFileAsync("C:\\temp\\downloaded file.pdf",uploadedFileS3Link );

I am getting exception

Amazon.S3.AmazonS3Exception: The specified key does not exist. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at ......

removed remaining exception for brevity

The file does exist inside bucket. In-fact i can copy and paste the s3url (i.e. the value of uploadedFileS3Link variable) and download the file via browser.

(Note that in reality i am trying to download 1000+ files that are already uploaded with spaces in their name. So removing the spaces while uploading is not an option)

Update 1 i noticed S3 browser Url Encode the file name

I tried downloading the file using the encoded file path https://mybucket.s3-us-west-2.amazonaws.com/development/test/my%20test%20file.pdf but it still did not work

score 7 · Accepted Answer · answered Mar 16 '18 at 17:40

so finally i found what was the issue. I am using AmazonS3Uri class to parse the given S3 url and get the key, bucket and region. The AmazonS3Uri returns my key as development/test/my%20test%20file.pdf

Because internally AmazonS3Uri is using System.Uri to build Uri and then returns AbsolutePath which returns Encoded path as Key (Should it return Local Path as Key?)

I don't know why but AmazonS3Client does not like it, and it throws exception if you pass Encoded key.

So to fix the issue i decode the key using System.Net.WebUtility.UrlDecode(s3Uri.Key). So new download method looks like

    public async Task DownloadFileAsync(string destinationFilePath, string s3Url)
    {
        var s3Uri = new S3UrlParser(s3Url);
        var s3Client = new AmazonS3Client(s3Uri.Region);
        GetObjectRequest getObjectRequest = new GetObjectRequest
        {
            BucketName = s3Uri.Bucket,
            Key = System.Net.WebUtility.UrlDecode(s3Uri.Key)
        };

        // dispose the underline stream when writing to local file system is done
        using (var getObjectResponse = await s3Client.GetObjectAsync(getObjectRequest).ConfigureAwait(false))
        {
            await getObjectResponse.WriteResponseStreamToFileAsync(destinationFilePath, false, default(System.Threading.CancellationToken)).ConfigureAwait(false);
        }
    }

score 0 · Answer 2 · answered Mar 31 '22 at 16:04

I had this same issue parsing filename with space and forward slashes in the name. The problem with unescaped forward slashes is most tools treat each forward slash as a directory delimiter within S3.

So a simple name like myfilename = "file/name.pdf" appended to a virtual path would turn into; "directory/subdirectory/myfile/name.pdf" (here it will now have a directory called myfile where it was not intended)

It can be mitigated by escaping the filename "directory/subdirectory/myfile%2fname.pdf" at the point of upload and then decoding as suggested in the answer of this post.

After a experiencing errors from erroneous slashes and blank folder names in AWS Explorer caused by extra double slashes e.g. "//" (described further here: https://webapps.stackexchange.com/a/120135)

I've come to the conclusion it's better to completely strip forward slashes altogether from any filename proportion (not the virtual directory) rather than relying on escaping on upload and unescaping it on download. Treating the forward slash as a reserved character as others have similarly experienced: https://stackoverflow.com/a/19515741/1165173

In the case of an escaped whitespace symbol "+" it is usually safe to unescape the character and it will work as intended.

Filename proportion can be encoded using

System.Web.HttpUtility.Encode(input);

List of non-acceptable filename characters can be compared at the point of file creation using Path.GetInvalidPathChars()

Documentation: https://learn.microsoft.com/en-us/dotnet/api/system.io.path.getinvalidfilenamechars

Amazon S3 client could not download file with spaces or hash?

2 Answers2