1

I have to upload a file with a presigned URL that S3 creates for me. To save up some space and time I am using a stream that compresses the file into .gz-format. So the S3 just receives the gz-file.

I use the AWS S3 CPP SDK. The AWS::HttpClient for the HTTP request and boost for the compression.

I am asking myself how can I set the content length for the HTTP request correctly? On one hand it is a stream, on the other hand the stream will be compressed (gz) which makes it not easier. I have no idea how to do this, sadly.

I tried to get the length with seekg() and tellg() but I couldn't solve my problem.

auto&& f = std::fstream{ filepath.string(), std::ios_base::in | std::ios_base::binary };

auto in = std::make_shared<boost::iostreams::filtering_stream<boost::iostreams::bidirectional>>();
in->push(boost::iostreams::combine(boost::iostreams::gzip_compressor(), boost::iostreams::gzip_decompressor()));
in->push(f);

req->AddContentBody(in);
//req->SetContentLength(); <-- How?
req->SetContentType("application/octet-stream");

auto res = MyHttpClient->MakeRequest(req);  //upload stream
smac89
  • 39,374
  • 15
  • 132
  • 179
Michael Kemmerzell
  • 4,802
  • 4
  • 27
  • 43

1 Answers1

1

The Content-Length must be the actual number of bytes being sent. If you are compressing data dynamically while sending it then obviously you can't know the final compressed size up front to set a Content-Length correctly. Fortunately, you don't have to, as S3 has a multipart upload API that supports streaming uploads. See Can I stream a file upload to S3 without a content-length header?.

Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770