3

On this web page, it is claimed that we can "Begin an upload before you know the final object size - You can upload an object as you are creating it."

But I did not find any example to do it. I have an input stream that I want to parse and then write results of the parsing in an output stream towards S3. I would like to avoid storing chunks of results on disk to upload classically in S3. I want to start writing in S3 as soon as I start receiving the input stream. I don't know by advance the size of the output stream I am going to write and it may be big > 100 GB. I found this method. But it is clearly highlighted that the content length must be specified. Otherwise AWS SDK will try to put everything in memory before sending it to S3.

I am using AWS SDK 1.9.40 (Gradle dependency : compile "com.amazonaws:aws-java-sdk:1.9.40").

Comencau
  • 1,084
  • 15
  • 35
  • 1
    I think you need to use the low-level Java API https://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html – tgdavies Apr 30 '18 at 11:35
  • @tgdavies Thank you. I did not notice the low-level API. However, even if I deal with the management of the small chunks (parts) to be sent, I have to specify a length for each part. I cannot simply write in an output stream towards S3 without knowing the length.I will try to hack with that. In fact, I am migrating a piece of code that get data as input stream and write it in HDFS to write in S3 instead of HDFS. With HDFS I can create an output stream towards a location in HDFS (FSDataOutputStream os = fs.create(path)). I was expecting the same thing for S3. But apparently, it does not exist. – Comencau Apr 30 '18 at 14:26
  • Possible duplicate of https://stackoverflow.com/questions/8653146/can-i-stream-a-file-upload-to-s3-without-a-content-length-header (or at least highly related) – lmsurprenant Jul 10 '20 at 21:30
  • Looks like buffering + multipart uploads is the only way to go; even if you do chunked, you have to [include the final expected content length](https://forums.aws.amazon.com/message.jspa?messageID=561616#jive-message-543097) in advance – Janaka Bandara Sep 19 '20 at 02:25

0 Answers0