0

I would to generate a big file (several TB) with special format using my C# logic and persist it to S3. What is the best way to do this. I can launch a node in EC2 and then write the big file into EBS and then upload the file from the EBS into S3 using the S3 .net Clinent library.

Can I stream the file content as I am generating in my code and directly stream it to S3 until the generation is done specially for such large file and out of memory issues. I can see this code help with stream but it sounds like the stream should have already filled up with. I obviously can not put such a mount of data to memory and also do not want to save it as a file to the disk first. 

PutObjectRequest request = new PutObjectRequest();
        request.WithBucketName(BUCKET_NAME);
        request.WithKey(S3_KEY);
        request.WithInputStream(ms);
        s3Client.PutObject(request);

What is my best bet to generate this big file ans stream it to S3 as I am generating it?

iCode
  • 4,308
  • 10
  • 44
  • 77
  • 1
    I think this may help you http://stackoverflow.com/questions/8653146/can-i-stream-a-file-upload-to-s3-without-a-content-length-header – Alex Jul 09 '12 at 10:14

1 Answers1

1

You certainly could upload any file up to 5 TB that's the limit. I recommend using the streaming and multipart put operations. Uploading a file 1TB could easily fail in the process and you'd have to do it all over, break it up into parts when you're storing it. Also you should be aware that if you need to modify the file you would need to download the file, modify the file and re-upload. If you plan on modifying the file at all i recommend trying to split it up into smaller files.

http://docs.amazonwebservices.com/AmazonS3/latest/dev/UploadingObjects.html

bwight
  • 3,300
  • 17
  • 21