3

Can Anyone please suggest me the way or sample code to write data (line-by-line) into S3.

Its like a stream coming in to some file, as soon as my program receives each line, it has to be written in S3. So I need only one buckets for that and one file in s3.

Sample input :

String lines[] = ["This is first line","This is Second line", "This is Third Line"]

s3 Initialization,...etc
itererator i = lines;  
while (i.hasnext())
{
    s3Service.putObject(testBucket, i); // Write each line in S3
}

Please check the flow and suggest me with code sample.

Very much appreciated...

Richard Heap
  • 48,344
  • 9
  • 130
  • 112
Avis
  • 496
  • 1
  • 5
  • 18
  • 1
    You need to use the multipart API, see [this SO post](http://stackoverflow.com/a/8881939/1415732) – Alden Apr 20 '14 at 16:43
  • Hi Alden, Thanks for you quick reply, I've doubt, If I make use of this API to upload stream data, wouldn't it cause performance issues. Looks like it was designed to upload files which are very big in size but in my case I planning to buffer 1024 bytes(as the data comes in from another process) every time and upload it. Is it okay to use in my case? Please suggest or else Is there a way to open a http file handle remotely and manage in my code? Please Let me know. Thanks in advance. – Avis Apr 22 '14 at 12:06
  • Can anyone tried uploading streams to S3 ? If yes , please help me with the procedure. I understand that we can do using AWS SDKs but it seems like some limitation with Multipart upload APIs (min 5MB of stream data is required) but In my case I want to upload 1024 Bytes as and when the I receive stream. Appreciate your help! – Avis Apr 25 '14 at 11:38
  • If you're only dealing with 1024 bytes, you could string join your lines with `\n` and then make one `putObject` call with everything – Alden Apr 25 '14 at 13:22
  • Hi Avis, what was your final solution then ? – SBD Jan 17 '22 at 18:51

1 Answers1

0

Noticed that this is a question 3 years ago, now you could use AWS Kinesis Firehose to achieve this easily, you could simply write that string into firehose and firehose will help you aggregate by time or size, and then deliver to S3.

Fez
  • 29
  • 4