0

I have list of java objects which I need to upload to [Amazon] s3. Currently I am uplaoding each object one by one which is quite inefficient. Each object in the list is first serialized to JSON and separated by a newline, i.e. \n

private ByteBufferOutputStream serialize(final Object object) {
    try {
        final ByteBufferOutputStream outputStream = new ByteBufferOutputStream(128, true, false);
        Json.writeValueAsToOutputStream(object, outputStream);
    
        outputStream.write('\n');
        return outputStream;
    } catch (final Exception e) {
        log.error("Error serializing data {}", object, e);
        return null;
    }
}
for (Data event : events) {
    final ByteBufferOutputStream serializedEvent = serialize(event);
        if (serializedEvent == null) {
            continue;
        }
        PutObjectRequest objectRequest = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .build();
        try {
            s3Client.putObject(objectRequest, RequestBody.fromByteBuffer(serializedEvent.toByteBuffer()));
        } catch (S3Exception e) {
            log.error("Failed to send raw data events to S3", e);
        }
}

How can I upload all the objects in a list only once after serialization ?

Abra
  • 19,142
  • 7
  • 29
  • 41
CyberPunk
  • 1,297
  • 5
  • 18
  • 35
  • into individual files (https://stackoverflow.com/q/31686093/995891 maybe) or into one that should contain all of them? I would also assume your `Json` has a way to produce strings directly, bytearraybuffer sounds very inconvenient, or is that something you made yourself? – zapl Sep 30 '22 at 22:45
  • yes Json can also produce strings. I want one file that should contain all the events seperate by \n. so one file should contains Data which is a list of java objects seperated by \n – CyberPunk Oct 01 '22 at 06:50
  • then make it `serialize(Object[] objects)` or `serialize(Collection objects)` and build a single long serialized thing instead of sending individual bits in a loop – zapl Oct 03 '22 at 20:59

0 Answers0