0

I am trying to read around 1.5GB size from s3 using ruby sdk for aws. S3 gem is - gem 'aws-sdk' ( https://github.com/aws/aws-sdk-ruby )

most_recent_s3_object = s3_resource.bucket(bucket_name).objects.max_by(&:last_modified)

Here most_recent_s3_object class is Aws::S3::ObjectSummary When I do most_recent_s3_object.get I get broken pipe error. Is there a way around to get this file in a whole for this size?

Praveen Kumar
  • 332
  • 4
  • 11
  • 1
    Which S3 gem are you using? What is the `class` of `most_recent_s3_object`? – Jared Beck Aug 30 '21 at 19:03
  • @JaredBeck updated the question with these details. – Praveen Kumar Aug 30 '21 at 19:17
  • `objects.max_by(&:last_modified)` is downloading every object from the entire bucket in order to find the most recently modified one. Unfortunately this kind of thing is unavailable server side for s3. See https://github.com/aws/aws-cli/issues/1104 and https://stackoverflow.com/questions/45429556/how-list-amazon-s3-bucket-contents-by-modified-date – melcher Aug 30 '21 at 22:13
  • You can use `s3api list-objects` but it's still client side filtering and only works up to a limited # of objects – melcher Aug 30 '21 at 22:14

0 Answers0