Can S3 Select search multiple objects?

Question

I'm testing out S3 Select and as far as I understand from the examples, you can treat a single object (CSV or JSON) as a data store.

I wanted to have a single JSON document per S3 object and search the entire bucket as a 'database'. I'm saving each 'file' as <ID>.json and each file has JSON documents with the same schema.

Is it possible to search multiple objects in a single call? i.e. Find all JSON documents where customerId = 123 ?

We have added a script which can loop through the folder and run the S3 Select on files and write it to a file: https://link.medium.com/6vKrErw9qR — Nagaraj Tantri, Oct 30 '18 at 13:07

score 28 · Answer 1 · edited Feb 25 '19 at 14:04

28

It appears that Amazon S3 Select operates on only one object.

You can use Amazon Athena to run queries across paths, which will include all files within that path. It also supports partitioning.

edited Feb 25 '19 at 14:04

SColvin

11,584
6
57
71

answered Jun 06 '18 at 22:08

John Rotenstein

241,921
22
380
470

score -5 · Answer 2 · answered Apr 28 '20 at 09:50

-5

Simple, just iterate over the folder key in which you have all the files and grab the key and use the same to leverage S3 Select.

answered Apr 28 '20 at 09:50

Avish Saha

77
4

This is not what is asked. – Sigex Jan 28 '22 at 14:45

Can S3 Select search multiple objects?

2 Answers2

Linked