AWS CLI for S3 Select

Question

I have the following code, which is used to run a SQL query on a keyfile, located in a S3 bucket. This runs perfectly. My question is, I do not wish to have the output written over to an output file. Could I see the output on the screen (my preference #1)? If not, what about an ability to append to the output file, rather than over-write it (my preference #2). I am using the AWS-CLI binaries to run this query. If there is another way, I am happy to try (as long as it is within bash)

aws s3api select-object-content \
    --bucket "project2" \
    --key keyfile1 \
    --expression "SELECT * FROM s3object s where Lower(s._1) = 'email@search.com'" \
    --expression-type 'SQL' \
    --input-serialization '{"CSV": {"FieldDelimiter": ":"}, "CompressionType": "GZIP"}' \
    --output-serialization '{"CSV": {"FieldDelimiter": ":"}}' "OutputFile"

I see you put a bounty on the question. Does it mean that the answer with `/dev/stdout` does not work? Can you provide an info why it does not work? Any errors? — Marcin, Aug 24 '20 at 02:30
Hi Marcin. thanks for checking in. i tried it on cygwin but it didnt work. No errors, no output. I could try on my ubuntu machine as well, in case u think this could be OS specific — rogerwhite, Aug 24 '20 at 06:06
No problem. You could comment at @jellycsc so that he gets notified. Maybe he knows how to do it on `cygwin`? But on ubuntu or any other linux I don't see a reason why it would not work. Thus I asked in the first place. — Marcin, Aug 24 '20 at 09:31

jellycsc · Answer 1 · 2021-04-28T01:28:23.090

10

Of course, you can use AWS CLI to do this since stdout is just a special file in linux.

aws s3api select-object-content \
--bucket "project2" \
--key keyfile1 \
--expression "SELECT * FROM s3object s where Lower(s._1) = 'email@search.com'" \
--expression-type 'SQL' \
--input-serialization '{"CSV": {"FieldDelimiter": ":"}, "CompressionType": "GZIP"}' \
--output-serialization '{"CSV": {"FieldDelimiter": ":"}}' /dev/stdout

Note the /dev/stdout in the end.

edited Apr 28 '21 at 01:28

answered Aug 17 '20 at 13:33

jellycsc

10,904
2
15
32

@ jellycsc - could you pls confirm if this would work on cygwin? – rogerwhite Aug 24 '20 at 13:23

score 0 · Answer 2 · answered Aug 17 '20 at 06:26

0

The AWS CLI does not offer such options.

However, you are welcome to instead call it via an AWS SDK of your choice.

For example, in the boto3 Python SDK, there is a select_object_content() function that returns the data as a stream. You can then read, manipulate, print or save it however you wish.

answered Aug 17 '20 at 06:26

John Rotenstein

241,921
22
380
470

Thanks John. I am trying to move away from Python (long story why). Is there any other workaround here? Could I append to a single output file maybe? – rogerwhite Aug 17 '20 at 07:25
You can use any of the other programming languages, too: C++, Go, Java, .Net, Node, PHP, Ruby. FYI, the AWS CLI is written in Python and simply calls the above command. – John Rotenstein Aug 17 '20 at 07:35

score -1 · Answer 3 · answered Nov 16 '20 at 05:14

-1

I think it opens /dev/stdout twice causing kaos.

answered Nov 16 '20 at 05:14

Geoff Johnson

1

AWS CLI for S3 Select

3 Answers3