2

Because of Glacier Deep's expensive support for small objects, I am writing an archiver. It would be most helpful to me to be able to ask boto3 to give me a list of objects in the bucket which are not already in the desired storage class. Thanks to this answer, I know I can do this in a shell:

aws s3api list-objects --bucket $BUCKETNAME --query 'Contents[?StorageClass!=`DEEP_ARCHIVE`]'

Is there a way to pass that query parameter into boto3? I haven't dug into the source yet, but I thought it was essentially a wrapper on the command line tools- but I can't find docs or examples anywhere using this technique.

smac2020
  • 9,637
  • 4
  • 24
  • 38
Frank
  • 459
  • 2
  • 9

2 Answers2

4

Is there a way to pass that query parameter into boto3?

Sadly, you can't do this, as --query option is specific to AWS CLI. But boto3 is Python AWS SDK, so you very easily post-process its outputs to obtain the same results as from CLI.

The --query option is based on jmespath. So if you really want to use jmespath in your python, you can use jmespath package .

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • 1
    so what you're saying is that in fact, the `aws` command above is pulling back JSON for all the objects rather than doing an intelligent query, and it's being post-processed by the CLI tool? – Frank Aug 29 '21 at 04:46
  • 3
    @Frank Yes, exactly. You can do the same using `jq` to post-process AWS CLI output, instead of using `--query`. – Marcin Aug 29 '21 at 04:47
  • 1
    That's hugely helpful- if there's no intelligent filtering at the head end, then I'll just do whatever processing I want myself. Did you find that out by digging through source, or is this buried in documentation somewhere that I missed? How do I get to your level of familiarity? – Frank Aug 29 '21 at 04:49
  • 1
    Thanks. Its also stated in docs ([Client-side filtering](https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html#cli-usage-filter-client-side)) that query is client side only. I also learn as I go. By browsing SO I often learn something new about AWS:-) – Marcin Aug 29 '21 at 04:52
1

Query S3 Inventory size column with Athena.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html

vdm
  • 481
  • 4
  • 8