1

I am trying to rename a large amount of files located in an AWS S3 bucket. What I am doing is using psql cli within my docker container image.

I did docker exec -it [dbname] bash

Then did psql -c -d [dbname] select * from t1

How can I access the results from the psql query above and also loop through it so that for each iteration, I will use AWS CLI command operations to move and rename object in bucket A to bucket B

E.g
//Results from query
**NAME**
George
Kim
Sam

**S3 Bucket A**
file_one.pdf
file_two.pdf
file_three.pdf

**S3 Bucket B (need to rename and move here)**
file_one_george.pdf
file_one_kim.pdf
file_one_same.pdf

I saw something like this aws s3 --recursive mv s3://<bucketname>/<folder_name_from> s3://<bucket>/<folder_name_to> here stackoverflow

I need to run an AWS command similar to that post but need to loop through the psql query results. Not sure about this but do I also need some kind of logic for each file in the AWS Bucket A. Such as

if (file one name in bucket A == psql query result row[x] name)
  //run aws cli command to rename file and move to new bucket
hnhl
  • 127
  • 8

1 Answers1

0

You need to write the results from your query to a file. You can do this with the redirection operator >. The following will run the query and put the result into output.txt

psql -d [dbname] -c 'select * from t1' > output.txt

Even though we got the query to store in the output.txt, it is probably still on the docker image you initially logged into. You will need to mount a local directory as a volume to have file output or pipe the output of the docker query to file. For Example, for getting the table accounts in a DB named my_db with a user named postgres, the command would be:

docker exec -it my_db bash -c "psql -U postgres -d my_db -c 'select * from accounts'" > output.txt

Your command might be (although you might need to change the -U postgres for the username for your DB):

docker exec -it [dbname] bash -c "psql -U postgres -d [dbname] -c 'select * from t1'" > output.txt

You can then load the query results from the file to run a script that will rename the S3 buckets with the aws command.

DogEatDog
  • 2,899
  • 2
  • 36
  • 65
  • Thank you for the suggestion. For the last part where I need to load the query results from the file and run a script, do I do that with the AWS cli? Meaning does the AWS cli have a utility to process all the rows in the file and make a new name/destination for them? – hnhl Jan 12 '22 at 17:41