1

I have text a file with a list of s3 objects, in the form of:

prefix_x/prefix_y/file_name_1 
prefix_w/prefix_z/file_name_88
etc...

I wrote a bash script to delete all of these objects, as follows:

#! /bin/bash

LIST_OF_PATHS=$1

while read FILE_PATH; do
  aws s3 rm s3://bucket-name/$FILE_PATH
done < $LIST_OF_PATHS

The script doesn't seem to delete the objects (they appear in the UI and in the terminal when lsing them with the CLI).

Further details and things I've already tried:

  • deleting the objects manually with similar command in the CLI - it works.
  • adding ls command to the loop provides no output, whereas typing this command manually on the very same files does give output.
  • adding sleep 0.1 to each iteration of the loop didn't help either.
  • of course the script runs - I see the output: delete: s3://bucket-name/prefix_x/prefix_y/file_name_1, but the file doesn't actually get deleted.
  • running a simpler bash script with the same command and a specific file name (not inside a loop) does delete successfully.

What might be the problem?

localhost
  • 461
  • 4
  • 19
  • 2
    Put `set -x` at the beginning of the script so you can see the commands as they're executing. – Barmar Jun 28 '21 at 08:29
  • Probably not the problem here, but quote your variables and use `read -r` as suggested by https://www.shellcheck.net/. – Socowi Jun 28 '21 at 08:42
  • 1
    Does the text file have any weird formatting (like [DOS/Windows line endings](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings)) or invisible "gremlin" characters in it? Try printing it with `LC_ALL=C cat -vet /path/to/textfile` -- you should see a "$" at the end of each line, but anything else indicates trouble (DOS/Windows line endings will look like "^M$"). P.s. I second the `set -x` suggestion. – Gordon Davisson Jun 28 '21 at 08:59
  • @Benjamin : Could it be that you have spaces in your `FILE_PATH`? But it is weird that you don't get any error message..... – user1934428 Jun 28 '21 at 08:59

2 Answers2

1

Solved!

The issue was that in a script, bash expects the newline char to be '\n', but my input file contained '\r' at the end of each line. More on this can be found here: https://superuser.com/questions/489180/remove-r-from-echoing-out-in-bash-script/489191

Many thanks to @Barmar, whose comment helped me see this and debug the issue.

I simply changed input file itself, and the script ran perfectly as it was.

localhost
  • 461
  • 4
  • 19
0
  1. Check your AWS permissions on the S3 service.
  2. Check your AWS command line tool configuration.
  3. Check your s3 bucket configuration.
  4. Run your AWS s3 command without using a loop, see if the command is run successfully, before using a loop
Shaqil Ismail
  • 1,794
  • 1
  • 4
  • 5