Renaming multiple files in bash by removing prefix and suffix containing special characters %?=

Question

I recently make a request against the Google Cloud Service API endpoint and wget a lot of files into one single folder. Owing to the fact that all sub-directories separator 0/ are being replaced by %2F with the addition of ?alt=media, all the downloaded files are contaminated with these strings. e.g.

hg38%2Fv0%2FHomo_sapiens_assembly38.dict?alt=media
hg19%2Fv0%2FHomo_sapiens_assembly19.fasta.alt?alt=media

I tried to test the following in bash and it returned the result i wanted:

echo "$hg19%2Fv0%2FHomo_sapiens_assembly19.fasta.alt?alt=media" | sed -e "s/^$hg19%2Fv0%2F//" -e "s/\?.*//g"

i.e. Homo_sapiens_assembly19.fasta.alt. Unfortunately when I scaled it up using,

for file in *; do 
    mv "$file" '$(echo "$file" | sed -e "s/^$hg19%2Fv0%2F//" -e "s/\?.*//g")' ; 
done

all the files turned into 1 file named "$file". I couldnt figure out why.

Please can anyone provide a solution to my problem? And if some of the files contain different repeats of "%2F", how can I elegantly only keep the string after the last "%2F" and string the "?alt=media" from the end in the same line?

Thank you in advance.

Do you have the `rename` command? It can do this without writing a script. — Barmar, Jun 22 '23 at 00:07
`$()` is not expanded inside single quotes, only double quotes. — Barmar, Jun 22 '23 at 00:07
It shouldn't have renamed them to `$file`, it should have renamed them to `$(echo "$file" | sed -e "s/^$hg19%2Fv0%2F//" -e "s/\?.*//g")` — Barmar, Jun 22 '23 at 00:08
I don't see how your initial test could have worked. It will try to use a variable named `$hg19`, which doesn't exist. — Barmar, Jun 22 '23 at 00:10
I posted an answer several minutes ago showing how to do it. — Barmar, Jun 22 '23 at 00:19

score 1 · Answer 1 · answered Jun 22 '23 at 00:13

actually to removing all occurrences of %2F except for the last one, you can do like this:

echo "hg38%2Fv0%2FHomo_sapiens_assembly38.dict?alt=media" | sed -e "s/.*%2F\([^%]*\)\?alt.*/\1/"

".*%2F" matches any characters followed by the last occurrence of "%2F".
"([^%]*)" captures any characters that are not "%".
"?alt.*" matches the string "?alt" followed by any characters.

result is :

Homo_sapiens_assembly38.dict

and about the for loop something like this :

for file in *
 do mv "$file" "$(echo "$file" | sed -e "s/^$hg19%2Fv0%2F//" -e "s/\?.*//g")"
done

Thank youuu for providing an answer to my question. Appreciate it!!! — newstudenttoprogramming, Jun 22 '23 at 00:26

score 1 · Accepted Answer · answered Jun 22 '23 at 00:15

Use .* to match everything up to the last %2F.

Put the command substitution inside double quotes, not single quotes. See Difference between single and double quotes in Bash

Don't put $ before hg at the beginning.

It's not a requirement, but sed commands are usually put in single quotes, unless you're using variables in the substitution.

for file in *; do 
    mv "$file" "$(echo "$file" | sed -e 's/^hg.*%2F//' -e 's/\?.*//g')" ; 
done

Renaming multiple files in bash by removing prefix and suffix containing special characters %?=

2 Answers2