Bash script to change file names of different extension

Question

we are working on a angularjs project, where the compiled output contains lot of file extensions like js,css, woff, etc.. along with individual dynamic hashing as part of file name.

I am working on simple bash script to search the files belonging to the mentioned file extensions and moving to some folder with hashing removed by searching for first instance of '.'.

Please note file extension .woff and .css should be retained.

/src/main.1cc794c25c00388d81bb.js   ==> /dst/main.js  
/src/polyfills.eda7b2736c9951cdce19.js ==> /dst/polyfills.js  
/src/runtime.a2aefc53e5f0bce023ee.js ==> /dst/runtime.js  
/src/styles.8f19c7d2fbe05fc53dc4.css ==> /dst/styles.css  
/src/1.620807da7415abaeeb47.js  ==> /dst/1.js  
/src/2.93e8bd3b179a0199a6a3.js  ==> /dst/2.js  
/src/some-webfont.fee66e712a8a08eef580.woff  ==> /dst/some-webfont.woff  
/src/Web_Bd.d2138591460eab575216.woff ==> /dst/Web_Bd.woff  

Bash code:

#!/bin/bash
echo Process web binary files!

echo Processing the name change for js files!!!!!!!!!!!!!
sfidx=0;
SFILES=./src/*.js #{js,css,voff}
DST=./dst/

for files in $SFILES
do
        echo $(basename $files)
        cp $files ${DST}"${files//.*}".js  
        sfidx=$((sfidx+1))
done
echo Number of target files detected in srcdir $sfidx!!!!!!!!!!

The above code has 2 problems, Need to add file extensions in for loop at a common place, instead of running for each extension. However, this method fails, not sure this needs to be changed.

SFILES=./src/*.{js,css,voff}
cp: cannot stat `./src/*.{js,css,voff}': No such file or directory

Second, the cp cmd fails due to below reason, need some help to figure out correct syntax.

cp $files ${DST}"${files//.*}".js  
1.620807da7415abaeeb47.js
cp: cannot create regular file `./dst/./src/1.620807da7415abaeeb47.js.js': No such file or directory

Answer 1

Here is a relatively simple command to do it:

find ./src -type f \( -name \*.js -o -name \*.css -o -name \*.woff \) -print0 | 
while IFS= read -r -d $'\0' line; do
    dest="./dst/$(echo $(basename $line) | sed -E 's/(\..{20}\.)(js|css|woff)/\.\2/g')"
    echo Copying $line to $dest 
    cp $line $dest
done

Answer 2

This is based on the original code and is Shellcheck-clean:

#!/bin/bash

shopt -s nullglob       # Make globs that match nothing expand to nothing

echo 'Process web binary files!'

echo 'Processing the name change for js, css, and woff files!!!!!!!!!!!!!'
srcfiles=( src/*.{js,css,woff} )
destdir=dst

for srcpath in "${srcfiles[@]}" ; do
    filename=${srcpath##*/}
    printf '%s\n' "$filename"

    nohash_base=${filename%.*.*}    # Remove the hash and suffix from the end
    suffix=${filename##*.}          # Remove everything up to the final '.'
    newfilename=$nohash_base.$suffix

    cp -- "$srcpath" "$destdir/$newfilename"
done

echo "Number of target files detected in srcdir ${#srcfiles[*]}!!!!!!!!!"

The code uses an array instead of a string to hold the list of files because it is easier (and generally safer because it can handle file with names that contain spaces and other special characters). See Arrays [Bash Hackers Wiki] for information about using arrays in Bash.

See Removing part of a string (BashFAQ/100 (How do I do string manipulation in bash?)) for information about using ${var##pattern} etc. for extracting parts of strings.

See Correct Bash and shell script variable capitalization for an explanation of why it is best to avoid uppercase variable names (such as SFILES).

shopt -s nullglob prevents strange things happening if the glob pattern(s) fail to match. See Why is nullglob not default? for more information.

See Bash Pitfalls #2 (cp $file $target) for why it's generally better to use cp -- instead of plain cp (though it's not necessary in this case (since neither argument can begin with '-')).

It's best to keep Bash code Shellcheck-clean. When run on the code in the question it identifies the key problem, and recommends the use of arrays as a way to fix it. It also identifies several other potential problems.

Answer 3

Your problem is precedence of the expansions. Here is my solution:

#!/bin/bash
echo Process web binary files!

echo Processing the name change for js files!!!!!!!!!!!!!
sfidx=0;
SFILES=$(echo ./src/*.{js,css,voff})
DST=./dst/

for file in $SFILES
do
        new=${file##*/}                  # basename
        new="${DST}${new%\.*}.js"        # escape \ the .            
        echo "copying $file to $new"     # sanity check
        cp $file "$new"
        sfidx=$((sfidx+1))
done
echo Number of target files detected in srcdir $sfidx!!!!!!!!!!

With three files in ./src all named "gash" I get:

Process web binary files!
Processing the name change for js files!!!!!!!!!!!!!
copying ./src/gash.js to ./dst/gash.js
copying ./src/gash.css to ./dst/gash.js
copying ./src/gash.voff to ./dst/gash.js
Number of target files detected in srcdir 3!!!!!!!!!!

(You might be able to get around using eval, but that can be a security issue)

new=${file##*/} - remove the longest string on the left ending in / (remove leading directory names, as basename). If you wanted to use the external non-shell basename program then it would be new=$(basename $file).

${new%\.*} - remove the shortest string on the right starting . (remove the old file extension)

Answer 4

A possible approach is to have the find command generate a shell script, then execute it.

src=./src
dst=./dst
find "$src" \( -name \*.js -o -name \*.woff -o -name \*.css \) \
  -printf 'p="%p"; f="%f"; cp "$p" "'"${dst}"'/${f%%%%.*}.${f##*.}"\n'

This will print the shell commands you want to execute. If they are what you want, just pipe the output to a shell:

find "$src" \( -name \*.js -o -name \*.woff -o -name \*.css \) \
  -printf 'p="%p"; f="%f"; cp "$p" "'"${dst}"'/${f%%%%.*}.${f##*.}"\n'|bash

(or |bash -x if you want to see what is going on.)

If you have files named, e.g., ./src/dir1/a.xyz.js and ./src/dir2/a.uvw.js they will both end up as ./dst/a.js, the second overwriting the first. To avoid this, you might want to use cp -i instead of cp.

If you are absolutely sure that there will never be spaces or other strange characters in your pathnames, you can use less quotes (to the horror of some shell purists)

find $src \( -name \*.js -o -name \*.woff -o -name \*.css \) \
  -printf "p=%p; f=%f; cp \$p ${dst}/\${f%%%%.*}.\${f##*.}\\n"|bash

Some final remarks:

• %p and %f are expanded by -printf as the full pathname and the basename of the processed file. they enable us to avoid the basename command. Unfortunately, the is no such directive for the file extension, so we must use brace expansion in the shell to compose the final name.

• In the -printf argument, we must use %% to write a single percent character. Since we need two of them, there have to be four...

• ${f%%.*} expands in the shell as the value of $f with everything removed from the first dot onwards

• ${f##*.} expands in the shell as the value of $f with everything removed up to the last dot (i.e., it expands to the file extension)