Shell script calls Jar with arguments - List of filenames treated as one String

Question

I have a java app used to automate deployment of files. I would like to create a shell script to make the deployment from svn easier.

Currently, my script checks out the files, gets the filenames, asks for some deployment info then passes the file names to the Jar.

Usually, I can pass multiple files after the -f option, and the Jar collects each file and if the file exists, adds them into an array for processing. If the filename contains a space, I add quotation marks around the filename.

This is the input that usually works with my Jar:

java -jar deploy.jar -f 'Cancellation V2.xsl' 'Final Price.xsl' -e test

My shell script takes the file name from the end of the SVN url, adds quotation marks around the name(s) and passes to the Jar. I get this error:

'Final Price.xsl' 'Cancellation V2.xsl'  not found!
 Exception in thread "main" com.DeployException: 
     at com.ConsoleUtils.parseCommandLineArgs(ConsoleUtils.java:42)
     at com.Main.main(Main.java:23)

This Exception is thrown when the Java app uses if(file.exists) on each filename before adding it to the array. This log shows that both file names are being passed as a single string, even though the shell script has formatted them correctly.

Here is my full Shell script:

#!/bin/bash
now=`date +%Y%m%d%H%M`
mkdir 'deployment-'$now
cd 'deployment-'$now

echo 'Please type tag name to deploy from:'
   read tag

#svn export svn://------[removed]------
#This checks out a list of SVN urls to deploy.

j=1
for i in `cat deploy-filelist`
do
   filename=$i
   $j = $j++
   svn export $filename
   echo $filename | rev | cut -d/ -f1 | rev >> files.txt
done

echo 'loaded' $j 'files'

#svn export svn://------[removed]------
#This checks out the Jar used to deploy the files

files=''
for f in `cat files.txt`
do
   filename="${f/\%20/ }"
   files+="'$filename' "
done

echo $files >> files.txt

echo "Please type environment to deploy to [devl | test | etc]:"
  read env

java -jar deploy.jar -f "$files" -e $env

Can anyone give me some advice on handling the file names? Why is the variable read as one String when the same format is usually ok?

Other comments aprreciated.

Thanks for reading

Answer 1

The last line of the code shows "$files", which means that the following is performed to the strings:

"'filename1' 'filename2'"

This is therefore read as one string by the JVM and passed through as a single argument.

To fix this, remove the quotes around $files as Mubin also suggests.

Further tip, try echoing your output at the different stages of the Shell so that you can see what's happening to the variables throughout the program. Helps massively when debugging Shell.

Answer 2

There really isn't a good way in "classic" Bourne shell to quote stuff with internal quotes and interpolate it later. Since you are using Bash on the shebang line, the simple and straightforward workaround is to have your list of file names in an array instead.

IFS=$'\n' urls=($(cat deploy-filelist))
for u in "${urls[@]}"; do
    svn export "$u"
done
echo "Exported ${#urls[@]} files"

IFS=$'\n' basenames=($(sed 's%.*/%%;s/%20/ /g' deploy-filelist))

Notice how this does away with some of the rather tortured shell script you had around writing and rewriting the files.txt file, which I assume you don't need for anything else (creating it as a byproduct is obviously trivial if you need it for other reasons).

(I renamed the arrays to urls and basenames to keep them self-documenting and unique. The first one apparently contains the SVN repo URLs while the second contains just the nondirectory part, with any %20 normalized back to a space.)

Now you can interpolate the array, with quoting preserved, like

java -jar deploy.jar -f "${basenames[@]}" -e $env

(This still looks slightly funny to me. Is the -f option a boolean and you supply two file names, or is -f an option which requires a file name argument, and you should have two of them?)

Answer 3

The problem starts here:

`for f in `cat files.txt`

You might assume that this will give you one f per line, but it won't, it will give you one f per word. So if your files have spaces, it will split them.

You can replace it with:

cat files.txt | while read -r
do
    f="$REPLY"
    # rest of your code here
done

Then still drop the double quotes at the jar invocation line.

The read command will read each line from stdin, where we are piping the list of files, one per line, and store it in the $REPLY variable. It does not split words like the for does.

Answer 4

a quick rewrite, incorporating much of the given advice, adding a little of my own:

#!/bin/bash
dir=deployment-$(date +%Y%m%d%H%M)
mkdir "$dir" || { echo "could not mkdir $dir"; exit 1; }
cd "$dir"

read -p 'Please type tag name to deploy from: ' tag

#svn export svn://------[removed]------
#This checks out a list of SVN urls to deploy.

j=1
files=()
while read -r filename; do
    if [[ -f "$filename" ]]; then
        ((j++))
        svn export "$filename"
        files+=( "$(basename "$filename")" )
    else
        echo "warning: no such file: $filename"
    fi
done < deploy-filelist

echo "loaded $j files"
echo ${files[@]}" >> files.txt

#svn export svn://------[removed]------
#This checks out the Jar used to deploy the files

while :; do
    read -p "Please type environment to deploy to [devl | test | etc]: " env
    case $env in
        devl|test|etc) break ;;
    esac
done

java -jar deploy.jar -f "${files[@]}" -e $env