1

I need a loop in a Bash script (analysis-run.sh) for running many queries. As I have many queries I can't run them manually so I need a way to automate them. So far, I created a file inputs.txt with all my queries and at the end of the bash script file I added the following:

while read f ; do
  ./analysis-run.sh $f 
done < imputs.txt

With that loop, analysis-run is only running the first query of inputs.txt over and over again. I am really new to this, so any help would be appreciated.

The content of imputs.txt is:

bones

muscles

blood

saliva

and so on..

The content of analysis-run.sh is:

Execute this script as ./analysis-run.sh [query] [group]

query=$1

group=$2

if [ $group = "clean" ]; then

    cluster=A

else

    cluster=B

fi

adamo-obtain_bundance.py - query $query -ref combined_$cluster.$group.align -splits 1 -group $group

adamo-obtain_structure.py -i $query.combined_$query$group.csv -o $query.$group -cutoff 0.5 -group $group

Community
  • 1
  • 1
Oscar
  • 21
  • 4
  • can you post the bash code you have so far? also if you want to run multiple queries individually you will have to delimit the file some how so you can process one query at a time. what are the queries, sql queries, shell commands? – Jpsh Apr 01 '17 at 17:05
  • In your question, the loop is fed `samples.txt`. Should that instead be the `inputs.txt` file mentioned in the question? – Fred Apr 01 '17 at 17:10
  • Also, please provide sample content from your `inputs.txt` file. Does it contain one query per line? – Fred Apr 01 '17 at 17:11
  • Yes, it should be inputs.txt. The content is one query per line: query1 query2 query3.. etc... each of them in a line – Oscar Apr 01 '17 at 18:00
  • 1
    Beware the unquoted `$f` if any line in `samples.txt` have characters meaningful to Bash. Unquotes lines will glob and word split. Beware of `read` without `-r` that will mangle escapes. A simple run through https://www.shellcheck.net will flag these errors. – dawg Apr 01 '17 at 18:02
  • 1
    Please post the contents of `samples.txt` and the method that `analysis-run.sh` uses to read its command line – dawg Apr 01 '17 at 19:14
  • I added a subsample of imputs.txt and the content of the script. Thanks – Oscar Apr 01 '17 at 19:36
  • If you only have 1 item per line in `samples.txt` then `group=$2` will always be a null value. – dawg Apr 02 '17 at 00:48

2 Answers2

0

Using loops in Bash can sometimes work, but it is loaded with perils.

Using xargs is usually the cleanest, most robust approach...

<inputs.txt xargs --max-args=1 do_something

The command to execute could be provided as a Bash function...

function do_something
   {
   echo value=${1}
   }

Although the call to xargs is somewhat more involved when taking that approach. See: Calling functions with xargs within a bash script

About xargs

xargs takes a list of arguments (usually file names), which are provided as an input file or stream, and it places those arguments on the command-line for another specified command or function. If the command can handle multiple input arguments, you can drop the --max-args=1 option.

Community
  • 1
  • 1
Brent Bradburn
  • 51,587
  • 17
  • 154
  • 173
  • Using this approach, you can do even more fun things -- like executing your file processing in parallel using the `--max-procs=max-procs` option. [xargs manual](https://www.gnu.org/software/findutils/manual/html_node/find_html/xargs-options.html) – Brent Bradburn Apr 01 '17 at 17:24
  • 1
    While the xargs could be fun, it isn't fun when here are spaces in the input. You need to use the `-0` and in pipes not all commands supports the `\0`. Even more, unfortunately the `GNU` variants are difer from BSD. (in command line switches - eg. need to use `-n1` instead of the `--max-args=1`.. So, sure it is cool - but not as easy as it sounds on the 1st... :) – clt60 Apr 01 '17 at 18:14
  • 1
    ... and also the claim about the function usage isn't true. You can't use function, in way as you answered. (you must export the function and call `bash -c`) – clt60 Apr 01 '17 at 18:23
  • 1
    @jm666: Good point about the function usage -- the call to `xargs` is [more involved](http://stackoverflow.com/q/11003418/86967) in that case. Spaces in the filenames are a common problem in Bash programming, but `xargs` is smart enough to deal with that if you have some [unambiguous delimiter](http://stackoverflow.com/questions/19204531/can-xargs-default-delimiter-be-changed), or use quoting. – Brent Bradburn Apr 01 '17 at 19:09
  • 1
    As I said - the xargs is cool! :) (i love it) - just need to know how to use it, and sometimes it is hard especially for the beginners. Nice links in the comment! – clt60 Apr 01 '17 at 19:14
0

With that loop, analysis-run is only running the first query of inputs.txt over and over again.

The problem (probably) is that you need to quote $f:

while read -r f ; do
  ./analysis-run.sh "$f" 
done < samples.txt

Without the quotes, the line read from samples.txt will be subject to word splitting and glob expansion.

Read http://tldp.org/LDP/abs/html/quotingvar.html

And run your scripts though ShellCheck

dawg
  • 98,345
  • 23
  • 131
  • 206
  • I added the -r and quote $f but the problem remains the same. Many thanks for info about shell checking though. – Oscar Apr 01 '17 at 19:04