Bash script to complete 21 process by running 4 process parallel

Question

I am new to scripting. I have 21 directories to compress and copy to another location. I want to run 4 process in parallel and if one process complete then start another one. like wise 21 directories should complete.

I got some idea from following post. bash script to check running process

I need this to add, if number of running service < 4, then start another process.

Please help me regarding this matter.

My script is as follows. Here I used ping command to test process. If this is working I can arrange it to copy command.

job.sh

#!/bin/bash

cat my.txt |while read line
do
 run_count=`ps eax | grep ping | grep -v grep | wc -l`
 if [ 4 -gt ${run_count} ]
 then
  /home/cms.sh $line &
 fi
done

cms.sh

#!/bin/bash
value=$1
cmd=ping
host=yahoo.com

$cmd -c $1 $host >> log-$1.txt

my.txt

If I run this, it is starting to run 4 process using first 4 lines in my.txt. After finish that initial 4 processes it will not continue remain values in my.txt.

Please let me know where I did mistake.

Thank you in advance!!!

possible duplicate of [Run bash commands in parallel, track results and count](http://stackoverflow.com/questions/6384013/run-bash-commands-in-parallel-track-results-and-count) — Mat, Jul 03 '12 at 06:38
What have you tried? Do you have any code to show us? How would you do this with one service? — John Lyon, Jul 03 '12 at 06:40
Please see [Process Management](http://mywiki.wooledge.org/ProcessManagement). — Dennis Williamson, Jul 03 '12 at 11:18

score 2 · Answer 1 · answered Jul 03 '12 at 07:34

Since you are new to scripting, take a look at xargs -P. It is a command worth becoming familiar with even if you are not running processes in parallel and it be somewhat more likely to be pre-installed on your system than parallel mentioned in the answer pointed to by Mat.

score 1 · Answer 2 · answered Jul 04 '12 at 23:13

There is pbzip2: parallel bzip2 compression for this kind of purpose. Also, 7z has multi-threading options, but I haven't tried them

I use this kind of setup usually:

tar -cp somedir_tobackup/ | pbzip2 | mbuffer -m 180M | netcat <host> <port>

Obviously, you can use cpio instead, drop the mbuffer, save to a local archive file first, use rsync with it etc. but the idea is there.

score 0 · Answer 3 · answered Jan 17 '14 at 22:26

Using GNU Parallel you can do:

cat my.txt | parallel /home/cms.sh

Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (http://www.gnu.org/software/parallel/parallel_tutorial.html). You command line with love you for it.

Mark Setchell · Answer 4 · 2014-01-18T12:46:22.637

You need a second loop just after your "do" that sleeps repeatedly for 1 second until there are not 4 processes running, and it then breaks out of the loop and starts a new process before reading another line from your control file. Something like this:

#!/bin/bash

while read line
do
   while :
   do
      run_count=`ps eax | grep  ping | grep -v grep | wc -l`
      [ 4 -gt ${run_count} ] && break;
      sleep 1
   done
   ./somejob $line &
done < my.txt

score -1 · Answer 5 · edited Jul 05 '12 at 14:23

I reckon that you got quad cores thus 4 processes. Wrong solution. Instead, count number of cores (this way it can be extended to dual cores, quad cores, 6 cores, etc, without manually changing process variable). This example is the best I can in running bash parallel. It is made by dividing the task in how many cores you got, clone script.cfg in how many cores you got, runs them in background. In this way, each core will compress only a part of those folders.

Create a folder let's say bash. Inside, create other 2 folders (backup and directory). Put those 21 folders inside directory. Create 2 files (make .sh file executable).

start.sh

#!/bin/bash
date=$(date '+%d_%B_%Y')
mkdir backup/$date
cores=$(ls  /proc/acpi/processor/ | wc -l)
cd directory
ls > ../list.result
cd ..
file_lines=$(cat list.result | wc -l)
let "split_lines=$file_lines/$cores"
let "split_lines=$split_lines+1"
split -d --lines=$split_lines list.result CPU
for script in `seq 1 $cores`
do
    let sufix=script-1
    cat script.cfg > "CPU0$sufix.script"
    chmod +x CPU0$sufix.script
    sed -i 's/filenumber/CPU0'$sufix'/g' CPU0$sufix.script
    ./CPU0$sufix.script &
done
exit

and script.cfg

#!/bin/bash
date=$(date '+%d_%B_%Y')
lenght=$(cat filenumber |wc -l)
counter=1
for x in `seq 1 $lenght`
do
    value="directory/"$(sed "${x}!d" filenumber)
    tar -zcvf $value.tar.gz $value
    mv directory/*.tar.gz backup/$date
done
rm *.script CPU* list.result
exit

All files in directory folder will be compressed using all cores in CPU and moved to backup folder. This technique runs all cores at 100% so take care of CPU temperature.

Bash script to complete 21 process by running 4 process parallel

5 Answers5