1

I have a task flow which can be divided into step_1 and step_2, both running in bash. step_1 needs run a bash script in folder_a. when I submit my jobs into multiple hosts, there are multiple threads which will trigger the bash script in folder_a, this will break the function.

I want my jobs run as following :

  1. every job's step_1 has to run one by one, otherwise it will break function in the folder_a.
  2. every job's step_2 can run parallelly for speed.

How to lock the folder if the bash in folder_a is running and refuse to run for other thread ? Maybe I can only lock the step_1 bash top script inside folder_a, instead of the folder.

jacobi
  • 13
  • 3
  • 5
    see https://man.cx/flock or https://man.cx/lockfile – pynexj Oct 28 '20 at 10:34
  • You can use **GNU Parallel** as a mutex... https://stackoverflow.com/a/37303133/2836621 You can also use **GNU Parallel** to run your jobs in parallel, on multiple hosts. – Mark Setchell Oct 31 '20 at 12:58

1 Answers1

0

If you are working under a Unix-like system (GNU/Linux, BSD...) you probably have flock and you could wrap the critical jobs in flock calls:

step1_job: | folder_a/.lock
    flock $| command [arguments]

folder_a/.lock:
    touch $@

See man flock for the details.

Renaud Pacalet
  • 25,260
  • 3
  • 34
  • 51
  • I tested flock, it works for my needs, added flock -x . before the command which needs to be locked for multiple process. example in my top task shell script : 1. cd folder_a; 2. flock -x . command_for_step1 – jacobi Nov 02 '20 at 09:17