How to start multiple job instance in a shell script to process multiple files in a directory?

Question

#!/bin/bash

      data_dir=./all
      for file_name in "$data_dir"/*
      do
        echo "$file_name"
        python process.py "$file_name"
      done

For example, this script processes the files sequentially in a directory in a 'for' loop. Is it possible to start multiple process.py instances to process files concurrently? I want to do this in a shell script.

score 1 · Answer 1 · answered May 11 '22 at 20:43

1

It's better to use os.listdir and subprocess.Popen to start new processes.

answered May 11 '22 at 20:43

Yehor Smoliakov

326
3
13

So, if I use the subprocess.open() to start a new process for each file in a list, it one process won't wait for the other to complete? I want multi python instances to process at the same time. – marlon May 11 '22 at 20:47
Could you write an answer based on the link? – marlon May 11 '22 at 20:49
1

Yes, all processes will work independently. You can use `communicate()` method to block the next processes until a running one will not be finished. – Yehor Smoliakov May 11 '22 at 20:49
So I shouldn't use 'communicate()' for concurrency? – marlon May 11 '22 at 20:52
subprocess.Popen(["python process.py", "all"]), but this gives an error. 'all' is my data directory. – marlon May 11 '22 at 20:55

score 0 · Accepted Answer · answered May 11 '22 at 21:09

0

I have another possibility for you, if still needed. It uses the screen command to create a new detached process with the supplied command.

Here is an example:

#!/bin/bash

data_dir=./all
for file_name in "$data_dir"/*
do
  echo "$file_name"
  screen -dm python process.py "$file_name"
done

answered May 11 '22 at 21:09

cfgn

201
2
10

Does this process files concurrently for each file, to start its own program instance? – marlon May 11 '22 at 21:17
Yes, it should be totally different processes. – cfgn May 11 '22 at 21:18
Those can then be seen (while running) through `screen -li`. – cfgn May 11 '22 at 21:18
This looks very simple, and I will try it! – marlon May 11 '22 at 21:19
How to add a more parameter, integer, like 10? – marlon May 11 '22 at 21:26
Adding the parameter after the command should allow the parameter to be fed to it like so: `screen -dm python process.py "$file_name" 10`. – cfgn May 11 '22 at 21:31

Mark Setchell · Answer 3 · 2022-05-11T22:03:52.977

With GNU Parallel, like this:

parallel python process.py {} ::: all/*

It will run N jobs in parallel, where N is the number of CPU cores you have, or you can specify -j4 to run on just 4, for example.

Many, many options for:

logging,
splitting/chunking inputs,
tagging/separating output,
staggering job starts,
massaging input parameters,
fail and retry handling,
distributing jobs and data to other machines
and so on...

Try putting [gnu-parallel] in the StackOverflow search box.

How to start multiple job instance in a shell script to process multiple files in a directory?

3 Answers3