running list of bash commands in a file in parallel

Question

I have a simple text file (command_script.txt) that I am executing as a bash script.

The content of command_script.txt is something like this:

#!/bin/bash
some_command -flags input1 output1
some_command -flags input2 output2
some_command -flags input3 output3
...
some_command -flags input1000 output1000

I execute it on the command line (./command_script.txt) and it runs through one line at a time. Is there an easy way I can run 20 lines at a time in parallel?

Thanks!

Look at **GNU Parallel** Remove the shebang from the first line and try `parallel --dry-run -j20 -a YOURFILE` — Mark Setchell, Feb 27 '22 at 18:53

score 1 · Answer 1 · answered Feb 27 '22 at 18:55

1

By appending "&" at the end of a line a process is launched in background. Hence, the next one is started immediately such that both are running in parallel.

answered Feb 27 '22 at 18:55

miro

67
6

3

It is possible that starting 1000 processes at the same time will cause problems. – Cyrus Feb 27 '22 at 19:00

Ole Tange · Answer 2 · 2022-03-01T17:36:39.380

1

#!/usr/bin/parallel --shebang -j20
some_command -flags input1 output1
some_command -flags input2 output2
some_command -flags input3 output3
...
some_command -flags input1000 output1000

If the commands are literaly the ones given, you can instead do:

seq 1000 | parallel -j20 some_command -flags input{} output{}

edited Mar 01 '22 at 17:36

answered Feb 28 '22 at 12:28

Ole Tange

31,768
5
86
104

running list of bash commands in a file in parallel

2 Answers2