Bash script to read ids from file and spawn process in a controlled manner

Question

I need to read hundreds of ids from a file and pass those to another shell script as parameters to be spawned as separate child requests. [Done]

But we cannot spawn more than 6 child requests i.e., not more than 6 requests can be running at a given point in time.

I have gone through this site (references given below) and others and came to know that you can get the PID of the spawned process using $! but am faraway from implementing it as I do not know how to store them in an array and delete it once the spawned process is complete.

Forking / Multi-Threaded Processes | Bash

How to wait in bash for several subprocesses to finish and return exit code !=0 when any subprocess ends with code !=0?

#!/bin/bash
file="/usr/share/nginx/html/cron/userids.txt" //file containing the userids that needs to be spawned

MAXCOUNT=6 //maximum number of child request that can be spawned

while IFS= read -r line
do
    #submit a background job here

    sh fetch.sh $line & //this is the shell script that needs to be submitted

    //check if the spawned request count is less than MAXCOUNT
    //If it is then wait for one or more request to finish 
    //If the number of child requests is less than MAXCOUNT then spawn another request 
    //if all the lines are read then wait till all the child process completes and then exit

done <"$file"

Please be cognizant that I am newbie and do not know much about the shell process.

Will appreciate any directions and feedback.

Thanks

Answer 1

You can use xargs to spawn a maximum number of processes passing the arguments read from stdin

xargs -n 1 -P 6 fetch.sh < "$file"

Answer 2

You can use GNU Parallel for this:

parallel -j6 -a "$file" fetch.sh

It has lots of options for handling failures, progress bars, logging etc.