proper use of && and wait in bash for mutitasking

Question

here is a shell script which takes domain and its parameters to find status code . this runs way faster due to threading but misses lot of requests.

while IFS= read -r url <&3; do
    while IFS= read -r uri <&4; do
    urlstatus=$(curl -o /dev/null --insecure --silent --head --write-out  '%{http_code}' "${url}""${uri}" --max-time 5 ) &&
    echo "$url  $urlstatus $uri" >> urlstatus.txt &
done 4<uri.txt 
done 3<url.txt

if i ran normally it process all requests but the speed is very low. is there a way through which speed is maintained and it also not misses all requests .

There is no threading here. You are running some background processes. — tripleee, Apr 05 '17 at 03:45
Actually what I am feeling is that all requests are gone into memory to process and then it processes some of them, probably a wait for all jobs is sufficient I suppose. — user7423959, Apr 05 '17 at 04:33
You are throwing away the failures with the `&&`. Probably you want to change that to just `;` — tripleee, Apr 05 '17 at 04:36

score 1 · Answer 1 · edited May 23 '17 at 12:10

The symptom you describe sounds like some resource outside of your program can't cope with the parallelism, but you have not provided any diagnostics to help troubleshoot this problem. If your local network bandwidth is sufficient but the remote server is throttling you, there isn't much you can do to persuade them to stop doing that, while of course if your problem is caused by local resource depletion, increasing bandwidth through the choke point (your DNS server?) should allow you to proceed without any actual code changes.

A common workaround is to throttle things on your end. Use xargs or parallel to run a controlled number of parallel processes instead of unleashing them all at the same time to fiercely compete over your finite resources.

See e.g. Bash: limit the number of concurrent jobs? for how to do that in practice.

Maybe add a small sleep between runs to calm things down if that alone is insufficient.

proper use of && and wait in bash for mutitasking

1 Answers1

Linked