Bash concurrency when writing to file

Question

I want to run a script agains a long subset of items, and each of them run concurrently, only when every iteration finishes, write it to a file.

For some reason, it writes to the file without finishing the function:

#!/bin/bash

function print_not_semver_line() {
    echo -n "$repo_name,"
    git tag -l | while read -r tag_name;do
        semver $tag_name > /dev/null || echo -n "$tag_name "
    done
    echo ""
}
csv_name=~/Scripts/all_repos/not_semver.csv
echo "Repo Name,Not Semver Versions" > $csv_name
while read -r repo_name;do
    cd $repo_dir
    print_not_semver_line >> $csv_name &    
done < ~/Scripts/all_repos/all_repos.txt

of course without &, it does what it supposed to do, but with it, it gets all messed up.

Ideas?

Does this answer your question? [two processes write to one file, prevent mixing the output](https://stackoverflow.com/questions/39061647/two-processes-write-to-one-file-prevent-mixing-the-output) — Joe, Jun 09 '20 at 13:31
maybe you can use gnu parallel. it has all sorts of concurrency tricks for you — on8tom, Jun 09 '20 at 13:45
Please explain *"only when every iteration finishes, write it to a file."* you have output per each iteration in both loops — alecxs, Jun 09 '20 at 13:52
if you want sequencial output in only one single file you must write into different files and merge into *not_semver.csv* — alecxs, Jun 09 '20 at 14:02
@alecxs every iteration outputs a single line, but as you see, there are multiple echoes. in reality every iteration needs to be in its own line (no importance of the order), but what it does is writes whatever is echoed straight to the file without order — Moshe, Jun 09 '20 at 14:14
i can only see every iteration is writing into **same** file *csv_name=~/Scripts/all_repos/not_semver.csv* what do you expect when writing all echos in **parallel**? — alecxs, Jun 09 '20 at 14:17
please explain the difference between *"no importance of the order"* and *"without order"* — alecxs, Jun 09 '20 at 14:24
first I echo `$repo_name`, in the same line I should have `$tag_name`, what happens when I run the script is i see in one line all `$repo_name` in the same line and the tags are scattered on a lot of consecutive lines — Moshe, Jun 09 '20 at 14:25
`repo1,repo2,repo3,repo4` is the first line. `tag1,tag20,tag5,tag123,tag9` is the second line, then another 10 blank lines, then `tag4530`, next is `tag4588,tag0939`... you get the point — Moshe, Jun 09 '20 at 14:28
ok i understand know, you want to preserve order within each line but don't care about order of the lines - in this case write each line from one single echo — alecxs, Jun 09 '20 at 14:41
@alecxs how so? see I have a `while` loop inside the function. I'm writing conditionally — Moshe, Jun 09 '20 at 14:58
yes i know, but you can't use 3x echos when running multiple instances in parallel, each echo will open+close the file. write string into variable (or tmp file) before — alecxs, Jun 09 '20 at 15:03
But this is why I added all `echo`es to a function, then I redirect all of the function output to a file. — Moshe, Jun 09 '20 at 15:05
please update the question with a sample of output, including an example of a single `$repo_name` with multiple `$tag_name`s (`repo1, tag1 tag2` or `repo1, tag1\nrepo1, tag2`?) in the meantime, use an intermediate variable to store the various values and then issue a single `echo` ... this should reduce the splitting of `repo` and `tag` but won't address multiple processes printing to same line — markp-fuso, Jun 09 '20 at 15:36

r2evans · Answer 1 · 2020-06-10T14:50:17.913

Here's an alternative that uses xargs for its natural parallelization, and a quick script that determines all of the non-semver tags and outputs at the end of the repo.

The premise is that this script does nothing fancy, it just loops over its provided directories and does one at a time, where you can parallelize outside of the script.

#!/bin/bash

log() {
    now=$(date -Isec --utc)
    echo "${now} $$ ${*}" > /dev/stderr
}

# I don't have semver otherwise available, so a knockoff replacement
function is_semver() {
    echo "$*" | egrep -q "^v?[0-9]+\.[0-9]+\.[0-9]+$"
}

log "Called with: ${@}"
for repo_dir in ${@} ; do
    log "Starting '${repo_dir}'"
    bad=$(
        git -C "${repo_dir}" tag -l | \
            while read tag_name ; do
                is_semver "${tag_name}" || echo -n "${tag_name} "
            done
       )
    log "Done '${repo_dir}'"
    echo "${repo_dir},${bad}"
done
log "exiting"

I have a project directory with various cloned github repos, I'll run it using xargs here. Notice a few things:

I am demonstrating calling the script with -L2 two directories per call (not parallelized) but -P4 four of these scripts running simultaneously
everything left of xargs in the pipe should be your method of determining what dirs/repos to iterate over
the first batch of processes starts with PIDs 17438, 17439, 17440, and 17442, and only when one of those quits (17442 then 17439) are new processes started
if you are not concerned with too many things running at once, you might use xargs -L1 -P9999 or something equally ridiculous :-)

$ find . -maxdepth 2 -iname .git | sed -e 's,/\.git,,g' | head -n 12 | \
    xargs -L2 -P4 ~/StackOverflow/5783481/62283574_2.sh > not_semver.csv
2020-06-09T17:51:39+00:00 17438 Called with: ./calendar ./callr
2020-06-09T17:51:39+00:00 17439 Called with: ./docker-self-service-password ./ggnomics
2020-06-09T17:51:39+00:00 17438 Starting './calendar'
2020-06-09T17:51:39+00:00 17440 Called with: ./ggplot2 ./grid
2020-06-09T17:51:39+00:00 17439 Starting './docker-self-service-password'
2020-06-09T17:51:39+00:00 17442 Called with: ./gt ./keyring
2020-06-09T17:51:39+00:00 17440 Starting './ggplot2'
2020-06-09T17:51:39+00:00 17442 Starting './gt'
2020-06-09T17:51:39+00:00 17442 Done './gt'
2020-06-09T17:51:40+00:00 17442 Starting './keyring'
2020-06-09T17:51:40+00:00 17438 Done './calendar'
2020-06-09T17:51:40+00:00 17438 Starting './callr'
2020-06-09T17:51:40+00:00 17439 Done './docker-self-service-password'
2020-06-09T17:51:40+00:00 17439 Starting './ggnomics'
2020-06-09T17:51:40+00:00 17442 Done './keyring'
2020-06-09T17:51:40+00:00 17439 Done './ggnomics'
2020-06-09T17:51:40+00:00 17442 exiting
2020-06-09T17:51:40+00:00 17439 exiting
2020-06-09T17:51:40+00:00 17515 Called with: ./knitr ./ksql
2020-06-09T17:51:40+00:00 17518 Called with: ./nanodbc ./nostalgy
2020-06-09T17:51:40+00:00 17515 Starting './knitr'
2020-06-09T17:51:40+00:00 17518 Starting './nanodbc'
2020-06-09T17:51:41+00:00 17438 Done './callr'
2020-06-09T17:51:41+00:00 17438 exiting
2020-06-09T17:51:42+00:00 17440 Done './ggplot2'
2020-06-09T17:51:42+00:00 17440 Starting './grid'
2020-06-09T17:51:43+00:00 17518 Done './nanodbc'
2020-06-09T17:51:43+00:00 17518 Starting './nostalgy'
2020-06-09T17:51:43+00:00 17518 Done './nostalgy'
2020-06-09T17:51:43+00:00 17518 exiting
2020-06-09T17:51:43+00:00 17440 Done './grid'
2020-06-09T17:51:43+00:00 17440 exiting
2020-06-09T17:51:44+00:00 17515 Done './knitr'
2020-06-09T17:51:44+00:00 17515 Starting './ksql'
2020-06-09T17:51:55+00:00 17515 Done './ksql'
2020-06-09T17:51:55+00:00 17515 exiting

The output, in not_semver.csv:

./gt,
./calendar,
./docker-self-service-password,2.7 2.8 3.0
./keyring,
./ggnomics,
./callr,
./ggplot2,ggplot2-0.7 ggplot2-0.8 ggplot2-0.8.1 ggplot2-0.8.2 ggplot2-0.8.3 ggplot2-0.8.5 ggplot2-0.8.6 ggplot2-0.8.7 ggplot2-0.8.8 ggplot2-0.8.9 ggplot2-0.9.0 ggplot2-0.9.1 ggplot2-0.9.2 ggplot2-0.9.2.1 ggplot2-0.9.3 ggplot2-0.9.3.1 show
./nanodbc,
./nostalgy,
./grid,0.1 0.2 0.5 0.5-1 0.6 0.6-1 0.7-1 0.7-2 0.7-3 0.7-4
./knitr,doc v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 v0.8 v0.9 v1.0 v1.1 v1.10 v1.11 v1.12 v1.13 v1.14 v1.15 v1.16 v1.17 v1.18 v1.19 v1.2 v1.20 v1.3 v1.4 v1.5 v1.6 v1.7 v1.8 v1.9
./ksql,0.1-pre1 0.1-pre10 0.1-pre2 0.1-pre4 0.1-pre5 0.1-pre6 0.1-pre7 0.1-pre8 0.1-pre9 0.3 v0.2 v0.2-rc0 v0.2-rc1 v0.3 v0.3-rc0 v0.3-rc1 v0.3-rc2 v0.3-rc3 v0.3-temp v0.4 v0.4-rc0 v0.4-rc1 v0.5 v0.5-rc0 v0.5-rc1 v4.1.0-rc1 v4.1.0-rc2 v4.1.0-rc3 v4.1.0-rc4 v4.1.1-rc1 v4.1.1-rc2 v4.1.1-rc3 v4.1.2-beta180719000536 v4.1.2-beta3 v4.1.2-rc1 v4.1.3-beta180814192459 v4.1.3-beta180828173526 v5.0.0-beta1 v5.0.0-beta10 v5.0.0-beta11 v5.0.0-beta12 v5.0.0-beta14 v5.0.0-beta15 v5.0.0-beta16 v5.0.0-beta17 v5.0.0-beta18 v5.0.0-beta180622225242 v5.0.0-beta180626015140 v5.0.0-beta180627203620 v5.0.0-beta180628184550 v5.0.0-beta180628221539 v5.0.0-beta180629053850 v5.0.0-beta180630224559 v5.0.0-beta180701010229 v5.0.0-beta180701053749 v5.0.0-beta180701175910 v5.0.0-beta180701205239 v5.0.0-beta180702185100 v5.0.0-beta180702222458 v5.0.0-beta180706202823 v5.0.0-beta180707005130 v5.0.0-beta180707072142 v5.0.0-beta180718203558 v5.0.0-beta180722214927 v5.0.0-beta180723195256 v5.0.0-beta180726003306 v5.0.0-beta180730183336 v5.0.0-beta19 v5.0.0-beta2 v5.0.0-beta20 v5.0.0-beta21 v5.0.0-beta22 v5.0.0-beta23 v5.0.0-beta24 v5.0.0-beta25 v5.0.0-beta26 v5.0.0-beta27 v5.0.0-beta28 v5.0.0-beta29 v5.0.0-beta3 v5.0.0-beta30 v5.0.0-beta31 v5.0.0-beta32 v5.0.0-beta33 v5.0.0-beta5 v5.0.0-beta6 v5.0.0-beta7 v5.0.0-beta8 v5.0.0-beta9 v5.0.0-rc1 v5.0.0-rc3 v5.0.0-rc4 v5.0.1-beta180802235906 v5.0.1-beta180812233236 v5.0.1-beta180824214627 v5.0.1-beta180826190446 v5.0.1-beta180828173436 v5.0.1-beta180830182727 v5.0.1-beta180902210116 v5.0.1-beta180905054336 v5.0.1-beta180909000146 v5.0.1-beta180909000436 v5.0.1-beta180911213156 v5.0.1-beta180913003126 v5.0.1-beta180914024526 v5.0.1-beta181008233543 v5.0.1-beta181018200736 v5.0.1-rc1 v5.0.1-rc2 v5.0.1-rc3 v5.0.2-beta181116204629 v5.0.2-beta181116204811 v5.0.2-beta181116205152 v5.0.2-beta181117022246 v5.0.2-beta181118024524 v5.0.2-beta181119063215 v5.0.2-beta181119185816 v5.0.2-beta181126211008 v5.1.0-beta180611231144 v5.1.0-beta180612043613 v5.1.0-beta180612224009 v5.1.0-beta180613013021 v5.1.0-beta180614233101 v5.1.0-beta180615005408 v5.1.0-beta180618191747 v5.1.0-beta180618214711 v5.1.0-beta180618223247 v5.1.0-beta180618225004 v5.1.0-beta180619025141 v5.1.0-beta180620180431 v5.1.0-beta180620180739 v5.1.0-beta180620183559 v5.1.0-beta180622181348 v5.1.0-beta180626014959 v5.1.0-beta180627203509 v5.1.0-beta180628064520 v5.1.0-beta180628184841 v5.1.0-beta180630224439 v5.1.0-beta180701010040 v5.1.0-beta180701175749 v5.1.0-beta180702063039 v5.1.0-beta180702063440 v5.1.0-beta180702214311 v5.1.0-beta180702220040 v5.1.0-beta180703024529 v5.1.0-beta180706202701 v5.1.0-beta180707004950 v5.1.0-beta180718203536 v5.1.0-beta180722215127 v5.1.0-beta180723023347 v5.1.0-beta180723173636 v5.1.0-beta180724024536 v5.1.0-beta180730185716 v5.1.0-beta180812233046 v5.1.0-beta180820223106 v5.1.0-beta180824214446 v5.1.0-beta180828022857 v5.1.0-beta180828173516 v5.1.0-beta180829024526 v5.1.0-beta180905054157 v5.1.0-beta180911213206 v5.1.0-beta180912202326 v5.1.0-beta180917172706 v5.1.0-beta180919183606 v5.1.0-beta180928000756 v5.1.0-beta180929024526 v5.1.0-beta201806191956 v5.1.0-beta201806200051 v5.1.0-beta34 v5.1.0-beta35 v5.1.0-beta36 v5.1.0-beta37 v5.1.0-beta38 v5.1.0-beta39 v5.1.0-rc1 v6.0.0-beta181009070836 v6.0.0-beta181009071126 v6.0.0-beta181009071136 v6.0.0-beta181011024526

To reduce verbosity, you could remove logging and such, most of this output was intended to demonstrate the timing and running.

As another alternative, consider something like this:

log() {
    now=$(date -Isec --utc)
    echo "${now} ${*}" > /dev/stderr
}

# I don't have semver otherwise available, so a knockoff replacement
function is_semver() {
    echo "$*" | egrep -q "^v?[0-9]+\.[0-9]+\.[0-9]+$"
}

function print_something() {
    local repo_name=$1 tag_name=
    bad=$(
        git tag -l | while read tag_name ; do
            is_semver "${tag_name}" || echo -n "${tag_name} "
        done
       )
    echo "${repo_name},${bad}"
}

csvdir=$(mktemp -d not_semver_tempdir.XXXXXX)
csvdir=$(realpath "${csvdir}")/
log "Temp Directory: ${csvdir}"
while read -r repo_dir ; do
    log "Starting '${repo_dir}'"
    (
        if [ -d "${repo_dir}" ]; then
            repo_name=$(basename "${repo_dir}")
            tmpfile=$(mktemp -p "${csvdir}")
            tmpfile=$(realpath "${tmpfile}")
            cd "${repo_dir}"
            print_something "${repo_name}" > "${tmpfile}" 2> /dev/null
        fi
    ) &
done
wait
outfile=$(mktemp not_semver_XXXXXX.csv)
cat ${csvdir}* > "${outfile}"
# rm -rf "${csvdir}" # uncomment when you're comfortable/confident
log "Output: ${outfile}"

I don't like it as much, admittedly, but its premise is that it creates a temporary directory in which each repo process will write its own file. Once all backgrounded jobs are complete (i.e., the wait near the end), all files are concatenated into an output.

Running it (without xargs):

$ find . -maxdepth 2 -iname .git | sed -e 's,/\.git,,g' | head -n 12 | \
   ~/StackOverflow/5783481/62283574.sh
2020-06-10T14:48:18+00:00 Temp Directory: /c/Users/r2/Projects/github/not_semver_tempdir.YeyaNY/
2020-06-10T14:48:18+00:00 Starting './calendar'
2020-06-10T14:48:18+00:00 Starting './callr'
2020-06-10T14:48:18+00:00 Starting './docker-self-service-password'
2020-06-10T14:48:18+00:00 Starting './ggnomics'
2020-06-10T14:48:18+00:00 Starting './ggplot2'
2020-06-10T14:48:19+00:00 Starting './grid'
2020-06-10T14:48:19+00:00 Starting './gt'
2020-06-10T14:48:19+00:00 Starting './keyring'
2020-06-10T14:48:19+00:00 Starting './knitr'
2020-06-10T14:48:19+00:00 Starting './ksql'
2020-06-10T14:48:19+00:00 Starting './nanodbc'
2020-06-10T14:48:19+00:00 Starting './nostalgy'
2020-06-10T14:48:38+00:00 Output: not_semver_CLy098.csv

r2@d2sb2 MINGW64 ~/Projects/github
$ cat not_semver_CLy098.csv
keyring,
ksql,0.1-pre1 0.1-pre10 0.1-pre2 0.1-pre4 0.1-pre5 0.1-pre6 0.1-pre7 0.1-pre8 0.1-pre9 0.3 v0.2 v0.2-rc0 v0.2-rc1 v0.3 v0.3-rc0 v0.3-rc1 v0.3-rc2 v0.3-rc3 v0.3-temp v0.4 v0.4-rc0 v0.4-rc1 v0.5 v0.5-rc0 v0.5-rc1 v4.1.0-rc1 v4.1.0-rc2 v4.1.0-rc3 v4.1.0-rc4 v4.1.1-rc1 v4.1.1-rc2 v4.1.1-rc3 v4.1.2-beta180719000536 v4.1.2-beta3 v4.1.2-rc1 v4.1.3-beta180814192459 v4.1.3-beta180828173526 v5.0.0-beta1 v5.0.0-beta10 v5.0.0-beta11 v5.0.0-beta12 v5.0.0-beta14 v5.0.0-beta15 v5.0.0-beta16 v5.0.0-beta17 v5.0.0-beta18 v5.0.0-beta180622225242 v5.0.0-beta180626015140 v5.0.0-beta180627203620 v5.0.0-beta180628184550 v5.0.0-beta180628221539 v5.0.0-beta180629053850 v5.0.0-beta180630224559 v5.0.0-beta180701010229 v5.0.0-beta180701053749 v5.0.0-beta180701175910 v5.0.0-beta180701205239 v5.0.0-beta180702185100 v5.0.0-beta180702222458 v5.0.0-beta180706202823 v5.0.0-beta180707005130 v5.0.0-beta180707072142 v5.0.0-beta180718203558 v5.0.0-beta180722214927 v5.0.0-beta180723195256 v5.0.0-beta180726003306 v5.0.0-beta180730183336 v5.0.0-beta19 v5.0.0-beta2 v5.0.0-beta20 v5.0.0-beta21 v5.0.0-beta22 v5.0.0-beta23 v5.0.0-beta24 v5.0.0-beta25 v5.0.0-beta26 v5.0.0-beta27 v5.0.0-beta28 v5.0.0-beta29 v5.0.0-beta3 v5.0.0-beta30 v5.0.0-beta31 v5.0.0-beta32 v5.0.0-beta33 v5.0.0-beta5 v5.0.0-beta6 v5.0.0-beta7 v5.0.0-beta8 v5.0.0-beta9 v5.0.0-rc1 v5.0.0-rc3 v5.0.0-rc4 v5.0.1-beta180802235906 v5.0.1-beta180812233236 v5.0.1-beta180824214627 v5.0.1-beta180826190446 v5.0.1-beta180828173436 v5.0.1-beta180830182727 v5.0.1-beta180902210116 v5.0.1-beta180905054336 v5.0.1-beta180909000146 v5.0.1-beta180909000436 v5.0.1-beta180911213156 v5.0.1-beta180913003126 v5.0.1-beta180914024526 v5.0.1-beta181008233543 v5.0.1-beta181018200736 v5.0.1-rc1 v5.0.1-rc2 v5.0.1-rc3 v5.0.2-beta181116204629 v5.0.2-beta181116204811 v5.0.2-beta181116205152 v5.0.2-beta181117022246 v5.0.2-beta181118024524 v5.0.2-beta181119063215 v5.0.2-beta181119185816 v5.0.2-beta181126211008 v5.1.0-beta180611231144 v5.1.0-beta180612043613 v5.1.0-beta180612224009 v5.1.0-beta180613013021 v5.1.0-beta180614233101 v5.1.0-beta180615005408 v5.1.0-beta180618191747 v5.1.0-beta180618214711 v5.1.0-beta180618223247 v5.1.0-beta180618225004 v5.1.0-beta180619025141 v5.1.0-beta180620180431 v5.1.0-beta180620180739 v5.1.0-beta180620183559 v5.1.0-beta180622181348 v5.1.0-beta180626014959 v5.1.0-beta180627203509 v5.1.0-beta180628064520 v5.1.0-beta180628184841 v5.1.0-beta180630224439 v5.1.0-beta180701010040 v5.1.0-beta180701175749 v5.1.0-beta180702063039 v5.1.0-beta180702063440 v5.1.0-beta180702214311 v5.1.0-beta180702220040 v5.1.0-beta180703024529 v5.1.0-beta180706202701 v5.1.0-beta180707004950 v5.1.0-beta180718203536 v5.1.0-beta180722215127 v5.1.0-beta180723023347 v5.1.0-beta180723173636 v5.1.0-beta180724024536 v5.1.0-beta180730185716 v5.1.0-beta180812233046 v5.1.0-beta180820223106 v5.1.0-beta180824214446 v5.1.0-beta180828022857 v5.1.0-beta180828173516 v5.1.0-beta180829024526 v5.1.0-beta180905054157 v5.1.0-beta180911213206 v5.1.0-beta180912202326 v5.1.0-beta180917172706 v5.1.0-beta180919183606 v5.1.0-beta180928000756 v5.1.0-beta180929024526 v5.1.0-beta201806191956 v5.1.0-beta201806200051 v5.1.0-beta34 v5.1.0-beta35 v5.1.0-beta36 v5.1.0-beta37 v5.1.0-beta38 v5.1.0-beta39 v5.1.0-rc1 v6.0.0-beta181009070836 v6.0.0-beta181009071126 v6.0.0-beta181009071136 v6.0.0-beta181011024526
knitr,doc v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 v0.8 v0.9 v1.0 v1.1 v1.10 v1.11 v1.12 v1.13 v1.14 v1.15 v1.16 v1.17 v1.18 v1.19 v1.2 v1.20 v1.3 v1.4 v1.5 v1.6 v1.7 v1.8 v1.9
calendar,
ggplot2,ggplot2-0.7 ggplot2-0.8 ggplot2-0.8.1 ggplot2-0.8.2 ggplot2-0.8.3 ggplot2-0.8.5 ggplot2-0.8.6 ggplot2-0.8.7 ggplot2-0.8.8 ggplot2-0.8.9 ggplot2-0.9.0 ggplot2-0.9.1 ggplot2-0.9.2 ggplot2-0.9.2.1 ggplot2-0.9.3 ggplot2-0.9.3.1 show
nostalgy,
callr,
docker-self-service-password,2.7 2.8 3.0
grid,0.1 0.2 0.5 0.5-1 0.6 0.6-1 0.7-1 0.7-2 0.7-3 0.7-4
ggnomics,
nanodbc,
gt,

alecxs · Answer 2 · 2020-06-10T14:09:25.203

use a variable or temp file for buffering lines. random file name is used
($0 = script name, $! = most recently background PID)

make sure you have write permissions. if you are worried about eMMC Flash Memory wear-out or write speeds you can also use shared-memory /run/shm

#!/bin/bash

print_not_semver_line() {
    # random file name for line buffering
    local tmpfile="${0%.*}${!:-0}.tmp~"
    touch "$tmpfile" || return 1

    # redirect stdout into different tmp file
    echo -n "$repo_name," > "$tmpfile"
    git tag -l | while read -r tag_name;do
        semver $tag_name > /dev/null || echo -n "$tag_name " >> "$tmpfile"
    done
    echo "" >> "$tmpfile"

    # print the whole line from one single ride
    cat "$tmpfile" && rm "$tmpfile" && return 0
}

however, it is recommended to limit the maximum number of background processes. for the above example you can count open files with lsof

this function is waiting for given file name. it will check for similar file names and wait until number of open files is below allowed maximum. use it in your loop

first argument is mandatory file name
second argument is optional limit (default 4)
third argument is optional frequency for lsof

usage: wait_of <file> [<limit>] [<freq>]

# wait for open files (of)
wait_of() {
    local pattern="$1" limit=${2:-4} time=${3:-1} path of

    # check path
    path="${pattern%/*}"
    pattern="${pattern##*/}"
    [ "$path" = "$pattern" ] && path=.
    [ -e "$path" ] && [ -d "$(realpath "$path")" ] || return 1

    # convert file name into regex
    pattern="${pattern//[0-9]/0}"
    while [[ "$pattern" =~ "00" ]]
      do
        pattern="${pattern//00/0}"
    done
    pattern="${pattern//0/[0-9]*}"
    pattern="${pattern//[[:space:]]/[[:space:]]}"

    # check path with regex for open files > 4 and wait
    of=$(lsof -t "$path"/$pattern 2> /dev/null | wc -l)
    while (( ${of:-0} > $limit ))
      do
        of=$(lsof -t "$path"/$pattern 2> /dev/null | wc -l)
        sleep $time
    done

    return 0
}

# make sure only give one single tmp file name
wait_of "${0%.*}${!:-0}.tmp~" || exit 2
print_not_semver_line >> $csv_name &

have added limit function similar to *xargs -P4* by @r2evans — alecxs, Jun 10 '20 at 13:19

Bash concurrency when writing to file

2 Answers2