Bash while loop from txt file?

Question

i trying to make a script to organize a pair of list i have, and process with other programs, but im a little bit stuck now.

I want from a List in Txt process every line first creating a folder to each line in the list and then process due to different scripts i have.

But my problem is is the list i give to the script is like 3-4 elements works great and create there own directory, but if i put a list with +1000 lines, then my script process only a few elements thru the scripts.

EDIT: the process are like 30-35 scripts, different language python,bash,python and golang

Any suggestions?

cat $STORES+NEW.txt | while read NEWSTORES 
do
    cd $STORES && mkdir $NEWSTORES && cd $NEWSTORES && mkdir .Files
    python3 checkstatus.py -n $NEWSTORES
    checkemployes $NEWSTORES -status
    storemanagers -s $NEWSTORES -o $NEWSTORES+managers.txt
    curl -s https://redacted.com/store?=$NEWSTORES | grep -vE "<|^[\*]*[\.]*$NEWSTORES" | sort -u | awk 'NF' > $NEWSTORES+site.txt
     ..
     ..
     ..
     ..
     ..
     ..
    cd ../..
done

I can't see any reason why it would only process the first few lines of the file. — Barmar, Dec 29 '21 at 23:15
Unless `checkstatus.py` or `checkemployes` tries to read from standard input. Then they'll read the rest of the input file. — Barmar, Dec 29 '21 at 23:17
Aside: You can create both directories at once with `mkdir -p "$NEWSTORES/.Files"` — Barmar, Dec 29 '21 at 23:18
@joop : Complementing the comment by _Barmar_: You have left out many commands from your loop body, and replaced them by just `..`. Did you verify that they don't eat your standard input? — user1934428, Dec 30 '21 at 09:40
With `set -e` any command that returns non-zero (with some caveats) will **cause the script to exit**. Long discussion about it at [bashfaq 105](http://mywiki.wooledge.org/BashFAQ/105) — glenn jackman, Dec 30 '21 at 20:02
Hello @glennjackman i understand, but the set -e is before the loop — joop, Dec 31 '21 at 15:54
Do you turn it off with `set +e` before entering the loop? If not perhaps you should. — glenn jackman, Dec 31 '21 at 16:04

konsolebox · Answer 1 · 2021-12-30T10:35:10.437

I'm not supposed to give an answer yet but I mistakenly answered my what should be a comment reply. Anyway here a few things I can suggest:

Avoid unnecessary use of cat.
Open your input file using another FD to prevent commands that read input inside the loop from eating the input: IFS= read -ru 3 NEWSTORES; do ...; done 3< "$STORES+NEW.txt" or { IFS= read -ru "$FD" NEWSTORES; do ...; done; } {FD}< "$STORES+NEW.txt". Also see https://stackoverflow.com/a/28837793/445221.
Not completely related but don't use while loop in a pipeline since it will execute in a subshell. In the future if you try to alter a variable and expect it to be saved outside the loop, it won't. You can use lastpipe to avoid it but it's unnecessary most of the time.
Place your variable expansions around double quotes to prevent unwanted word splitting and filename expansion.
Use -r option unless you want backslashes to escape characters.
Specify IFS= before read to prevent stripping of leading and trailing spaces.
Using readarray or mapfile makes it more convenient: readarray -t ALL_STORES_DATA < "$STORES+NEW.txt"; for NEWSTORES IN "${ALL_STORES_DATA[@]}"; do ...; done
Use lowercase characters on your variables when you don't use them in a global manner to avoid conflict with bash's variables.

Bash while loop from txt file?

1 Answers1