Bash For Loop Iteration for More than 1 at a time?

Question

i have a list/array of per line quoted strings in a list.json file. this file is dynamically generated and fluctuates from 1 to inf in size. currently 127 elements. i need to process every line no matter if its 1 or inf.

[
  "p1",
  "p2",
  ...,
  "p127"
]

these params need to be tacked onto a command flag within an array eg. command --params=["p1","p2","...","p20"] it can take up to a max of 20 per command.

currently im just going through generating one command for every param in my list.json, using a simple for loop. eg below

lines="$(cat list.json)"

   for l in ${lines} ;
        do
          command --params=[${l}] 
        done

eg. currently generates and runs 127 loops: command --params=["p1"] command --params=["p2"] ... command --params=["p127"] etc. and i save the outputs.

this is the obvs basic, and very inefficient & slow method. so ideally i want to reduce my total number of commands so its using max params eg. 20 in each loop eg. 1st loop = command --params=["p1","p2",..."p20"] 2nd loop = command --params=["p21","p22",..."p40"] etc.

this has two problems i need to address.

1. i dunno how to do that in a for loop...looking at a bunch of examples of for loops for specifying some logic for sequence & increments which is obvs enough, but im really not sure what are you supposed to do where you want it to batch multiple lines per loop, eg like a range of 20 lines at a time? seems simple enough to me, but havent had to do it before.

2. potentially dealing with any comma separation situation from the array of params. eg. splitting the elements 20 at a time, into their own not malformed json array ie. without trailing commas etc.

Answer 1

You would first join 20 elements with comma, then run your command with these as arguments.

xargs -n20 sh -c 'IFS=,; echo "$*"' - < lines.json |
xargs -I{} command --params=[{}] 

I guess --params= is expected to be a JSON array. I would suggest just using jq to loop and split it into an arrays of 20 elements. How to split an array into chunks with jq?

jq -c '_nwise(20)' lines.json |
xargs -d '\n' -I{} command --params={}

Answer 2

It's not clear how you want to use the lines, and xargs is almost certainly what you want to use, but you could read several lines with something like:

#!/bin/sh

line_count=5
while for ((i=0; i < $line_count; i++)); do
    eval read line$i || break;
    done;
    test $i = $line_count
do
        for ((i=0; i < $line_count; i++)); do
                eval "echo line $i = \$line$i"
        done
done

Answer 3

Here's an xargs idea/demo:

$ seq 43 > testdata
$ n=5
$ xargs -L $n xargs bash -c '
    [ $# = '$n' ] || exit
    echo args = "$@"
' -- <testdata
args = 1 2 3 4 5
args = 6 7 8 9 10
args = 11 12 13 14 15
args = 16 17 18 19 20
args = 21 22 23 24 25
args = 26 27 28 29 30
args = 31 32 33 34 35
args = 36 37 38 39 40

• note that '$n' is actually unquoting the variable
• -- is a comment (see bash -c documentation)
• replace echo with something more interesting

Answer 4

mapfile -t params < <(jq '.[]' params.json)

IFS=,
for ((i=0; i < ${#params[@]}; i+=20)); do
    echo command --params="[${params[*]: i:20}]"
done

Run this, and verify that the generated command(s) look OK, then remove echo to run for real.

For file params.json:

[
  "1",
  "2",
  "3",
...
  "24",
  "25"
]

The commands generated are:

command --params=["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20"]
command --params=["21","22","23","24","25"]

Remove echo, and these commands will be executed. command will see the commas and double quotes.