3

I have a number of files (in the same folder) all with the same number of lines:

a.txt

20
3
10
15
15

b.txt

19
4
5
8
8

c.txt

2
4
9
21
5

Using Bash, I'd like to create an array of arrays that contain the value of each line in every file. So, line 1 from a.txt, b.txt, and c.txt. The same for lines 2 to 5, so that in the end it looks like:

[
   [20, 19, 2],
   [3, 4, 4],
   ...
   [15, 8, 5]
]

Note: I messed up the formatting and wording. I've changed this now.

I'm actually using jq to get these lists in the first place, as they're originally specific values within a JSON file I download every X minutes. I used jq to get the values I needed into different files as I thought that would get me further, but now I'm not sure that was the way to go. If it helps, here is the original JSON file I download and start with.

I've looked at various questions that somewhat deal with this:

Among others. But none of these deal with taking the value of the same line from various files. I don't know Bash well enough to do this and any help is greatly appreciated.

duxk.gh
  • 161
  • 1
  • 8

3 Answers3

3

Here’s one approach:

$ jq -c -n '[$a,$b,$c] | transpose'  --slurpfile a a.txt  --slurpfile b b.txt  --slurpfile c c.txt

Generalization to an arbitrary number of files

In the following, we'll assume that the files to be processed can be specified by *.txt in the current directory:

jq -n -c '
  [reduce inputs as $i ({}; .[input_filename] += [$i]) | .[]]
  | transpose' *.txt
peak
  • 105,803
  • 17
  • 152
  • 177
1

Use paste to join the files, then read the input as raw text, splitting on the tabs inserted by paste:

$ paste a.txt b.txt c.txt | jq -Rc 'split("\t") | map(tonumber)'
[20,19,2]
[3,4,4]
[10,5,9]
[15,8,21]
[15,8,5]

If you want to gather the entire result into a single array, pipe it into another instance of jq in slurp mode. (There's probably a way to do it with a single invocation of jq, but this seems simpler.)

$ paste a.txt b.txt c.txt | jq -R 'split("\t") | map(tonumber)' | jq -sc
[[20,19,2],[3,4,4],[10,5,9],[15,8,21],[15,8,5]]
chepner
  • 497,756
  • 71
  • 530
  • 681
0

I could not come up with a simple way, but here's one I got to do this.

1. Join files and create CSV-like file

If your machine have join, you can create joined records from two files (like join command in SQL). To do this, make sure your file is sorted. The easiest way I think is just numbering each lines. This works as Primary ID in SQL.

$ cat a.txt | nl > a.txt.nl
$ cat b.txt | nl > b.txt.nl
$ cat c.txt | nl > c.txt.nl

Now you can join sorted files into one. Note that join can join only two files at once. This is why I piped output to next join.

$ join a.txt.nl b.txt.nl | join - c.txt.nl > conc.txt

now conc.txt is:

1 20 19 2
2 3 4 4
3 10 5 9
4 15 8 21
5 15 8 5

2. Create json from the CSV-like file

It seems little complicated.

jq -Rsn '
    [inputs
     | . / "\n"
     | (.[] | select((. | length) > 0) | . / " ") as $input
     | [$input[1], $input[2], $input[3] ] ]
' <conc.txt

Actually I do not know detailed syntex or usage of jq, it seems like doing:

  • split input file by \n
  • split a given line by space, then select valid data
  • put splitted records in appropriate location by their index

I used this question as a reference: https://stackoverflow.com/a/44781106/10675437

toshim
  • 1
  • 2