Join multiple files in gawk

Question

I have a large number of files (around 500). Each file contain two columns. The first column is same for every file. I want to join all the files into a single file using gawk.
For example,

File 1  
a 123  
b 221  
c 904

File 2 
a 298  
b 230  
c 102

and so on. I want a final file like as below:

Final file
a 123 298  
b 221 230  
c 904 102

I have found scripts that can join two files, but I need to join multiple files.

jaypal singh · Accepted Answer · 2014-04-19T18:45:52.900

5

For given sample files:

$ head f*
==> f1 <==
a 123
b 221
c 904

==> f2 <==
a 298
b 230
c 102

==> f3 <==
a 500
b 600
c 700

Method 1:

$ awk '{a[FNR]=((a[FNR])?a[FNR]FS$2:$0)}END{for(i=1;i<=FNR;i++) print a[i]}' f*
a 123 298 500
b 221 230 600
c 904 102 700

Method 2: (Will probably be faster as your are not loading 500 files in memory)

Using paste and awk together. (Assuming first column is same and present in all files). Doing paste f* will give you the following result:

$ paste f*
a 123   a 298   a 500
b 221   b 230   b 600
c 904   c 102   c 700

Pipe that to awk to remove extra columns.

$ paste f* | awk '{printf "%s ",$1;for(i=2;i<=NF;i+=2) printf "%s%s",$i,(i==NF?RS:FS)}'
a 123 298 500
b 221 230 600
c 904 102 700

You can re-direct the output to another file.

edited Apr 19 '14 at 18:45

answered Apr 19 '14 at 18:38

jaypal singh

74,723
23
102
147

1

Worked perfectly for me. – userpmondal Apr 20 '14 at 15:17
@userpmondal : don't hesitate to upvote for the answers that you find useful ;) – Idriss Neumann Apr 20 '14 at 15:54
@userpmondal Don't worry. You'll soon get there! `:)` – jaypal singh Apr 20 '14 at 16:10
@userpmondal arf, I didn't know sorry. – Idriss Neumann Apr 20 '14 at 16:31

HighEnergy · Answer 2 · 2016-05-20T03:59:03.580

1

I have encountered this problem very frequently.

I strongly encourage you to check into the getline function in gawk.

getline var < filename

is the command syntax and can be used to solve your problem.

I would suggest utilizing another language that solves this problem much more easily. Typically I invest about 5 lines of code to solve this standard problem.

j=1;
j=getline x < "filename";
if(j==0) {
      break;
}
... (Commands involving x such as split and print).

edited May 20 '16 at 03:59

answered May 20 '16 at 03:19

HighEnergy

47
12

1

How would you use `getline` exactly? As it stands, this is not a very complete answer. Notice that none of the other answers use `getline`, so you'd be adding a new method with a more complete answer. – Benjamin W. May 20 '16 at 03:50

score 0 · Answer 3 · answered Apr 19 '14 at 18:36

0

You could try something like :

$ ls
f1.txt  f2.txt  f3.txt
$ awk '($0 !~ /#/){a[$1]=a[$1]" "$2} END {for(i in a){print i""a[i]}}' *.txt
a 123 298 299
b 221 230 231
c 904 102 103

answered Apr 19 '14 at 18:36

Idriss Neumann

3,760
2
23
32

1

You can do `awk '{a[$1]=(a[$1])?a[$1] FS $2:$2}END{for(i in a){print i,a[i]}}' *.txt` too but this will not preserve the ordering of files. Final output will be random due to `in` operator (`for(i in a)`). – jaypal singh Apr 19 '14 at 18:41

score 0 · Answer 4 · edited May 23 '17 at 12:24

0

awk 'FNR==NR{arr[$1]=$2; next;}{printf "%s%s%s%s%s",$1,OFS,arr[$1],OFS,$2; print"";}' file1 file2

based on this

edited May 23 '17 at 12:24

Community

1
1

answered Apr 19 '14 at 18:50

a5hk

7,532
3
26
40

This is good for 2 files only. OP wants around 500 files to be joined together. – jaypal singh Apr 19 '14 at 18:56

Join multiple files in gawk

4 Answers4

Method 1:

Method 2: (Will probably be faster as your are not loading 500 files in memory)