0

I have about 450 csv files in a folder with names in the format (sample names below):

1_a.csv

1_b.csv

1_c.csv

...

1_h.csv

2_a.csv

2_b.csv

...

2_h.csv

...

42_a.csv

...

42_h.csv

I wish to combine all files of the type "1_xxx.csv" into "1.csv", all files of the format "2_xxx.csv" into "2.csv" and so on.

I tried using cat but I am only able to merge one set of files at a time. Is there a way to run the command in loop in linux? Any Python3 based solutions are also welcome.

Sagar Joshi
  • 11
  • 2
  • 2
    https://stackoverflow.com/questions/13613336/python-concatenate-text-files – mnistic Oct 20 '17 at 17:29
  • Hi mnistic, thanks for pointing it out. The link however talks simply about merging the files. My question is regarding grouping 462 files into 42 sets of 11 each based on the starting part of their name and then merging them. – Sagar Joshi Oct 21 '17 at 00:16

1 Answers1

0

For a python solution the comment provided by mnistic links to a solution. If you want a bash solution you can use the find command with the -exec flag like

find . -name "1_*.csv" -exec cat {} >> 1.csv \;

That command should find all the csv files beginning with 1_ and then appending them to the file 1.csv.

For more information on find check out https://ss64.com/bash/find.html

EDIT: For a loop

for i {1..450}
do
    find . -name "$i_*.csv" -exec cat {} >> $i.csv \;
done

this should work

Source: http://go2linux.garron.me/bash-for-loop/

Garrigan Stafford
  • 1,331
  • 9
  • 20
  • Hi Garrigan, thanks for the quick response. I'm however looking for a solution that works in loop. With this code, I still need to run it 42 times for files starting with 1 and then with 2 and so on. I am looking for a solution that can repeat it 42 times itself for each of the different name types. – Sagar Joshi Oct 21 '17 at 00:13
  • You could use a bash loop and iterate from 1->42 and just use the variable in the command – Garrigan Stafford Oct 21 '17 at 00:23
  • Yes Garrigan, exactly what I'm looking for but definitely something I do not know how to write a code for. Can you maybe please add the loop code to the solution? – Sagar Joshi Oct 21 '17 at 00:27
  • Added it in the answer, you just need to put that into a bash file and call bash filename – Garrigan Stafford Oct 21 '17 at 00:34
  • Hi, I tried using this script by typing it out in vi text editor but it given the below error: Test.sh: line 1: syntax error near unexpected token `{1..52}' Test.sh: line 1: `for i {1..52}' I have also tried copy pasting from here, typing it out in notepad and using dos2unix, typing the code manually in vi text editor and then using the dos2unix on top of that as well after it still gave the error. Using dos2unix removed an earlier error of '\r'. Can't seem to get rid of this one. – Sagar Joshi Oct 21 '17 at 05:03
  • Try some of the other loop structures in the link, also don't forget the #!/bin/bash at the top of the file – Garrigan Stafford Oct 21 '17 at 05:23