2

I have a folder of a little over 10,000 .csv files that I want to combine into one master file. They are all categorized the same way (Column A B C D E F are the same thing in each file). I'd prefer to do it in a shell script.

I tried

cat *.csv > Everything.csv

and it returns that Argument is too long

I also tried 

copy *.csv > Everything.csv

and it returns the same error.

How do I get it to combine about 10,000 files into one Master file?

Prradep
  • 5,506
  • 5
  • 43
  • 84
user3736201
  • 99
  • 2
  • 6

4 Answers4

1

This question discusses the error you're seeing: Argument list too long error for rm, cp, mv commands

One possible solution would be something like:

find . -name "*.csv" -exec cat '{}' >> ./Everything.csv ';'
Community
  • 1
  • 1
khan
  • 444
  • 3
  • 8
0

I've done it using:

cat *.csv >> /tmp/master_file_name.csv

Single '>' will re-write the file, not append it,

Konark Modi
  • 739
  • 6
  • 8
0

You can use a simple for loop:

for file in `ls`; do
    cat $file >> master_file
done
mirks
  • 3
  • 3
0

find . -name "*.csv" | xargs -I{} cat '{}' > Everything.csv

(edit) Ah well, beat to the punch...

JohnH
  • 2,713
  • 12
  • 21
  • Returns error "cat: ./Everything.csv: input file is output file" – user3736201 Jun 25 '14 at 19:13
  • True, and everything is in the Everything.csv when done (i.e., you still get the right answer). It's a warning, really. If your output file didn't end in .csv, or if it were in a different directory above the current one, you would not get that warning. – JohnH Jun 25 '14 at 21:21