1

I have been following the directions to concatenate files in a directory as directed in this post: Unix: merge many files, while deleting first line of all files, but it is not working correctly. To sum, this is the command being used:

find . -name "*.extension" | xargs -n 1 tail -n +2

I attempt to attach a name to this file and redirect like this:

find . -name "*.extension" | xargs -n 1 tail -n +2 > concatenated.csv

but this only grabs the first file. Am I doing something wrong? I am very unfamiliar with these commands.

Community
  • 1
  • 1
derigible
  • 964
  • 5
  • 15
  • 32
  • 1
    This should work fine if none of your filenames have whitespace or quotes in them. Do you get any error messages? does `find . -name "*.extension"` output all the files you expect? – that other guy Feb 15 '13 at 21:43
  • They do have whitespace and I get errors. Do I just need to remove the whitespace? – derigible Feb 15 '13 at 21:47
  • 1
    You have two working answers below that do not require renaming. Friendly tip for the future: include any error messages you get, as this make it much easier to determine what the problem is. – that other guy Feb 15 '13 at 21:54

3 Answers3

3

The problem is that your files contain whitespace or quotes, which xargs by default doesn't like. If you use the GNU versions of find and xargs, you can use 

find . -name "*.extension" -print0 | xargs -0 -n 1 tail -n +2 > concatenated.csv

to safely handle all file names.

that other guy
  • 116,971
  • 11
  • 170
  • 194
  • 1
    I also found that if you are going to use this method you must either place the output in a different directory or it will add the already concatenated file to itself. Just a heads up on that. – derigible Feb 15 '13 at 22:27
  • Definitely a good point. You can exclude the output with `-name "*.csv" ! -name "concatenated.csv" -print0"` if you want to write to the same directory. – that other guy Feb 16 '13 at 01:34
2

For a simple command like this, you can skip xargs and just use the -execdir primary:

find . -name "*.extension" -execdir tail -n +2 '{}' \; > concatenated.csv

Inspired by Kevin's answer, you can also use

find . -name "*.extension" -execdir tail -q -n +2 '{}' + > concatenated.csv

which will only run tail one time on all the files, instead of once per file.

chepner
  • 497,756
  • 71
  • 530
  • 681
  • I couldn't get this one to work. It tells me that `-execdir` is not a valid predicate. – derigible Feb 15 '13 at 21:57
  • Hm, that's odd. You can try with `-exec` as well, which may be more standard. In this example, there should be no difference between the two. – chepner Feb 15 '13 at 22:00
1

Instead of using tail on each file individually, you may want to use another tool that can process more files at once. e.g. awk:

find . -name '*.extension' -exec awk 'FNR > 1' + >concatenated.csv
Kevin
  • 53,822
  • 15
  • 101
  • 132
  • You can do the same with `tail -q` (which I wouldn't have thought to check if you hadn't posted this answer, so +1). – chepner Feb 15 '13 at 22:02