1

I have a directory of 40 or so csv's. Each csv file has an extra 10 lines at the top that I don't need. I'm new to bash commands, but I have found that I can use 

tail -n +10 oldfile.csv > newfile.csv

to cut 10 lines from a file one at a time. How can I do this across all csv's in the directory? I have tried doing this:

for filename in *foo*; do echo tail -n +10 \"$filename\" > \"${filename}\"; done

From what I've read, I thought this would pass in every csv containing foo in its name, run the formula, and leave the filename alone. Where am I going wrong?

James
  • 209
  • 3
  • 13

3 Answers3

2

You cannot use the same file as input and ouput.

With sed, your can edit the file in place with the -i flag:

for f in *.csv; do
    sed -i '1,10d' "$f"
done

or as one-liner for the command line:

for f in *.csv; do sed -i '1,10d' "$f"; done

As a side note, your tail should be tail -n +11 to output 11th line to end of file.

SLePort
  • 15,211
  • 3
  • 34
  • 44
  • Getting an "invalid command code" error when I run this. The way I'm running this is by 'cd' to the directory, for f in *.csv; do [Enter], >sed -i '1,10d' "$f" [Enter], > done [Enter].. back to $.. then I get invalid command code – James Jan 12 '17 at 06:34
  • Multiline is more for scripts. I added a one-liner for the command line. – SLePort Jan 12 '17 at 06:54
  • Works! But... had to add '' in between -i and '1,10d'.. credit: http://stackoverflow.com/questions/29081799/sed-1-invalid-command-code-f (I'm on a Mac). Final: for f in *.csv; do sed -i '' '1,10d' "$f"; done – James Jan 12 '17 at 07:07
1

Use a proper loop as below. Am using the native ex editor which Vim uses internally to in-place replacement, so you don't have to move the files back again using mv or any other command. 

for file in *.csv
do
    ex -sc '1d10|x' "$file"
done

The command moves to first line, selects 10 lines from first, deletes it and saves & closes the file.

Use a command-line friendly version in a single line as

for file in *.csv; do ex -sc '1d10|x' "$file"; done

The ex command is POSIX compatible and can work on all major platforms and distros.

Inian
  • 80,270
  • 14
  • 142
  • 161
0

In awk:

$ awk 'FNR>10{ print > "new-" FILENAME }' files*

Explained:

  • FNR>10 if current record number in current file is greater than 10, condition is true.
  • print well, output
  • > "new-" FILENAME redirect output to a new file named new-file, for example.

Edited to writing output to multiple files. Original which just outputed to screen was awk 'FNR>10' files*

James Brown
  • 36,089
  • 7
  • 43
  • 59