Deleting lines in multiple csv files in a folder with Python

Question

I want to delete top and bottom lines in multiple csv files in a folder at once. Then delete the second line after that. My code works well with one file but i have 1000s of files to go through and i cannot specify each file name. How can i modify this to make it work?

#Remove top and bottom lines from all files 
lines = open('My/local/Drive/*.csv').readlines()
open('My/local/Drive/*.csv', 'w').writelines(lines[2:-4]);

#Remove 2nd line from all files
lines = []
with open('My/local/Drive/*.csv', 'r') as f:
    lines = f.readlines()
with open('My/local/Drive/*.csv', 'w') as f:
    f.writelines(lines[:1] + lines[2:]);

could you specify exactly what are the lines you want to remove/keep? — Allan, Mar 20 '19 at 11:30
I am removing the first two lines before the header, the line after the header and the last four lines in the dataset. — enufeffizy, Mar 20 '19 at 11:32
if it were linux I would have solved it in 1 command line, anyway I have answered ;-) — Allan, Mar 20 '19 at 12:12

score 1 · Answer 1 · answered Mar 20 '19 at 12:10

>>>TAKE A BACKUP OF YOUR FILES BEFORE RUNNING THIS<<<

You can use the following code, where you define path following your needs.

import glob

path='.'
for filename in glob.iglob(path+'/*.csv'):
  with open(filename, 'r') as f:
    lines = f.read().split("\n")
    f.close()
    if len(lines) >= 7:
      lines = lines[2:-5]
      lines = lines[:1] + lines[2:]
      o = open(filename, 'w')
      for line in lines:
        o.write(line+'\n')
      o.close()

in my target folder (2 csv files):

$ ls *.csv
input1.csv  input2.csv

file1:

to delete1
to delete2
ID;TAG;GROUP
todelete;B00;AB0
niub12617500;B01;AB4
niub16371500;B01;AB3
todelete;B00;AB0
todelete;B00;AB0
todelete;B00;AB0
todelete;B00;AB0

file2:

to delete1
to delete2
ID;TAG;GROUP
todelete;B00;AB0
niub12677500;B00;AB2
niub16377500;B01;AB0
todelete;B00;AB0
todelete;B00;AB0
todelete;B00;AB0

After execution:

$ cat input1.csv
ID;TAG;GROUP
niub12617500;B01;AB4
niub16371500;B01;AB3
$ cat input2.csv
ID;TAG;GROUP
niub12677500;B00;AB2
niub16377500;B01;AB0

@enufeffizy: if you are satisfied by my answer, can you vote up and accept? Thank you — Allan, Mar 20 '19 at 12:48

score 0 · Answer 2 · answered Mar 20 '19 at 11:30

Well, you can try and list all files in a directory (having a folder with all your .csv files is a great option here!). This will list you ALL files:

import os
for filename in os.listdir(os.getcwd()):

Or you can just list the .csv files!

import glob
for filename in glob.glob('*.txt'):

Having that, you can do whatever you need to delete those files!

Check this out if you need more insight on listing and operating with that!

Dividing your problem in small problems, everything has an easier solution: how to open every file in a folder

Deleting lines in multiple csv files in a folder with Python

2 Answers2