How to loop through a csv file in python and output each part of the csv file into a new file?

Question

I have a csv file in excel that contains 2000 rows of data. I would like to output 100 lines of the data to different text files. However I have no idea of how to do this. All I can do is output the file into a single file. I have read the CSV file data in Python Pyscripter and then wrote the file to a single file like this:

def read_csv(self):
    with open(self.data, newline='') as f:
        reader = csv.reader(f)
        for row in reader:
            self.content.append(row)

def write_txt(self):
    f = open(self.txtoutput, 'w')
    for row in self.content:
        f.write(', '.join(row) + '\n')
    f.close()

However, I would like each 100 rows of the 2000 row data to be outputted to different text files.Can anyone point me to the right direction. Note:I am using Python3. Thanks in advance.

score 2 · Answer 1 · edited May 23 '17 at 10:34

Iterate over the csv file in chunks of 100 rows at a time and write each chunk to a separate file:

with open(csv_filename, newline='') as file:
    chunks = zip(*[csv.reader(file)] * 100) # assume nrows % 100 == 0
    for i, rows in enumerate(chunks):
       with open("out%d.csv" % (i,), 'w', newline='') as output_file:
           csv.writer(output_file).writerows(rows)

See What is the most “pythonic” way to iterate over a list in chunks?

score 0 · Answer 2 · answered Dec 05 '12 at 22:07

0

For example: You have a counter that you increase with one for each line, and once it reaches a hundred you close the output file and open a new one.

answered Dec 05 '12 at 22:07

Lennart Regebro

167,292
41
224
251

John · Answer 3 · 2012-12-05T22:21:10.747

0

Something like

def write_txt(self):

    for index, row in enumerate(self.content):

        if index % 100 == 0:
            f = open(self.txtoutput + str(index) + ".txt", 'w')
            if index > 0:
                f.close()
        f.write(', '.join(row) + '\n')

    f.close()

edited Dec 05 '12 at 22:21

answered Dec 05 '12 at 22:11

John

13,197
7
51
101

Won't this write the first line to self.txtoutput, and then write the next 100 to file_0.txt? I don't think he wanted the single line self.txtoutput... – Travis Griggs Dec 05 '12 at 22:17

score 0 · Answer 4 · answered Dec 05 '12 at 22:11

Something like the following should work:

def write_txt(self):
    i = 0
    while i < len(self.content):
        with open(self.txtoutput + str(i/100), 'w') as f:
            for row in self.content[i:i+100]:
                f.write(', '.join(row) + '\n')
        i += 100

Since you didn't specify how the different text files should be named, I just appended an increasing number to the end of self.txtoutput.

Travis Griggs · Answer 5 · 2012-12-05T22:37:34.907

def writeText(self):
    for index, offset in enumerate(range(0, len(self.content), 100)):
        with open(self.txtoutput + '{:03}'.format(index) + '.txt', 'w') as file:
            for eachRow in self.content[offset, offset+100]:
                file.write(', '.join(eachRow) + '\n')

No extra variables is fun sometimes. This is a while-less version of @F.J's solution. I formatted the incrementing index with leading 0's so they'd sort conveniently in file listings.

A list comprehension solution with tuneable rowCount might look like (haven't tested this):

def writeText(self):
    rowCount = 100
    for index, eachGlump in enumerate(self.content[i:i+rowCount] for i in range(0, len(self.content), rowCount)):
        with open(self.txtoutput + '{:03}'.format(index) + '.txt', 'w') as file:
            for eachRow in eachGlump:
                file.write(', '.join(eachRow) + '\n')

How to loop through a csv file in python and output each part of the csv file into a new file?

5 Answers5