-1

Background is the csv file going to grow into huge size after many columns added, so prefer not to use pandas dataframe.to_csv to write the whole matrix from memory. and also the data need to write into the same file instead of generating a new files as historic topic as tried code as below. might be pandas to_csv append mode, from new column, but not sure how to write.

data1,data2 data3,data4
1,4,2,4
2,32,1,4
3,3,1,5
4,3,1,5
5,2,22,9
6,3,34,9
7,5,4,9

import csv
def add_col_to_csv(csvfile,fileout,new_list):
    with open(csvfile, 'r') as read_f, \
        open(fileout, 'w', newline='') as write_f:
        csv_reader = csv.reader(read_f)
        csv_writer = csv.writer(write_f)
        i = 0
        for row in csv_reader:
            row.append(new_list[i])
            csv_writer.writerow(row)
            i += 1 



new_list1 = ['new_col',4,4,5,5,9,9,9]
add_col_to_csv('input.csv','output.csv',new_list1)
JayPeerachai
  • 3,499
  • 3
  • 14
  • 29
GreatShark
  • 3
  • 2
  • You can't write back to the same file if you don't want to read the whole thing into memory first. Write to a new file, then rename it. – Barmar Dec 12 '22 at 15:25
  • delete and rename is a way, but I thought Python could be smarter... – GreatShark Dec 13 '22 at 15:46
  • You can use the [`fileinput`](https://docs.python.org/3/library/fileinput.html) module to read and update a file in place, but I don't think you can combine it with CSV parsing. Internally it works by copying the original file to a backup, then reading from that so it can overwrite the original file. – Barmar Dec 13 '22 at 16:28
  • See https://stackoverflow.com/questions/5453267/is-it-possible-to-modify-lines-in-a-file-in-place for other ways to do in-place editing of a file. Maybe the [`in_place`](https://pypi.org/project/in-place/) module is better suited. – Barmar Dec 13 '22 at 16:28
  • thanks for your comments, Pandas can improve their .to_csv method. – GreatShark Dec 13 '22 at 17:21

1 Answers1

0

you can use something like this

df = pd.DataFrame(new_list1).to_csv(f'output.csv', mode='a', index=False, header=False)
del df
del new_list1
new_list1 = []

this will append it and delete it from memory right after. You can enable index and header based on the values in you're array how ever this is a very weird and bad way to append to csv files try json instead.

d-dutchview
  • 54
  • 5