1

I am using pandas for read and write csv file. i want to extend column in csv. i am reading csv in chunks, when first chunk read then extend new column data to it and then write that chunk to csv, then receive 2nd chunk add new column data and then write to csv. Below is what i want.

SN, Name, City
1, John, Washington
2, Eric, Los Angeles
3, Brad, Texas 

it append the new column data to new rows rather then appending to existing row.

SN, Name, 
1, John, 
2, Eric, 
3, Brad, 
4   City
5  Washington
6  Los Angeles
7  Texas

Here is the code.

with open('exampleCsv.csv', 'r') as fieldnames:
    csv_input = pd.read_csv(fieldnames, chunksize=4)
    list_1 = [5,6,7,8,1,2,3,4]
    list_len = len(list_1)//2

    for i in csv_input:
        if list_len%2 == 0:   
            if list_iter == 1:
                i['channel_name'] = list_1[list_len:]
                i.to_csv('exampleCsv.csv', mode='a', index=True)

            elif list_iter == 2:
                i['channel_name'] = list_1[:list_len]
                i.to_csv('exampleCsv.csv', header=None, mode='a', index=True)
        elif list_len%2 == 1:
            i['channel_name'] = list1
            i.to_csv('exampleCsv.csv', header=None, mode='a', index=True)
        list_iter = list_iter + 1
  • The data could be in Tbs. or may b more. – Ahsan Akram Sep 23 '19 at 15:30
  • Possible duplicate of [How to read a 6 GB csv file with pandas](https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas) – Vishnudev Krishnadas Sep 29 '19 at 10:09
  • @Vishnudev Thanks for reply, Reading Data in chunks is working,, but when i add new column data into it then write to effect those specific row in csv,, this append new row in the end of csv file,, i want new column in next to city column, – Ahsan Akram Sep 29 '19 at 11:57
  • @AhsanAkram Have you solved this problem? – Sergei Jan 12 '23 at 22:07

0 Answers0