Python: append dict to existing csv

Question

I want to append dictionary data to an existing csv, without writing again headers.

I tried every thing from this issue: append new row to old csv file python

I tried pandas and csv solutions.

   try:
       if os.path.isfile(csv_filepath) is False:
          df = pandas.DataFrame.from_dict(dict_data, orient="index")
          df.to_csv(csv_filepath)
       else:
           list_values = []
           list_keys = []
           for key, value in dict_data.items():
               list_values.append(value)
               list_keys.append(key)
           with open(csv_filepath, 'a', newline='') as fd:
               writer = csv.DictWriter(fd, fieldnames=list_keys)
               writer.writerow(dict_data)
   except Exception as e:
       raise e

I also tried

with open(csv_filtepath, 'a') as f:
    writer = csv.writer(f)
    writer.writerow(dict_data)

but both erase and write the new data without appending.

I also tried to to load the both csv in dataframes then append the second to the first but it added headers two times

EDIT:

For exemple, I first create a csv with: {'toto': 1, 'tata': 2, 'titi': 3}

Then I want to append the created csv {'toto': 2, 'tata': 1, 'titi': 5}

After this operation, I want to have 1 and 2 for in toto column, 2 and 1 in the tata column and 3 and 5 in the titi column.

EDIT 2

I tried this:

 df1 = pd.read_csv(csv_filepath)
 df2 = pd.DataFrame(dict_data)
 df3 = pd.concat([df1, df2], axis=0)
 df3.to_csv(csv_filepath)

but I have an error message : 'Error tokenizing data. C error: Expected 2 fields in line 20, saw 9'

EDIT 3

list_keys = []
for key, value in dict_data.items():
    list_keys.append(key)
with open(csv_filepath, 'a', newline='') as fd:
    writer = csv.DictWriter(fd, fieldnames=list_keys)
    writer.writerow([data['toto'], data['tata'], data['titi']])

And I have this error message: 'list' object has no attribute 'keys'

I want to add to an existing csv file, data from a dictionary with same keys.

provide the data to work with...and the output that you got ..and the output you want — geekzeus, Jul 25 '19 at 16:52
Thanks @geekzeus for your answer, I just have edited my question — Malkovitch John, Jul 25 '19 at 16:57
The `open(csv_filtepath, 'a')` approach works; in fact, it just worked for me without truncation. Note that you create a `writer`, not `DictWriter`; the former accepts lists, not dicts. But if you're appending to an existing file, you have to follow its order of columns explicitly anyway, so a `DictWriter` won't work. You will have to do something like `.writerow([data['toto'], data['tata'], data['titi']])`, assuming that it's the column order of the file you're appending to. — 9000, Jul 25 '19 at 17:18
Thanks @9000, can you please make an answer ? I will edit right now my question with the try from your proposition — Malkovitch John, Jul 25 '19 at 17:32

score 3 · Accepted Answer · edited Feb 12 '21 at 22:49

The open(csv_filtepath, 'a') approach works; in fact, it just worked for me without truncation.

Note that you create a writer, not DictWriter; the former accepts lists, not dicts.

But if you're appending to an existing file, you have to follow its order of columns explicitly anyway, so a DictWriter won't work. You will have to do something like below, assuming that it's the column order of the file you're appending to.

import csv

with open(csv_file, 'a') as f:
  writer = csv.writer(f)  # Note: writes lists, not dicts.
  for data in rows_to_append:  # Maybe your df, or whatever iterable.
    writer.writerow([data['toto'], data['tata'], data['titi']])

Knowing the column order may be tricky, too. If you're not sure about it, you can e.g. read only the first line of the (presumably huge) first CSV file if that line contains column names. You can use the normal CSV reader for it:

with open(csv_file) as f:
  reader = csv.reader(f)
  column_names = next(reader)
# Here the file is closed after reading just one line.

Thanks but your anwser isn't complete, you should add how to declare a writer then I will validate your answer — Malkovitch John, Jul 25 '19 at 18:02

geekzeus · Answer 2 · 2019-07-25T17:52:43.060

0

df1 = pd.DataFrame({'toto': [1], 'tata': [2], 'titi': [3]})
df2 = pd.DataFrame({'toto': [2], 'tata': [1], 'titi': [5]})

DF = pd.concat([df1, df2], axis=0)

Is this what you are looking for

EDIT: Memory Efficient snippet for your example

filepaths = ['pathtocsvfolder/df1.csv','pathtocsvfolder/df2.csv']
only_head = True

for i in filepaths:
    df = pd.read_csv(i)
    if only_head:
        df.to_csv('final.csv', index=False)
        only_head = False
    else:
        df.to_csv('final.csv', index=False, header=False, mode='a')

edited Jul 25 '19 at 17:52

answered Jul 25 '19 at 17:07

geekzeus

785
5
14

This would need to read both parts into RAM; this may be not very desirable. – 9000 Jul 25 '19 at 17:13
it is not really what I look for because in my case I want to append. Using your logique I tried something, I edit my question. – Malkovitch John Jul 25 '19 at 17:21
try `sep='delimiter', header=None` – geekzeus Jul 25 '19 at 17:31

Python: append dict to existing csv

2 Answers2