Overwritten data into a JSON data structure in python

Question

I want to convert a CSV file containing data into JSON format to import it into another software.

I have issues regarding how Python handles JSON data.

Here's what CSV file looks like :

Date;Message;login
29/01/2019 15:38:55;Login successfull;user1
29/01/2019 15:38:44;Logout;user1
29/01/2019 15:31:12;Login successfull;user1

Here's my Code :

#-*- coding: UTF-8 -*- 
from collections
import namedtuple
import csv
import json
import hashlib
import datetime
import os

headers = namedtuple ("Headers", "Date, Message, Login")

csv_file_name = 'test.csv'
json_file_name = 'output.json'
json_data = []
json_line = {}

if (os.path.lexists(json_file_name)):
    os.remove(json_file_name)

json_file = open (json_file_name,'w')

with open(csv_file_name, encoding="UTF-8", newline='') as csvfile:
    next(csvfile)
    csv_data = csv.reader(csvfile, delimiter=';')
    for header in map(headers._make, csv_data):

        logDatetime = datetime.datetime.strptime(header.Date, '%d/%m/%Y %H:%M:%S')
        Message = header.Message
        login = hashlib.sha512(bytes(header.Login, 'UTF-8')).hexdigest()
        json_line["Date"] = logDatetime.strftime('%d-%m-%Y %H:%M:%S')
        json_line["Message"] = Message
        json_line["Login"] = login

        json_data.append(json_line)

    with open (json_file_name,'w') as output :
        json.dump(json_data, output, indent=4)
        output.write('\n')

Here's the Data structure generated by code :

[
    {
        "Date": "29-01-2019 15:31:12",
        "Message": "Login successfull",
        "Login": "9ec62c20118ff506dac139ec30a521d12b9883e55da92b7d9adeefe09ed4e0bd152e2a099339871424263784f8103391f83b781c432f45eccb03e18e28060d2f"
    },
    {
        "Date": "29-01-2019 15:31:12",
        "Message": "Login successfull",
        "Login": "9ec62c20118ff506dac139ec30a521d12b9883e55da92b7d9adeefe09ed4e0bd152e2a099339871424263784f8103391f83b781c432f45eccb03e18e28060d2f"
    },
    {
        "Date": "29-01-2019 15:31:12",
        "Message": "Login successfull",
        "Login": "9ec62c20118ff506dac139ec30a521d12b9883e55da92b7d9adeefe09ed4e0bd152e2a099339871424263784f8103391f83b781c432f45eccb03e18e28060d2f"
    } ]

I can't figure why my code overwrites the previous entries in the JSON data with the new one. I would be gratefull if someone could explain this to me.

Possible duplicate of [How do I pass a variable by reference?](https://stackoverflow.com/questions/986006/how-do-i-pass-a-variable-by-reference) — Rach Sharp, Mar 26 '19 at 10:54
This is because of passing by reference vs. passing by value — Rach Sharp, Mar 26 '19 at 10:55
so if I understand correctly, output.json has some lines already and you want to add to that file, is that correct? — nickyfot, Mar 26 '19 at 10:57
output.json is a fresh file i created each time the code is executed. I want to translate csv data into JSON data and save it into a file after having executed some treatement on the data. — Aurelien, Mar 26 '19 at 11:56
`json_data` is empty at first and the only operation that your code performs oon it is `.append`. What exactly is being overwritten? What do you mean by "the previous entries "? Previous to what? — Stop harming Monica, Mar 26 '19 at 13:47
@Goyo My code parses a CSV file and adds the content to `json_data` via `.append`. The problem i encounter is that the previous CSV lines saved in the variable are overwritten by the current one. — Aurelien, Mar 26 '19 at 13:54
Oh, that is because you are appending always the same dictionary. Thus you end up with a list containing the same dictionary three times. — Stop harming Monica, Mar 26 '19 at 13:59

score 1 · Accepted Answer · answered Mar 26 '19 at 14:03

What happens:

A dict is a mutable object. You use the same dict for every row in the csv file, and append that single object many times to a list. At the end, the list has multiple references on that unique object that contains the values for the last line

How to fix:

Simple: create a new dict for every line:

...
json_data = []

if (os.path.lexists(json_file_name)):
    os.remove(json_file_name)

json_file = open (json_file_name,'w')

with open(csv_file_name, encoding="UTF-8", newline='') as csvfile:
    next(csvfile)
    csv_data = csv.reader(csvfile, delimiter=';')
    for header in map(headers._make, csv_data):

        json_line = {}
        logDatetime = datetime.datetime.strptime(header.Date, '%d/%m/%Y %H:%M:%S')
        ...

Thanks. I made it with another way : `json_data.append(json_line.copy())` instead of `json_data.append(json_line)` — Aurelien, Mar 26 '19 at 14:21

score 0 · Answer 2 · answered Mar 26 '19 at 11:18

0

Assuming you have some data present in output.json If that's the case first you need to read that file & append data in list

json_data

then save the file

answered Mar 26 '19 at 11:18

Sachin Patel

499
2
12

My code aims to delete the present file with same name, then create a new one with the parsed data. I corrected my code which was incomplete because of a bad Copy/paste. – Aurelien Mar 26 '19 at 11:47

Overwritten data into a JSON data structure in python

2 Answers2