python json to csv，csv repeat loop write

Question

I need to convert a large JSON file into a csv file and read it line by line, but the code will write to the csv file in a loop, the output of the 10MB JSON file gets a 1GB csv file, and the csv file is still increasing, and the code can only be stopped manually after running.

my code

import pandas as pd
import json
all_data = []
with open('G:\\1.json',encoding='utf-8') as f_json:
    for line in f_json:
        line = line.strip()
        if line:
            all_data.append(json.loads(line))
            df = pd.json_normalize(all_data)
            df.to_csv('G:\\1.csv', index=None,encoding='utf-8', mode='a')

my JSON file

{"_index":"core-bvd-dmc","_type":"_doc","_id":"e22762d5c4b81fbcad62b5c1d77226ec","_score":1,"_source":{"a_id":"P305906272","a_id_type":"Contact ID","a_name":"Mr Chuanzong Chen","a_name_normal":"MR CHUANZONG CHEN","a_job_title":"Executive director and general manager","relationship":"Currently works for (Executive director and general manager)","b_id":"CN9390051924","b_id_type":"BVD ID","b_name":"Yantai haofeng trade co., ltd.","b_name_normal":"YANTAI HAOFENG TRADE CO","b_country_code":"CN","b_country":"China","b_in_compliance_db":false,"b_nationality":"CN","b_street_address":"Bei da jie 53hao 1609shi; Zhi fu qu","b_city":"Yantai","b_postcode":"264000","b_region":"East China|Shandong","b_phone":"+86 18354522200","b_email":"18354522200@163.com","b_latitude":37.511873,"b_longitude":121.396883,"b_geo_accuracy":"Community","b_national_ids":{"Unified social credit code":["91370602073035263P"],"Trade register number":["370602200112047"],"NOC":["073035263"]},"dates":{"date_of_birth":null},"file_name":"/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json","b_geo_point":{"lat":37.511873,"lon":121.396883}}}

output CSV

_index,_type,_id,_score,_source.a_id,_source.a_id_type,_source.a_name,_source.a_name_normal,_source.a_job_title,_source.relationship,_source.b_id,_source.b_id_type,_source.b_name,_source.b_name_normal,_source.b_country_code,_source.b_country,_source.b_in_compliance_db,_source.b_nationality,_source.b_street_address,_source.b_city,_source.b_postcode,_source.b_region,_source.b_phone,_source.b_email,_source.b_latitude,_source.b_longitude,_source.b_geo_accuracy,_source.b_national_ids.Unified social credit code,_source.b_national_ids.Trade register number,_source.b_national_ids.NOC,_source.dates.date_of_birth,_source.file_name,_source.b_geo_point.lat,_source.b_geo_point.lon
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
_index,_type,_id,_score,_source.a_id,_source.a_id_type,_source.a_name,_source.a_name_normal,_source.a_job_title,_source.relationship,_source.b_id,_source.b_id_type,_source.b_name,_source.b_name_normal,_source.b_country_code,_source.b_country,_source.b_in_compliance_db,_source.b_nationality,_source.b_street_address,_source.b_city,_source.b_postcode,_source.b_region,_source.b_phone,_source.b_email,_source.b_latitude,_source.b_longitude,_source.b_geo_accuracy,_source.b_national_ids.Unified social credit code,_source.b_national_ids.Trade register number,_source.b_national_ids.NOC,_source.dates.date_of_birth,_source.file_name,_source.b_geo_point.lat,_source.b_geo_point.lon
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
_index,_type,_id,_score,_source.a_id,_source.a_id_type,_source.a_name,_source.a_name_normal,_source.a_job_title,_source.relationship,_source.b_id,_source.b_id_type,_source.b_name,_source.b_name_normal,_source.b_country_code,_source.b_country,_source.b_in_compliance_db,_source.b_nationality,_source.b_street_address,_source.b_city,_source.b_postcode,_source.b_region,_source.b_phone,_source.b_email,_source.b_latitude,_source.b_longitude,_source.b_geo_accuracy,_source.b_national_ids.Unified social credit code,_source.b_national_ids.Trade register number,_source.b_national_ids.NOC,_source.dates.date_of_birth,_source.file_name,_source.b_geo_point.lat,_source.b_geo_point.lon
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
_index,_type,_id,_score,_source.a_id,_source.a_id_type,_source.a_name,_source.a_name_normal,_source.a_job_title,_source.relationship,_source.b_id,_source.b_id_type,_source.b_name,_source.b_name_normal,_source.b_country_code,_source.b_country,_source.b_in_compliance_db,_source.b_nationality,_source.b_street_address,_source.b_city,_source.b_postcode,_source.b_region,_source.b_phone,_source.b_email,_source.b_latitude,_source.b_longitude,_source.b_geo_accuracy,_source.b_national_ids.Unified social credit code,_source.b_national_ids.Trade register number,_source.b_national_ids.NOC,_source.dates.date_of_birth,_source.file_name,_source.b_geo_point.lat,_source.b_geo_point.lon
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
_index,_type,_id,_score,_source.a_id,_source.a_id_type,_source.a_name,_source.a_name_normal,_source.a_job_title,_source.relationship,_source.b_id,_source.b_id_type,_source.b_name,_source.b_name_normal,_source.b_country_code,_source.b_country,_source.b_in_compliance_db,_source.b_nationality,_source.b_street_address,_source.b_city,_source.b_postcode,_source.b_region,_source.b_phone,_source.b_email,_source.b_latitude,_source.b_longitude,_source.b_geo_accuracy,_source.b_national_ids.Unified social credit code,_source.b_national_ids.Trade register number,_source.b_national_ids.NOC,_source.dates.date_of_birth,_source.file_name,_source.b_geo_point.lat,_source.b_geo_point.lon
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
_index,_type,_id,_score,_source.a_id,_source.a_id_type,_source.a_name,_source.a_name_normal,_source.a_job_title,_source.relationship,_source.b_id,_source.b_id_type,_source.b_name,_source.b_name_normal,_source.b_country_code,_source.b_country,_source.b_in_compliance_db,_source.b_nationality,_source.b_street_address,_source.b_city,_source.b_postcode,_source.b_region,_source.b_phone,_source.b_email,_source.b_latitude,_source.b_longitude,_source.b_geo_accuracy,_source.b_national_ids.Unified social credit code,_source.b_national_ids.Trade register number,_source.b_national_ids.NOC,_source.dates.date_of_birth,_source.file_name,_source.b_geo_point.lat,_source.b_geo_point.lon
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
_index,_type,_id,_score,_source.a_id,_source.a_id_type,_source.a_name,_source.a_name_normal,_source.a_job_title,_source.relationship,_source.b_id,_source.b_id_type,_source.b_name,_source.b_name_normal,_source.b_country_code,_source.b_country,_source.b_in_compliance_db,_source.b_nationality,_source.b_street_address,_source.b_city,_source.b_postcode,_source.b_region,_source.b_phone,_source.b_email,_source.b_latitude,_source.b_longitude,_source.b_geo_accuracy,_source.b_national_ids.Unified social credit code,_source.b_national_ids.Trade register number,_source.b_national_ids.NOC,_source.dates.date_of_birth,_source.file_name,_source.b_geo_point.lat,_source.b_geo_point.lon
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883
core-bvd-dmc,_doc,e22762d5c4b81fbcad62b5c1d77226ec,1,P305906272,Contact ID,Mr Chuanzong Chen,MR CHUANZONG CHEN,Executive director and general manager,Currently works for (Executive director and general manager),CN9390051924,BVD ID,"Yantai haofeng trade co., ltd.",YANTAI HAOFENG TRADE CO,CN,China,False,CN,Bei da jie 53hao 1609shi; Zhi fu qu,Yantai,264000,East China|Shandong,+86 18354522200,18354522200@163.com,37.511873,121.396883,Community,['91370602073035263P'],['370602200112047'],['073035263'],,/media/hedwig/iforce/data/BvD/s3-transfer/SuperTable_v3_json/dmc/part-00020-7b09c546-2adc-413e-9e68-18b300e205cf-c000.json,37.511873,121.396883

davidverweij · Answer 1 · 2022-02-04T12:30:58.500

The output .csv shows that it is incrementally printing the output as each line comes in. In other words, the second time you write to csv, it also prints the first with it. The third time, it prints the first and second as well (etc). This is because your script loads each line separately (which is fine), but write the whole dataframe df each time. Instead, you could try to load everything into the dataframe first, then only write to .csv once. E.g.,

import pandas as pd
import json
all_data = []
with open('G:\\1.json',encoding='utf-8') as f_json:
    for line in f_json:
        line = line.strip()
        if line:
            all_data.append(json.loads(line))

# now we have all the data in 'all_data'
# only write once
df = pd.json_normalize(all_data)
df.to_csv('G:\\1.csv', index=None,encoding='utf-8', mode='a')

Alternatively, you could use Python's built in csv module instead, see also How can I convert JSON to CSV?

It converts fine, but when loading a large JSON file, I get: return {**top_dict_, **nested_dict_} MemoryError — jeri teri, Feb 04 '22 at 13:05
Do you get the error with your approach? Or using the csv module as suggested as an alternative? If your goal is solely to convert it, not perform any transformations or math with it, I'd suggest looking at the .csv writer - as this works line-by-line (no memory errors). Pandas is also very heavy on the system, which is worth it in some cases, but not here I'd say. — davidverweij, Feb 04 '22 at 13:09

python json to csv，csv repeat loop write

1 Answers1

Linked