0

I am using big csv data in spyder python to convert csv to json but it shows error field larger than field limit (131072).

Script for conversion:

import csv
import json


file = r'abcdata.csv'
json_file = r'abcdata.json'


#Read CSV File
def read_CSV(file, json_file):
    csv_rows = []
    with open(file) as csvfile:
        reader = csv.DictReader(csvfile)
        field = reader.fieldnames
        for row in reader:
            csv_rows.extend([{field[i]:row[field[i]] for i in range(len(field))}])
        convert_write_json(csv_rows, json_file)


#Convert csv data into json
def convert_write_json(data, json_file):
    with open(json_file, "w") as f:
        f.write(json.dumps(data, sort_keys=False, indent=1, separators=(',', ': '))) #for pretty
        f.write(json.dumps(data))

read_CSV(file, json_file)
CristiFati
  • 38,250
  • 9
  • 50
  • 87
NAk
  • 1
  • 1
  • 2
  • Please add the stacktrace. And also add some more data about the data you're using, row/column number, file size, and so on. – CristiFati Jan 04 '19 at 16:24

1 Answers1

1

You must have large columns of data. The default limit for the data in a single column is csv.field_size_limit(). It can be changed:

>>> import csv
>>> csv.field_size_limit()
131072
>>> csv.field_size_limit(256<<10)
131072
>>> csv.field_size_limit()
262144

You could also be reading the .CSV incorrectly.

Mark Tolonen
  • 166,664
  • 26
  • 169
  • 251