Difficult CSV import

Question

I have a .csv file with rows in the following format

1234, "Hello, Im text. "Thats a silly way to do it" is what the guy said.", 5678

The content of some of the entries contains double quotes and commas. However they are never next to each other. Some of the text entries are up to 524,288 characters long

pandas.read_csv('file.csv', dtype={'a': np.uint16, 'b': 'S524288', \
'c', np.uint16}, delimiter=',', quotechar='"', engine='python')

is giving me an error.

ParserError: field is larger than field limit (131072)

Any ideas?

The error is definitely not from the line you've posted. Please look at how to provide a [mcve]. — cs95, Jul 13 '17 at 16:59
Possible duplicate of [\_csv.Error: field larger than field limit (131072)](https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072) — Bharath M Shetty, Jul 13 '17 at 17:00
@cs95 Au contraire, that line probably tells us exactly what's wrong. The data is not well-formed CSV so the CSV parser gets into a state where it's reading in the rest of the file into a single field looking for the missing closing double quote which never arrives. — tripleee, Feb 19 '21 at 06:18

score 1 · Accepted Answer · answered Jul 13 '17 at 17:05

One Side Note:

In Python, regardless of where the " are, having them within a string is bothersome.

When you set

x = "He said "hi there!" to me"

This will return an error, because it parses the string "He said " and then errors out with hi there!. This is an issue you may run into depending on how you parse your csv file.

This would return a different error than you are receiving, so it might not pose an immediate problem, but if the row you uploaded in your question actually has "Hello, Im text. "Thats a silly way to do it" is what the guy said." in it you might run into issues down the road.

Back to the problem at hand:

To solve the error you are receiving, you can try running:

import sys
import csv

csv.field_size_limit(sys.maxsize)

This should increase the size of the field read_csv() accepts.

Hope it helps!

Source

Difficult CSV import

1 Answers1