How to ignore NULL byte when reading a csv file

Question

I'm reading a csv file generated from an equipment and I got this error message:

Error: line contains NULL byte

I opened the csv file in text editor and I do see there're some NUL bytes in the header section, which I don't really care about. How can I make the csv reader function to ignore the NUL byte and just goes through the rest of the file?

There're two blank lines between the header section and the data, maybe there's way to skip the whole header?

My code for reading the csv file is

with open(FileName, 'r', encoding='utf-8') as csvfile:
  csvreader = csv.reader(csvfile)

Potential duplicate with http://stackoverflow.com/questions/1706198/python-how-to-ignore-comment-lines-when-reading-in-a-file/1706204, http://stackoverflow.com/questions/1730649/more-pythonic-way-of-skipping-header-lines and http://stackoverflow.com/questions/9578580/skip-first-couple-of-lines-while-reading-lines-in-python-file — Artem Sokolov, Aug 25 '16 at 17:25

score 14 · Answer 1 · answered Aug 25 '16 at 17:34

14

This will replace your NULL bytes

csvreader = csv.reader(x.replace('\0', '') for x in csvfile)

answered Aug 25 '16 at 17:34

Marlon Abeykoon

11,927
4
54
75

I'm dealing with large files and have to clean few lines and this seems to be way to do it. – majTheHero Jan 15 '20 at 15:11
will it work if the csv contains value that have new lines – KennyXu Jul 21 '20 at 11:08

score 5 · Answer 2 · answered Feb 08 '18 at 19:40

When reading in a csv, you have to remove those.

There was an article I read a while ago, it was called the taco bell method of programming. In it, the article posits that taco bell really only has 8 ingredients, but from it, makes all their chalupas, and bean thingy's, and other inedible food products.

Probably should add doritos to that ingredient list. Still, the point remains.

wget, awk, sed, etc. Those should be used when you can. No point in making it overly complicated and bringing in all these libs to do it in one language.

So, I ask, can you do it in UNIX first? And you can.

UNIX

tr < file-in -d '\000' > file-out

it's quick and will work.

...Now, get back to tacos.

score 0 · Answer 3 · answered Aug 15 '19 at 03:39

0

This worked. replaced 'Null' character to '' in a file

sed -i 's/\x0//g' test.csv

answered Aug 15 '19 at 03:39

DennisLi

3,915
6
30
66

How to ignore NULL byte when reading a csv file

3 Answers3