Error with fread in R--embedded nul in string: '\0'

Question

I am trying to read a csv file >4GB, However, when I use fread command it produces and error

library(data.table)
csv1 <- fread("cleaned.csv",sep = ",",colClasses = "character",showProgress = TRUE)

Error: embedded nul in string: '\0'

After some looking I found that you could use sed function such as in this stackoverflow Question But I have no clue how to use it in my scenario. Please help!

UPDATE: I have attempted to use the sed function as described below in comments, however, they throw an error.

sed couldn't flush stdout no space left on device

UPDATE2: I have solved it with the help of some colleagues.However, I am still looking to automate this activity since I had to repeat the process for each file. Expected Automation would either be from within the R or using a BASH Script. Any Suggestions?

sed is one of the bash tool which is installed by default in all linux distributions. — Avinash Raj, Jul 29 '15 at 13:12
Why don't you try this `fread("sed 's/\\0//g' mycsv.csv")` ? — Avinash Raj, Jul 29 '15 at 13:12
then open the terminal and run `sed 's/\\0//g' /src/path/mycsv.csv > /dest/path/mycsv.csv` — Avinash Raj, Jul 29 '15 at 13:13
possible duplicate of ['Embedded nul in string' error when importing huge csv with fread](http://stackoverflow.com/questions/22643372/embedded-nul-in-string-error-when-importing-huge-csv-with-fread) — nrussell, Jul 29 '15 at 13:23
@AvinashRaj, which terminal linux terminal or do I need to type this command in r terminal ? — Shoaibkhanz, Jul 29 '15 at 13:23
run `sed -i 's/\\0//g' /src/path/mycsv.csv` command in linux terminal. — Avinash Raj, Jul 29 '15 at 14:28
@AvinashRaj, this hasnt solved my problem, I have tried multiple ways through multiple functions(readr package, data.table package), even the sed function gives up;Spit out error `sed couldn't flush stdout no space left on device` — Shoaibkhanz, Aug 07 '15 at 07:37
since `sed` doesn't seem to be working, you may want to give `tr` a try. `tr < file-with-nulls -d '\000' > file-without-nulls` — scribbles, Aug 07 '15 at 13:08

score 4 · Accepted Answer · answered Aug 07 '15 at 18:14

4

The csv files were populated with ^@ and they were placed within the blank values, somehow they couldn't be searched or replaced via sed commands to solve the problem, I followed the following solution.

In linux, follow to the file directory and use vim command such as,

vim filename.csv

:%s/CTRL+2//g

ESC #TO SWITCH FROM INSERT MODE

:wq # TO SAVE THE FILE

I had to do this manually for every file. However, I still looking for a way to automate this either within R or using from BASH script.

answered Aug 07 '15 at 18:14

Shoaibkhanz

1,942
3
24
41

2

Vim scripts (gvim,vim) can be pretty easy to adapt `vi -s edit.vim filename.txt` where edit.vim contains (the :wq is optional) `:%s/CTRL+2//g :wq` you can also use the `:argdo `command to run a command on all files in the argument. – scribbles Aug 07 '15 at 19:46

Error with fread in R--embedded nul in string: '\0'

1 Answers1

Linked