0

I'm trying to use the "chunked" package to read a big .csv file with read_csv_chunkwise but there seems to be a problem in my data where the number of columns are more than the number of elements in the first, second or n-th row. This creates an error:

>chunked::read_csv_chunkwise(file1, sep = ",", dec = ".", chunk_size = 300000, header = T)

    Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec,  : line 1 did not have 13 elements

Can this be fixed somehow ? I heard that other packages like "data.table" have an argument fill = TRUE that fixes this problem

adl
  • 1,390
  • 16
  • 36
  • Well, what's actually going on in your data file? Is it malformed? Do you know why the number of columns change? It would be helpful to have some sort of minimal [reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) to know what the problem really is. – MrFlick Oct 03 '18 at 15:34
  • there are empty elements that by default are not turned into NA but an error is returned because the length of the columns (number of columns) and length of elements in the first observations (or any other observation) are not the same. I found this example with "data.table" : read.table( text= "Element1 Element2 Element5 Element6 Element7" , fill = FALSE , header = FALSE ) – adl Oct 03 '18 at 15:46

0 Answers0