0

I am using fread function for importing ".dat" file [file size 3.5 GB]. The issue with the file is some fields have embedded separator[I got to know as the same file is being used for loading via SSIS ETL tool] in it. 

data <- fread("xyz.dat", sep = '|', encoding = "UTF-8",showProgress = T, select = ord_shp_col, fill = TRUE, sep2 = "|")

Tried sep2 argument to handle as per the R document and even tried with only limited column, so that such columns could be skipped. However, ending with same error again n again.

Read 0.0% of 1712440 rowsError in fread("xyz.dat", sep = "|", encoding = "UTF-8", : Expecting 118 cols, but line 2143 contains text after processing all cols. Try again with fill=TRUE. Another reason could be that fread's logic in distinguishing one or more fields having embedded sep='|' and/or (unescaped) '\n' characters within unbalanced unescaped quotes has failed. If quote='' doesn't help, please file an issue to figure out if the logic could be improved.

Any help is highly appreciated.

nikn8
  • 1,016
  • 8
  • 23
  • When asking for help, you should include a simple [reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) with sample input and desired output that can be used to test and verify possible solutions. – MrFlick Mar 28 '18 at 14:41
  • @MrFlick I understand your concern, but the issue is with data load itself. After seeing the ways of reproducible example, I don't think I can help here as data is confidential. Could you please advise some ways to handle such scenario?? However I am trying hard if I could share some sample data. – nikn8 Mar 29 '18 at 11:14

0 Answers0