Cannot read unicode .csv into R

Question

I have a .csv file, which contains the following data:

"Ա","Բ"
1,10
2,20

I cannot read it into R so that the column names are displayed like they are in the file.

d <- read.csv("./Data/1.csv", fileEncoding="UTF-8")
head(d)

Produces the following:

> d <- read.csv("./Data/1.csv", fileEncoding="UTF-8")
Warning messages:
1: In read.table(file = file, header = header, sep = sep, quote = quote,  :
  invalid input found on input connection './Data/1.csv'
2: In read.table(file = file, header = header, sep = sep, quote = quote,  :
  incomplete final line found by readTableHeader on './Data/1.csv'
> head(d)
[1] X.
<0 rows> (or 0-length row.names)

Meanwhile, doing the same without specifying the fileEncoding produces this:

> d <- read.csv("./Data/1.csv")
> head(d)
  Ô. Ô²
1  1 10
2  2 20

When I run the "file" utility to find out the encoding of the file, it says it is UTF-8:

Data\1.csv: UTF-8 Unicode text, with CRLF line terminators

I am using RStudio, Windows 7, R version 2.15.2, 32-bit.

Thanks in advance.

Works perfectly for me with R 2.15.3 on Linux. (The second warning is probably just a missing "enter" on the last line of the file) — Spacedman, May 30 '13 at 14:24
I can replicate the issue. I've never used these params, but using `encoding` instead, i.e. `read.csv(..., encoding="UTF-8")` reads in the file but the header is not shown as letters, but as `X.U.FEFF..U.0531. X.U.0532.` instead. — eddi, May 30 '13 at 15:57
an observation: `> a = "Ա"; > a; [1] "Ա"` works ok, but this doesn't: `data.frame(a); # a #1 ` — eddi, May 30 '13 at 16:32
Possible work around solution: read the data in and add the names manually. **names(data) <- c("Ա","Բ")** Problem comes when you wanna write it out. I could not write it out, but maybe there is a solution. — alap, Sep 05 '13 at 10:04
I'm having a similar problem trying to import a .csv of tweets. The `user_name` and `tweet` fields both have non-English characters, etc. [Here's a gist with a sample of the data.](https://gist.github.com/ccheaton/7490347) Any ideas how to properly import this file and set the correct encoding? — Clay, Nov 15 '13 at 19:44

score 13 · Answer 1 · edited May 23 '17 at 12:32

I wrote a longer answer on the same issue here: R on Windows: character encoding hell .

Quick answer, using the parameter encoding instead of fileEncoding should fix your first issue. You will not be able to read it possibly in either console or table view in RStudio, but you will be able to use it in formulaes.

d <- read.csv("./Data/1.csv", encoding="UTF-8")
head(d)

Having saved your table into a UTF-8 file:

> test2 <- read.csv("test2.csv", header = FALSE, sep = ",", quote = "\"", dec = ".", fill = TRUE, comment.char = "", encoding = "UTF-8")
Warning message:
In read.table(file = file, header = header, sep = sep, quote = quote,  :
  incomplete final line found by readTableHeader on 'test2.csv'

This gives you how it looks like in the console and RStudio view

> test2
        V1       V2
1 <U+0531> <U+0532>
2        1       10
3        2       20

However importantly you are able to manipulate this within R. Thus in my case it is possible to see that the script window input Ա has UTF-8 encoding, and a grep correctly finds this encoding in your table.

> Encoding("Ա")
[1] "UTF-8"
> grep("Ա", as.character(test2[1,1]))
[1] 1

You may need to find suitable encoding variants that work on your settings, or possibly change them. Unfortunately I am not sure where it is done.

You might not be able to make it pretty in all stages, but it is definitely possible to get it to work also in Windows 7 environment.

I believe `fileEncoding` is the correct argument for reading from files, whilst `encoding` is used for direct text. — Sam Brightman, May 17 '19 at 07:50

score 0 · Answer 2 · answered Sep 20 '13 at 16:31

I tried two ways to replicate your problem.

I copied the characters above into RStudio, saved it to a csv with this code:

write.csv(c("Ա","Բ",
             1,10,
             2,20), "test.csv")

df <- read.csv("test.csv")

This worked fine.

Then I thought, well maybe R is cheating when I save it to CSV with R? So I just pasted the characters to a text file and save it as a CSV. This approach doesn't have problems either.

Here's my session info:

sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
[1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C               LC_TIME=en_CA.UTF-8       
[4] LC_COLLATE=en_CA.UTF-8     LC_MONETARY=en_CA.UTF-8    LC_MESSAGES=en_CA.UTF-8   
[7] LC_PAPER=C                 LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats4    grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] party_1.0-9       modeltools_0.2-21 strucchange_1.4-7 sandwich_2.2-10   zoo_1.7-10       
[6] GGally_0.4.4      reshape_0.8.4     plyr_1.8          ggplot2_0.9.3.1  

loaded via a namespace (and not attached):
[1] coin_1.0-23        colorspace_1.2-2   dichromat_2.0-0    digest_0.6.3      
[5] gtable_0.1.2       labeling_0.2       lattice_0.20-23    MASS_7.3-29       
[9] munsell_0.4.2      mvtnorm_0.9-9995   proto_0.3-10       RColorBrewer_1.0-5
[13] reshape2_1.2.2     scales_0.2.3       splines_3.0.1      stringr_0.6.2

score 0 · Answer 3 · edited Oct 05 '16 at 12:57

0

I had the same problem and found out that the file was corrupted.

I opened the file with OpenOffice and saved it back using "UTF8" character set (you need to click the edit filter settings box) and then imported it with the read.csv()(no encoding or filencoding option) and it worked fine.

edited Oct 05 '16 at 12:57

John Paul

12,196
6
55
75

answered Aug 21 '14 at 11:35

Julien Colomb

508
4
20

Cannot read unicode .csv into R

3 Answers3

Linked

Related