Error in tolower() invalid multibyte string

Question

This is the error that I receive when I try to run tolower() on a character vector from a file that cannot be changed (at least, not manually - too large).

Error in tolower(m) : invalid multibyte string X

It seems to be French company names that are the problem with the É character. Although I have not investigated all of them (also not possible to do so manually).

It's strange, because my thought was that encoding issues would have been identified during read.csv(), rather than during operations after the fact.

Is there a quick way to remove these multibyte strings? Or, perhaps a way to identify and convert? Or even just ignore them entirely?

`tolower("École")` gives me `"école"` which seems right (I'm French :-)) — flodel, Nov 02 '12 at 01:23
This question might be helpful: http://stackoverflow.com/questions/4993837/r-invalid-multibyte-string — thelatemail, Nov 02 '12 at 01:38
I should also mention that I don't get the same error on Windows (or at least, when I had Windows) — Brandon Bertelsen, Nov 02 '12 at 01:49

score 23 · Accepted Answer · edited Nov 21 '17 at 21:21

23

Here's how I solved my problem:

First, I opened the raw data in a texteditor (Geany, in this case), clicked properties and identified the Encoding type.

After which I used the iconv() function.

x <- iconv(x,"WINDOWS-1252","UTF-8")

To be more specific, I did this for every column of the data.frame from the imported CSV. Important to note that I set stringsAsFactors=FALSE in my read.csv() call.

dat[,sapply(dat,is.character)] <- sapply(
    dat[,sapply(dat,is.character)],
    iconv,"WINDOWS-1252","UTF-8")

edited Nov 21 '17 at 21:21

Michael Ohlrogge

10,559
5
48
76

answered Nov 02 '12 at 03:50

Brandon Bertelsen

43,807
34
160
255

4

An alternative for those, like me, too lazy to identify the encoding would be `x <- enc2utf8(x)` – Martin Smith Apr 17 '18 at 13:09
Thanks man! this helped me a lot, I was banging my head here n there. – Rana Usman Mar 19 '19 at 05:28

Onur Ece · Answer 2 · 2020-04-23T09:16:24.680

8

I was getting the same error. However, in my case it wasn't when I was reading the file, but a bit later when processing it. I realised that I was getting the error, because the file wasn't read with the correct encoding in the first place.

I found a much simpler solution (at least for my case) and wanted to share. I simply added encoding as below and it worked.

read.csv(<path>, encoding = "UTF-8")

edited Apr 23 '20 at 09:16

answered Jan 18 '19 at 15:49

Onur Ece

105
1
6

2

Or "ISO-8859-1" or "Windows-1252". Great addition Onur! – Brandon Bertelsen Jan 18 '19 at 21:48

dmt · Answer 3 · 2022-02-08T12:30:50.907

5

library(tidyverse)

data_clean = data %>%
    mutate(new_lowercase_col = tolower(enc2utf8(as.character(my_old_column))))

Where new_lowercase_col is the name of the new column I'm making out of the old uppercase one, which was called my_old_column.

edited Feb 08 '22 at 12:30

answered Jun 24 '20 at 14:46

dmt

2,113
3
24
23

score 4 · Answer 4 · edited Dec 09 '16 at 15:24

4

I know this has been answered already but thought I'd share my solution to this as I experienced the same thing.

In my case, I used the function str_trim() from package stringr to trim whitespace from start and end of string.

com$uppervar<-toupper(str_trim(com$var))

edited Dec 09 '16 at 15:24

histelheim

4,938
6
33
63

answered Jul 22 '14 at 19:26

user3866101

41
1

score 0 · Answer 5 · answered Jun 08 '19 at 19:48

0

# to avoid datatables warning: error in tolower(x) invalid multibyte string
# assuming all columns are char
new_data <- as.data.frame(
  lapply(old_data, enc2utf8),
  stringsAsFactors = FALSE
)

answered Jun 08 '19 at 19:48

Edgar Manukyan

1,153
10
21

score 0 · Answer 6 · answered Jan 26 '20 at 06:41

My solution to this issue

library(dplyr) # pipes
library(stringi) # for stri_enc_isutf8

#Read in csv data
old_data<- read.csv("non_utf_data.csv", encoding = "UTF-8")

#despite specifying utf -8, the below columns are not utf8:
all(stri_enc_isutf8(old_data$problem_column))

#The below code uses regular expressions to cleanse. May need to tinker with the last 
#portion that selects the grammar to retain

utf_eight_data<- old_data %>% 
  mutate(problem_column = gsub("[^[:alnum:][:blank:]?&/\\-]", "", old_data$problem_column)) %>%
rename(solved_problem = problem_column)

#this column is now utf 8.

all(stri_enc_isutf8(utf_eight_data$solved_problem))

Error in tolower() invalid multibyte string

6 Answers6

Linked