21

Since MySQL's utf8 doesn't support 4 byte characters, I'm looking for a way to detect and eliminate any 4 byte utf8 characters from a string in Ruby. I understand that I can update my table to use utf8m4 but for a couple reasons that's not possible or the desired solution.

Simply encoding the string to ASCII will remove these characters but will also remove all other non-ASCII characters, which is not good.

JZC
  • 470
  • 5
  • 12

2 Answers2

37

The following seems to work for me in Ruby 1.9.3:

input.each_char.select{|c| c.bytes.count < 4 }.join('')

For example:

input = "hello \xF0\xA9\xB6\x98 world"                  # includes U+29D98
input.each_char.select{|c| c.bytes.count < 4 }.join('') # 'hello  world'
John Ledbetter
  • 13,557
  • 1
  • 61
  • 80
  • Thanks! Seems obvious now that you've suggested it. I was so deep in thinking about encodings, I didn't think to simply look at the byte count of each character. – JZC May 10 '13 at 17:48
  • How is the performance of this with a long string? 5000+ chars? – Arnold Roa Jul 17 '14 at 03:14
  • thank you thank you thank you ... not personally worried about performance, for the moment I'm happy to have a working solution – steve Apr 14 '16 at 20:51
  • That is not efficient at all, is there any other way to do that? – raquelhortab Oct 08 '21 at 09:56
1

Another option (tested in ruby 2.7) is to use a regular expression with gsub:

input = "hello \xF0\xA9\xB6\x98 world"    # includes U+29D98
input.gsub(/[\u{10000}-\u{10FFFF}]/, "?") # 'hello ? world'
James Healy
  • 14,557
  • 4
  • 33
  • 43