Ruby and MySQL UTF-8 characters

Question

I'm switching a Sinatra application from SQLite3 to MySQL. For some reason, which I can't understand, when I extract data from the MySQL using Ruby and Sequel the characters appear in 8-BIT ASCII instead of UTF-8.

The deployment environment is a FreeBSD 9.1 and MySQL 5.6.12, with a system-wide ruby19 installed from FreeBSD ports. RVM ruby-2.0p247 produces the same result though.

My my.cnf is the following:

# The following options will be passed to all MySQL clients
[client]
default-character-set=utf8
#password = your_password
port    = 3306
socket    = /tmp/mysql.sock
# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port    = 3306
socket    = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 128M 
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 2M
myisam_sort_buffer_size = 32M
thread_cache_size = 4
query_cache_size= 8M 
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 2

# encoding issues
character-set-server=utf8
collation-server=utf8_general_ci

log-bin=mysql-bin
binlog_format=mixed
server-id = 1
[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
safe-updates

[myisamchk]
key_buffer_size = 64M
sort_buffer_size = 64M
read_buffer = 1M
write_buffer = 1M

[mysqlhotcopy]
interactive-timeout

All my files use the shebang line along with the UTF-8 encoding like this script I use to test entries:

#!/usr/bin/env ruby
# encoding: UTF-8

require 'sequel'

msql = Sequel.connect(adapter: 'mysql', host: 'localhost', database: 'metrosignage', user: 'atma', password: 'toola697', encoding: 'utf8')

b = msql[:drama_addressbook]
b.each do |entry|
  p entry
  # p entry[:city].force_encoding("utf-8")
end

If I use entry[:city].force_encoding("utf-8") the output is correct, Greek UTF-8 characters are displayed fine. However I don't understand why I can't extract UTF-8 directly.

The table I'm reading the data is created using the following SQL:

CREATE TABLE `drama_addressbook` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `address` varchar(255) DEFAULT NULL,
  `address_no` int(11) DEFAULT NULL,
  `address_description` varchar(255) DEFAULT NULL,
  `phone` varchar(255) DEFAULT NULL,
  `city` varchar(255) DEFAULT NULL,
  `country` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=30 DEFAULT CHARSET=utf8;

So the database is UTF-8 and the data is UTF-8. My questions are:

Am I doing something wrong?
Why does Ruby need force_encoding?

did you create the DB like this? `CREATE DATABASE metrosignage DEFAULT CHARSET utf8;` — Patrick Oscity, Jul 09 '13 at 21:28
possible duplicate of [sequel never returns utf-8, just ascii-8bit](http://stackoverflow.com/questions/14070281/sequel-never-returns-utf-8-just-ascii-8bit) — Patrick Oscity, Jul 09 '13 at 21:28

score 4 · Accepted Answer · answered Jul 10 '13 at 16:12

4

Try using the mysql2 adapter instead of the mysql adapter, as I believe the mysql2 driver handles encodings while the mysql driver does not.

answered Jul 10 '13 at 16:12

Jeremy Evans

11,959
27
26

2

Thanks that worked fine! I can't believe that the default adapter doesn't support UTF-8, but anyway :-) thanks a lot! – patm Jul 10 '13 at 17:37
1

I am using mysql2 itself, but I having the issue. Following is the code I used: `Sequel.mysql2('some_db', :user => 'user', :password => 'password', :host => 'localhost', :encoding => 'utf8', :loggers => [logger])` – Jikku Jose Oct 15 '14 at 10:48

Ruby and MySQL UTF-8 characters

1 Answers1

Linked