8

I have everything set to utf8mb4 in my DB :

mysql> show variables like "%character%";show variables like "%collation%";
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8mb4                    |
| character_set_connection | utf8mb4                    |
| character_set_database   | utf8mb4                    |
| character_set_filesystem | binary                     |
| character_set_results    | utf8mb4                    |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

+----------------------+--------------------+
| Variable_name        | Value              |
+----------------------+--------------------+
| collation_connection | utf8mb4_general_ci |
| collation_database   | utf8mb4_general_ci |
| collation_server     | utf8mb4_general_ci |
+----------------------+--------------------+```

My database.yml [ I added encoding/collation after it was created ]. But I did however update all my tables and database to utf8mb4.

development:
  adapter: mysql2
  encoding: utf8mb4
  collation: utf8mb4_general_ci

When I run any rails db:migrate command, the export always looks something like this for every table :

-/*!50503 SET character_set_client = utf8mb4 */;
+/*!40101 SET character_set_client = utf8 */;

I can check any of those individual tables and see that they are indeed utf8mb4 ( but only on some rows. Perhaps that's a problem? ) :

mysql> show full columns from ab_experiment_buckets;
+--------------------+--------------+--------------------+------+-----+---------+----------------+---------------------------------+---------+
| Field              | Type         | Collation          | Null | Key | Default | Extra          | Privileges                      | Comment |
+--------------------+--------------+--------------------+------+-----+---------+----------------+---------------------------------+---------+
| id                 | int(11)      | NULL               | NO   | PRI | NULL    | auto_increment | select,insert,update,references |         |
| foobar_id   | int(11)      | NULL               | NO   |     | NULL    |                | select,insert,update,references |         |
| foobar_nam        | varchar(255) | utf8mb4_general_ci | NO   |     | NULL    |                | select,insert,update,references |         |
| foobar_index       | int(11)      | NULL               | NO   |     | NULL    |                | select,insert,update,references |         |
| foobar_probability | float        | NULL               | NO   |     | NULL    |                | select,insert,update,references |         |
| foobar_value       | text         | utf8mb4_general_ci | YES  |     | NULL    |                | select,insert,update,references |         |
| created_at         | datetime     | NULL               | YES  |     | NULL    |                | select,insert,update,references |         |
| updated_at         | datetime     | NULL               | YES  |     | NULL    |                | select,insert,update,references |         |
| foobar_is_default         | tinyint(1)   | NULL               | YES  |     | NULL    |                | select,insert,update,references |         |
+--------------------+--------------+--------------------+------+-----+---------+----------------+---------------------------------+---------+
mu is too short
  • 426,620
  • 70
  • 833
  • 800
Trip
  • 26,756
  • 46
  • 158
  • 277
  • Which version of Rails are you using? It's [utf8mb4 character set by default for MySQL database](https://github.com/rails/rails/pull/33608) from Rails 6. – eux Mar 30 '21 at 01:55
  • @eux rails 6. i ended up reinstalled sql and mysql-client entirely and that has gotten me pretty much the rest of the way there.. – Trip Mar 30 '21 at 19:11
  • Had to reinstall mySQL and now I'm in the same boat again. – Trip Apr 05 '22 at 17:20
  • Please show the query that exemplifies the problem. – Rick James Apr 06 '22 at 17:15
  • Hey @RickJames, thanks for the reply and answer. The query itself is just `rails db:migrate` . The structure.sql that's dumped is basically what _my_ database is. But is it? Why would it be exporting UTF-8 if every single detail points to utf8-mb4. – Trip Apr 07 '22 at 18:30
  • @Trip - MySQL's "utf8mb4" is equivalent to the rest of the worlds "UTF-8" – Rick James Apr 08 '22 at 01:16
  • in addition to encoding and collation try adding **charset: utf8mb4** to your database.yml – David Apr 08 '22 at 13:59
  • @David I believe that was clearly stated in the original question. – Trip Apr 19 '22 at 10:59
  • The diff shows a regression of changing from the newer "utf8mb4" to the older "utf8". Was that what you intended? – Rick James Apr 27 '22 at 21:42
  • the goal is no diff. As my SQL config and database is on utf8mb4, and the code I'm pulling from is on the same. So the problem originates from doing a `mysqldump` in which my SQL config and database are ignored, and it dumps the SQL in utf8 format incorrectly. And presumably via an older version of mysql that I'm not even using. – Trip Apr 28 '22 at 12:59

1 Answers1

0

Do you have something like this?

  def configure_charsets
    response.headers["Content-Type"] = "text/html; charset=UTF-8"
    suppress(ActiveRecord::StatementInvalid) do
      ActiveRecord::Base.connection.execute 'SET NAMES utf8mb4'
    end
  end

and/or

dataSource:
    dbCreate: "update"
    url: "jdbc:mysql://localhost:8889/mydbname?useUnicode=true&zeroDateTimeBehavior=convertToNull&autoReconnect=true"

The "UTF-8" above is deliberate; it should not be "utf8mb4" and vice versa.

Do you use rake db:reset? It may cause trouble.

Trouble with UTF-8 characters; what I see is not what I stored may help with a specific symptom.

Rick James
  • 135,179
  • 13
  • 127
  • 222
  • Not particularly interested in monkeywrenching the output. I _did_ have it working by reinstalling my mysql-client in the past. But I bought a new M1, reinstalled all the same sql 7.5.3 and mysql-client 8. Exact same database. Literally just ported over. Same version sql and client. And back to the old problem again. – Trip Apr 07 '22 at 18:34
  • Since you're a SQL genius though. I think this is the big hint. `-/*!50503 SET NAMES utf8mb4 /; +/!40101 SET NAMES utf8mb4 */;` When I run the migration it's replacing the version min `50503` with `40101`. Which makes me think that the version of the client or sql is being either being misinterpreted, ignored, or I'm using a much older version ( somehow ) then I am led to believe. What would cause that rewrite other than an older SQL? I'm using versions mysql 5.7.36 and mysql-client 8.0.28 – Trip Apr 07 '22 at 18:38
  • @Trip - The `SET NAMES` command was added in 4.1.1 ("40101"), but `utf8mb4` was not added until 5.5.3. Either SET would be executed by 4.7 and 8.0. What product generated `/!40101 SET NAMES utf8mb4 */` ? – Rick James Apr 08 '22 at 01:22
  • The product is Rails' ActiveRecord. And it would be specifically this file that is doing the ORM'ing https://github.com/rails/rails/blob/main/activerecord/lib/active_record/connection_adapters/abstract_mysql_adapter.rb – Trip Apr 19 '22 at 11:06
  • Or if you're asking what product of mysql, the version is `mysql 5.7.36` – Trip Apr 19 '22 at 11:20
  • Ignoring Rails all together, if I do just `mysqldump` , I am still provided with the `40101`.. and not `50503` .. how would I make it `50503` ? What determines that number? – Trip Apr 19 '22 at 11:55
  • @Trip - If you are _loading_ the dump on mysql 5.7, the `SET NAMES` will be 'performed. If you are loading onto the very old mysql 4.0, it would ignore that `SET` (My "4.7" looks like a typo for "5.7".) – Rick James Apr 19 '22 at 13:34
  • I found your answer here https://dba.stackexchange.com/questions/137620/how-to-make-mysqldump-use-40101-set-character-set-client-utf8mf4 and it looks like you believe it is a bug that it exports this instead of `50503` for the version I'm using. – Trip Apr 27 '22 at 14:30