-1

Working on an application and it's a bit of a mess with its encoding protocol.

The application currently uses php_value default_charset ISO-8859-1 but also in places does <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

The MySql charset is Latin-1 (ISO-8859-1) so explains why the default_charset is being done.

There's also a wide range of encoding being done everywhere utf8_encode, json_encode, mb_convert_encoding (less prevalent)

The biggest issue we are seeing is with our mobile app REST API. People submitting emojis and such can cause some really strange behavior. Fields being emptied on display, etc.

Is there a standard protocol for handling this type of encoding to get a more uniform approach?

shenn
  • 859
  • 4
  • 17
  • 47
  • 4
    I would imagine picking an encoding that meets your requirements and sticking to it across all layers of your architecture would be optimal; is there more specific guidance you’re looking for? – esqew Feb 15 '23 at 15:45
  • @esqew that's what I was thinking about but wasn't sure if there was a recommendation. So you would suggest UTF8 on the mobile, web, server and database from the top down? I would imagine supporting ISO-8859-1 everywhere would be quite tricky – shenn Feb 15 '23 at 15:55
  • That "standard protocol" could be UTF8 - or what do you mean by protocol? – Nico Haase Feb 15 '23 at 15:59
  • 1
    ISO-8859-1 doesn't support emojis or lots of international characters. I think UTF-8 is a good solution. It's widely supported, it encodes everything ISO-8859-1 can encode and also all the special characters from every language. – Bill Karwin Feb 15 '23 at 16:01

2 Answers2

1

I have been in that hell too (self created by the way).

What the lazy textbook answer is: Redo the whole softwarestack in UTF-8. But that isn't always feasable (finance are tight, time is an issue, etc)

My practical advice would be:

Start with your database and MAKE it understand unicode (UTF-8). The database is at the core of your application, and schould be able to store unicode. This sounds maybe scary, but if it already uses LATIN-1 you can easily convert the relevant columns to unicode.

The PHP code schould not notice the difference.

From there on, make sure your PHP is using UTF-8

php_value default_charset ISO-8859-1 <-- change that

And last, you'll have the messy job to gradually look through all the code, and remove all convertions to LATIN-1.

I hope you can set up some kind of test environment, because hacking like this in a production environment is bad for your mental wellbeing. :-)

Good luck.

Erwin Moller
  • 2,375
  • 14
  • 22
0

Erwin's answer is not complete.

MySQL's charset for UTF-8 is called utf8mb4, not simply utf8. You must use utf8mb4 to allow for Emoji.

"ISO-8859-1" is the same as MySQL's "latin1". But if you claim that it is UTF-8, a mess ensues. See Trouble with UTF-8 characters; what I see is not what I stored , especially the parts on "truncate", which seems to be your main symptom.

ALTER TABLE can be used in either of 2 ways to change the charset of individual or all columns. However, if the data has been truncated, it is lost and cannot be recovered without reloading. So, I suggest you change the table definitions and connection parameters and start over.

Do not use any mb conversion routines, that will make it even harder to debug.

Rick James
  • 135,179
  • 13
  • 127
  • 222