PHP How to Remove/Replace Unknown question mark diamond characters

Question

I am using PHP to access data on old machines and output them.

Putty shows:

▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒
▒NONE.
▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒

Its the weird formatting in a attempt to show data in a more clean way

PHP echo-ed chrome shows:

������
�NONE. �
������

I have tried:

$Str1 = str_replace("▒","",$Str1);

But it doesn't filter them out. The output is already utf 8.

Does anyone know how to filter out these things? Maybe identify what � is to php?

Figure out what the actual *byte value* is, possibly convert the encoding to UTF-8. — deceze, Oct 03 '17 at 09:44
Possible duplicate of [PHP: Convert any string to UTF-8 without knowing the original character set, or at least try](https://stackoverflow.com/questions/7979567/php-convert-any-string-to-utf-8-without-knowing-the-original-character-set-or) — Martin, Oct 03 '17 at 09:46
So its utf 8 already. I could try to find the byte value but I am not even sure how to go about doing that. — MoonEater916, Oct 03 '17 at 10:30

score 18 · Accepted Answer · edited Sep 09 '19 at 11:23

18

Try this:

$Str1 = preg_replace('/[\x00-\x1F\x7F-\xFF]/', '', $Str1);

edited Sep 09 '19 at 11:23

Mayank Pandeyz

25,704
4
40
59

answered Oct 03 '17 at 09:55

Er Nilay Parekh

569
5
17

2

Worked Perfectly. but why \x00-\x1F\x7F-\xFF? – MoonEater916 Oct 03 '17 at 10:37
1

its remove unwanted character from string – Er Nilay Parekh Oct 03 '17 at 12:22
2

Thank you. Worked like a charm. – mrmut Apr 04 '18 at 09:56
This worked for me too, how did you specifically know to use '/[\x00-\x1F\x7F-\xFF]/' ? Do you have docs you could link to? – logos_164 Jul 19 '18 at 20:43
7

it also removes other letters like ç, ğ – temirbek Jun 28 '19 at 10:08
2

@temirbek exactly, therefore, this is not a valid solution – James Jan 11 '20 at 08:30
1

Thank you. Finally I can sleep now :) – hassanrazadev Apr 30 '21 at 19:19

score 2 · Answer 2 · answered Jan 08 '22 at 21:07

The problem with a regex like '/[\x00-\x1F\x7F-\xFF]/' is that it simply demolishes all UTF-8. So, about only 1% or less of all possible characters will work with this. Full Working Demo

Why not prevent what is causing this?

With a fully utf-8-configured DB and proper headers, this problem can happen if you use:

substr() — Instead, use mb_substr($utfstring, 0, 10, 'utf-8');.
htmlspecialchars() — Instead, use htmlspecialchars($utfstring, ENT_QUOTES, 'UTF-8');.
preg_replace() — Instead, use mb_ereg_replace().

PHP How to Remove/Replace Unknown question mark diamond characters

2 Answers2

Why not prevent what is causing this?

Linked