UTF-8 safe equivalent of ord or charCodeAt() in PHP

Question

I need to be able to use ord() to get the same value as javascript's charCodeAt() function. The problem is that ord() doesn't support UTF8.

How can I get Ą to translate to 260 in PHP? I've tried some uniord functions out there, but they all report 256 instead of 260.

Thanks a lot for any help!

Regards

Answer 1

mbstring version:

function utf8_char_code_at($str, $index)
{
    $char = mb_substr($str, $index, 1, 'UTF-8');

    if (mb_check_encoding($char, 'UTF-8')) {
        $ret = mb_convert_encoding($char, 'UTF-32BE', 'UTF-8');
        return hexdec(bin2hex($ret));
    } else {
        return null;
    }
}

using htmlspecialchars and htmlspecialchars_decode for getting one character:

function utf8_char_code_at($str, $index)
{
    $char = '';
    $str_index = 0;

    $str = utf8_scrub($str);
    $len = strlen($str);

    for ($i = 0; $i < $len; $i += 1) {

        $char .= $str[$i];

        if (utf8_check_encoding($char)) {

            if ($str_index === $index) {
                return utf8_ord($char);
            }

            $char = '';
            $str_index += 1;
        }
    }

    return null;
}

function utf8_scrub($str)
{
    return htmlspecialchars_decode(htmlspecialchars($str, ENT_SUBSTITUTE, 'UTF-8'));
}

function utf8_check_encoding($str)
{
    return $str === utf8_scrub($str);
}

function utf8_ord($char)
{
    $lead = ord($char[0]);

    if ($lead < 0x80) {
        return $lead;
    } else if ($lead < 0xE0) {
        return (($lead & 0x1F) << 6) 
      | (ord($char[1]) & 0x3F);
    } else if ($lead < 0xF0) {
        return (($lead &  0xF) << 12)
     | ((ord($char[1]) & 0x3F) <<  6)
     |  (ord($char[2]) & 0x3F);
    } else {
        return (($lead &  0x7) << 18)
     | ((ord($char[1]) & 0x3F) << 12)
     | ((ord($char[2]) & 0x3F) <<  6)
     |  (ord($char[3]) & 0x3F);
    }
}

PHP extension version:

#include "ext/standard/html.h"
#include "ext/standard/php_smart_str.h"

const zend_function_entry utf8_string_functions[] = {
    PHP_FE(utf8_char_code_at, NULL)
    PHP_FE_END
};

PHP_FUNCTION(utf8_char_code_at)
{
    char *str;
    int len;
    long index;

    unsigned int code_point;
    long i;
    int status;
    size_t pos = 0, old_pos = 0;

    if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "sl", &str, &len, &index) == FAILURE) {
        return;
    }

    for (i = 0; pos < len; ++i) {
        old_pos = pos;
        code_point = php_next_utf8_char((const unsigned char *) str, (size_t) len, &pos, &status);

        if (i == index) {
            if (status == SUCCESS) {
                RETURN_LONG(code_point);
            } else {
                RETURN_NULL();
            }

        }

    }

    RETURN_NULL();
}

Answer 2

ord() works byte per byte (as most of PHPs standard string functions - if not all). You would need to convert it your own, for example with the help of the multibyte string extension:

$utf8Character = 'Ą';
list(, $ord) = unpack('N', mb_convert_encoding($utf8Character, 'UCS-4BE', 'UTF-8'));
echo $ord; # 260

Answer 3

Try:

function uniord($c) {
        $h = ord($c{0});
        if ($h <= 0x7F) {
            return $h;
        } else if ($h < 0xC2) {
            return false;
        } else if ($h <= 0xDF) {
            return ($h & 0x1F) << 6 | (ord($c{1}) & 0x3F);
        } else if ($h <= 0xEF) {
            return ($h & 0x0F) << 12 | (ord($c{1}) & 0x3F) << 6
                                     | (ord($c{2}) & 0x3F);
        } else if ($h <= 0xF4) {
            return ($h & 0x0F) << 18 | (ord($c{1}) & 0x3F) << 12
                                     | (ord($c{2}) & 0x3F) << 6
                                     | (ord($c{3}) & 0x3F);
        } else {
            return false;
        }
    }
    echo uniord('Ą');

Answer 4

This should be the equivalent to JavaScript’s charCodeAt() based of @hakre’s work but corrected to actually work the same as JavaScript (in every way I could think of to test):

function charCodeAt($string, $offset) {
  $string = mb_substr($string, $offset, 1);
  list(, $ret) = unpack('S', mb_convert_encoding($string, 'UTF-16LE'));
  return $ret;
}

(This requires the PHP extension "mbstring" to be installed and activated.)

Answer 5

Since PHP 7.2 there is mb_ord(). Using this one can get an JS equivalent to charCodeAt() as

function jsCharCodeAt($string, $index)
{
     return mb_ord(mb_substr($string, $index, 1));
}

This seems to work just fine for all UTF-16 characters. However, the behavior of charCodeAt() for non UTF-16 characters is a little bit tricky and the functions are not equivalent on them.

Answer 6

There is one ord_utf8 function here : https://stackoverflow.com/a/42600959/7558876

This function looks like this (accept string and return integer)

<?php

function ord_utf8($s){
return (int) ($s=unpack('C*',$s[0].$s[1].$s[2].$s[3]))&&$s[1]<(1<<7)?$s[1]:
($s[1]>239&&$s[2]>127&&$s[3]>127&&$s[4]>127?(7&$s[1])<<18|(63&$s[2])<<12|(63&$s[3])<<6|63&$s[4]:
($s[1]>223&&$s[2]>127&&$s[3]>127?(15&$s[1])<<12|(63&$s[2])<<6|63&$s[3]:
($s[1]>193&&$s[2]>127?(31&$s[1])<<6|63&$s[2]:0)));
}

And one fast chr_utf8 here : https://stackoverflow.com/a/42510129/7558876

This function looks like this (accept integer and return a string)

<?php

function chr_utf8($n,$f='C*'){
return $n<(1<<7)?chr($n):($n<1<<11?pack($f,192|$n>>6,1<<7|191&$n):
($n<(1<<16)?pack($f,224|$n>>12,1<<7|63&$n>>6,1<<7|63&$n):
($n<(1<<20|1<<16)?pack($f,240|$n>>18,1<<7|63&$n>>12,1<<7|63&$n>>6,1<<7|63&$n):'')));
}

Please check links if you want one example…