What is the replacement of deprecated array of unicode chars?

Question

An array of unicode characters can be used as a mutable string :

import array

ins = "Aéí"
ms = array.array('u', ins)
ms[0] = "ä"
outs = ms.tounicode()
# äéí

But type 'u' is deprecated since Python 3.3. What is the modern replacement?

I could do:

ms = list(ins)
# mutate
outs = ''.join(ms)

But I find a list of characters very memory inefficient compared to the array.

Alternatively:

ms = array.array('L', (ord(ch) for ch in ins))
ms[0] = ord("ä")
outs = "".join(chr(ch) for ch in ms)

But it is far less readable than the deprecated original.

Answer 1

This is similar to your last example, but initializes the array in a more readable and efficient way. You must choose an array size that is four bytes in size. I is the code for unsigned int and is four bytes on most OSes. For portability you may want to choose this value programmatically.

#!coding:utf8
import array
import sys

# Verifying the item size.
assert array.array('I').itemsize == 4

# Choose encoding base on native endianness:
encoding = 'utf-32le' if sys.byteorder == 'little' else 'utf-32be'

ins = "Aéí"
ms = array.array('I',ins.encode(encoding))
ms[0] = ord('ä')
print(ms.tobytes().decode(encoding))

Output:

äéí

Timings for a 1000-element string show it is quite a bit faster:

In [7]: s = ''.join(chr(x) for x in range(1000))

In [8]: %timeit ms = array.array('I',s.encode('utf-32le'))
1.77 µs ± 14.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [9]: %timeit ms = array.array('I',(ord(x) for x in s))
167 µs ± 5.35 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [21]: %timeit outs = "".join(chr(x) for x in ms)
194 µs ± 4.98 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [23]: %timeit outs = ms.tobytes().decode('utf-32le')
3.92 µs ± 97 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

But you may be overthinking it. I don't know what string sizes you are dealing with, but just using list(data) is faster, if less memory efficient, and it isn't that bad. Here's a list of non-BMP characters (~1M), and timings for immutable string slicing, mutating an array, and mutating a list:

In [67]: data = ''.join(chr(x) for x in range(0x10000,0x110000))

In [68]: ms = array.array('I',data.encode('utf-32le'))

In [69]: %%timeit global data
    ...: data = data[:500] + 'a' + data[501:]
    ...:
3.33 ms ± 235 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [70]: %timeit ms[500] = ord('a')
73.6 ns ± 0.433 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [71]: %%timeit v = list(data)
    ...: v[500] = 'a'
    ...:
28.7 ns ± 0.144 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [72]: sys.getsizeof(data)
Out[72]: 4194380

In [73]: sys.getsizeof(ms)
Out[73]: 4456524

In [74]: sys.getsizeof(list(data))
Out[74]: 9437296

Mutating a list is straightforward, 3x faster than mutating the array, and only uses a little more that 2x the memory.

Answer 2

The replacement in Python3 is str, the builtin type. Strings are immutable sequences of Unicode code points.

As a result, there is no need to use array('u', ...), str is enough.

By the way, the reason of removing 'u' (use PyUnicode* internally) is the Py_UNICODE* representation is deprecated and inefficient; it should be avoided in performance- or memory-sensitive situations.

Answer 3

Python 3.3+ has deprecated the use of Unicode separately. Because of PEP393 and that is why everything can be expressed and used as the str.

Here is a nice How to about the Unicode handling in post-Python 3.3 interpreters. By definition -

The Unicode standard describes how characters are represented by code points. A code point is an integer value, usually denoted in base 16. In the standard, a code point is written using the notation U+12CA to mean the character with value 0x12ca (4,810 decimal). The Unicode standard contains a lot of tables listing characters and their corresponding code points:

And finally, I am not sure what you want to achieve, but does the following do the work?

ins = "Aéí"
m = "ä" + ins[1:]
print(m)

In a Python 3.5.2 version it gives me this output -

äéí

Please let me know if this helps

---

As a final note, if you are wondering about the + operator, then here is a nice SO post about that. The gist of the most voted answer is : If you are using Cpython, you better use + than append join etc.

---

If you want to use your apprach, there is still the entire array module available with all the things you needed to run our code as it is. I tested it in python3.5 and it works.

There is this way also-

a = "ä"
p = b"".join([a.encode('utf-8'), ins[1:].encode('utf-8')])
print(p.decode('utf-8'))