11

I have a string in unicode and I need to return the first N characters. I am doing this:

result = unistring[:5]

but of course the length of unicode strings != length of characters. Any ideas? The only solution is using re?

Edit: More info

unistring = "Μεταλλικα" #Metallica written in Greek letters
result = unistring[:1]

returns-> ?

I think that unicode strings are two bytes (char), that's why this thing happens. If I do:

result = unistring[:2]

I get

M

which is correct, So, should I always slice*2 or should I convert to something?

SilentGhost
  • 307,395
  • 66
  • 306
  • 293
Jon Romero
  • 4,062
  • 6
  • 36
  • 34

3 Answers3

7

When you say:

unistring = "Μεταλλικα" #Metallica written in Greek letters

You do not have a unicode string. You have a bytestring in (presumably) UTF-8. That is not the same thing. A unicode string is a separate datatype in Python. You get unicode by decoding bytestrings using the right encoding:

unistring = "Μεταλλικα".decode('utf-8')

or by using the unicode literal in a source file with the right encoding declaration

# coding: UTF-8
unistring = u"Μεταλλικα"

The unicode string will do what you want when you do unistring[:5].

Thomas Wouters
  • 130,178
  • 23
  • 148
  • 122
  • You'll need "#coding: utf-8" before the .decode() example as well, and the file must be actually saved in utf-8. Python 2.x defaults to ASCII when decoding scripts. Any use of non-ASCII characters requires the #coding line to declare the encoding used to save the file. – Mark Tolonen Jan 29 '10 at 11:20
  • 1
    In Python 2.5 and later you need the coding declaration on any source file with non-ASCII contents, yes. (Before then it's just a warning.) The coding declaration will however not change the meaning of the code, since it's just bytes in a bytestring. – Thomas Wouters Jan 29 '10 at 13:23
  • 2
    -1 This is not correct u"Some Unicode test"[:5] May give illegal sequence, because UTF-16 is variable width encoding, so cutting "Unicode" string is not correct as cutting utf-8 string – Artyom Jan 29 '10 at 13:43
  • 1
    You seem to be confused between UTF-16 and Unicode. Python only uses UTF-16 for Unicode in UCS-2 builds (which is mostly just on Windows.) In UCS-4 builds, slicing unicode works fine (which is why you should use UCS-4 builds.). In UCS-2 builds, it works fine for any BMP character, which is what the OP was using. – Thomas Wouters Jan 29 '10 at 14:00
  • 1
    By default Python is build with UCS-2, even on linux. I know that it is possible to use UCS-4 builds, but they are not common. So in any case assuming that the string can be cut "as-is" is wrong. Unless you work in BMP only. In any case this is wrong approach. See my answer below for the reason why. – Artyom Jan 29 '10 at 14:47
  • 1
    The default is UCS-2, but most linux distributions in fact use UCS-4. Just take a look at sys.maxunicode on a typical system. When using a UCS-2 build and non-BMP characters, there's no good way to slice, regardless. – Thomas Wouters Jan 29 '10 at 14:56
7

Unfortunately for historical reasons prior to Python 3.0 there are two string types. byte strings (str) and Unicode strings (unicode).

Prior to the unification in Python 3.0 there are two ways to declare a string literal: unistring = "Μεταλλικα" which is a byte string and unistring = u"Μεταλλικα" which is a unicode string.

The reason you see ? when you do result = unistring[:1] is because some of the characters in your Unicode text cannot be correctly represented in the non-unicode string. You have probably seen this kind of problem if you ever used a really old email client and received emails from friends in countries like Greece for example.

So in Python 2.x if you need to handle Unicode you have to do it explicitly. Take a look at this introduction to dealing with Unicode in Python: Unicode HOWTO

LennyLip
  • 1,683
  • 19
  • 19
Tendayi Mawushe
  • 25,562
  • 6
  • 51
  • 57
  • "Μεταλλικα" is not an ASCII string. It is a byte string in the encoding used to save the script. – Mark Tolonen Jan 29 '10 at 11:15
  • 2
    You are right Mark it is more correct to refer to these as byte strings rather than ASCII strings, I have updated the answer accordingly. What I was really trying to express was that ASCII text (or equivalent byte string depending on the code pages on your computer) is the only thing that can be safely manipulated with byte strings. – Tendayi Mawushe Jan 29 '10 at 12:48
4

There is no correct straight-forward approach with any type of "Unicode string".

Even Python "Unicode" UTF-16 string has variable length characters so, you can't just cut with ustring[:5]. Because some Unicode Code points may use more then one "character" i.e. Surrogate pairs.

So if you want to cut 5 code points (note these are not characters) so you may analyze the text, see http://en.wikipedia.org/wiki/UTF-8 and http://en.wikipedia.org/wiki/UTF-16 definitions. So you need to use some bit masks to figure out boundaries.

Also you still do not get characters. Because for example. Word "שָלוֹם" -- peace in Hebrew "Shalom" consists of 4 characters and 6 code points letter "shin", vowel "a" letter "lamed", letter "vav" and vowel "o" and final letter "mem".

So character is not code point.

Same for most western languages where a letter with diacritics may be represented as two code points. Search for example for "unicode normalization".

So... If you really need 5 first characters you have to use tools like ICU library. For example there is ICU library for Python that provides characters boundary iterator.

Tomer Cohen
  • 374
  • 6
  • 13
Artyom
  • 31,019
  • 21
  • 127
  • 215