Comparing unicode with unicode in python

Question

I am trying to count the number of same words in an Urdu document which is saved in UTF-8.

so for example I have document containing 3 exactly same words separated by space

خُداوند خُداوند خُداوند

I tried to count the words by reading the file using the following code:

        file_obj = codecs.open(path,encoding="utf-8")
        lst = repr(file_obj.readline()).split(" ")
        word = lst[0]
        count =0
        for w in lst:
            if word == w:
                count += 1
        print count

but the value of count I am getting is 1 while I should get 3.

How does one compare Unicode strings?

What does `lst` print as? I have `[u'\u062e\u064f\u062f\u0627\u0648\u0646\u062f', u'\u062e\u064f\u062f\u0627\u0648\u0646\u062f', u'\u062e\u064f\u062f\u0627\u0648\u0646\u062f']` and those are exactly identical (your code works). But if there are any denormalized forms then they won't be identical. — Martijn Pieters, Nov 03 '13 at 10:14
See [Normalizing Unicode](http://stackoverflow.com/q/16467479) for the proper way to handle Unicode values with denormalized codepoints. — Martijn Pieters, Nov 03 '13 at 10:15
And remove the `repr()`. You just added `u'` and `'` to the start and end of the string. So `word` is now `"u'\u062e\u064f\u062f\u0627\u0648\u0646\u062f`, `lst[1]` is `'\u062e\u064f\u062f\u0627\u0648\u0646\u062f'` and `list[2]` is `"\u062e\u064f\u062f\u0627\u0648\u0646\u062f'"`. These strings are *obviously* not equal. — Martijn Pieters, Nov 03 '13 at 10:25
removing repr() gives me an error: File "C:\Python27\lib\encodings\cp437.py", line 12, in encode return codecs.charmap_encode(input,errors,encoding_map) UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-7: character maps to — mdanishs, Nov 03 '13 at 10:29
Not on that line it won't. Are you trying to *print* these values somewhere too? — Martijn Pieters, Nov 03 '13 at 10:30
Your Windows *console* cannot handle these characters when printing. That is a *output* problem, not a problem with the input. — Martijn Pieters, Nov 03 '13 at 10:30
oh i see, so removing the print statements and repr() I got my code working, but the count remains 1, when printing using repr() I got same output as 3rd comment. I have now removed repr() so without u' and ' this should work fine now. — mdanishs, Nov 03 '13 at 10:36

score 3 · Accepted Answer · edited May 23 '17 at 11:51

Remove the repr() from your code. Use repr() only to create debug output; you are turning a unicode value into a string that can be pasted back into the interpreter.

This means your line from the file is now stored as:

>>> repr(u'خُداوند خُداوند خُداوند\n').split(" ")
["u'\\u062e\\u064f\\u062f\\u0627\\u0648\\u0646\\u062f", '\\u062e\\u064f\\u062f\\u0627\\u0648\\u0646\\u062f', "\\u062e\\u064f\\u062f\\u0627\\u0648\\u0646\\u062f\\n'"]

Note the double backslashes (escaped unicode escapes) and the first string starts with u' and the last string ends with \\n'. These values are obviously never equal.

Remove the repr(), and use .split() without arguments to remove the trailing whitespace too:

lst = file_obj.readline().split()

and your code will work:

>>> res = u'خُداوند خُداوند خُداوند\n'.split()
>>> res[0] == res[1] == res[2]
True

You may need to normalize the input first; some characters can be expressed either as one unicode codepoint or as two combining codepoints. Normalizing moves all such characters to a composed or decomposed state. See Normalizing Unicode.

score 1 · Answer 2 · answered Nov 03 '13 at 10:16

1

Try removing the repr?

lst = file_obj.readline().split(" ")

The point is that you should at least print variables like lst and w to see what they are.

answered Nov 03 '13 at 10:16

satoru

31,822
31
91
141

Even *with* the `repr()` the sample input still works; of course the OP should not be using that, however. – Martijn Pieters Nov 03 '13 at 10:22
Ah, no, the first string will have `u'`, the second has no quotes, the third a trailing `'`. – Martijn Pieters Nov 03 '13 at 10:23
Your opening sentence *Try removing the `repr`?* suggested otherwise. – Martijn Pieters Nov 03 '13 at 10:29

score 0 · Answer 3 · answered Nov 03 '13 at 10:21

0

Comparing unicode strings in Python:

a = u'Artur'
print(a)
b = u'\u0041rtur'
print(b)

if a == b:
    print('the same')

result:

Artur
Artur
the same

answered Nov 03 '13 at 10:21

Artur

7,038
2
25
39

Comparing unicode with unicode in python

3 Answers3