Unicode string not being detected in data structure

Asked Mar 15 '17 at 18:06

Active Mar 15 '17 at 18:06

Viewed 81 times

Using python 2, I'm unable to detect if my unicode string is detected in a list, set, or dictionary.

For example I'm generating the list by iterating through file names

res = []
for root, dir, file in os.walk('./'):
    for name in file:
        res.append(name.replace('/', '_'))

One of the file's name is Município

When I print the item out I get Munici\xcc\x81pio. That's not an issue, but when I try to check if Município is in the list, it returns false. I check if both are unicode and they are.

asked Mar 15 '17 at 18:06

user3145912

Is this windows? What does `sys.getfilesystemencoding()` say? – tdelaney Mar 15 '17 at 18:11
OSX. It prints out `utf-8` – user3145912 Mar 15 '17 at 18:12
What does `print repr(name)` show? It seems like that file name wasn't encoded properly when it was created. – tdelaney Mar 15 '17 at 18:18
Ah, I get `Munici\xcc\x81pio` – user3145912 Mar 15 '17 at 18:24
You don't get `'Munici\xcc\x81pio'`? That's strange. When you simply print a string it does encoding which can mask the real problem. `repr` can be more definitive. – tdelaney Mar 15 '17 at 18:30
Oh i do. forgot the quotes! My bad. – user3145912 Mar 15 '17 at 18:31
1

`name.decode('utf-8') == u"Município"` should work, but you may need to add `# coding=utf-8` to the top of your script (second line, if you use a shebang). – tdelaney Mar 15 '17 at 19:19
1

Watch out for different Unicode normalisation forms: [Unicode encoding for filesystem in Mac OS X not correct in Python?](http://stackoverflow.com/questions/9757843/unicode-encoding-for-filesystem-in-mac-os-x-not-correct-in-python) – roeland Mar 16 '17 at 01:50
Thank you both! @roeland that link helped! Did not know OSX differently encodes. – user3145912 Mar 16 '17 at 02:03

Unicode string not being detected in data structure

0 Answers0