1

I am creating some files using Python, with unicode text, and saving the content in sqlite3 DB. Later, I fetch these files and make some changes. Problem is that Python doesn't think these two strings are the same, even though they look the same.

Here's an example:

Str1 : "Copa América 2019"
Str2 : "Copa América 2019"

While both the strings look exactly the same, the program doesn't treat them the same.

I'm seeing the same behavior in other languages as well (Korean, Japanese etc.), and chose French example here as it's easier to read for me. Korean / Japanese I matched the characters by how they look.

I also did the comparisons online (for ex. https://www.quickdiff.com/) and found the same result... so it's not Python specific but some other issue.

What operation can I do on the strings so the code sees these two strings as same?

Tried explicit conversions to str / utf-8 encoding individually but none of it helped.

a = "Copa América 2019"
b = "Copa América 2019"
if a == b:
    print 'y'

type(a)
    <type 'str'>
type(b)
    <type 'str'>

I would like the String comparison for these strings to result in True.

Blorgbeard
  • 101,031
  • 48
  • 228
  • 272
sudcha
  • 623
  • 6
  • 18
  • Unicode can encode characters like `é` in different ways. Check `len(a)` and `len(b)`. You could try normalizing both strings: https://stackoverflow.com/questions/16467479/normalizing-unicode – Blorgbeard Jul 22 '19 at 23:59
  • Thank you. Yes, they are different. len(a) = 18; len(b) = 19. What I'm not able to understand is how to solve this problem. I tried b1=b.encode('utf-8') and a1=a.encode('utf-8'), but they are different as well (of course, because they are different characters). – sudcha Jul 23 '19 at 00:01
  • `'Copa Ame\xcc\x81rica 2019'` vs `'Copa Am\xc3\xa9rica 2019'` – Selcuk Jul 23 '19 at 00:02
  • 1
    Update: normalizing seems to work fine: https://repl.it/repls/RepentantRepulsiveHandwritingrecognition – Blorgbeard Jul 23 '19 at 00:03

1 Answers1

1

Full demonstration of @Blorgbeard's answer.

import unicodedata

a = "é"
b = "é"

print(len(a)) # 1
print(len(b)) # 2

print(len(unicodedata.normalize("NFC", a))) # 1
print(len(unicodedata.normalize("NFC", b))) # 1

print(len(unicodedata.normalize("NFD", a))) # 2
print(len(unicodedata.normalize("NFD", b))) # 2

This question may be flagged as a duplicate, Let this stay here just in case.

Rockybilly
  • 2,938
  • 1
  • 13
  • 38
  • Thank you so much! I'm usually good at finding similar questions, but looks like in this case I didn't know what to search for, really. This solves my issue! – sudcha Jul 23 '19 at 00:11