How to get Python to recognize German symbols, like umlaut?

Question

I'm reading my dataframe from a CSV file, using pd.read_csv()

The \x9f should be an umlaut:

'Heiner Dr\x9fke "Weil, Gotshal & Manges"'

I tried to no avail:

person1.encode('utf-8')

UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 9: ordinal not in range(128)

TRIED

I get this when i use macroman person1.decode('macroman')
Out[511]:
u'Heiner Dr\xfcke "Weil, Gotshal & Manges"'

However, when I print person1.decode('macroman') does print out the umlaut. How do I capture this into a string?

person1.decode("cp1251")
Out[512]:
u'Heiner Dr\u045fke "Weil, Gotshal & Manges"'

have you tried using https://docs.python.org/2/library/codecs.html? — Rafael Barros, Jun 26 '14 at 22:22
no it is macroman encoding ... bad ... not sure how he got that encoded like that — Joran Beasley, Jun 26 '14 at 22:33
read http://www.gossamer-threads.com/lists/python/python/760326 — Rachel Gallen, Jun 27 '14 at 01:47

Joran Beasley · Accepted Answer · 2014-06-27T02:49:58.247

4

somehow you are encoded to macroman ... you shouldnt be

>>> print 'Heiner Dr\x9fke "Weil, Gotshal & Mages"'.decode("macroman")
Heiner Drüke "Weil, Gotshal & Mages"

this will decode it to unicode that python understands ...

if you want to encode it for a google search

'Heiner Dr\x9fke "Weil, Gotshal & Mages"'.decode("macroman").encode('ascii', 'xmlcharrefreplace')

should work fine

edited Jun 27 '14 at 02:49

answered Jun 26 '14 at 22:31

Joran Beasley

I get this when i use macroman person1.decode('macroman') Out[511]: u'Heiner Dr\xfcke "Weil, Gotshal & Manges"' – user3314418 Jun 27 '14 at 00:56
You have to `print person1.decode('macroman')` to see the character in Python 2.X. `u'\xfc'` is the Unicode hex escape character for `ü`. – Mark Tolonen Jun 27 '14 at 01:36
is there a way to capture the 'ü' for search? I'm trying to use that string to search on google, but I can only use 'Dr\xfcke' – user3314418 Jun 27 '14 at 01:45
see update answer that should give you a path forward ... you did not specify that criteria however in your initial question... but this method should work fine – Joran Beasley Jun 27 '14 at 02:50
cause I googled umlaut \x9f and found it to be that encoding ... it encodes it to ascii using xml character replacement on any non-ascii characters ... in otherwords it just works ;P – Joran Beasley Jun 27 '14 at 03:56

Rachel Gallen · Answer 2 · 2018-07-08T23:13:08.117

1

u = u"profileDir_(\u00fc)" (u umlaut) according to this reference

edited Jul 08 '18 at 23:13

answered Jun 27 '14 at 01:52

Rachel Gallen

I dont think this really answers his question ... his string is encoded as macRoman ... when properly decoded he does get his desired unicode result – Joran Beasley Jun 27 '14 at 02:52

2 Answers2