5

I have a hex-string made from a unicode string with that function:

def toHex(s):
    res = ""
    for c in s:
        res += "%02X" % ord(c) #at least 2 hex digits, can be more
    return res

hex_str = toHex(u"...")

This returns a string like this one:

"80547CFB4EBA5DF15B585728"

That's a sequence of 6 chinese symbols.
But 

u"Knödel"

converts to

"4B6EF664656C"

What I need now is a function to convert this back to the original unicode. The chinese symbols seem to have a 2-byte representation while the second example has 1-byte representations for all characters. So I can't just use unichr() for each 1- or 2-byte block.

I've already tried

binascii.unhexlify(hex_str)

but this seems to convert byte-by-byte and returns a string, not unicode. I've also tried

binascii.unhexlify(hex_str).decode(...)

with different formats. Never got the original unicode string.

Thank you a lot in advance!

Robert
  • 227
  • 1
  • 3
  • 9

3 Answers3

4

This seems to work just fine:

binascii.unhexlify(binascii.hexlify(u"Knödel".encode('utf-8'))).decode('utf-8')

Comes back to the original object. You can do the same for the chinese text if it's encoded properly, however ord(x) already destroys the text you started from. You'll need to encode it first and only then treat like a string of bytes.

viraptor
  • 33,322
  • 10
  • 107
  • 191
  • Thanks for your answer! My problem is, that I get the Hex-String from a 3rd party software and thus I have no influence to the encoding... – Robert Jul 21 '11 at 09:01
  • Ask them what encoding is it then. Otherwise, you won't be able to decode it correctly... – viraptor Jul 21 '11 at 09:29
  • Also, it seems the text might be utf-16. `binascii.unhexlify("80547CFB4EBA5DF15B585728").decode('utf-16')` seems to return something asian (can't verify what though) – viraptor Jul 21 '11 at 09:31
  • Thanks alot! It really seems to be UTF-16 with Big Endian byte order! `(codecs.BOM_UTF16_BE + binascii.unhexlify("80547CFB4EBA5DF15B585728")).decode('utf-16')` seems to do the right thing. Thanks again for all your answers! – Robert Jul 21 '11 at 10:21
1

Can't be done. Using %02X loses too much information. You should be using something like UTF-8 first and converting that, instead of inventing a broken encoding.

>>> u"Knödel".encode('utf-8').encode('hex')
'4b6ec3b664656c'
Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • Thanks for your answer! My problem is, that I get the Hex-String from a 3rd party software and thus I have no influence to the encoding... So you mean, that there is no way to surely reconstruct what "80547CFB4EBA5DF15B585728" means? – Robert Jul 21 '11 at 09:01
  • That is correct. Unless you agree on a consistent protocol, which brute-force encoding as the code gives is not. – Ignacio Vazquez-Abrams Jul 21 '11 at 09:25
  • @Robert: if it's made with a known encoding, you can get the text e.g. UTF-8 or UTF-16. – JeremyP Jul 21 '11 at 10:33
0

When I was working with Unicode in a VB app a while ago the first 1 or 2 digits would be removed if they were a "0". Meaning "&H00A2" would automatically be converted to "&HA2", I just created a small function to check the length of the string and if it was less than 4 chars add the missing 0's. I'm not sure if this is what's happening to you, but I thought I would give bit of information as something to be aware of.

Tony
  • 9
  • 1