Replacing Unicode Characters with actual symbols

Question

string = "At Donald Trump<U+2019>s Properties, a Showcase for a Brand and a President-Elect"

I want to get rid of the <U + 2019> and replace it with '. Is there a way to do this in python?

Edit : I also have instances of <U + 2014>, <U + 201C> etc. Looking for something which can replace all of this with appropriate values

JosefZ · Answer 1 · 2022-03-01T17:42:48.787

Here's my solution for all code points denoted as U+0000 through U+10FFFF ("U+" followed by the code point value in hexadecimal, which is prepended with leading zeros to a minimum of four digits):

import re
def UniToChar(unicode_notation):
    return chr(int(re.findall(r'<U\+([a-hA-H0-9]{4,5})>',unicode_notation)[0],16))

xx= '''
At Donald<U+2019>s <U+2016>Elect<U+2016> in <U+2017>2019<U+2017>
<U+00C0> la Donald<U+2019>s friend <U+1F986>. <U+1F929><U+1F92A><U+1F601>
'''
for x in xx.split('\n'):
    abc =  re.findall(r'<U\+[a-hA-H0-9]{4,5}>',x)
    if len(abc) > 0:
        for uniid in set(abc): x=x.replace(uniid, UniToChar(uniid))
    
    print(repr(x).strip("'"))

Output: 71307293.py

At Donald’s ‖Elect‖ in ‗2019‗
À la Donald’s friend .

In fact, private range from U+100000 to U+10FFFD (Plane 16) isn't detected using above simplified regex… Improved code follows:

import re
def UniToChar(unicode_notation):
    aux = int(re.findall(r'<U\+([a-hA-H0-9]{4,6})>',unicode_notation)[0],16)
    # circumvent the "ValueError: chr() arg not in range(0x110000)"
    if aux <= 0x10FFFD:
        return chr(aux)
    else:
        return chr(0xFFFD) # Replacement Character

xx= '''
At Donald<U+2019>s <U+2016>Elect<U+2016> in <U+2017>2019<U+2017>
<U+00C0> la Donald<U+2019>s friend <U+1F986>. <U+1F929><U+1F92A><U+1F601>
Unassigned: <U+05ff>; out of Unicode range: <U+110000>.
'''
for x in xx.split('\n'):
    abc =  re.findall(r'<U\+[a-hA-H0-9]{4,6}>',x)
    if len(abc) > 0:
        for uniid in set(abc): x=x.replace(uniid, UniToChar(uniid))
    
    print(repr(x).strip("'"))

Output: 71307293.py

At Donald’s ‖Elect‖ in ‗2019‗
À la Donald’s friend . 
Unassigned: \u05ff; out of Unicode range: �.

Thanks for replying, I will implement this solution and will get back to you in a couple of hours — Aditya Verma, Mar 01 '22 at 17:20

Mark Tolonen · Accepted Answer · 2022-03-02T00:31:23.057

Replace them all at once with re.sub:

import re

string = "testing<U+2019> <U+2014> <U+201C>testing<U+1F603>"

result = re.sub(r'<U\+([0-9a-fA-F]{4,6})>', lambda x: chr(int(x.group(1),16)), string)
print(result)

Output:

testing’ — “testing

The regular expression matches <U+hhhh> where hhhh can be 4-6 hexadecimal characters. Note that Unicode defines code points from U+0000 to U+10FFFF so this accounts for that. The lambda replacement function converts the string hhhh to an integer using base 16 and then converts that number to a Unicode character.

PW1990 · Answer 3 · 2022-03-01T11:25:58.810

0

You can replace using .replace()

print(string.replace('<U+2019>', "'"))

Or if your year changes, you can use re. But make it more attractive than mine.

import re

string = "At Donald Trump<U+2019>s Properties, a Showcase for a Brand and a President-Elect"

rep = re.search('[<][U][+]\d{4}[>]', string).group()

print(string.replace(rep, "'"))

edited Mar 01 '22 at 11:25

answered Mar 01 '22 at 10:56

PW1990

393
1
8

Thanks for replying, I have slightly edited the question for more understanding – – Aditya Verma Mar 01 '22 at 11:09
I have edited my answer, still use replace but incorporate regex. – PW1990 Mar 01 '22 at 11:26

score 0 · Answer 4 · answered Mar 01 '22 at 10:57

0

I guess this solves the problem if its just one or two of these characters.

>>> string = "At Donald Trump<U+2019>s Properties, a Showcase for a Brand and a President-Elect"
>>> string.replace("<U+2019>","'")
"At Donald Trump's Properties, a Showcase for a Brand and a President-Elect"

If there are many if these substitutions to be done, consider using 'map()' method.
Source: Removing \u2018 and \u2019 character

answered Mar 01 '22 at 10:57

CrYbAbY

92
1
1
11

Thanks for replying, I have slightly edited the question for more understanding – Aditya Verma Mar 01 '22 at 11:08

yassine ben · Answer 5 · 2022-03-01T12:49:13.737

what version of python are u using?

I edited my answer so it can bee used with multiple code point in the same string

well u need to convert the unicode's code point that is between < >, to unicode char

I used regex to get the unicode's code point and then convert it to the corresponding uniode char

import re

string = "At Donald Trump<U+2019>s Properties, a Showcase for a Brand and a President<U+2014>Elect"

repbool = re.search('[<][U][+]\d{4}[>]', string)

while repbool:
  rep = re.search('[<][U][+]\d{4}[>]', string).group()
  
  string=string.replace(rep, chr(int(rep[1:-1][2:], 16)))
 
  repbool = re.search('[<][U][+]\d{4}[>]', string)
  

print(string)

Replacing Unicode Characters with actual symbols

5 Answers5