How to replace unicode characters in string with something else python?

Question

I have a string that I got from reading a HTML webpage with bullets that have a symbol like "•" because of the bulleted list. Note that the text is an HTML source from a webpage using Python 2.7's urllib2.read(webaddress).

I know the unicode character for the bullet character as U+2022, but how do I actually replace that unicode character with something else?

I tried doing str.replace("•", "something")

but it does not appear to work... how do I do this?

What is the type of the string, and which version of Python are you using? — Fred Foo, Oct 26 '12 at 20:14
I'm sorry, I'm not going to download a webpage using `urllib2` now. What is the `type`? `str` or `unicode`? — Fred Foo, Oct 26 '12 at 20:16
Have you tried u.encode('ascii', 'replace') and then replacing '?' ? — r_31415, Oct 26 '12 at 20:16
How do I tell what the "type" is? I just assume that whatever text I get from the webpage download would just be a normal string. — Rolando, Oct 26 '12 at 20:17
@Damascusi: `type(str)`. There is, unfortunately,no "normal string" type in Python 2.x; there are two string types. — Fred Foo, Oct 26 '12 at 20:18
I tried print type(varcontainingstring) and the output I got was — Rolando, Oct 26 '12 at 20:19
if your python code contains utf-8 characters, you should use the 'magic comment' `# coding=utf8` in the first or the second line of your code. — Kinjal Dixit, Oct 15 '13 at 12:09

score 82 · Accepted Answer · answered Oct 26 '12 at 20:23

82

Decode the string to Unicode. Assuming it's UTF-8-encoded:
```
str.decode("utf-8")
```
Call the replace method and be sure to pass it a Unicode string as its first argument:
```
str.decode("utf-8").replace(u"\u2022", "*")
```

Encode back to UTF-8, if needed:

str.decode("utf-8").replace(u"\u2022", "*").encode("utf-8")

(Fortunately, Python 3 puts a stop to this mess. Step 3 should really only be performed just prior to I/O. Also, mind you that calling a string str shadows the built-in type str.)

answered Oct 26 '12 at 20:23

Fred Foo

355,277
75
744
836

I get this error: `TypeError: Can't convert 'bytes' object to str implicitly` – Kristada673 May 24 '18 at 05:33
11

could you please elaborate in what way "Python 3 puts a stop to this mess"? How would I do this in Python 3 then? – cryanbhu Apr 09 '19 at 10:48
1

I get: `.decode("utf-8")` -> `AttributeError: 'str' object has no attribute 'decode'. Did you mean: 'encode'?` – Dimitri Kopriwa Jul 03 '23 at 10:19

RParadox · Answer 2 · 2012-10-26T20:36:15.830

15

Encode string as unicode.

>>> special = u"\u2022"
>>> abc = u'ABC•def'
>>> abc.replace(special,'X')
u'ABCXdef'

edited Oct 26 '12 at 20:36

answered Oct 26 '12 at 20:24

RParadox

6,393
4
23
33

what is "special"? I get a nameError: name 'special' is not defined. – Rolando Oct 26 '12 at 20:26
@Rolando Notice 'u' has been prefixed in string, that makes it unicode string. – igauravsehrawat Feb 06 '15 at 06:27

David · Answer 3 · 2012-10-26T20:29:46.153

3

import re
regex = re.compile("u'2022'",re.UNICODE)
newstring = re.sub(regex, something, yourstring, <optional flags>)

edited Oct 26 '12 at 20:29

answered Oct 26 '12 at 20:16

David

6,462
2
25
22

1

It is not an asterisk, it is a bullet (circle shape) – Rolando Oct 26 '12 at 20:18
When trying: re.sub(u'2022', varcontainingstring, ''), it makes the string empty with nothing in it. – Rolando Oct 26 '12 at 20:25
@Damascusi fixed - try it now – David Oct 26 '12 at 20:30
@NullUserException Why is it a bad idea to use a regex to replace fixed strings? – Teodor Anton Sep 28 '18 at 07:59
@AntonTeodor Regex is less efficient than a simple string search and replace. It will work though – NullUserException Oct 03 '18 at 16:17

score 3 · Answer 4 · answered Sep 10 '21 at 01:14

3

Try this one.

you will get the output in a normal string

str.encode().decode('unicode-escape')

and after that, you can perform any replacement.

str.replace('•','something')

answered Sep 10 '21 at 01:14

Rahul Kumar Gupta

151
1
5

This proves useful when the `\u` sequence is actually present as is in the source string. – asu Dec 23 '22 at 15:17

score -1 · Answer 5 · answered Jun 20 '22 at 23:23

str1 = "This is Python\u500cPool"

Encode the string to ASCII and replace all the utf-8 characters with '?'.

str1 = str1.encode("ascii", "replace")

Decode the byte stream to string.

str1 = str1.decode(encoding="utf-8", errors="ignore")

Replace the question mark with the desired character.

str1 = str1.replace("?"," ")

score -2 · Answer 6 · edited Nov 19 '18 at 12:15

-2

Funny the answer is hidden in among the answers.

str.replace("•", "something")

would work if you use the right semantics.

str.replace(u"\u2022","something")

works wonders ;) , thnx to RParadox for the hint.

edited Nov 19 '18 at 12:15

Sahil Mahajan Mj

11,033
8
53
100

answered Nov 19 '18 at 12:00

Mafketel

117
1

Khánh Pluto · Answer 7 · 2021-10-01T07:40:28.680

If you want to remove all \u character. Code below for you

def replace_unicode_character(self, content: str):
    content = content.encode('utf-8')
    if "\\x80" in str(content):
        count_unicode = 0
        i = 0
        while i < len(content):
            if "\\x" in str(content[i:i + 1]):
                if count_unicode % 3 == 0:
                    content = content[:i] + b'\x80\x80\x80' + content[i + 3:]
                i += 2
                count_unicode += 1
            i += 1

        content = content.replace(b'\x80\x80\x80', b'')
    return content.decode('utf-8')

How to replace unicode characters in string with something else python?

7 Answers7

Linked