1

Python has some good libraries to convert Unicode accent characters to its closest Ascii character, as well as libraries to encode codepoint to its Unicode character.

However, what options are there to check whether a string has unicode codepoint or HTML escape? For example, this string:

Rialta te Venice&#199

Has the &#199, which translates to a latin capital letter C. Is there a python library that detects codepoints/escape within a string and outputs the Unicode equivalent?

Elie
  • 35
  • 1
  • 2
  • 6

1 Answers1

3

It's not quite clear to me what you're asking, but here is my best try:

  1. &#199 is an HTML escape, which you can unescape like so:

    >>> s = 'Rialta te Venice&#199'
>>> import html
>>> s2 = html.unescape(s); s2
'Rialta te VeniceÇ'

  2. As you've said, there are libraries for normalizing/removing accents:

    >>> import unidecode
>>> unidecode.unidecode(s2)
'Rialta te VeniceC'

    You don't really need to check if it has Unicode codepoints, as this function won't change non-accented characters. But you could check anyway using s2.isascii().

So the complete solution is to use unidecode.unidecode(html.unescape(s)).

xjcl
  • 12,848
  • 6
  • 67
  • 89
  • 1
    My question was essentially how to "undo" the ```Ç```, which I mistakenly thought was codepoint (when it is actually HTML escape, as you pointed out). Your answer works beautifully, as ```unescape(s)``` detects where an HTML escape exists - thank you! – Elie Nov 29 '20 at 19:47