File contain \u00c2\u00a0, convert to characters

Question

I have a JSON file which contains text like this

 .....wax, and voila!\u00c2\u00a0At the moment you can't use our ...

My simple question is how CONVERT (not remove) these \u codes to spaces, apostrophes and e.t.c...?

Input: a text file with .....wax, and voila!\u00c2\u00a0At the moment you can't use our ...

Output: .....wax, and voila!(converted to the line break)At the moment you can't use our ...

Python code

def TEST():
        export= requests.get('https://sample.uk/', auth=('user', 'pass')).text

        with open("TEST.json",'w') as file:
            file.write(export.decode('utf8'))

What I have tried:

• Using .json()
• any different ways of combining .encode().decode() and e.t.c.

Edit 1

When I upload this file to BigQuery I have - Â symbol

Bigger Sample:

{
    "xxxx1": "...You don\u2019t nee...",
    "xxxx2": "...Gu\u00e9rer...",
    "xxxx3": "...boost.\u00a0Sit back an....",
    "xxxx4": "\" \u306f\u3058\u3081\u307e\u3057\u3066\"",
    "xxxx5": "\u00a0\n\u00a0",
    "xxxx6": "It was Christmas Eve babe\u2026",
    "xxxx7": "It\u2019s xxx xxx\u2026"
}

Python code:

import json
import re
import codecs


def load():
    epos_export = r'{"xxxx1": "...You don\u2019t nee...","xxxx2": "...Gu\u00e9rer...","xxxx3": "...boost.\u00a0Sit back an....","xxxx4": "\" \u306f\u3058\u3081\u307e\u3057\u3066\"","xxxx5": "\u00a0\n\u00a0","xxxx6": "It was Christmas Eve babe\u2026","xxxx7": "It\u2019s xxx xxx\u2026"}'
    x = json.loads(re.sub(r"(?i)(?:\\u00[0-9a-f]{2})+", unmangle_utf8, epos_export))

    with open("TEST.json", "w") as file:
        json.dump(x,file)

def unmangle_utf8(match):
    escaped = match.group(0)                   # '\\u00e2\\u0082\\u00ac'
    hexstr = escaped.replace(r'\u00', '')      # 'e282ac'
    buffer = codecs.decode(hexstr, "hex")      # b'\xe2\x82\xac'

    try:
        return buffer.decode('utf8')           # '€'
    except UnicodeDecodeError:
        print("Could not decode buffer: %s" % buffer)



if __name__ == '__main__':
    load()

Answer 1

I have made this crude UTF-8 unmangler, which appears to solve your messed-up encoding situation:

import codecs
import re
import json

def unmangle_utf8(match):
    escaped = match.group(0)                   # '\\u00e2\\u0082\\u00ac'
    hexstr = escaped.replace(r'\u00', '')      # 'e282ac'
    buffer = codecs.decode(hexstr, "hex")      # b'\xe2\x82\xac'

    try:
        return buffer.decode('utf8')           # '€'
    except UnicodeDecodeError:
        print("Could not decode buffer: %s" % buffer)

Usage:

broken_json = '{"some_key": "... \\u00e2\\u0080\\u0099 w\\u0061x, and voila!\\u00c2\\u00a0\\u00c2\\u00a0At the moment you can\'t use our \\u00e2\\u0082\\u00ac ..."}'
print("Broken JSON\n", broken_json)

converted = re.sub(r"(?i)(?:\\u00[0-9a-f]{2})+", unmangle_utf8, broken_json)
print("Fixed JSON\n", converted)

data = json.loads(converted)
print("Parsed data\n", data)
print("Single value\n", data['some_key'])

It uses regex to pick up the hex sequences from your string, converts them to individual bytes and decodes them as UTF-8.

For the sample string above (I've included the 3-byte character € as a test) this prints:

Broken JSON
 {"some_key": "... \u00e2\u0080\u0099 w\u0061x, and voila!\u00c2\u00a0\u00c2\u00a0At the moment you can't use our \u00e2\u0082\u00ac ..."}
Fixed JSON
 {"some_key": "... ’ wax, and voila!  At the moment you can't use our € ..."}
Parsed data
 {'some_key': "... ’ wax, and voila!\xa0\xa0At the moment you can't use our € ..."}
Single value
 ... ’ wax, and voila!  At the moment you can't use our € ...

The \xa0 in the "Parsed data" is caused by the way Python outputs dicts to the console, it still is the actual non-breaking space.

Answer 2

The hacky approach is to remove the outer layer of encoding:

import re
# Assume export is a bytes-like object
export = re.sub(b'\\\u00([89a-f][0-9a-f])', lambda m: bytes.fromhex(m.group(1).decode()), export, flags=re.IGNORECASE)

This matches the escaped UTF-8 bytes and replaces them with actual UTF-8 bytes . Writing the resulting bytes-like object to disk (without further decoding!) should result in a valid UTF-8 JSON file.

Of course this will break if the file contains genuine escaped unicode characters in the UTF-8 range, like \u00e9 for an accented "e".

Answer 3

As you try to write this in a file named TEST.json, I will assume that this string is a part of a larger json string.

Let me use an full example:

js = '''{"a": "and voila!\\u00c2\\u00a0At the moment you can't use our"}'''
print(js)

{"a": "and voila!\u00c2\u00a0At the moment you can't use our"}

I would first load that with json:

x = json.loads(js)
print(x)

{'a': "and voila!Â\xa0At the moment you can't use our"}

Ok, this now looks like an utf-8 string that was wrongly decoded as Latin1. Let us do the reverse operation:

x['a'] = x['a'].encode('latin1').decode('utf8')
print(x)
print(x['a'])

{'a': "and voila!\xa0At the moment you can't use our"}
and voila! At the moment you can't use our

Ok, it is now fine and we can convert it back to a correct json string:

print(json.dumps(x))

{"a": "and voila!\\u00a0At the moment you can\'t use our"}

meaning a correctly encoded NO-BREAK SPACE (U+00A0)

TL/DR: what you should do is:

# load the string as json:
js = json.loads(request)

# identify the string values in the json - you probably know how but I don't...
...

# convert the strings:
js[...] = js[...].encode('latin1').decode('utf8')

# convert back to a json string
request = json.dumps(js)