UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 8-8: Non-BMP character not supported in Tk

Question

Keep in mind I just started Python today so I'm terrible.

Hello, I am programming a bot for Discord, and I'm running into a problem when I run it. I'm trying to get it online and I've been getting the same error. I don't know where the error is coming from though. Can someone please help?

Here is my code so far:

import discord

from discord.ext.commands import bot

from discord.ext import commands

import asyncio

import time

Client = discord.Client()

client = commands.Bot(command_prefix = "~")

@client.event
async def on_ready():

    print("I'm up on some BOOF!" + client.user.id)
    print("I am the" + client.user.name)

@client.event
async def on_message(message):

    if message.content == "Boof":
        await client.send_message(message.channel, ":b:")


client.run("<redacted>")

The error I am getting:

Ignoring exception in on_ready
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/discord/client.py", line 307, in _run_event
    yield from getattr(self, event)(*args, **kwargs)
  File "/Users/johnathanhelsel/Documents/Boof Bot/BoofBot.py", line 13, in on_ready
    print("I am the" + client.user.name)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/run.py", line 362, in write
    return self.shell.write(s, self.tags)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/rpc.py", line 604, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/rpc.py", line 216, in remotecall
    return self.asyncreturn(seq)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/rpc.py", line 247, in asyncreturn
    return self.decoderesponse(response)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/rpc.py", line 267, in decoderesponse
    raise what
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 8-8: Non-BMP character not supported in Tk

I am absolutely stuck! I have tried every solution posted and nothing worked. Please help if you can!

PS, I already changed the token, don't even try.

Thanks- Johnathan

Answer 1

There's likely no problem with your code, but rather a problem with IDLE. If you run the same code from the console, or from a different IDE like PyCharm or Spyder, it will work just fine.

As of 3.7, IDLE internally handles all strings as UCS-2, an obsolete encoding that can only handle the first 65536 characters in Unicode.¹

So, what happens when you try to pass it a string that has an "astral character", a character outside that range? Well, it depends on various details, but usually it's a UnicodeEncodeError like this one.

Which means that if you have a client whose name includes, say, a character from the supplemental CJK range—which many Chinese last names do—or an Emoji, this line will raise that exception:

print("I am the" + client.user.name)

… as IDLE tries to take the print output and convert it to display on is tkinter output.

---

To work around the problem, you need to do something to explicitly remove or substitute any astral characters before printing them:

def clean_remove(s):
    return ''.join(c for c in s if ord(c) < 65536)
def clean_replace_question(s):
    return ''.join(c if ord(c) < 65536 else '?' for c in s)
def clean_replace_unicode(s):
    return ''.join(c if ord(c) < 65536 else '\ufffd' for c in s)
print("I am the" + clean_whichever(client.user.name))

You can write this a bit more efficiently, but it's unlikely to matter (displaying text in tkinter windows is much slower anyway), and hopefully this way is easy to understand.

You can also monkeypatch Python with a custom print function that does this filtering before passing through to the builtin, or a custom replacement for sys.stdout that filters things for you.

Or you can just not run your code in IDLE.

---

_{1. UCS-2 has been replaced by UTF-16, many years back in fact, but IDLE can't be updated until someone can fix the string-handling code interfacing with every supported GUI platform (mainly Windows) and every supported Tcl/Tk version, which hasn't happened yet.}

Answer 2

Your problem is that you're trying to run your code inside the IDLE editor. The IDLE editor uses UCS-2 encoding so when you try to print your user name string with weird unicode characters and emoji, the IDLE window can't display it.

I suggest debugging using the ascii() of the strings, that will print the python representation of the unicode instead of the actual codepoint (that can't be printed depending on the destination window):

print(ascii("I am the" + client.user.name))