21

I have this Python3 code to attempt to read and print from a utf-8 encoded file:

f = open('mybook.txt', encoding='utf-8')
for line in f:
    print(line)

When I build using Sublime Text 3 I get the following error:

UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 18: ordinal not in range(128)

However, it works file when I just execute my code in the terminal with python3.

My build configuration is

{
"cmd": ["/usr/local/bin/python3", "$file"]
, "selector": "source.python"
, "file_regex": "file \"(...*?)\", line ([0-9]+)"
}

If I change it to:

f = open('mybook.txt', encoding='utf-8')
for line in f:
    print(line.encode('utf-8'))

Then it does print the utf-8 encoded byte string (I think that's what's happening).

b'Hello\n'
b'\xc2\xab\xe2\x80\xa2\n'
b'Goodbye'

But I also don't know how to go from this to printing the unicode characters on the screen...

Also, if I try changing this env variable as per A python program fails to execute in sublime text 3, but success in bash it still doesn't fix it.

Community
  • 1
  • 1
allstar
  • 1,155
  • 4
  • 13
  • 29
  • Try `print(line.decode())` – lycuid Sep 19 '16 at 15:01
  • 2
    It seems like SublimeText output device only handles ASCII... check SublimeText options to see if you can do something about that. – Bakuriu Sep 19 '16 at 15:03
  • 1
    decode is not a method in string. I have updated above with another bit of info. – allstar Sep 19 '16 at 15:52
  • I still believe this is a Sublime Text problem: it is telling the python interpreter to use ASCII to display the output. To solve this problem you should find the configurations for SublimeText's terminal and choose the encoding there, or properly set the environment variables. – Bakuriu Sep 19 '16 at 16:24
  • I think so too - just haven't been able to find it yet so I'm hoping there are some sublime text ninjas around :) – allstar Sep 19 '16 at 16:26
  • Does the same error occur if you duplicate the ST3 Python build file and just modify it to run python3 instead of python? – OdatNurd Sep 19 '16 at 21:00
  • @OdatNurd I wanted to do this but I can't seem to find the default sublime-build files for ST3 in the packages folder. I have ST2 as well, and I find them for ST2 - the Python one is almost identical to what I had. I tried modifying that one for Python 3 and no change. – allstar Sep 19 '16 at 23:03
  • Sublime ninja here :) I have a UTF-8-encoded text file and a `.py` file remarkably similar to yours (except I used the more Pythonic `with` context manager) both on my Dropbox, so I'm using the exact same input. On OS X I'm getting the UnicodeEncodeError, but on Linux (in a VM) it works fine. Haven't booted up any Windows VMs to check yet. Let me see if I can figure out what's different about Linux. Are you running Linux or OS X? – MattDMo Sep 19 '16 at 23:07
  • @MattDMo OS X Yosemite 10.10.5 – allstar Sep 19 '16 at 23:11
  • Does the file run OK on the command line? Mine does... – MattDMo Sep 19 '16 at 23:15
  • Yep, runs fine on command line. – allstar Sep 19 '16 at 23:17
  • OK, I figured it out. Writing up an answer now. – MattDMo Sep 19 '16 at 23:19

3 Answers3

42

The answer was actually in the question linked in your question - PYTHONIOENCODING needs to be set to "utf-8". However, since OS X is silly and doesn't pick up on environment variables set in Terminal or via .bashrc or similar files, this won't work in the way indicated in the answer to the other question. Instead, you need to pass that environment variable to Sublime.

Luckily, ST3 build systems (I don't know about ST2) have the "env" option. This is a dictionary of keys and values passed to exec.py, which is responsible for running build systems without the "target" option set. As discussed in our comments above, I indicated that your sample program worked fine on a UTF-8-encoded text file containing non-ASCII characters when run with ST3 (Build 3122) on Linux, but not with the same version run on OS X. All that was necessary to get it to run was to change the build system to enclude this line:

"env": {"PYTHONIOENCODING": "utf8"},

I saved the build system, hit ⌘B, and the program ran fine.

BTW, if you'd like to read exec.py, or Packages/Python/Python.sublime-build, or any other file packed up in a .sublime-package archive, install PackageResourceViewer via Package Control. Use the "Open Resource" option in the Command Palette to pick individual files, or "Extract Package" (both are preceded by "PackageResourceViewer:", or prv using fuzzy search) to extract an entire package to your Packages folder, which is accessed by selecting Sublime Text → Preferences → Browse Packages… (just Preferences → Browse Packages… on other operating systems). It is located on your hard drive in the following location:

  • Linux: ~/.config/sublime-text-3/Packages
  • OS X: ~/Library/Application Support/Sublime Text 3/Packages
  • Windows Regular Install: C:\Users\YourUserName\AppData\Roaming\Sublime Text 3\Packages
  • Windows Portable Install: InstallationFolder\Sublime Text 3\Data\Packages

Once files are saved to your Packages folder (if you just view them via the "Open Resource" option and close without changing or saving them, they won't be), they will override the identically-named file contained within the .sublime-package archive. So, for instance, if you want to edit the default Python.sublime-build file in the Python package, your changes will be saved as Packages/Python/Python.sublime-build, and when you choose the Python build system from the menu, it will only use your version.

MattDMo
  • 100,794
  • 21
  • 241
  • 231
12

It works, thanks, the complete build system script for Sublime Text 3

Tool -> Build System -> New Build System 

{
    "shell_cmd": "python \"$file\"",
    "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
    "selector": "source.python",
    "env": {"PYTHONIOENCODING": "utf8"}
}
Sunding Wei
  • 1,803
  • 17
  • 12
2

Note if you use venv with S3 you will need to update the Python + Virtualenv.sublime-build resource file.

{
    "target": "virtualenv_exec",
    "shell_cmd": "python -u \"$file\"",
    "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
    "selector": "source.python",
    "env": {"PYTHONIOENCODING": "utf-8"}
}
Jongware
  • 22,200
  • 8
  • 54
  • 100
Matt Moore
  • 61
  • 5