0

MCVE

In terminal while loading IPython/Jupyter Notebook as .json:

$ python
Python 3.5.2 | packaged by conda-forge | (default, Jul 26 2016, 01:32:08)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> import io
>>> with io.open('Untitled.ipynb','r',encoding='utf-8') as sf:
...     data = json.load(sf)    # error on this line
...     print(data)
...
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' in position 189292: ordinal not in range(128)

How to make this work?

Also:

>>> print(sys.getdefaultencoding())
utf-8
>>> print(sys.stdout.encoding)
ANSI_X3.4-1968    # I assume this is the main reason why it does not work

What have I tried?

  1. export PYTHONIOENCODING=UTF-8 does not work.
  2. import importlib; importlib.reload(sys); sys.setdefaultencoding("UTF-8") does not work.
  3. sys.stdout = codecs.getwriter('utf-8')(sys.stdout.buffer, 'strict') gives TypeError: write() argument must be str, not bytes

Related: 1, 2, 3

alex
  • 10,900
  • 15
  • 70
  • 100
  • If your error is on the line `data = json.load(sf)` then it has nothing to do with the output encoding, that's a different matter. `Untitled.ipynb` simply isn't utf-8 in this case. – mata Nov 23 '17 at 11:24
  • Are you sure the error isn't on the next line? – mata Nov 23 '17 at 11:36
  • @mata I am sure of the line, because `print` isn't even there in the original function (and also I just double checked that in debugger). About `Untitled.ipynb` encoding - I checked it in quite a few different ways ([1](https://stackoverflow.com/questions/11018967/how-can-i-be-sure-of-the-file-encoding), [2](https://nlp.fi.muni.cz/projects/chared/), ...), so I'm ~90% sure it actually is utf-8. – alex Nov 23 '17 at 12:29
  • Try `data = open('Untitled.ipynb','r',encoding='utf-8').read()`. If that fails with an encodig error, then it's 100% not utf-8. – mata Nov 23 '17 at 13:22
  • @mata In terminal your code gives no error, but then `print(data)` fails. More interesting here is the fact that your code fails in [pudb](https://pypi.python.org/pypi/pudb). It is the same error. – alex Nov 23 '17 at 15:58

0 Answers0