Python - yet another encoding issue. Windows 8.1, both latest pythons. pip throws exception

Question

I have installed on my Windows 8.1 machine a clean new version of python. I tried Python 2.7.8 and Python 3.4.

I run python get-pip.py from command prompt - this installed the pip correctly. I had it from here.

Afterwards I run pip install mongo-connector. This downloads mongo connector and throws an exception like this:

Downloading/unpacking mongo-connector
  Downloading mongo-connector-1.3.tar.gz
  Running setup.py (path:c:\users\imon~1\appdata\local\temp\pip_build_èimon\mong
o-connector\setup.py) egg_info for package mongo-connector

Cleaning up...
Exception:
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\pip\basecommand.py", line 122, in main
    status = self.run(options, args)
  File "C:\Python27\lib\site-packages\pip\commands\install.py", line 278, in run

    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundl
e=self.bundle)
  File "C:\Python27\lib\site-packages\pip\req.py", line 1234, in prepare_files
    req_to_install.assert_source_matches_version()
  File "C:\Python27\lib\site-packages\pip\req.py", line 464, in assert_source_ma
tches_version
    % (display_path(self.source_dir), version, self))
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8a in position 55: ordinal
 not in range(128)

Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\pip.exe\__main__.py", line 9, in <module>
  File "C:\Python27\lib\site-packages\pip\__init__.py", line 185, in main
    return command.main(cmd_args)
  File "C:\Python27\lib\site-packages\pip\basecommand.py", line 161, in main
    text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8a in position 70: ordinal
 not in range(128)

In case of Python 3.4 it throws:

Downloading/unpacking mongo-connector
  Downloading mongo-connector-1.3.tar.gz
Cleaning up...
Exception:
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\pip\basecommand.py", line 122, in main
    status = self.run(options, args)
  File "C:\Python34\lib\site-packages\pip\commands\install.py", line 278, in run

    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundl
e=self.bundle)
  File "C:\Python34\lib\site-packages\pip\req.py", line 1229, in prepare_files
    req_to_install.run_egg_info()
  File "C:\Python34\lib\site-packages\pip\req.py", line 292, in run_egg_info
    logger.notify('Running setup.py (path:%s) egg_info for package %s' % (self.s
etup_py, self.name))
  File "C:\Python34\lib\site-packages\pip\log.py", line 102, in notify
    self.log(self.NOTIFY, msg, *args, **kw)
  File "C:\Python34\lib\site-packages\pip\log.py", line 162, in log
    consumer.write(write_content)
  File "C:\Python34\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line
 116, in write
    self.write_and_convert(text)
  File "C:\Python34\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line
 143, in write_and_convert
    self.write_plain_text(text, cursor, len(text))
  File "C:\Python34\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line
 148, in write_plain_text
    self.wrapped.write(text[start:end])
  File "C:\Python34\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0160' in position
70: character maps to <undefined>

Traceback (most recent call last):
  File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python34\Scripts\pip.exe\__main__.py", line 9, in <module>
  File "C:\Python34\lib\site-packages\pip\__init__.py", line 185, in main
    return command.main(cmd_args)
  File "C:\Python34\lib\site-packages\pip\basecommand.py", line 168, in main
    logger.fatal('Storing debug log for failure in %s' % log_file_fn)
  File "C:\Python34\lib\site-packages\pip\log.py", line 111, in fatal
    self.log(self.FATAL, msg, *args, **kw)
  File "C:\Python34\lib\site-packages\pip\log.py", line 162, in log
    consumer.write(write_content)
  File "C:\Python34\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line
 116, in write
    self.write_and_convert(text)
  File "C:\Python34\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line
 140, in write_and_convert
    self.write_plain_text(text, cursor, start)
  File "C:\Python34\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line
 148, in write_plain_text
    self.wrapped.write(text[start:end])
  File "C:\Python34\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0160' in position
42: character maps to <undefined>

====

Running just pip without any parameters throws this:

Traceback (most recent call last):
  File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python34\Scripts\pip.exe\__main__.py", line 9, in <module>
  File "C:\Python34\lib\site-packages\pip\__init__.py", line 177, in main
    cmd_name, cmd_args = parseopts(initial_args)
  File "C:\Python34\lib\site-packages\pip\__init__.py", line 148, in parseopts
    parser.print_help()
  File "C:\Python34\lib\optparse.py", line 1646, in print_help
    file.write(self.format_help())
  File "C:\Python34\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0160' in position
1235: character maps to <undefined>

Running pip in Python 2.7.8 throws this:

Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\pip.exe\__main__.py", line 9, in <module>
  File "C:\Python27\lib\site-packages\pip\__init__.py", line 177, in main
    cmd_name, cmd_args = parseopts(initial_args)
  File "C:\Python27\lib\site-packages\pip\__init__.py", line 148, in parseopts
    parser.print_help()
  File "C:\Python27\lib\optparse.py", line 1670, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
  File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8a in position 1210: ordin
al not in range(128)

This never happened on any other machine I used. I found many similar issues, but none of them addressed what I need to do with my case.

This thread tells me to run chcp 65001. When I do it and run pip afterwards, it throws this:

C:\Python27\Scripts>pip
Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\pip.exe\__main__.py", line 9, in <module>
  File "C:\Python27\lib\site-packages\pip\__init__.py", line 177, in main
    cmd_name, cmd_args = parseopts(initial_args)
  File "C:\Python27\lib\site-packages\pip\__init__.py", line 148, in parseopts
    parser.print_help()
  File "C:\Python27\lib\optparse.py", line 1670, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
LookupError: unknown encoding: cp65001

According to this thread maybe the problem is, that the username is "Šimon" containing an unicode character. However, I do not know, what to do with it. I am not to edit any python files to different encoding, that seems to be just a very difficult and temporary workaround. (windows did the user name for me - another bad win idea, not letting me choose the username for me.)

This is the second post I've seen with the same issue today. Yes, the [umlaut](http://en.wikipedia.org/wiki/Umlaut_%28linguistics%29) in your username is causing the issue. If you can, install these packages via binary. — Anshul Goyal, Aug 23 '14 at 20:32
Does that mean I cannot use python and pip from command line alltogether? How will I run the script later anyway? Is there no other way to deal with this? Can you not send the link to the other issue you've seen? — Ev0oD, Aug 23 '14 at 20:47
Just google for the error messages `UnicodeDecodeError: 'ascii' codec can't decode byte` and `UnicodeEncodeError: 'charmap' codec can't encode character` and you will find a good number of posts around the error. — Anshul Goyal, Aug 23 '14 at 20:56
I've done it multiple times and none of the issue solutions were working for me. I tried changing chcp for multiple of numbers (1250, 1252, 65001) and it did not help. I also wrote about it in my question. — Ev0oD, Aug 24 '14 at 08:05

score 4 · Answer 1 · edited Dec 04 '15 at 20:43

4

because python do not know "cp65001" is alias of "utf-8" you can set this in windows:

set PYTHONIOENCODING=utf-8

edited Dec 04 '15 at 20:43

Scott Stafford

43,764
28
129
177

answered May 18 '15 at 02:55

YRUsoDiao

951
6
10

score 1 · Answer 2 · answered Sep 05 '14 at 14:55

This is rather a workaround, not a solution for the problem. My username in windows was automatically "Šimon". The "Š" was an issue. I found a way to rename the username, but afterwards many things stopped working. Windows registry was full of "Šimon" scattered around the place, had to rename them all at once with a special tool and many links in user folder stopped working. Including %APPDATA% and other similar folders - thus a lot of software stopped working, throwing unexpected errors at startup, including Dropbox, XMind etc.

So it would be good if the problem was fixed at its root, since this workaround is rather difficult to execute correctly and it still is NOT solving the problem.

score 0 · Answer 3 · answered Nov 16 '15 at 20:07

This is a pip bug that is still not properly fixed after two years. See https://github.com/pypa/pip/issues/1291. Setting PYTHONIOENCODING as suggested by @YRUsoDiao will not fix the problem because that env var only affects stdin/stdout/stderr. For now you'll have to change your windows account to use CP65001 (UTF-8) before running "pip install" if the setup.py module contains utf-8 sequences other than pure ASCII.

Python - yet another encoding issue. Windows 8.1, both latest pythons. pip throws exception

3 Answers3