24

In what encoding are the elements of sys.argv, in Python? are they encoded with the sys.getdefaultencoding() encoding?

sys.getdefaultencoding(): Return the name of the current default string encoding used by the Unicode implementation.

PS: As pointed out in some of the answers, sys.stdin.encoding would indeed be a better guess. I would love to see a definitive answer to this question, though, with pointers to solid sources!

PPS: As Wim pointed out, Python 3 solves this issue by putting str objects in sys.argv (if I understand correctly). The question remains open for Python 2.x, though. Under Unix, the LC_CTYPE environment variable seems to be the correct thing to check, no? What should be done with Windows (so that sys.argv elements are correctly interpreted whatever the console)?

Eric O. Lebigot
  • 91,433
  • 48
  • 218
  • 260

7 Answers7

8

I'm guessing that you are asking this because you ran into issue 2128. Note that this has been fixed in Python 3.0.

Wim Coenen
  • 66,094
  • 13
  • 157
  • 251
6

A few observations:

(1) It's certainly not sys.getdefaultencoding.

(2) sys.stdin.encoding appears to be a much better bet.

(3) On Windows, the actual value of sys.stdin.encoding will vary, depending on what software is providing the stdio. IDLE will use the system "ANSI" code page, e.g. cp1252 in most of Western Europe and America and former colonies thereof. However in the Command Prompt window, which emulates MS-DOS more or less, the corresponding old DOS code page (e.g. cp850) will be used by default. This can be changed by using the CHCP (change code page) command.

(4) The documentation for the subprocess module doesn't provide any suggestions on what encoding to use for args and stdout.

(5) One trusts that assert sys.stdin.encoding == sys.stdout.encoding never fails.

John Machin
  • 81,303
  • 11
  • 141
  • 189
  • The observations seem to be correct, I have also observed the same. Do you have any idea of what exactly the sys.getdefaultencoding returns? – Ankit Jaiswal Oct 25 '10 at 09:55
  • "It returns the name of the current default string encoding used by the Unicode implementation." I think it means that Python uses the defaultencoding() in its console. You can override the defaultencoding() by prepending `u'` by the way. Great answer +1 – soulseekah Oct 25 '10 at 11:38
  • 4
    I agree about (2)--I thought of it later. (5) is actually not true: under Unix, `python test.py > test.txt` can for instance have UTF-8 for the stdin encoding and None for the stdout encoding. – Eric O. Lebigot Oct 25 '10 at 15:32
5

I don't know if this helps or not but this is what I get in DOS mode:

C:\Python27>python Lib\codingtest.py нер
['Lib\\codingtest.py', '\xed\xe5\xf0']

C:\Python27>python Lib\codingtest.py hello
['Lib\\codingtest.py', 'hello']

In IDLE:

>>> print "hello"
hello
>>> "hello"
'hello'
>>> "привет"
'\xef\xf0\xe8\xe2\xe5\xf2'
>>> print "привет"
привет
>>> sys.getdefaultencoding()
'ascii'
>>>

What can we deduce from this? I don't know yet... I'll comment in a little bit.

A little bit later: sys.argv is encoded with sys.stdin.encoding and not sys.getdefaultencoding()

soulseekah
  • 8,770
  • 3
  • 53
  • 58
  • \xef is the UNICODE CP1251 Cyrillic representation of SMALL LETTER PE ('п'), thus I'm beginning to believe that `sys.argv` is encoded with `sys.stin.encoding` and not `sys.getdefaultencoding()` – soulseekah Oct 25 '10 at 08:02
4

On Unix systems, it should be in the user's locale, which is (strangely) not tied to sys.getdefaultencoding. See http://docs.python.org/library/locale.html.

In Windows, it'll be in the system ANSI codepage.

(By the way, those elementary school teachers who told you not to end a sentence with a preposition were lying to you.)

Glenn Maynard
  • 55,829
  • 10
  • 121
  • 131
  • Dangling prepositions is something up with which I shall not put. The supposed stricture against the dangling preposition apparently evolved from an observation on style. To wit, the first and last words of a sentence are those which have the most natural impact. Thus it was considered to be stylistically weak for a mere preposition to be placed in such a strategically important location. – Jim Dennis Nov 05 '10 at 21:16
  • @Jim: Style is all well and good, but some people seem to have this silly notion that it's ungrammatical, leading to such goofiness as the title of this question. – Glenn Maynard Nov 05 '10 at 21:28
  • The title of this question seems clear enough though I might have suggested the use of *which* rather than "what." A more precise phrasing might be: "Which encoding is used for processing sys.argv?" The whole issue of text encoding has gotten rather complicated by all these attempts to accommodate both International character sets while preserving some of the simple ASCII string handling. The terminology surrounding the whole affair has become similarly convoluted. – Jim Dennis Nov 06 '10 at 12:13
  • @Jim: The point--which was nothing but an amused aside, of course--was that writing that sentence naturally is perfectly fine: "What encoding is sys.argv in?". "In what encoding" isn't unclear, it's just peculiar and unnatural. – Glenn Maynard Nov 06 '10 at 13:30
  • 1
    For reference: I guess that this answer refers to `locale.getdfaultlocale()[1]` (https://docs.python.org/2/library/locale.html#locale.getdefaultlocale). – Eric O. Lebigot Apr 12 '15 at 08:54
4

"What should be done with Windows (so that sys.argv elements are correctly interpreted whatever the console)?"

For Python 2.x, see this comment on issue2128.

(Note that no encoding is correct for the original sys.argv, because some characters may have been mangled in ways that there is not enough information to undo; for example, if the ANSI codepage cannot represent Greek alpha then it will be mangled to 'a'.)

Daira Hopwood
  • 2,264
  • 22
  • 14
1

As per https://docs.python.org/3/library/sys.html#sys.argv

argv is encoded with sys.getfilesystemencoding() using sys.getfilesystemencodeerrors().

See also https://www.python.org/dev/peps/pep-0383/ which explains the tricky way of how non-UTF8 sequences are encoded within that (UTF-8), when encoding="utf-8" ... by using surrogateescape as error handler.

Of intrest might also be os.fsdecode and os.fsencode.

calestyo
  • 327
  • 2
  • 7
0

sys.getfilesystemencoding() works for me, at least on Windows. On Windows it is actually 'mbcs', and 'utf-8' on *nix.

GPCracker
  • 151
  • 1
  • 4
  • Problem is, Windows has TWO correct codepages. `GetACP()` for GUI programs and `GetOENCP()` for text programs. For many languages those have the same values, but not for all... – Arioch 'The Jul 02 '23 at 13:55