29

I want my Python script to be able to read Unicode command line arguments in Windows. But it appears that sys.argv is a string encoded in some local encoding, rather than Unicode. How can I read the command line in full Unicode?

Example code: argv.py

import sys

first_arg = sys.argv[1]
print first_arg
print type(first_arg)
print first_arg.encode("hex")
print open(first_arg)

On my PC set up for Japanese code page, I get:

C:\temp>argv.py "PC・ソフト申請書08.09.24.doc"
PC・ソフト申請書08.09.24.doc
<type 'str'>
50438145835c83748367905c90bf8f9130382e30392e32342e646f63
<open file 'PC・ソフト申請書08.09.24.doc', mode 'r' at 0x00917D90>

That's Shift-JIS encoded I believe, and it "works" for that filename. But it breaks for filenames with characters that aren't in the Shift-JIS character set—the final "open" call fails:

C:\temp>argv.py Jörgen.txt
Jorgen.txt
<type 'str'>
4a6f7267656e2e747874
Traceback (most recent call last):
  File "C:\temp\argv.py", line 7,
in <module>
    print open(first_arg)
IOError: [Errno 2] No such file or directory: 'Jorgen.txt'

Note—I'm talking about Python 2.x, not Python 3.0. I've found that Python 3.0 gives sys.argv as proper Unicode. But it's a bit early yet to transition to Python 3.0 (due to lack of 3rd party library support).

Update:

A few answers have said I should decode according to whatever the sys.argv is encoded in. The problem with that is that it's not full Unicode, so some characters are not representable.

Here's the use case that gives me grief: I have enabled drag-and-drop of files onto .py files in Windows Explorer. I have file names with all sorts of characters, including some not in the system default code page. My Python script doesn't get the right Unicode filenames passed to it via sys.argv in all cases, when the characters aren't representable in the current code page encoding.

There is certainly some Windows API to read the command line with full Unicode (and Python 3.0 does it). I assume the Python 2.x interpreter is not using it.

Community
  • 1
  • 1
Craig McQueen
  • 41,871
  • 30
  • 130
  • 181
  • Does Japanese Windows use raster fonts on the console by default? This would likely limit it to displaying characters in the Windows-932 code page. See http://support.microsoft.com/kb/Q99795 (this is a separate issue to reading the args, but may have some bearing) – McDowell May 11 '09 at 08:28
  • Check this question asked here on Stack Overflow, it should provide the answer to your question: http://stackoverflow.com/questions/477061/how-to-read-unicode-input-and-compare-unicode-strings-in-python – AlbertoPL May 11 '09 at 05:52
  • Yes, appears to be an exact dup. – Dana the Sane May 11 '09 at 05:56
  • That question and its answers talk about raw_input() (even though the question mentions "command line" once). I'm interested in the command line, e.g. sys.argv. – Craig McQueen May 11 '09 at 06:02
  • Well essentially, you would loop on sys.argv like this: for arg in sys.argv: print arg.decode("utf-8") I used print, but you'd do whatever you needed to. You'd also pick the correct encoding you need. – AlbertoPL May 11 '09 at 06:15

4 Answers4

30

Here is a solution that is just what I'm looking for, making a call to the Windows GetCommandLineArgvW function:
Get sys.argv with Unicode characters under Windows (from ActiveState)

But I've made several changes, to simplify its usage and better handle certain uses. Here is what I use:

win32_unicode_argv.py

"""
win32_unicode_argv.py

Importing this will replace sys.argv with a full Unicode form.
Windows only.

From this site, with adaptations:
      http://code.activestate.com/recipes/572200/

Usage: simply import this module into a script. sys.argv is changed to
be a list of Unicode strings.
"""


import sys

def win32_unicode_argv():
    """Uses shell32.GetCommandLineArgvW to get sys.argv as a list of Unicode
    strings.

    Versions 2.x of Python don't support Unicode in sys.argv on
    Windows, with the underlying Windows API instead replacing multi-byte
    characters with '?'.
    """

    from ctypes import POINTER, byref, cdll, c_int, windll
    from ctypes.wintypes import LPCWSTR, LPWSTR

    GetCommandLineW = cdll.kernel32.GetCommandLineW
    GetCommandLineW.argtypes = []
    GetCommandLineW.restype = LPCWSTR

    CommandLineToArgvW = windll.shell32.CommandLineToArgvW
    CommandLineToArgvW.argtypes = [LPCWSTR, POINTER(c_int)]
    CommandLineToArgvW.restype = POINTER(LPWSTR)

    cmd = GetCommandLineW()
    argc = c_int(0)
    argv = CommandLineToArgvW(cmd, byref(argc))
    if argc.value > 0:
        # Remove Python executable and commands if present
        start = argc.value - len(sys.argv)
        return [argv[i] for i in
                xrange(start, argc.value)]

sys.argv = win32_unicode_argv()

Now, the way I use it is simply to do:

import sys
import win32_unicode_argv

and from then on, sys.argv is a list of Unicode strings. The Python optparse module seems happy to parse it, which is great.

Craig McQueen
  • 41,871
  • 30
  • 130
  • 181
  • Yes, that will work. Just get rid of the ".encode('utf-8')" at the end – Ryan Ginstrom May 11 '09 at 07:30
  • This piece of code does not work for me when I drag and drop file to py file. However, this code works when I type the file name in command prompt. And I wrote a C++ program to call GetCommandLineW, the program can show the file name correctly if I drag and drop the file to the program. – Franz Wong May 01 '12 at 12:17
  • @franziga: What does "not work" mean specifically? It is necessary to [enable drag-and-drop onto Python files](http://stackoverflow.com/q/142844/60075). It's been a while (and different company) since I've done this, but I think I must have enabled long filenames. – Craig McQueen May 01 '12 at 22:54
  • @CraigMcQueen I didn't enable that. But my python program is still able to accept drag and drop file. My program just accepts the file names as parameter. Then it will show the file names in hexadecimal form. I found that some characters become 0x3f ('?'). – Franz Wong May 10 '12 at 02:03
  • @franziga, I suggest you ask that as a separate question on StackOverflow, and include sample code that demonstrates the problem you're having. – Craig McQueen May 10 '12 at 04:57
  • It fails if a paramater ends with `\"`, in that case this code appends `"` to the parameter. Can you fix it? – Smit Johnth Apr 16 '15 at 21:40
  • @SmitJohnth: What should it do instead? Can you provide an example of such a command line, and what you expect to see in `argv`? – Craig McQueen Apr 16 '15 at 23:08
  • @CraigMcQueen parameter `"1\"`, the code shows it as `1"`. Backslash doesn't escape `"` on windows, does it? – Smit Johnth Apr 17 '15 at 05:11
  • @CraigMcQueen http://blogs.msdn.com/b/oldnewthing/archive/2010/09/17/10063629.aspx How to fix it? AFAIR `\` is not the windows escape character. – Smit Johnth Apr 17 '15 at 15:46
  • @CraigMcQueen unfortunately it doesn't work with PyScripter and other programs which patch sys.argv. Workarounds and monkey patches have their limitations. – Smit Johnth Apr 23 '15 at 23:54
  • do I understand corectly that the code above makes sense only for Python 2? I tried it with Python 3 and got an error about `xrange()`; when I changed it to `range()`, the returned value is not unicode. – Vasily A Apr 23 '16 at 04:47
  • @VasilyA: That's correct. Python 3 already reads the command line as Unicode strings, so there is no need for this code. – Craig McQueen Apr 24 '16 at 07:30
  • 2
    I have integrated this code into the development version of my `win-unicode-console` package: https://github.com/Drekin/win-unicode-console. – user87690 Jun 14 '16 at 09:10
  • @user87690 does it change Python behavior globally? =/ – anatoly techtonik Apr 01 '17 at 11:06
  • @anatolytechtonik: Depends on how you apply it. You may enable it in sitecustomize, which has the global effect and is the intended usage. – user87690 Apr 02 '17 at 09:41
  • @user87690, ouch. It is an anti-pattern.People will develop code and debug it on their systems with this global hook installed and will have a lot of problems trying to debug it in production. – anatoly techtonik Apr 09 '17 at 07:12
  • @anatolytechtonik: It's not a feature package, it's an external fix to the Python interpreter itself. Fortunately it's not needed in Python 3.6. Ideally it should be transparent to Python. Compare it to extensions like pyreadline. Also you may disable it when your testing interactis with it. – user87690 Apr 09 '17 at 13:32
  • I used this code to fix unicode based cmd line arguments used with PyPy – Robert_Jordan Dec 16 '21 at 13:33
  • @CraigMcQueen PyPy3 does not seem to read the cmd line as Unicode, so your sample helped me to fix that. – Robert_Jordan Dec 16 '21 at 13:40
12

Dealing with encodings is very confusing.

I believe if your inputing data via the commandline it will encode the data as whatever your system encoding is and is not unicode. (Even copy/paste should do this)

So it should be correct to decode into unicode using the system encoding:

import sys

first_arg = sys.argv[1]
print first_arg
print type(first_arg)

first_arg_unicode = first_arg.decode(sys.getfilesystemencoding())
print first_arg_unicode
print type(first_arg_unicode)

f = codecs.open(first_arg_unicode, 'r', 'utf-8')
unicode_text = f.read()
print type(unicode_text)
print unicode_text.encode(sys.getfilesystemencoding())

running the following Will output: Prompt> python myargv.py "PC・ソフト申請書08.09.24.txt"

PC・ソフト申請書08.09.24.txt
<type 'str'>
<type 'unicode'>
PC・ソフト申請書08.09.24.txt
<type 'unicode'>
?日本語

Where the "PC・ソフト申請書08.09.24.txt" contained the text, "日本語". (I encoded the file as utf8 using windows notepad, I'm a little stumped as to why there's a '?' in the begining when printing. Something to do with how notepad saves utf8?)

The strings 'decode' method or the unicode() builtin can be used to convert an encoding into unicode.

unicode_str = utf8_str.decode('utf8')
unicode_str = unicode(utf8_str, 'utf8')

Also, if your dealing with encoded files you may want to use the codecs.open() function in place of the built-in open(). It allows you to define the encoding of the file, and will then use the given encoding to transparently decode the content to unicode.

So when you call content = codecs.open("myfile.txt", "r", "utf8").read() content will be in unicode.

codecs.open: http://docs.python.org/library/codecs.html?#codecs.open

If I'm miss-understanding something please let me know.

If you haven't already I recommend reading Joel's article on unicode and encoding: http://www.joelonsoftware.com/articles/Unicode.html

monkut
  • 42,176
  • 24
  • 124
  • 155
2

Try this:

import sys
print repr(sys.argv[1].decode('UTF-8'))

Maybe you have to substitute CP437 or CP1252 for UTF-8. You should be able to infer the proper encoding name from the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage\OEMCP

pts
  • 80,836
  • 20
  • 110
  • 183
0

The command line might be in Windows encoding. Try decoding the arguments into unicode objects:

args = [unicode(x, "iso-8859-9") for x in sys.argv]
a paid nerd
  • 30,702
  • 30
  • 134
  • 179