Passing a unicode string (Japanese char) as a commandline argument

Question

I will just ask if anyone knows using python 2.7, how will i be able to pass a unicode string (e.g. Japanese filename) as a commandline argument of my python script. Once this filename is passed to the function/method correctly, some file processing will be done (e.g. metadata extraction / retrieval) by some engine (a DLL, which is identified to have unicode support). I've tried the following but unfortunately, python crashes:

Passing of the filename to the method that will process the file:

processingMethod(unicode(argv[1], "utf-8", errors="ignore").encode("utf-8"))

on the method, this is how I decode the string passed:

unicode(file_path).decode("utf-8")

Any feedback will be of great help. Thanks a lot!

correction: processingMethod(unicode(argv[1], "utf-8", errors="ignore").encode("utf-8")) — jaysonpryde, Feb 24 '12 at 05:36
I've made the edit for you. In the future, remember that you can always edit your own posts. — Adam Mihalcin, Feb 24 '12 at 05:43
Command line arguments are generated by the OS. Which OS are you using? — Mark Ransom, Feb 24 '12 at 06:03
Calling `.decode()` on a `unicode` is almost certainly wrong. — Ignacio Vazquez-Abrams, Feb 24 '12 at 06:27

score 0 · Accepted Answer · edited May 23 '17 at 12:27

unicode(argv[1], "utf-8"

Unfortunately, the encoding used by the Windows command prompt is never(*) UTF-8. It's a locale-specific encoding, so you can only pass Japanese characters in an argument on a Japanese Windows install.

If you want to be able to read Unicode characters in arguments reliably from Python 2, you will have to sniff to detect you're running on Windows, and use the Windows-specific APIs to read args instead of the standard C library ones that rely on the locale encoding. See this answer for an example of doing it with ctypes.

(*: well, unless you do chcp 65001, but that causes lots of other stuff to fall over so is best avoided.)

Passing a unicode string (Japanese char) as a commandline argument

1 Answers1