How to accept command-line args ending in backslash

Question

Here's a barebones Python app that simply prints the command-line arguments passed in:

import sys
if __name__ == "__main__":
    print "Arguments:"
    for i in range(len(sys.argv)):
        print "[%s] = %s" % (i, sys.argv[i])

And here's some sample runs:

python args.py hello world
Arguments:
[0] = args.py
[1] = hello
[2] = world

python args.py "hello world"
Arguments:
[0] = args.py
[1] = hello world

python args.py "hello\world"
Arguments:
[0] = args.py
[1] = hello\world

So far so good. But now when I end any argument with a backslash, Python chokes on it:

python args.py "hello\world\"
Arguments:
[0] = args.py
[1] = hello\world"

python args.py "hello\" world "any cpu"
Arguments:
[0] = args.py
[1] = hello" world any
[2] = cpu

I'm aware of Python's less-than-ideal raw string behavior via the "r" prefix (link), and it seems clear that it's applying the same behavior here.

But in this case, I don't have control of what arguments are passed to me, and I can't enforce that the arguments don't end in a backslash. How can I work around this frustrating limitation?

--

Edit: Thanks to those who pointed out that this behavior isn't specific to Python. It seems to be standard shell behavior (at least on Windows, I don't have a Mac at the moment).

Updated question: How can I accept args ending in a backslash? For example, one of the arguments to my app is a file path. I can't enforce that the client sends it to me without a trailing backslash, or with the backslash escaped. Is this possible in any way?

Answer 1

That's likely the shell treating \ as an escape character, and thus escaping the character. So the shell sends \" as " (because it thinks you are trying to escape the double quote). The solution is to escape the escape character, like so: $ python args.py "hello\world\\".

Answer 2

The Microsoft Parameter Parsing Rules

These are the rules for parsing a command line passed by CreateProcess() to a program written in C/C++:

1. Parameters are always separated by a space or tab (multiple spaces/tabs OK)
2. If the parameter does not contain any spaces, tabs, or double quotes, then all the characters in the parameter are accepted as is (there is no need to enclose the parameter in double quotes).
3. Enclose spaces and tabs in a double quoted part
4. A double quoted part can be anywhere within a parameter
5. 2n backslashes followed by a " produce n backslashes + start/end double quoted part
6. 2n+1 backslashes followed by a " produce n backslashes + a literal quotation mark
7. n backslashes not followed by a quotation mark produce n backslashes
8. If a closing " is followed immediately by another ", the 2nd " is accepted literally and added to the parameter (This is the undocumented rule.)

For a detailed and clear description see http://www.daviddeley.com/autohotkey/parameters/parameters.htm#WINCRULESDOC

Answer 3

The backslash at the end is interpreted as the start of an escape sequence, in this case a literal double quote character. I had a similar problem with handling environment parameters containing a path that sometimes ended with a \ and sometimes didn't.
The solution I came up with was to always insert a space at the end of the path string when calling the executable. My executable then uses the directory path with the slash and a space at the end, which gets ignored. You could possibly trim the path within the program if it causes you issues.

If %SlashPath% = "hello\"

python args.py "%SlashPath% " world "any cpu"
Arguments:
[0] = args.py
[1] = hello\ 
[2] = world
[3] = any cpu

If %SlashPath% = "hello"

python args.py "%SlashPath% " world "any cpu"
Arguments:
[0] = args.py
[1] = hello 
[2] = world
[3] = any cpu

Hopefully this will give you some ideas of how to get around your problem.

Answer 4

The backslash 'escapes' the character following it. This means that the closing quotation marks become a part of the argument, and don't actually terminate the string.

This is the behaviour of the shell you're using (presumably bash or similar), not Python (although you can escape characters within Python strings, too).

The solution is to escape the backslashes:

python args.py "hello\world\\"

Your Python script should then function as you expect it to.

Answer 5

The backslash (\) is escaping the ". That's all. That is how it is supposed to work.

Answer 6

If this is on Windows, then you are not using a standard Windows command prompt (or shell). This must be bash doing this. The Windows command prompt doesn't treat backslash as an escape character (since it's the file path separator).

Extra trivia point: the quoting character in Windows command prompts is caret: ^

Answer 7

When the user passes your function a string "hello\", regardless of what their intention was, they sent the actual string hello", just like if a user passed a filepath like "temp\table" what they have really typed, intentionally or not, is "temp able" (tab in the middle).

This being said, a solution to this problem means that if a user inputs "temp\table" and honestly means "temp able", you are going to process this into "temp\table" and now you've programmatically destroyed the users input.

With this warning in mind, if you still want to do this, you can look for the string representation of these escaped-characters and replace them. As a really easy example, something like this:

def allow_tabs(str_w_tab):
    str_w_tab.replace('\t','\\t')
    print str_w_tab

Now if you want to handle all the other escape characters, you'll have to do something similar for each one. As for being able to do this for the example: "hello\", the user passed you the string hello", and whether they intended to or not, they never closed the double-quote, so this is what your program sees.

Answer 8

On 'nix based systems, this is a fundamental shell limitation, as others have said here. So, just suck it up. That said, it's really not that important because you don't often need backslashes in arguments on those platforms.

On Windows, however, backslashes are of critical value! A path ending in one would explicitly denote a directory vs a file. I have seen the documentation for MS C (see: https://learn.microsoft.com/en-us/previous-versions/17w5ykft(v=vs.85) ), and within the Python source (e.g. in subprocess.list2cmd https://github.com/python/cpython/blob/master/Lib/subprocess.py), explaining this problem with quoting a process argument and have it not able to end with a backslash. So, I forgive the Python developers for keeping the logic the same - but not the MS C ones! This is not a cmd.exe shell issue or a universal limitation for arguments in Windows! (The caret ^ is the equivalent escape character in that natural shell.)

Batch Example (test.bat):

@echo off
echo 0: %0 
echo 1: %1 
echo 2: %2
echo 3: %3 

Now execute it (via cmd.exe):

test.bat -t "C:\test\this path\" -v

Yields:

0: test.bat
1: -t
2: "C:\test\this path\"
3: -v

As you can see - a simple batch file implicitly understands what we want!

But... let's see what happens in Python, when using the standard argparse module (https://docs.python.org/3/library/argparse.html), which is intertwined with sys.argv initial parsing by default:

broken_args.py

import os
import argparse # pip install argparse

parser = argparse.ArgumentParser( epilog="DEMO HELP EPILOG" ) 
parser.add_argument( '-v', '--verbose', default=False, action='store_true', 
                     help='enable verbose output' )
parser.add_argument( '-t', '--target', default=None,
                     help='target directory' )                           
args = parser.parse_args()                       
print( "verbose: %s" % (args.verbose,) )
print( "target: %s" % (os.path.normpath( args.target ),) )

Test that:

python broken_args.py -t "C:\test\this path\" -v

Yields these bad results:

verbose: False
target: C:\test\this path" -v

And so, here's how I solved this. The key "trick" is first fetching the full, raw command line for the process via the Windows api:

fixed_args.py

import sys, os, shlex
import argparse # pip install argparse

IS_WINDOWS = sys.platform.startswith( 'win' )
IS_FROZEN  = getattr( sys, 'frozen', False )
    
class CustomArgumentParser( argparse.ArgumentParser ):
    if IS_WINDOWS:
        # override
        def parse_args( self ):
            def rawCommandLine():
                from ctypes.wintypes import LPWSTR
                from ctypes import windll
                Kernel32 = windll.Kernel32
                GetCommandLineW = Kernel32.GetCommandLineW
                GetCommandLineW.argtypes = ()
                GetCommandLineW.restype  = LPWSTR
                return GetCommandLineW()                            
            NIX_PATH_SEP = '/'                
            commandLine = rawCommandLine().replace( os.sep, NIX_PATH_SEP )
            skipArgCount = 1 if IS_FROZEN else 2
            args = shlex.split( commandLine )[skipArgCount:]        
            return argparse.ArgumentParser.parse_args( self, args )
 
parser = CustomArgumentParser( epilog="DEMO HELP EPILOG" ) 
parser.add_argument( '-v', '--verbose', default=False, action='store_true', 
                     help='enable verbose output' )
parser.add_argument( '-t', '--target', default=None,
                     help='target directory' )                           
args = parser.parse_args()                       
print( "verbose: %s" % (args.verbose,) )
print( "target: %s" % (os.path.normpath( args.target ),) )

Confirm the fix:

python fixed_args.py -t "C:\test\this path\" -v

Yields these good results:

verbose: True
target: C:\test\this path