Python module to shellquote/unshellquote?

Question

Is there anything in the Python standard library that will properly parse/unparse strings for using in shell commands? I'm looking for the python analog to perl's String::ShellQuote::shell_quote:

$ print String::ShellQuote::shell_quote("hello", "stack", "overflow's", "quite", "cool")
hello stack 'overflow'\''s' quite cool

And, even more importantly, something which will work in the reverse direction (take a string and decompose it into a list).

score 33 · Answer 1 · edited Jan 15 '15 at 16:41

33

Looks like

try:  # py3
    from shlex import quote
except ImportError:  # py2
    from pipes import quote

quote("hello stack overflow's quite cool")
>>> '"hello stack overflow\'s quite cool"'

gets me far enough.

edited Jan 15 '15 at 16:41

Stephen Fuhry

12,624
6
56
55

answered Jul 01 '09 at 18:57

YGA

9,546
15
47
50

1

Sure, `subprocess` is swell for actually starting processes in python, but for code generation, this was just what I needed! +1 – SingleNegationElimination Mar 27 '12 at 18:17
In Python 3, this is `shlex.quote`. – asmeurer Jan 12 '14 at 04:26
4

On windows, `subprocess.list2cmdline` is more accurate. `pipes.quote` always uses single quotes, which is not acceptable in Windows command line environment. – Rockallite Feb 24 '14 at 07:54
@asmeurer but pipes isn't deprecated in python3 – Ciasto piekarz May 03 '14 at 04:21
[*"Deprecated since version 2.7"*](https://docs.python.org/2/library/pipes.html#pipes.quote) [*"It is finally exposed publicly in Python 3.3 as the quote function in the shlex module."*](https://docs.python.org/3/library/shlex.html#shlex.quote) – Denilson Sá Maia Aug 05 '14 at 21:31

score 10 · Accepted Answer · answered Aug 05 '14 at 20:19

10

pipes.quote is now shlex.quote in python 3. It is easy enough to use that piece of code.

https://github.com/python/cpython/blob/master/Lib/shlex.py#L281

That version handles zero-length argument correctly.

answered Aug 05 '14 at 20:19

dnozay

23,846
6
82
104

score 8 · Answer 3 · answered Jan 26 '11 at 14:14

8

To unquote, try shlex.split()

answered Jan 26 '11 at 14:14

DomQ

4,184
38
37

score 7 · Answer 4 · answered Dec 10 '09 at 23:07

7

I'm pretty sure that pipes.quote is broken, and should not be used, because it does not handle zero-length arguments correctly:

>>> from pipes import quote
>>> args = ['arg1', '', 'arg3']
>>> print 'mycommand %s' % (' '.join(quote(arg) for arg in args))
mycommand arg1  arg3

I believe the result should be something like

mycommand arg1 '' arg3

answered Dec 10 '09 at 23:07

John Wiseman

3,081
1
22
31

1

fair enough. but then we need a better solution :-) – YGA Dec 14 '09 at 05:23
1

`print 'mycommand %s' % (' '.join(quote(arg) or "''" for arg in args))` ? – Day Aug 23 '12 at 08:58
4

At John's initiative, [this was fixed in Python 2.6.](http://bugs.python.org/issue7476) – Søren Løvborg Dec 14 '12 at 14:54
... but the use of `join` defeats most of the purpose of using `quote` here! – Sylvain Leroux Aug 08 '14 at 17:04
Why do you say that? – John Wiseman Aug 14 '14 at 08:44

score 6 · Answer 5 · answered Jul 31 '11 at 17:20

For shell quoting, this works: I've rigorously tested it on Posix. [I'm assuming that the list2cmdline function supplied by Python works as advertised on Windows]

# shell.py
import os
if os.name == 'nt':
    from subprocess import list2cmdline

    def quote(arg):
        return list2cmdline([arg])[0]
else:
    import re
    _quote_pos = re.compile('(?=[^-0-9a-zA-Z_./\n])')

    def quote(arg):
        r"""
        >>> quote('\t')
        '\\\t'
        >>> quote('foo bar')
        'foo\\ bar'
        """
        # This is the logic emacs uses
        if arg:
            return _quote_pos.sub('\\\\', arg).replace('\n',"'\n'")
        else:
            return "''"

    def list2cmdline(args):
        return ' '.join([ quote(a) for a in args ])

The tests are here, if anyone cares.

Counter example: string to be quoted contains `"\xC3\xA9"`, which is an é in UTF-8, and thus not uncommon in filenames. Code above puts backslashes in front of both characters, which is incorrect. `pipes.quote` will put it in single quotes. — greggo, Jan 26 '15 at 03:16

score 2 · Answer 6 · answered Aug 08 '14 at 17:08

The quotefunction is available for quite some time (Python 2.7?) -- the major drawback is it moved from pipe module to shlex between 3.2 and 3.3.

You have to be prepared to handle both cases while importing that function:

try:
    from shlex import quote
except ImportError:
    from pipes import quote

score 2 · Answer 7 · edited Dec 14 '11 at 17:46

2

The standard library module subprocess has the list2cmdline function which does this, albeit according to Microsoft rules so I am not sure how reliable it works in Unix-like environments for more complicated command lines.

edited Dec 14 '11 at 17:46

sjngm

12,423
14
84
114

answered Jan 10 '11 at 17:18

Alex

21
1

score 0 · Answer 8 · edited Jun 08 '09 at 23:25

0

You should never have to shell quote. The correct way to do a command is to not do shell quoting and instead use subprocess.call or subprocess.Popen, and pass a list of unquoted arguments. This is immune to shell expansion.

i.e.

subprocess.Popen(['echo', '"', '$foo'], shell=False)

If you want to unquote shell quoted data, you can use shlex.shlex like this:

list(shlex.shlex("hello stack 'overflow'\''s' quite cool"))

edited Jun 08 '09 at 23:25

Stephan202

59,965
13
127
133

answered Jun 08 '09 at 23:02

Jerub

41,746
15
73
90

12

What if I need to pass a command (that requires escaping) for ssh to execute once it reaches the other side? – Mike Boers Jun 09 '09 at 04:00
13

This is not a helpful answer (well it answers one half my question, so it's half helpful...). There are are any number of occasions when you need to shell quote -- Mike Boers gives just one great example (in fact, that's the one I'm running into) – YGA Jun 09 '09 at 16:54
actually even worse, the given example breaks: (Pdb) list(shlex.shlex("hello stack 'overflow'\''s' quite cool")) *** Error in argument: '(shlex.shlex("hello stack \'overflow\'\\\'\'s\' quite cool"))' – YGA Jun 09 '09 at 17:09
1

From subprocess documentation: If shell is True, it is recommended to pass args as a string rather than as a sequence. ... This includes, for example, quoting or backslash escaping filenames with spaces in them. - That's just another example where you need quoting. – Seppo Enarvi Oct 29 '12 at 13:43
This answer is not good, you may be generating a shell script to run later and not executing the command directly. – ideasman42 Oct 30 '12 at 03:22
5

My use case for shell quoting is: my Python program is using subprocess.Popen() as suggested, but when it does I'd like it to print something to the console which can be directly cut-and-pasted into a shell, for manual debugging should the command fail. – user23614 Jul 25 '14 at 09:06

Python module to shellquote/unshellquote?

8 Answers8

Linked

Related