0

tl;dr: Python 3 occasionally splits argv in weird places, unlike Python 2.

The discussion originally arose on the Git mailing list. After brief googling I was unable to find any documentation on the topic.

print_argv.py:

import sys
from pprint import pprint

pprint(sys.argv)

test.py:

import sys
import subprocess

subprocess.check_call([
    'python%s' % sys.argv[1],
    './print_argv.py',
    'similarity index 90%\nrename from file1.txt\nrename to file1-mv.txt\nindex 2bef330..f8fd673 100644\n',
])

Python 2, expected behavior:

$ python3 test.py 2
['./print_argv.py',
 'similarity index 90%\nrename from file1.txt\nrename to file1-mv.txt\nindex 2bef330..f8fd673 100644\n']

Python 3, broken behavior:

$ python3 test.py 3
['./print_argv.py',
 'similarity index 90%\n'
 'rename from file1.txt\n'
 'rename to file1-mv.txt\n'
 'index 2bef330..f8fd673 100644\n']

The same directly via shell. Python 2, expected behavior:

$ python2 print_argv.py 'similarity index 90%\nrename from file1.txt\nrename to file1-mv.txt\nindex 2bef330..f8fd673 100644\n'
['print_argv.py',
 'similarity index 90%\\nrename from file1.txt\\nrename to file1-mv.txt\\nindex 2bef330..f8fd673 100644\\n']

Python 3, very broken behavior:

$ python3 print_argv.py 'similarity index 90%\nrename from file1.txt\nrename to file1-mv.txt\nindex 2bef330..f8fd673 100644\n'
['print_argv.py',
 'similarity index 90%\\nrename from file1.txt\\nrename to '
 'file1-mv.txt\\nindex 2bef330..f8fd673 100644\\n']

Is this argv-splitting behavior a bug?

Pastafarianist
  • 833
  • 11
  • 27
  • It's not splitting- note there isn't a comma in your first Python3 example. The list still has only 2 elements- it's just formatted nicely – mbatchkarov Aug 30 '19 at 15:54
  • Your code in `test.py` is not valid, you can't have a literal newline in a string unless you triple-quote it. Was the line wrapping a copying error? – Barmar Aug 30 '19 at 16:01
  • Since your question is all about newlines, you should't do any line wrapping that isn't in the actual code or output. – Barmar Aug 30 '19 at 16:04
  • Line-wrapping was a copying error. I have corrected it. – Pastafarianist Aug 30 '19 at 16:05

1 Answers1

3

I've figured it out. It's not a bug, it's a peculiarity of pprint(). It splits strings to avoid long lines and abuses the fact that in Python strings split with whitespace are concatenated by the parser.

Also, in my example newlines are not parsed correctly in the shell. Escaping them, I get the exact same behavior as in subprocess.check_call():

$ python2 print_argv.py $'similarity index 90%\nrename from file1.txt\nrename to file1-mv.txt\nindex 2bef330..f8fd673 100644\n'
['print_argv.py',
 'similarity index 90%\nrename from file1.txt\nrename to file1-mv.txt\nindex 2bef330..f8fd673 100644\n']
$ python3 print_argv.py $'similarity index 90%\nrename from file1.txt\nrename to file1-mv.txt\nindex 2bef330..f8fd673 100644\n'
['print_argv.py',
 'similarity index 90%\n'
 'rename from file1.txt\n'
 'rename to file1-mv.txt\n'
 'index 2bef330..f8fd673 100644\n']

An SO question on the same topic: Can I make pprint in python3 not split strings, like in python2?

Pastafarianist
  • 833
  • 11
  • 27