Python subprocess.run not able to handle large argument string

Question

I need to invoke a powershell script and capture the output as generated from it.

Since I need to capture the output, I chose to use subprocess.run()

Powershell invocation

powershell DeleteResults -resultscsv '1111,2222,3333,4444'

Python(Python 3.5.2 :: Anaconda 4.1.1 (64-bit)) code

command = "powershell DeleteResults -resultscsv '{}'".format(resultscsv)
output = subprocess.run(command, stdout=subprocess.PIPE).stdout.decode('utf-8')

All goes fine as long as the length of command is less than 33K(approx)

However, subprocess.call() throws error when the length exceeds 33K (There is no issue on the powershell side as it works perfectly fine when invoked directly)

ERROR: [WinError 206] The filename or extension is too long
Traceback (most recent call last):
  File "D:\path\to\python\wrapper.py", line 154, in <module>
    output = subprocess.run(command, stdout=subprocess.PIPE).stdout.decode('utf-8')
  File "D:\Anaconda3\lib\subprocess.py", line 693, in run
    with Popen(*popenargs, **kwargs) as process:
  File "D:\Anaconda3\lib\subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "D:\Anaconda3\lib\subprocess.py", line 1224, in _execute_child
    startupinfo)

Any pointer will be great help.

Not sure if relevant - the python script is invoked via Control-M on a windows environment.

--Edit--

Adding this section to add more details in response to answer by Alex.

We don't own the ps script DeleteResults. So, we can't modify it. We just consume it.

As it is done today,

The resultscsv(80K chars) is stored in a results.ini file
A small piece of ps inline code parses .ini file and then invokes DeleteResults. Note: There is powershell call inside the outer powershell invocation(invocation below).
This approach works perfectly fine even if chars >80K.
However, we don't want the inline ini parser to be part of invocation - looks ugly.
So, the idea is to write a python wrapper which will parse .ini file and invoke the powershell


powershell -NoLogo -NonInteractive -Command "Get-Content 'results.ini' | foreach-object -begin {$h=@{}} -process { $k = [regex]::split($_,'='); if(($k[0].compareTo(' ') -ne 0) -and ($k[0].startswith('[') -ne $True)) {$h.Add($k[0], $k[1]) }}; powershell DeleteResults -resultscsv $h.Get_Item('resultscsv')"

So, wondering why the above ps 1-liner not hitting the char length limit ? Is it that the line powershell DeleteResults -resultscsv $h.Get_Item('resultscsv') is NOT actually expanded inline - thereby not hitting the char length limit ?

Suggestion: try [subprocess.Popen()](https://docs.python.org/2/library/subprocess.html) instead. You can definitely read the Python subprocess's "stdout" as an arbitrarily long stream... but "subprocess.run()" probably isn't the best way to do it. Given POpen a try, and post back what you find. — paulsm4, Nov 16 '19 at 06:12

score 2 · Answer 1 · answered Nov 16 '19 at 06:24

2

There is command line string limitation, it's value depends on OS version.

It is not possible to pass large data through command line arguments. Pass a filename instead.

Documentation and workaround https://support.microsoft.com/en-us/help/830473/command-prompt-cmd-exe-command-line-string-limitation

answered Nov 16 '19 at 06:24

rtxndr

872
1
9
20

Thanks for pointing the exact cause. However, I have added a section in the question describing what we do today.I am intrigued to know why the char limit is not breached ? Please have a check. – Soumya Nov 16 '19 at 18:57
I have zero knowledge in powershell scripting, but doesn't it look like it processes file line-by-line, instead of sending whole 80K at once? https://stackoverflow.com/a/33511982/1307996 – rtxndr Nov 16 '19 at 19:41
Are you sure `$h.Get_Item('resultscsv')` is that large? Maybe it's smaller than you think – rtxndr Nov 16 '19 at 19:45
The .ini file actually contains a single line(ignoring the section header line) like so resultscsv=111,222,333... – Soumya Nov 16 '19 at 20:49
$h is the hash which gets populated after reading the ini file. The $h actually contains more than 80K chars data.We use the one liner regularly....we simply want to move away from it.. – Soumya Nov 16 '19 at 20:52
if you replace `powershell DeleteResults -resultscsv` with some application that prints it's argv, does it contain all that 80K? Just to be sure – rtxndr Nov 16 '19 at 20:54
limits are 8k for cmd, and 32k for inner powershell call https://www.reddit.com/r/PowerShell/comments/8ylspm/fun_fact_windows_has_a_command_line_character/ – rtxndr Nov 16 '19 at 20:55
Maybe you can split data in few chunks before calling the script – rtxndr Nov 16 '19 at 21:00

Python subprocess.run not able to handle large argument string

1 Answers1