0

Error:

OSError: [Errno 7] Argument list too long

History:

Running a script on pvpython (vtk); however on older distributions pvpython < 5.0, the matplotlib modules are obsolete, thus making the use of that module not possible. To overcome this, another .py is used and the arguments passed in the terminal using subprocess as shown below and run on python but since the information passed is large, the above error is met.

The problematic code is:

import subprocess
command = ("python illustrations.py %s %s %s %s %s %s %s %s %s %s" % (str(post_processing), str(width), str(height), str(len(new_overall_lines)), str(reset_scale), str(str_rose_angle), str(str_damage), str(fname), str(fname1), str("ax=None")))
subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)

argv 6 and 7 are extremely lengthy (882770, 879326) characters each, as they are lists.

An alternate solution could be to pass the list into a temp file and load those in to python module. But I am trying to save on write and load time.

Thus am looking for a way to pass large arguments in terminal or how to pass arguments two environments, in this case pvpython and python.

ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
Aly Abdelaziz
  • 292
  • 4
  • 24
  • 5
    i would store that data in a file and pass the file names – Paul H Apr 17 '19 at 15:52
  • 1
    Would it not be an option to upgrade pvpython? – Sayse Apr 17 '19 at 15:55
  • Ignore shell=True and check you `getconf ARG_MAX` arg_max val – BetaDev Apr 17 '19 at 15:55
  • 1
    If you had only one large list you could make the Python program you try to run read from `sys.stdin` and pipe the list in. But if you have multiple large lists I don't think there is any way around saving them to temporary files and passing the paths of those files. – Graipher Apr 17 '19 at 15:57
  • 1
    Save to file using `pickle` or `json`, then read. Just a few lines of code and won't take much time either. Passing bible-sized lists as command line args is probably not a good idea. – Jussi Nurminen Apr 17 '19 at 15:58
  • @PaulH as mentioned, that is what I intended to do. Just wanted a much more robust way, rather then rewriting files. – Aly Abdelaziz Apr 17 '19 at 18:33
  • @Sayse the package 'paraview-pvpython' is system based, shipped with the OS. And is a downgraded version on Linux Mint 17. – Aly Abdelaziz Apr 17 '19 at 18:34

1 Answers1

0

An immediately visible possible cause is that you are not using proper quoting and escaping according to your system shell's syntax when constructing the command line. So if your argument's string representation has spaces or special shell characters, it will be misinterpreted as many arguments (or worse).

So, don't use shell=True (which is not recommended for use specifically for this reason!) and invoke your program via a proper argument list:

command = ["python", "illustrations.py"] +
          [str(v) for v in post_processing, width, height,
                            len(new_overall_lines), reset_scale,
                            str_rose_angle, str_damage,
                            fname, fname1] +
          ["ax=None"]

subprocess.Popen(command, stdout=subprocess.PIPE)

Besides, and since command line passing is OS-specific (e.g in Windows, your array will be combined into a single line under the hood anyway because that's how CreateProcess requires the command line to be passed) and individual command line arguments are also subject to length limits, a better solution is to pass arbitrarily long data by means other than the command line.

There are the following ways to pass information to a child process:

  • Command line
  • files (since filesystem is a shared resource, with due protection considerations from other processes)
  • IPC (pipes incl. standard streams, shared memory, sockets etc.)
  • things inherited via fork():
    • memory snapshot (only applies if the child is running the same program)
    • descriptors (standard streams also fall here; note that Python intentionally marks file descriptors other than standard streams non-inheritable by default)
    • environment variables

As you can see, command line parameters are far from being the only way.

ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152