2

I came across some bash (v5.1.16) behavior today that surprised me:

~ > export TEST=`python -c "print('a'*131067)"`  # This works.
~ > export TEST=`python -c "print('a'*131067)"`  # This does not.
-bash: /usr/bin/python: Argument list too long
~ > export TEST=`python -c "print('a'*131067)"`  # This works.
~ > export TEST=`python -c "print('a'*131067)"`  # This does not.
-bash: /usr/bin/python: Argument list too long

Note how the first line succeeds, and the second does not. The behavior repeats reliably: setting TEST works, then it doesn’t, then it does, then it doesn’t...

This is on Gentoo Linux 5.18.5. Interestingly, 131066 works always; on Mac it’s 259832 that works, and one more fails.

Related perhaps: Difference between single and double quotes in Bash

What’s going on here?

Addendum: Thanks @dave-thompson-085 for your answer below.

With some more digging I found these two related discussions: What is the maximum size of a Linux environment variable value? and How to get around the Linux "Too Many Arguments" limit. Also, reading the docs for execve() helps, particulalry the section Limits on size of arguments and environment.

Jens
  • 8,423
  • 9
  • 58
  • 78
  • I can't reproduce it on my platform (Cygwin, bash 4.4.12), and I don't see any quote-related problem in your code. You basically set a variable to an extremely long string, and when the shell starts Python, it needs to place the string into the environment of that child process. Perhaps there is a bug in the operating system itself, related to updating an already huge environment with another value of the same size. Do you see the same behaviour, when starting the Python process from, i.e., zsh, or from a C-program? – user1934428 Sep 09 '22 at 05:50
  • Another interesting experiment would be to let the environment aside: Start a fresh bash and do the same, but replace `export` by `local`. – user1934428 Sep 09 '22 at 05:52
  • @user1934428: cygwin is not actually Unix, although it tries hard to be Unix-like, and this is an OS (kernel) issue – dave_thompson_085 Sep 09 '22 at 06:39
  • @dave_thompson_085 : Correct. But if the OP compares Linux and MacOS, I think I can throw in Windoze as well. And, while I **suspected** that it is an OS issue, I was not sure, and if it would be a problem in the shell, it might have occured in Cygwin as well. – user1934428 Sep 09 '22 at 06:43

1 Answers1

2

Let's look step by step.

dthomps@virt8:~$ export TEST=$(python -c "print 'a'*131067")
dthomps@virt8:~$ echo ${#TEST}
131067

TEST is set to a string with length just short of 128k, and marked for export

dthomps@virt8:~$ echo foo
foo
dthomps@virt8:~$ /bin/echo foo
-bash: /bin/echo: Argument list too long

echo is a shell builtin and works, but /bin/echo is a program and trying to run a program fails because the 128k-length environment value can't be passed to the child process

dthomps@virt8:~$ export TEST=$(python -c "print 'a'*131067")
-bash: /usr/bin/python: Argument list too long
dthomps@virt8:~$ echo ${#TEST}
0

python is also a program and can't run, so it has no output and TEST is now set to empty

dthomps@virt8:~$ /bin/echo foo
foo
dthomps@virt8:~$ export TEST=$(python -c "print 'a'*131067")
dthomps@virt8:~$ echo ${#TEST}
131067

Running any progam -- /bin/echo or python -- with TEST="" works, and now TEST is again set to a long value.

Lather, rinse, repeat.

dave_thompson_085
  • 34,712
  • 6
  • 50
  • 70
  • Any idea why the error message refers to the **argument** list of python? Do environment and the argument vector for a new process share a common space? – user1934428 Sep 09 '22 at 06:45
  • 1
    @user1934428 Yes (at least in unix and Linux), so the more space is taken by environment variables, the less there is for arguments. The error message can be misleading, since it only mentions arguments. `getconf ARG_MAX` will show you the space available for the environment and arguments, and if you have the GNU version of xargs, `xargs --show-limits --no-run-if-empty – Gordon Davisson Sep 09 '22 at 06:50
  • @GordonDavisson : Interestingly, the Cygwin `getconf` shows me, that there **is** a limit for ARG_MAX to be 32000. Nevertheless I was able to create a variable `TEST` with 888888 characters and put it into the environment, where it stayed intact (`printenv TEST | wc -c`). I also can expand it on the command line (`echo $TEST | wc -c`). I wonder what that 32000-byte limit reported by Cygwin's `getconf` really means. – user1934428 Sep 09 '22 at 07:14
  • Thanks @dave-thompson-085! I added a few more references to the original question above. – Jens Sep 09 '22 at 10:27