31

How long can be a command line that can be passed to sh -c ''? (in bash and in bourne shell)

The limit is much lower than that from the OS (in case of modern Linux).

For example:

$ /bin/true $(seq 1 100000)
$ /bin/sh -c "/bin/true $(seq 1 100000)"
bash: /bin/sh: Argument list too long

And how could I circumvent this problem?

Update

I want to note that getconf can't help here (because that is not a system limit):

$ seq 1 100000 | wc -c
588895
$ getconf ARG_MAX
2097152

Update #2

Now I've understood what is the point here. That is not a shell limit, that is a system limit but for the length of each argument, not for the entire arglist.

$ /bin/true $(seq 1 100000)
$ /bin/true "$(seq 1 100000)"
bash: /bin/true: Argument list too long

Thank you, CodeGnome, for the explanation.

Todd A. Jacobs
  • 81,402
  • 15
  • 141
  • 199
Igor Chubin
  • 61,765
  • 13
  • 122
  • 144
  • Related: [Does "argument list too long" restriction apply to shell builtins?](https://stackoverflow.com/questions/47443380/does-argument-list-too-long-restriction-apply-to-shell-builtins) – codeforester Nov 22 '17 at 21:18
  • Also see [Argument list too long error for rm, cp, mv commands](https://stackoverflow.com/q/11289551/608639) – jww Nov 07 '19 at 15:36

4 Answers4

30

TL;DR

A single argument must be shorter than MAX_ARG_STRLEN.

Analysis

According to this link:

And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072). This might become relevant if you generate a long call like "sh -c 'generated with long arguments'".

This is exactly the "problem" identified by the OP. While the number of arguments allowed may be quite large (see getconf ARG_MAX), when you pass a quoted command to /bin/sh the shell interprets the quoted command as a single string. In the OP's example, it is this single string that exceeds the MAX_ARG_STRLEN limit, not the length of the expanded argument list.

Implementation Specific

Argument limits are implementation specific. However, this Linux Journal article suggests several ways to work around them, including increasing system limits. This may not be directly applicable to the OP, but it nonetheless useful in the general case.

Do Something Else

The OP's issue isn't actually a real problem. The question is imposing an arbitrary constraint that doesn't solve a real-world problem.

You can work around this easily enough by using loops. For example, with Bash 4:

for i in {1..100000}; do /bin/sh -c "/bin/true $i"; done

works just fine. It will certainly be slow, since you're spawning a process on each pass through the loop, but it certainly gets around the command-line limit you're experiencing.

Describe Your Real Problem

If a loop doesn't resolve your issue, please update the question to describe the problem you're actually trying to solve using really long argument lists. Exploring arbitrary line-length limits is an academic exercise, and not on-topic for Stack Overflow.

Todd A. Jacobs
  • 81,402
  • 15
  • 141
  • 199
  • That is not a system limit! that is shell limit! – Igor Chubin Jul 13 '12 at 18:00
  • 2
    I don't have `MAX_ARG_STRLEN ` on a system with `2.6.31` – Dennis Williamson Jul 13 '12 at 19:18
  • @DennisWilliamson: I also (3.2.0-rc7-686-pae); but anyway, argument length is the point. The only question is how to find it. – Igor Chubin Jul 13 '12 at 19:26
  • @CodeGnome: There are systems without `MAX_ARG_STRLEN` (my and the system of Dennis). So we need to find another way to find this value. – Igor Chubin Jul 13 '12 at 19:31
  • @DennisWilliamson execve(2) says "On kernel 2.6.23 and later...the limit per string is 32 pages (the kernel constant MAX_ARG_STRLEN), and the maximum number of strings is 0x7FFFFFFF." So it seems likely that your system does have it, but it's a kernel constant. – Todd A. Jacobs Jul 13 '12 at 19:34
  • Ah, it's just not visible to `getconf` and it's not in `limits.h` (but it is in the `man` page). Thanks. – Dennis Williamson Jul 13 '12 at 19:36
  • @DennisWilliamson Anytime. Just for the record, I also grepped the kernel headers and found that it's defined in `/usr/src/linux-headers-3.0.0-14/include/linux/binfmts.h` if one ever needs to redefine it. – Todd A. Jacobs Jul 13 '12 at 19:41
  • Or if you don't have the source installed and you just need to look it up: `/usr/include/linux/binfmts.h` – Dennis Williamson Jul 13 '12 at 19:45
  • @ToddA.Jacobs If you are correct, this would be another silly Linux deviation from the POSIX standard. POSIX only knows one limitation: The initial stack of a process must not exceed `ARG_MAX`. – schily Nov 09 '18 at 10:21
6

I don't get that error message. My secret? Single quotes:

/bin/sh -c '/bin/true $(seq 1 100000)'

If I use double quotes, I get that error with every shell:

$ /bin/sh -c "/bin/true $(seq 1 100000)"
-bash: /bin/sh: Argument list too long
$ /bin/bash -c "/bin/true $(seq 1 100000)"
-bash: /bin/bash: Argument list too long
$ /bin/ksh -c "/bin/true $(seq 1 100000)"
-bash: /bin/ksh: Argument list too long
$ /bin/zsh -c "/bin/true $(seq 1 100000)"
-bash: /bin/zsh: Argument list too long

The argument list gets expanded in the current shell when double quotes are used as evidenced by the fact that Bash is the one issuing the error "-bash: ..." regardless of the shell being used to run the command. On my system sh is Dash, by the way.

This holds true even for other "host" shells:

$ dash
$ /bin/bash -c '/bin/true $(seq 1 100000)'
$ /bin/bash -c "/bin/true $(seq 1 100000)"
dash: /bin/bash: Argument list too long

Patient: Doctor, it hurts when I do this."
Doctor: Don't do that.

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
  • 1
    Dennis, single/double quotes is not a point. I used `$(seq 1 100000)` here only for demonstration purposes; it doesn't matter how you produce this long string; of course I need double quotes here because I need very long string. The main question is how long can be the argument of `sh -c ''` – Igor Chubin Jul 13 '12 at 19:02
  • @IgorChubin: The answer is that if it matters, you're doing it wrong. On my system, by the way, it's [131071](http://stackoverflow.com/a/11475732/26428). – Dennis Williamson Jul 13 '12 at 19:12
  • @IgorChubin: See "Do Something Else" [here](http://stackoverflow.com/a/11475732/26428). – Dennis Williamson Jul 13 '12 at 19:19
  • What a good idea :) but anyway, I suppose that CodeGnome have found the right answer. The real point is the length of an argument. – Igor Chubin Jul 13 '12 at 19:24
  • Just for completeness: `/bin/bash -c '/bin/true "$(seq 1 100000)"'` breaks too because of how the shell tokenizes strings. +1 for solving for Y in this X/Y problem, though--I think it's useful to have a published solution for related issues. – Todd A. Jacobs Jul 13 '12 at 20:34
1

Make a file with #!/bin/sh as the first line, then the put rest of your command on subsequent lines? :)

More seriously, you can also read commands from STDIN using the -s option, so you can generate your long command line and pipe it in to /bin/sh -s

dannysauer
  • 3,793
  • 1
  • 23
  • 30
  • You can do this even without `-s`; yes I know that; probably it is the best solution except situations when you need to use shell stdin for something else – Igor Chubin Jul 13 '12 at 18:01
  • I think CodeGnome has found the solution. See his answer. – Igor Chubin Jul 13 '12 at 19:32
  • Sort of - except that there's still a maximum command-line length, and the "for $(command that generates lots of args); do; done" structure is one command line which can go over that length. Piping the commands in on STDIN (whether with `-s` or with `for arg in $( – dannysauer Jul 17 '12 at 19:07
0

on my os it's the max length obtained by dichotomy

/bin/sh -c "/bin/true $(perl -e 'print"a"x131061')"

so it gives 131071

but there is no reason to have a line as long ; if it's due to a large number of arguments, "$@" can be used instead; for example

/bin/sh -c 'command "$@"' -- arg1 arg2 .. argn
Nahuel Fouilleul
  • 18,726
  • 2
  • 31
  • 36