2

I wrote a simple shell recently. This problem came up while I implemented the pipe.

I knew the maximum length of command line argument in Ubuntu is 2097152. (By this)

#define MAX_CMD_LEN 2097152

But I wanted to know is there a maximum number of pipe commands? For example: (the number of n)

ps -aux |　grep "a.out" | awk '{print $5}' | top | ... | cat
1       |  2            | 3                | 4   | ... | n
Community
  • 1
  • 1
Kir Chou
  • 2,980
  • 1
  • 36
  • 48
  • 1
    I do not *think* there will be any limit on the maximum limit as the pipe itself does not impose any resource constraint. If you are talking about pipe buffer, then the answer is *yes*, we are constrained by the memory limit. – toddlermenot Oct 12 '14 at 06:07
  • 1
    Sure, but that's just heap, same as anything else. – Charles Duffy Oct 12 '14 at 06:08
  • Your `MAX_CMD_LEN` limit is system specific: it is different on my Debian/Linux running 3.17 kernel... – Basile Starynkevitch Oct 12 '14 at 06:09
  • 1
    Going off on a bit of a tangent, by the way -- if you're looking to write a script that generates very long pipelines, doing so recursively can be a source of fun and profit. See for instance my answer to http://stackoverflow.com/questions/9898939/handling-long-edit-lists-in-xmlstarlet – Charles Duffy Oct 12 '14 at 06:11
  • 1
    The limits are going to bigger than you want to construct, even if you use a program to write the pipeline. – Jonathan Leffler Oct 12 '14 at 06:55

2 Answers2

5

No, there is not such a limit. For a single process table entry, ARG_MAX applies (to combined environment and argv length). However, a pipeline spans separate process table entries, and each piece connects only to the process before it (via stdin) and the process after it (via stdout). There's no single place the entire pipeline needs to be stored or tracked as a unit inside the operating system itself (as opposed to the constructing shell).

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Technically, I believe the limiting factor is the number of processes. It can be limited with [setrlimit(2)](http://man7.org/linux/man-pages/man2/setrlimit.2.html) `RLIMIT_NPROC` – Basile Starynkevitch Oct 12 '14 at 06:07
  • Right -- but if (as is the case) the maximum number of processes you can have in a pipeline matches the maximum number of processes you could have individually with no connection between them, I'm calling it entirely fair to say that there is no limit on pipeline length. – Charles Duffy Oct 12 '14 at 06:09
1

There are not strict limits on the length of the pipeline (that is the number of commands composing it). However, the maximum number of processes can be limited with setrlimit(2) RLIMIT_NPROC and each system has some absolute limit. Likewise, the number of file descriptors can be limited with RLIMIT_NOFILE

Also, your shell will call pipe(2), and that syscall can fail (and this is practically limiting).

BTW, the maximal command line should better not be a hardwired limit (it can be changed somehow). The limitation is when execve(2) fails. And you could use sysconf(_SC_ARG_MAX) -see sysconf(3)- to query something related.

At last proc(5) could be used to query some related thresholds, e.g. using /proc/sys/fs/pipe-max-size, /proc/sys/kernel/core_pipe_limit, etc.

The bottom line is that you should not build in any wired-in limits in your shell. You should just manage resources and handle failure (of syscalls(2) and of standard functions like malloc(3) ...). Avoid defining arbitrary builtin limits like your MAX_CMD_LEN (BTW, the exact figure is not the same on my Debian/Sid/x86-64 running a kernel 3.17 which I did compile myself).

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
  • 1
    To be sure, a `pipe()` call can fail, or a `fork()` can fail. However, such failures are because no more FDs are available, or no more process table entries, or some other resource also used for things other than pipeline construction -- not because a limit _on pipeline length as such_ has been reached. – Charles Duffy Oct 12 '14 at 06:15
  • Actually, I knew `MAX_CMD_LEN` depends on kernel version. I've used a dynamic way to replace. Thanks your answer, I think some tough parts will help me not the present but the future. – Kir Chou Oct 12 '14 at 06:36
  • 1
    I believe even that your `MAX_CMD_LEN` depend upon the system state. In particular, it might become lower if the environment has additional variables (e.g. set with bash `export` builtin) – Basile Starynkevitch Oct 12 '14 at 07:29