Can the first argument to setpgid() be a session leader or a group leader?

Question

From The Linux Programming Interface

#include <unistd.h>
int setpgid(pid_t  pid , pid_t  pgid );

The pid argument may not specify a process that is a session leader. Violation of this rule results in the error EPERM .

1. Why can't pid be a session leader?

2. Can pid be a group leader and why?

   If yes, after the call to setpgid(), in which group will the other processes originally in the group originally led by process pid be:

   • the original group whose gid is pid which now has lost process pid and has no group leader, or
   • the new group pgid which process pid is changed to?

   I suspect the first might contradict to what the book described for setsid():

   The restriction against a process group leader being able to call setsid() is necessary because, without it, the process group leader would be able to place itself in another (new) session, while other members of the process group remained in the original session. (A new process group would not be created, since, by definition, the process group leader’s process group ID is already the same as its process ID.) This would violate the strict two-level hierarchy of sessions and process groups, whereby all members of a process group must be part of the same session.

Thanks.

Some related questions Can a leader of a process session or group leave for another existing session or group?

Answer 1

A session leader is always a process group leader. If you were to move it to another process group, it would no longer be the process group leader, which violates the preceding rule.

Answer 2

This behavior is specified by POSIX (which means it applies to everything calling itself "Unix", not just Linux). http://pubs.opengroup.org/onlinepubs/9699919799/functions/setpgid.html says:

ERRORS
    [EPERM] The process indicated by the pid argument is a session leader.

The spec doesn't say why this rule exists, but I believe the rationale stated by dbush is correct: a session leader must always be a process group leader; if it could move into another process group, it would cease to be a process group leader, violating the invariant.

However, a process that is only a process group leader, not a session leader, may put itself into another process group (ceasing to be a process group leader) and may then take itself back out of the process group and become a process group leader again. Job control shells actually have to do this under some circumstances: note the bit at the bottom of the RATIONALE section of the spec

One non-obvious use of setpgid() is to allow a job control shell to return itself to its original process group (the one in effect when the job control shell was executed). A job control shell does this before returning control back to its parent when it is terminating or suspending itself as a way of restoring its job control "state" back to what its parent would expect.

POSIX doesn't explain why the job control shell would have changed its process group in the first place, but the GNU C Library manual section on implementing job control fills in the gap:

When a shell program that normally performs job control is started, it has to be careful in case it has been invoked from another shell that is already doing its own job control.

A subshell that runs interactively has to ensure that it has been placed in the foreground by its parent shell before it can enable job control itself.

[...]

Once the subshell has been placed into the foreground by its parent shell, it can enable its own job control. It does this by calling setpgid to put itself into its own process group, and then calling tcsetpgrp to place this process group into the foreground.

And then, of course, it has to undo that again if the subshell is suspended (e.g. Bash has a suspend builtin that does this).

Also notice that a job control shell does not establish itself as a session leader, even if it's not a subshell of anything. The session is initialized by whatever program is responsible for setting up the controlling terminal; that program is typically the parent and pre-exec identity of the outermost shell running in the terminal. For instance, xterm calls setsid after opening a pseudoterminal and forking, but before exec-ing the program that will run in that terminal window.