ssh port forwarding ("ssh -fNL") doesn't work via expect spawn to automatically provide password

Question

I know that to do port forwarding, the command is ssh -L. I also use other options to decorate it. So for example, a final full command may look like this ssh -fCNL *:10000:127.0.0.1:10001 127.0.0.1. And everything just works after entering password.

Then, because there is not only one port need to be forwarded, I decide to leave the job to shell script and use expect(tcl) to provide passwords(all the same).

Although without a deep understanding of expect, I managed to write the code with the help of Internet. The script succeeds spawning ssh and provides correct password. But I end up finding there is no such process when I try to check using ps -ef | grep ssh and netstat -anp | grep 10000.

I give -v option to ssh and the output seems to be fine.

So where is the problem? I have searched through Internet but most of questions are not about port forwarding. I'm not sure whether it is proper to use expect while I just want to let script automatically provide password.

Here the script.

#!/bin/sh

# Port Forwarding

# set -x

## function definition
connection ()
{
    ps -ef | grep -v grep | grep ssh | grep $1 | grep $2 > /dev/null
    if [ $? -eq 0 ] ; then
        echo "forward $1 -> $2 done"
        exit 0
    fi

    # ssh-keygen -f "$HOME/.ssh/known_hosts" -R "127.0.0.1"

    /usr/bin/expect << EOF
set timeout 30
spawn /usr/bin/ssh -v -fCNL *:$1:127.0.0.1:$2 127.0.0.1
expect {
"yes/no" {send "yes\r" ; exp_continue}
"password:" {send "1234567\r" ; exp_continue}
eof
}
catch wait result
exit [lindex \$result 3]
EOF
    echo "expect ssh return $?"
    echo "forward $1 -> $2 done"
}

## check expect available
which expect > /dev/null
if [ $? -ne 0 ] ; then
    echo "command expect not available"
    exit 1
fi

login_port="10000"
forward_port="10001"

## check whether the number of elements is equal
login_port_num=$(echo ${login_port} | wc -w)
forward_port_num=$(echo ${forward_port} | wc -w)
if [ ${login_port_num} -ne ${forward_port_num} ] ; then
    echo "The numbers of login ports and forward ports are not equal"
    exit 1
fi
port_num=${login_port_num}

## provide pair of arguments to ssh main function
index=1
while [ ${index} -le ${port_num} ] ; do
    login_p=$(echo ${login_port} | awk '{print $'$index'}')
    forward_p=$(echo ${forward_port} | awk '{print $'$index'}')
    connection ${login_p} ${forward_p}
    index=$((index + 1))
done

Here the output from script

spawn /usr/bin/ssh -v -fCNL *:10000:127.0.0.1:10001 127.0.0.1
OpenSSH_7.2p2 Ubuntu-4ubuntu2.10, OpenSSL 1.0.2g  1 Mar 2016
...
debug1: Next authentication method: password
wang@127.0.0.1's password: 
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (password).
Authenticated to 127.0.0.1 ([127.0.0.1]:22).
debug1: Local connections to *:10000 forwarded to remote address 127.0.0.1:10001
debug1: Local forwarding listening on 0.0.0.0 port 10000.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on :: port 10000.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions@openssh.com
debug1: forking to background
expect ssh return 0
forward 10000 -> 10001 done

Answer 1

This should work for you:

spawn -ignore SIGHUP ssh -f ...

UPDATE:

Another workaround is:

spawn bash -c "ssh -f ...; sleep 1"

---

UPDATE 2 (a bit explanation):

ssh -f calls daemon() to make itself a daemon. See ssh.c in the souce code:

/* Do fork() after authentication. Used by "ssh -f" */
static void
fork_postauth(void)
{
        if (need_controlpersist_detach)
                control_persist_detach();
        debug("forking to background");
        fork_after_authentication_flag = 0;
        if (daemon(1, 1) == -1)
                fatal("daemon() failed: %.200s", strerror(errno));
}

daemon() is implemented like this:

int
daemon(int nochdir, int noclose)
{
        int fd;

        switch (fork()) {
        case -1:
                return (-1);
        case 0:
                break;
        default:
                _exit(0);
        }

        if (setsid() == -1)
                return (-1);

        if (!nochdir)
                (void)chdir("/");

        if (!noclose && (fd = open(_PATH_DEVNULL, O_RDWR, 0)) != -1) {
                (void)dup2(fd, STDIN_FILENO);
                (void)dup2(fd, STDOUT_FILENO);
                (void)dup2(fd, STDERR_FILENO);
                if (fd > 2)
                        (void)close (fd);
        }
        return (0);
}

There's a race condition ^{(not sure if its the correct term for here)} between _exit() in the parent process and setsid() in the child process. Here _exit() would always complete first since "the function _exit() terminates the calling process immediately" and setsid() is much more heavy weight. So when the parent process exits, setsid() is not effective yet and the child process is still in the same session as the parent process. According to the apue book (I'm referring to the 2005 edition, Chapter 10: Signals), SIGHUP "is also generated if the session leader terminates. In this case, the signal is sent to each process in the foreground process group."

In brief:

• Expect allocates a pty and runs ssh on the pty. Here, ssh would be running in a new session and be the session leader.
• ssh -f calls daemon(). The parent process (session leader) calls _exit(). At this time, the child process is still in the session so it'll get SIGHUP whose default behavior is to terminate the process.

How the workarounds works:

• The nohup way (spawn -ignore SIGHUP) is to explicitly ask the process to ignore SIGHUP so it'll not be terminated.
• For bash -c 'sshh -f ...; sleep 1', bash would be the session leader and sleep 1 in the end prevents the session leader from exiting too soon. So after sleep 1, the child ssh process's setsid() has already done and child ssh is already in a new process session.

---

UPDATE 3:

You can compile ssh with the following modification (in ssh.c) and verify:

static int
my_daemon(int nochdir, int noclose)
{
    int fd;

    switch (fork()) {
    case -1:
        return (-1);
    case 0:
        break;
    default:
        // wait a while for child's setsid() to complete
        sleep(1);
//      ^^^^^^^^
        _exit(0);
    }

    if (setsid() == -1)
        return (-1);

    if (!nochdir)
        (void)chdir("/");

    if (!noclose && (fd = open(_PATH_DEVNULL, O_RDWR, 0)) != -1) {
        (void)dup2(fd, STDIN_FILENO);
        (void)dup2(fd, STDOUT_FILENO);
        (void)dup2(fd, STDERR_FILENO);
        if (fd > 2)
            (void)close (fd);
    }
    return (0);
}

/* Do fork() after authentication. Used by "ssh -f" */
static void
fork_postauth(void)
{
    if (need_controlpersist_detach)
        control_persist_detach();
    debug("forking to background");
    fork_after_authentication_flag = 0;
    if (my_daemon(1, 1) == -1)
//      ^^^^^^^^^
        fatal("my_daemon() failed: %.200s", strerror(errno));
}