0

I'm working on small Raspberry PI cluster, my host-program creates IP packet fragments and sends them to multiple relay-programes. Relays receive those packet fragments and forward them to destination using raw sockets. Because of raw sockets my relay-programes must run with sudo permission. My setup involves RPi 3 B v2 and RPi 2 B v1. SSH is already set up, nodes can SSH-in without password, although I must run ssh-agent and ssh-add my keys on each node. I've managed to run program sending rank from one node to another(2 different RPis). I run MPI programs in MPMD-way, since I have only 2 RPis I run host and relay on node #1 and relay on node #2. Host program takes path to file to be sent as command line argument.

If I run:

mpirun --oversubscribe -n 1 --host localhost /home/pi/Desktop/host /some.jpeg : -n 2 --host localhost,rpi2 /home/pi/Desktop/relay

it runs, but obviously program fail because relays can't open raw sockets without sudo permission.

If I run:

mpirun --oversubscribe -n 1 --host localhost /home/pi/Desktop/host /some.jpeg : -n 2 --host localhost,rpi2 sudo /home/pi/Desktop/relay

relays report world size: 1 and host program hangs.

If I run:

mpirun --oversubscribe -n 1 --host localhost sudo /home/pi/Desktop/host /some.jpeg : -n 2 --host localhost,rpi2 sudo /home/pi/Desktop/relay

all relays and host reports world size 1.

I found similar problem here: OpenMPI / mpirun or mpiexec with sudo permission

Following short answer I run:

mpirun --oversubscribe -n 1 --host localhost /home/pi/Desktop/host /some.jpeg : -n 2 --host localhost,rpi2  sudo -E /home/pi/Desktop/relay

which results in:

[raspberrypi:00979] OPAL ERROR: Unreachable in file ext2x_client.c at line 109
[raspberrypi:00980] OPAL ERROR: Unreachable in file ext2x_client.c at line 109
*** An error occurred in MPI_Init
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[raspberrypi:00979] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[raspberrypi:00980] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[32582,1],1]
  Exit code:    1
--------------------------------------------------------------------------

I've run sudo visudo and my file on both nodes looks like that:

# User privilege specification
root    ALL=(ALL:ALL) ALL
pi      ALL = NOPASSWD:SETENV:  /etc/alternatives/mpirun
pi      ALL=NOPASSWD:SETENV:    /usr/bin/orterun
pi      ALL=NOPASSWD:SETENV:    /usr/bin/mpirun

When I run everything on one node it just works:

sudo mpirun --alow-run-as-root --oversubscribe -n 1 --host localhost /home/pi/Desktop/host /some.jpeg : -n 2 --host localhost,localhost /home/pi/Desktop/relay //host

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    int world_size = []() {
        int size;
        MPI_Comm_size(MPI_COMM_WORLD, &size);
        return size;
    }();

    int id = []() {
        int id;
        MPI_Comm_rank(MPI_COMM_WORLD, &id);
        return id;
    }();

    if (argc != 2) {
        std::cerr << "Filepath not passed\n";
        MPI_Finalize();
        return 0;
    }

    const std::filesystem::path filepath(argv[1]);
    if (not std::filesystem::exists(filepath)) {
        std::cerr << "File doesn't exist\n";
        MPI_Finalize();
        return 0;
    }

    std::cout << "World size: " << world_size << '\n';

    MPI_Finalize();
    return 0;
}

//relay
int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    int world_size = []() {
        int size;
        MPI_Comm_size(MPI_COMM_WORLD, &size);
        return size;
    }();

    int id = []() {
        int id;
        MPI_Comm_rank(MPI_COMM_WORLD, &id);
        return id;
    }();

    std::cout << "World size: " << world_size << '\n';

    MPI_Finalize();
    return 0;
}

How do I configure nodes to allow them to run MPI programs with sudo?

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Hana0xAF
  • 1
  • 3
  • what if you `sudo mpirun --alow-run-as-root` with several hosts ? note you must be able to ssh passwordless as root (e.g. `sudo ssh remotehost hostname` should just work). you might also consider setting the `suid` bit to your binary and run it without `sudo` (be aware of the security implications), or even use a helper that creates raw sockets for you. – Gilles Gouaillardet Jul 09 '19 at 00:30
  • If i run `sudo mpirun --alow-run-as-root` everything hangs. I tried to set suids of relay-programs, `sudo -i`, `chmod u+s relay_program` and run them one node but it still can't open. I can't wrap my head around configuring passwordless ssh access to root. What do you mean by helper? – Hana0xAF Jul 09 '19 at 21:28
  • Run everything from your user account, and before `MPI_Init()` you can `fork&exec sudo helper`. The helper purpose is to create raw sockets and then pass them to the MPI program via fd passing. An other option is to make your MPI app setuid **and** own by root, create the raw sockets and setuid to the user before `MPI_Init()`, and then mpirun from your user account. – Gilles Gouaillardet Jul 10 '19 at 14:47
  • I made small program which creates raw socket and then inits MPI https://pastebin.com/RRTX9kzr. Then `sudo -i` and compiled program with mpic++, changed suid with `chmod u+s a.out`, which made it root-owned and with sudo permission upon execution `-rwsr-xr-x 1 root root 36316 lip 9 23:28 a.out`. When I simply run it as it was normal program `./a.out` it works, but then when I run it with with `mpirun -n 1 /home/pi/Desktop/a.out` it crushes with error https://pastebin.com/x2Q8iGvH. If I run it with `--allow-run-as-root` it hangs, where did I miss something? – Hana0xAF Jul 10 '19 at 18:53
  • Okay, I missed the thing about setting uid back to user, basicly what I did was compile program with mpic++, `sudo -i`, then changed owner to root with `chown 0:0 a.out` and set suid to root with `chmod u+s a.out`. Repeated it on both nodes and run MPI program as user with `mpirun -n 2 --host localhost, rpi1 /home/pi/Desktop/a.out` and everything works. In code I changed uid just before MPI_Init by using `setreuid(geteuid(), getuid());`. Thats good solution and I'm happy I can develop and test my app further! Although I will have to resolve issue properly before it's ready for commercial use. – Hana0xAF Jul 10 '19 at 20:24
  • note you can use capabilities instead of setuid bit (`ping` used to be suid root, but is now using capabilities which is more secure: `$ getcap /usr/bin/ping /usr/bin/ping = cap_net_admin,cap_net_raw+p ` – Gilles Gouaillardet Jul 11 '19 at 02:04
  • I checked solution with capabilities, basically I set capabilities with `setcap program cap_net_raw,cap_net_admin+eip`. Everything works just fine and I don't have to set suid to user before MPI_Init. I think it's really elegant solution, thank you. – Hana0xAF Jul 11 '19 at 21:05

1 Answers1

0

The easiest way to resolve the problem is to set capabilities of file, it still pose security problem, but it's not as serious as setting suid of program to root. To set capabilities of program allowing to open raw socket: setcap program cap_net_raw,cap_net_admin+eip.

Hana0xAF
  • 1
  • 3