Troubles communicating with ROS2 node in docker container

Question

I'm studying ROS2. I've got a docker container with ROS2 foxy installation inside it.

This container has many other things installed, so it is preferable for me to deal with it instead of ones downloaded from DockerHub.

The container is based on Ubuntu 18.04, and my host runs Ubuntu 20.04.

Following doesn't work:

On host: $ docker run --net host -it <container name>

Inside container:

# env | grep ROS_
ROS_DOMAIN_ID=142
ROS_VERSION=2
ROS_LOCALHOST_ONLY=0
ROS_PYTHON_VERSION=3
ROS_DISTRO=foxy


# ros2 run examples_rclpy_minimal_publisher publisher_local_function
[INFO] [1611658788.451254349] [minimal_publisher]: Publishing: "Hello World: 0"
[INFO] [1611658788.930325228] [minimal_publisher]: Publishing: "Hello World: 1"
[INFO] [1611658789.430629464] [minimal_publisher]: Publishing: "Hello World: 2"
...

On the same host in another terminal:

$ source /opt/ros/foxy/setup.zsh
$ export ROS_DOMAIN_ID=142
$ env | grep ROS_
ROS_DISTRO=foxy
ROS_LOCALHOST_ONLY=0
ROS_PYTHON_VERSION=3
ROS_VERSION=2
ROS_DOMAIN_ID=142


$ ros2 run examples_rclpy_minimal_subscriber subscriber_member_function

No output from subscriber.

At the same time, I see open UDP ports:

$ sudo netstat -unlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 0.0.0.0:35379           0.0.0.0:*                           2103557/python3
udp        0      0 127.0.0.1:41750         0.0.0.0:*                           1867221/python3
udp        0      0 0.0.0.0:42900           0.0.0.0:*                           2103557/python3
udp        0      0 0.0.0.0:42900           0.0.0.0:*                           1867221/python3
udp        0      0 0.0.0.0:42912           0.0.0.0:*                           2103557/python3
udp        0      0 0.0.0.0:42913           0.0.0.0:*                           2103557/python3
udp        0      0 0.0.0.0:42916           0.0.0.0:*                           1867221/python3
udp        0      0 0.0.0.0:42917           0.0.0.0:*                           1867221/python3
udp        0      0 127.0.0.1:47375         0.0.0.0:*                           2103557/python3

PIDs, starting with 186xxxx belong to ros2_daemon on host, PIDs, starting with 210xxxx, belong to python, running in the container.

If I execute subscriber in another /bin/bash in the container, it works, that is, the subscriber prints messages that it receives from publisher.

Multicast UDP datagrams also work:

In container:

# ros2 multicast receive
Waiting for UDP multicast datagram...
Received from 106.xxx.xxx.xxx:45829: 'Hello World!'

On host:

$ ros2 multicast send
Sending one UDP multicast datagram...

UPDATE. I've tried pulling standard container osrf/ros:foxy-desktop... And examples work as expected.

Publisher in container:

$ docker pull osrf/ros:foxy-desktop
$ docker run --net host -it osrf/ros:foxy-desktop
# export ROS_DOMAIN_ID=142
# env | grep ROS_
ROS_VERSION=2
ROS_PYTHON_VERSION=3
ROS_DOMAIN_ID=142
ROS_LOCALHOST_ONLY=0
ROS_DISTRO=foxy
#ros2 run examples_rclpy_minimal_publisher publisher_local_function

[INFO] [1611670054.887068490] [minimal_publisher]: Publishing: "Hello World: 0"
[INFO] [1611670055.367854925] [minimal_publisher]: Publishing: "Hello World: 1"
...

Subscriber on host:

$ ros2 run examples_rclpy_minimal_subscriber subscriber_member_function
[INFO] [1611670073.075589355] [minimal_subscriber]: I heard: "Hello World: 7"
[INFO] [1611670073.540520496] [minimal_subscriber]: I heard: "Hello World: 8"
[INFO] [1611670074.040020703] [minimal_subscriber]: I heard: "Hello World: 9"
...

Update 2:

Getting back to original container. I see two UDP sockets with the same port number 7400 in netstat. Is it OK?

Update: Yes, it is: https://stackoverflow.com/a/1694148

The same phenomenon is observed in the output of netstat above, but port number is different.

$ sudo netstat -unlp
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
...
udp        0      0 0.0.0.0:39604           0.0.0.0:*                           2319288/python3
udp        0      0 0.0.0.0:7400            0.0.0.0:*                           2319288/python3
udp        0      0 0.0.0.0:7400            0.0.0.0:*                           2319267/python3
udp        0      0 0.0.0.0:7412            0.0.0.0:*                           2319267/python3
...

And processes:

$ ps axf
...
2319287 pts/4    S+     0:00      \_ /usr/bin/python3 /opt/ros/foxy/bin/ros2 run examples_rclpy_minimal_publisher publisher_local_function
2319288 pts/4    Sl+    0:01          \_ /usr/bin/python3 /opt/ros/foxy/lib/examples_rclpy_minimal_publisher/publisher_local_function
...
2319050 ?        Sl     0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id ae2da482416
2319075 pts/0    Ss+    0:00  \_ /bin/bash
2319266 pts/0    S      0:00      \_ /usr/bin/python3 /root/git/ros2_foxy/install/bin/ros2 run examples_rclpy_minimal_subscriber subscriber_member_function
2319267 pts/0    Sl     0:00          \_ /usr/bin/python3 /root/git/ros2_foxy/install/lib/examples_rclpy_minimal_subscriber/subscriber_member_function

Process with ID 2319288 is running from a host, I've accidentally cut output of ps.

Update 3

If I run docker container without --net=host, then I subscriber sees messages from publisher. I cannot afford this, because docker container is not seen in the network.
I've replaced subscriber in the container with netcat (netcat -l -u 42900) - and netcat in the container has received messages from the publisher that was working outside it. Container is run with --net=host

It suggests that everything is OK with the network in the container, but ROS2 uses it somehow incorrectly.

How do I correct it?

score 3 · Accepted Answer · answered Feb 12 '21 at 09:00

The last releases of Fast-DDS come with SharedMemory transport by default. Using --net=host implies both DDS participants believe they are in the same machine and they try to communicate using SharedMemory instead of UDP. Fast-DDS team will work to implement a mechanism to detect this kind of situation. Meanwhile, I can give you two solutions:

Using an XML to disable SharedMemory transport in one of the DDS participants.

    <?xml version="1.0" encoding="UTF-8" ?>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles" >
        <transport_descriptors>
            <transport_descriptor>
                <transport_id>CustomUdpTransport</transport_id>
                <type>UDPv4</type>
            </transport_descriptor>
        </transport_descriptors>

        <participant profile_name="participant_profile" is_default_profile="true">
            <rtps>
                <userTransports>
                    <transport_id>CustomUdpTransport</transport_id>
                </userTransports>

                <useBuiltinTransports>false</useBuiltinTransports>
            </rtps>
        </participant>
    </profiles>

Enable SharedMemory between host and container. For this you should share /dev/shm:

docker run -ti --net host -v /dev/shm:/dev/shm <DOCKER_IMAGE>

Also, both applications should be run with the same UID. In my case, my docker container's user is root (UID=0). Then I had to run the host application as root.

score 1 · Answer 2 · edited Aug 25 '23 at 09:42

I have also encountered this issue and haven't been able to come up with a fix, but some workaround. The error seems to be rooted in Fast-DDS trying to use SHM when using the --net=host option. The issue has also been described multiple times, e.g. https://github.com/eProsima/Fast-DDS/issues/3475, https://github.com/eProsima/Fast-DDS/issues/2956

My workaround to use SHM between ROS 2 nodes inside containers and on the host is to:

start the container with these options: --network=host --ipc=host --pid=host
run nodes inside the container with the same user id as nodes on the host

For convenience, I have created a repository to build (as root user) and run (with host's user id) ROS 2 packages/ nodes inside containers. This allows to use Shared Memory Transport (SHM) between ROS 2 nodes run inside the container and on the host.

Troubles communicating with ROS2 node in docker container

2 Answers2