ipyparallel/ipcluster inside docker container

Question

I am running an ipcluster on a linux host. The setup is very similar to this SO question (https://stackoverflow.com/a/31479269/2146052).

I have the following adjustments to my ipcontroller_config.py

myip = '*'
c.HubFactory.engine_ip = myip
c.HubFactory.client_ip = myip
c.HubFactory.monitor_ip = myip
c.HubFactory.ip = myip

c.HubFactory.iopub = (10001, 10002)
c.HubFactory.control = (10003,10004)
c.HubFactory.task = (10005,10006)
c.HubFactory.mux = (10007,10008)
c.HubFactory.regport = 10009
c.HubFactory.hb = (10010,10011)
c.HubFactory.notifier_port = 10012

I start the docker container as

docker run -it --rm -p 10000-10012:10000-10012 <myimg> /bin/bash
ipcluster start -n4

The ipcluster starts ok but when I try to connect I get Hub connection timeout. This error does not occur if I run the same configuration outside of docker. Do I need to make a further adjustment beyond the simple port forwarding?

Any luck with this? Seeing the same thing – frmdstryr Mar 15 '18 at 20:08 — frmdstryr, Mar 15 '18 at 20:08
No luck so far, unfortunately – bjonen Mar 15 '18 at 20:09 — bjonen, Mar 15 '18 at 20:09

bjonen · Accepted Answer · 2018-12-24T01:49:14.507

Finally here is the missing piece:

When connecting from a different docker container one needs to specify location inside the configuration:

c.IPControllerApp.location = '<name of service in docker-compose>'

This is documented under https://ipyparallel.readthedocs.io/en/latest/process.html#ports-and-addresses (see --location section for more information)

Also make sure that the two services are on the same docker network https://stackoverflow.com/a/38089080/2146052 and see https://docs.docker.com/compose/networking/

ipcontroller_config.py should therefore look like this (if running parallel for the first time one needs to create the default config by running: ipython profile create --parallel --profile=myprofile)

myip = '*'
c.HubFactory.engine_ip = myip
c.HubFactory.client_ip = myip
c.HubFactory.monitor_ip = myip
c.HubFactory.ip = myip
# c.IPControllerApp.location = '<name of service in docker-compose>'
c.IPControllerApp.location = 'ipyp'

c.HubFactory.iopub = (10001, 10002)
c.HubFactory.control = (10003,10004)
c.HubFactory.task = (10005,10006)
c.HubFactory.mux = (10007,10008)
c.HubFactory.regport = 10009
c.HubFactory.hb = (10010,10011)
c.HubFactory.notifier_port = 10012

Sample docker-compose.yml setup

version: '3.3'
services:
  ipyp:
    image: "ipyp"
    build: .
    ports:
     - "10000-10012:10000-10012"
    command: ipcluster start -n2 --profile=myprofile
    volumes:
      - configdata:/root/.ipython/
    restart: always

  ipyp_test:
    image: "ipyp"
    build: .
    ports:
     - "9999:9999"
    command: jupyter lab --allow-root --port 9999 --ip=0.0.0.0
    volumes:
      -configdata:/parallelconfig:ro
    restart: always

volumes:
  configdata:

Try out the setup by starting a notebook inside the ipyp_test service and running

import ipyparallel
fl = '/parallelconfig/profile_myprofile/security/ipcontroller-client.json'
rc = ipyparallel.Client(fl)
rc.ids

score 1 · Answer 2 · answered Jan 10 '23 at 21:54

There seems to be an updated set of parameters in ipcontroller_config.py for ipyparallel 8.4.1 I suggest an update so the settings will work for ipyparallel 8.4.1

c.IPController.ports = [10000,10001,10002,10003,10004]
c.IPController.client_ports = [10005,10006,10007,10008]
c.IPController.engine_ports = [10009,10010,10011,10012]
c.IPController.location = 'ipyp'

instead of the the following deprecated settings

c.IPController.iopub = (10001,10002)
c.IPController.control = (10003,10004)
c.IPController.task = (10005,10006)
c.IPController.mux = (10007,10008)
c.IPController.regport = 10009
c.IPController.hb = (10010,10011)
c.IPController.notifier_port = 10012

ipyparallel/ipcluster inside docker container

2 Answers2