How to run multiple Python scripts and an executable files using Docker?

Question

I want to create a container that is contained with two Python packages as well as a package consist of an executable file.

Here's my main package (dockerized_package) tree:

dockerized_project
├── docker-compose.yml
├── Dockerfile
├── exec_project
│   ├── config
│   │   └── config.json
│   ├── config.json
│   ├── gowebapp
├── pythonic_project1
│   ├── __main__.py
│   ├── requirements.txt
│   ├── start.sh
│   └── utility
│       └── utility.py
└── pythonic_project2
    ├── collect
    │   ├── collector.py
    ├── __main__.py
    ├── requirements.txt
    └── start.sh

Dockerfile content:

FROM ubuntu:18.04
        
RUN apt update
RUN apt-get install -y python3.6 python3-pip python3-dev build-essential gcc \
    libsnmp-dev snmp-mibs-downloader

RUN pip3 install --upgrade pip

RUN mkdir /app
WORKDIR /app
COPY . /app

WORKDIR /app/snmp_collector
RUN pip3 install -r requirements.txt
WORKDIR /app/proto_conversion
RUN pip3 install -r requirements.txt

WORKDIR /app/pythonic_project1
CMD python3 __main__.py

WORKDIR /app/pythonic_project2
CMD python3 __main__.py

WORKDIR /app/exec_project
CMD ["./gowebapp"]

docker-compose content:

version: '3'

services:
  proto_conversion:
      build: .
      image: pc:2.0.0
      container_name: proto_conversion
#      command: 
#        - "bash  snmp_collector/start.sh"
#        - "bash  proto_conversion/start.sh"
      restart: unless-stopped
      ports:
        - 8008:8008
      tty: true

Problem:

When I run this project with docker-compose up --build, only the last CMD command runs. Hence, I think the previous CMD commands are killed in Dockerfile because when I remove the last two CMD, the first CMD works well.

Is there any approach to run multiple Python scripts and an executable file in the background?

I've also tried with the bash files without any success either.

I don't know you mean. In my Dockerfile the running project procedure is clarified. My problem is running multiple python script in the Docker file with `CMD`. when next `CMD` ran the previous `CMD` killed, I tried also with `nohub`. — Benyamin Jafari, Dec 25 '18 at 09:08
FWIW you can pass a list of packages as arguments to `apt install`. Just `apt install -y package1 package2 package3` ... — tripleee, Dec 25 '18 at 10:14
I think you should have a look at the [interaction of `CMD` and `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact). You can only have one `CMD` in one Dockerfile (see [`CMD` documentation](https://docs.docker.com/engine/reference/builder/#cmd)) but you can combine `ENTRYPOINT` and `CMD` as shown in the first link. — FK82, Dec 25 '18 at 10:18
My understanding is that you should still prefer `apt-get` over `apt` for noninteractive use. — tripleee, Dec 26 '18 at 09:44
@tripleee Thanks for the suggestion. Even in `18.04` `apt-get` should be preferred? — Benyamin Jafari, Dec 26 '18 at 13:08
Question is same what exactly I Want to know. Question is nicely asked here..Thanks — Muralitharan Sathyamoorthy, Mar 06 '20 at 12:39

Farzad Vertigo · Accepted Answer · 2020-03-21T22:30:45.643

20

As mentioned in the documentation, there can be only one CMD in the docker file and if there is more, the last one overrides the others and takes effect. A key point of using docker might be to isolate your programs, so at first glance, you might want to move them to separate containers and talk to each other using a shared volume or a docker network, but if you really need them to run in the same container, including them in a bash script and replacing the last CMD with CMD run.sh will run them alongside each other:

#!/bin/bash

exec python3 /path/to/script1.py &
exec python3 /path/to/script2.py

Add COPY run.sh to the Dockerfile and use RUN chmod a+x run.sh to make it executable. CMD should be CMD ["./run.sh"]

edited Mar 21 '20 at 22:30

answered Dec 25 '18 at 09:48

Farzad Vertigo

2,458
1
29
32

1

The first `exec` is pointless; the shell already takes care of running the background process separately. – tripleee Dec 26 '18 at 11:22
And how about `gowebapp` executable file? – Benyamin Jafari Dec 30 '18 at 09:23
Maybe adding `&` at the end of the second python line and adding `./gowebapp` in the third line? – Farzad Vertigo Dec 31 '18 at 00:41

score 5 · Answer 2 · answered Dec 25 '18 at 08:54

5

try it via entrypoint.sh

ENTRYPOINT ["/docker_entrypoint.sh"]

docker_entrypoint.sh

#!/bin/bash

set -e

exec python3 not__main__.py &
exec python3 __main__.py

symbol & says that you run service as daemon in background

answered Dec 25 '18 at 08:54

frankegoesdown

1,898
1
20
38

each `__main__.py` placed in different folder. – Benyamin Jafari Dec 25 '18 at 09:09
use cd or python3 ../../folder/script.py – frankegoesdown Dec 25 '18 at 09:33
2

The `exec` statements seem useless here. The `set -e` offers no benefits for background jobs. Probably take it out along with both `exec` statements, leaving just two `python3` commands with different arguments. – tripleee Dec 26 '18 at 09:41
I would use `set -m`. See [set docs](https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html) – Evhz Aug 24 '23 at 12:10

score 2 · Answer 3 · answered Dec 25 '18 at 10:04

Best practice is to launch these as three separate containers. That's doubly true since you're taking three separate applications, bundling them into a single container, and then trying to launch three separate things from them.

Create a separate Dockerfile in each of your project subdirectories. These can be simpler, especially for the one that just contains a compiled binary

# execproject/Dockerfile
FROM ubuntu:18.04
WORKDIR /app
COPY . ./
CMD ["./gowebapp"]

Then in your docker-compose.yml file have three separate stanzas to launch the containers

version: '3'
services:
  pythonic_project1:
    build: ./pythonic_project1
    ports:
      - 8008:8008
    env:
      PY2_URL: 'http://pythonic_project2:8009'
      GO_URL: 'http://execproject:8010'
  pythonic_project2:
    build: ./pythonic_project2
  execproject:
    build: ./execproject

If you really can't rearrange your Dockerfiles, you can at least launch three containers from the same image in the docker-compose.yml file:

services:
  pythonic_project1:
    build: .
    workdir: /app/pythonic_project1
    command: ./__main__.py
  pythonic_project2:
    build: .
    workdir: /app/pythonic_project1
    command: ./__main__.py

There's several good reasons to structure your project with multiple containers and images:

If you roll your own shell script and use background processes (as other answers have), it just won't notice if one of the processes dies; here you can use Docker's restart mechanism to restart individual containers.
If you have an update to one of the programs, you can update and restart only that single container and leave the rest intact.
If you ever use a more complex container orchestrator (Docker Swarm, Nomad, Kubernetes) the different components can run on different hosts and require a smaller block of CPU/memory resource on a single node.
If you ever use a more complex container orchestrator, you can individually scale up components that are using more CPU.

How to run multiple Python scripts and an executable files using Docker?

Here's my main package (dockerized_package) tree:

Dockerfile content:

docker-compose content:

Problem:

3 Answers3

Linked