docker stop spark container from exiting

Question

I know docker only listens to pid 1 and in case that pid exits (or turns into a daemon) it thinks the program exited and the container is shut down.

When apache-spark is started the ./start-master.sh script how can I kept the container running?

I do not think: while true; do sleep 1000; done is an appropriate solution.

E.g. I used command: sbin/start-master.sh to start the master. But it keeps shutting down.

How to keep it running when started with docker-compose?

score 1 · Accepted Answer · edited May 23 '17 at 11:59

1

As mentioned in "Use of Supervisor in docker", you could use phusion/baseimage-docker as a base image in which you can register scripts as "services".

The my_init script included in that image will take care of the exit signals management.

And the processes launched by start-master.sh would still be running.
Again, that supposes you are building your apache-spark image starting from phusion/baseimage-docker.

As commented by thaJeztah, using an existing image works too: gettyimages/spark/~/dockerfile/. Its default CMD will keep the container running.

Both options are cleaner than relying on a tail -f trick, which won't handle the kill/exit signals gracefully.

edited May 23 '17 at 11:59

Community

1
1

answered Apr 07 '16 at 07:40

VonC

1,262,500
529
4,410
5,250

cool I will need to look a bit closer at this. Already using the phusion base image https://hub.docker.com/r/possibly/spark/ – Georg Heiler Apr 07 '16 at 07:42
Or use an existing image; https://hub.docker.com/r/gettyimages/spark/~/dockerfile/ (or use it as an example) – thaJeztah Apr 07 '16 at 09:12
@thaJeztah Good point. I have included your comment in the answer for more visibility. – VonC Apr 07 '16 at 09:35

score 1 · Answer 2 · answered Aug 09 '19 at 22:30

Here is another solution. Create a file spark-env.sh with the following contents and copy it into the spark conf directory.

SPARK_NO_DAEMONIZE=true

If your CMD in the Dockerfile looks like this:

CMD ["/spark/sbin/start-master.sh"]

the container will not exit.

score 0 · Answer 3 · answered Apr 07 '16 at 09:33

0

tail -f -n 50 /path/to/spark/logfile

This will keep the container alive and also provide useful info if you run -it interactive mode. You can run -d detached and it will stay alive.

answered Apr 07 '16 at 09:33

kliew

3,073
1
14
25

docker stop spark container from exiting

3 Answers3