3

I made a very small Phoenix Framework app (only slightly modified from what you get when you run: mix phoenix.new). I've been trying to deploy it in a Docker container. It works fine while the container is running, but it always dies within a minute of startup with a message of "Killed." Whether I make requests to it or not does not seem to matter. I tried watching the docker events, and got the following:

$ docker events
2016-04-09T16:24:02.538602484-04:00 container create 
ef45a768723c64125c717a7b40ee7513e477f27339c6266bd28cc3084c60e11f
(image=gcr.io/myprojectname/myapp:v2, name=amazing_bhabha)

2016-04-09T16:24:02.550438045-04:00 container attach
ef45a768723c64125c717a7b40ee7513e477f27339c6266bd28cc3084c60e11f   
(image=gcr.io/myprojectname/myapp:v2, name=amazing_bhabha)

2016-04-09T16:24:02.599731705-04:00 network connect 
c64a1439c8095f82ab0fea5c48b563c8aac7256d6064b3189b0bc8d052d69fe4
(name=bridge, type=bridge, container=ef45a768723c64125c717a7b40ee7513e477f27339c6266bd28cc3084c60e11f)

2016-04-09T16:24:02.600048755-04:00 container start
ef45a768723c64125c717a7b40ee7513e477f27339c6266bd28cc3084c60e11f 
(image=gcr.io/myprojectname/myapp:v2, name=amazing_bhabha)

2016-04-09T16:24:53.858352733-04:00 container die 
ef45a768723c64125c717a7b40ee7513e477f27339c6266bd28cc3084c60e11f 
(name=amazing_bhabha, image=gcr.io/myprojectname/myapp:v2)

2016-04-09T16:24:53.930349810-04:00 network disconnect
c64a1439c8095f82ab0fea5c48b563c8aac7256d6064b3189b0bc8d052d69fe4 
(container=ef45a768723c64125c717a7b40ee7513e477f27339c6266bd28cc3084c60e11f, name=bridge, type=bridge)

I'm still very new to Docker and Elixir, so I'm not sure what other research I can do into this. There is a similar sounding question here: I run a docker container,but after few minutes , it was killed by himself

But I'm not sure how or if the OP ever resolved it. Thanks in advance for any hints. Please let me know if there's any other information I can get that might help.

Edit 1: I learned that docker ps -a will actually tell me the exit code, which I haven't found elsewhere. All of my containers are exiting with a 137 error code. My docker VM has 4GB of memory, so I tried running with a -m=3g flag but got the same result. I also did not see any processes in the Windows process explorer approach 3GB.

Edit 2: I played around a bit more with the memory limit on my container and found that the time the container lives is directly correlated to how much memory I allow it. So I created a totally new project (mix --no-brunch --no-ecto phoenix.new),copied my Dockerfile, and tried to build and run it. It gave me exactly the same results. This leads me to believe that my problem is in my Dockerfile or how I'm running the app.

Dockerfile:

FROM marcelocg/phoenix
MAINTAINER Arcaten
RUN echo $PWD
#Copy source
ADD . ./
#Get dependencies
RUN yes | mix local.hex
RUN yes | mix deps.get
#compile
RUN yes | mix compile
RUN ls -l
EXPOSE 4000
#Run server
ENTRYPOINT yes | MIX_ENV=dev mix phoenix.server

Build:

docker build -t hello_phoenix .

Run:

docker run -p 4000:4000 -m=512m hello_phoenix

And with that, it runs for about 7 seconds and exits with a 137 error code.

Edit 3: Since I was getting "OOMKill":true in my containers, I tried moving in the other direction. I removed the memory cap from my run commands. I still get the same result, but now "OOMKill" is set to false and all of the memory numbers from my inspects now read 0. Also, the StopSignal is now set to "15"

Community
  • 1
  • 1
Arcaten
  • 33
  • 1
  • 5
  • 1
    do you have other containers running in your VM? it's still possible that the container is killed by the kernel if it's running out of memory. Does `docker inspect ` show if it is "oom killed"? – thaJeztah Apr 09 '16 at 21:39
  • I have no other containers. Doing `docker inspect hello_phoenix` did not seem to say anything about it being "oom killed." I can try to edit that info in if it will be helpful. – Arcaten Apr 09 '16 at 22:10
  • does `docker logs hello_phoenix` show anything useful, something that gives you a clue why it exited? – thaJeztah Apr 09 '16 at 22:11
  • I don't have any experience with phoenix, but does `mix phoenix.server` start a process in the foreground, or does it daemonize? (I'm at the airport now, with crappy wi-fi so not really able to run your Dockerfile – thaJeztah Apr 09 '16 at 22:19
  • docker logs just gives me: [info] Running HelloPhoenix.Endpoint with Cowboy using http on port 4000 Killed `mix phoenix.server` does start a process in the foreground, but I believe I can start it in a detached mode. – Arcaten Apr 09 '16 at 22:24
  • no, it needs to be in the foreground – thaJeztah Apr 09 '16 at 22:28
  • Alright. And I did find in the `docker inspect` output where "OOMKilled" is set to true. – Arcaten Apr 09 '16 at 22:43
  • So, in that case it looks like the VM ran out of memory, and the kernel decided to kill a process, which in this case was the container (probably because you gave it a memory limit. – thaJeztah Apr 09 '16 at 22:45
  • I tried increasing my VM memory to 8GB and removed the memory limit on the container, but it just increased the time it took to crash. I can't imagine it's the app taking up that much memory -- I don't see that kind of usage when I run it locally. – Arcaten Apr 09 '16 at 22:58

2 Answers2

5

The problem is the yes | part. The Erlang VM behaves differently from regular programs when it comes to the stdin and input. It will buffer whatever input you throw at it, and with yes | you give it an infinite stream of yeses. Those yeses are buffered and memory grows until the process is killed by the system, because there's no more memory left.

It is generally a bad idea to use yes | with anything using Elixir/Erlang, even more so with long running tasks - with short running ones you have a chance at completing them before you run out of memory, but it's still not a great idea.

michalmuskala
  • 11,028
  • 2
  • 36
  • 47
0

Not sure if this is still relevant but the whole infinite stream problem seems to be solved by just piping echo y.

ex:

echo y | mix compile

Although I'm not sure if there's something I'm missing that makes this a silly solution.

EDIT: this is probably better https://stackoverflow.com/a/25921514

Community
  • 1
  • 1
lrosa007
  • 1
  • 1