10

I am having some docker container which listens on RabbitMQ and process the message received. I have a code pipeline which kicks off the rebuilding of the image and updating the tasks when there is a code commit.

My problem here is the container will be killed abruptly during the message processing is there any way where I can stop the container killing until the process is finished and allow it to stop so that a new new container will be automatically created as I am ok with the current container processing the message with the old code. My container is running python code inside.

Mark Amery
  • 143,130
  • 81
  • 406
  • 459
Ram
  • 407
  • 2
  • 6
  • 12

2 Answers2

25

ECS by default sends a SIGTERM:

StopTask

Stops a running task.

When StopTask is called on a task, the equivalent of docker stop is issued to the containers running in the task. This results in a SIGTERM and a default 30-second timeout, after which SIGKILL is sent and the containers are forcibly stopped. If the container handles the SIGTERM gracefully and exits within 30 seconds from receiving it, no SIGKILL is sent.

Note

The default 30-second timeout can be configured on the Amazon ECS container agent with the ECS_CONTAINER_STOP_TIMEOUT variable. For more information, see Amazon ECS Container Agent Configuration in the Amazon Elastic Container Service Developer Guide.

Knowing this, you can add a simple check in your app to catch the SIGTERM, and react appropriately.

MrDuk
  • 16,578
  • 18
  • 74
  • 133
  • 1
    Thanks for the response so if i am able to catch SIGTERM and change the default timeout i can wait for the process to finish even it takes a longer time like max 30 min? – Ram Mar 30 '18 at 04:56
  • 1
    Worth noting that the maximum timeout is 120 seconds. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-stoptimeout – GregHNZ Jun 25 '21 at 01:54
-4

Two ways to achieve a graceful stop in Docker:

Using docker stop

When issuing a docker stop command to a container, Docker fires a SIGTERM signal to the process inside the container, and waits for 10 seconds before cleaning up the container.

You can specify the timeout other than 10s:

docker stop --time 30 <CONTAINER>

You need to ensure that the process handles the SIGTERM signal properly. Otherwise it will be rudely killed by a SIGKILL signal.

Using docker kill

By default the docker kill command sends a SIGKILL signal to the process. But you can specify another signal to be used:

docker kill --signal SIGQUIT <CONTAINER>

Also ensure that the process handles the specified signal properly. Worse than docker stop, the docker kill command does not have a timeout behavior.

Yuankun
  • 6,875
  • 3
  • 32
  • 34
  • 4
    The question specifically requested help with Amazon ECS (https://aws.amazon.com/ecs/), ECS manages the containers for you and does not provide access to raw docker cli commands. – Kevin Doveton Aug 11 '20 at 00:23