3

ECS' Container Definitions allow you to specify a memoryReservation for each container:

The soft limit (in MiB) of memory to reserve for the container. When system memory is under contention, Docker attempts to keep the container memory to this soft limit; however, your container can consume more memory when needed, up to either the hard limit specified with the memory parameter (if applicable), or all of the available memory on the container instance, whichever comes first. This parameter maps to MemoryReservation in the Create a container section of the Docker Remote API and the --memory-reservation option to docker run...If you specify memoryReservation, then that value is subtracted from the available memory resources for the container instance on which the container is placed; otherwise, the value of memory is used. For example, if your container normally uses 128 MiB of memory, but occasionally bursts to 256 MiB of memory for short periods of time, you can set a memoryReservation of 128 MiB, and a memory hard limit of 300 MiB. This configuration would allow the container to only reserve 128 MiB of memory from the remaining resources on the container instance, but also allow the container to consume more memory resources when needed.

I believe this is implemented using cgroups' "soft limits."

So that means, on ECS+EC2, the purpose of this option is to allow you to schedule a group of tasks on an instance which, if they all used their maximum memory at once, would exceed the total memory available on the instance, but you are predicting that they will not do that.

But what about on ECS+Fargate? Fargate has a task size and the combined hard limits ("memory" option) of a task's containers cannot exceed the task size's memory. [This last part was wrong; see my clarifying answer.] You are billed based on the task size regardless of the memory limits you set. So what benefit could you possibly get by setting a memoryReservation below your hard limit?

The only possibilities I can think of are:

  1. memoryReservation instructs AWS to pack your containers onto a smaller number of underlying VMs, risking performance issues at no benefit to you (since the price is the same).
  2. AWS just ignores memoryReservation for Fargate.

Have I missed any possibilities? If not , does Amazon say which one of the two it is?

Chris Hundt
  • 43
  • 1
  • 5

2 Answers2

1

What about the case where you have multiple containers in a single task. You might want to reserve some amount of memory for one container. Also you might have a case where different containers require a lot of memory at the same time and you don't want any of them to take too much memory and stall out the rest of the containers.

Soccergods
  • 440
  • 5
  • 17
  • I realized that there was an incorrect premise in my question, so it didn't make a lot of sense. But I think the second part of your answer is probably correct so I accepted it. I don't think the first part ("reserve some amount of memory for one container") is correct because that is not what memoryReservation actually does, despite its name. – Chris Hundt Jul 02 '19 at 12:43
  • What do you think the soft limit does then. I assumed that it just permanently reserves that amount of memory for a particular container so that it always has resources to run. Maybe you have three containers out of which one has to keep running continuously while the other two have bursts. If we assign say 1MB for the first task while not having limits for the other two tasks, the app will run properly. Why isn't this a use-case for soft limit? – Soccergods Jul 03 '19 at 04:38
  • This is described in the [kernel docs](https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt): "When the system detects memory contention or low memory, control groups are pushed back to their soft limits." That's all it is. In practice it is intended as a hint as to how much memory you expect your process to use in the long term. I think it is called a "reservation" because most orchestrators will use that value (as opposed to the hard limit) for counting how much of a node's capacity is "used" by each container. But there is no guarantee that the memory will actually be available. – Chris Hundt Jul 03 '19 at 13:11
  • Why do you say that there is no guarantee that the memory will actually be available? If they are pushed back to their soft limits, isn't that the guarantee? In other cases, it will get completely wiped out, but if you put a soft limit of say 1GB, then you'll always have 1GB reserved for the container at the minimum because that can never be "taken away". Hard limits are for when you don't want any single container to overcome all the non-reserved space. – Soccergods Jul 03 '19 at 16:07
  • No, the docs are saying that the kernel *will* try to reclaim memory if it is over the soft limit. It does not promise that it *won't* try to reclaim memory if it is under the soft limit. In fact, if you look at the docs, the very next sentence describes a situation where it will continue reclaiming past the soft limit. – Chris Hundt Jul 03 '19 at 17:04
  • Also, when I say that there is no guarantee the memory will be available, I also mean that the process may never be able to allocate it in the first place. For example, if the soft limit is 1G and the container has used 512MB so far, the kernel will not keep 512MB "free" or otherwise ensure that the container can get an additional 512MB in the future. – Chris Hundt Jul 03 '19 at 17:29
  • I'm not quite following your train of thought. Maybe [this document](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#standard_container_definition_params) will help. Soft limit is memory reservation and hard limit is just memory. This is mentioned while setting these variables as well. This what I'm trying to say. Sorry if I've explained it poorly – Soccergods Jul 06 '19 at 18:05
  • I agree that the kernel doc says what you have described here but since the AWS docs say otherwise and because the kernel doc itself says it is outdated, I have to trust the AWS docs. There was [another stackoverflow question](https://stackoverflow.com/questions/44764135/aws-ecs-task-memory-hard-and-soft-limits) asking something similar about the soft/hard limit. – Soccergods Jul 06 '19 at 18:20
  • I agree that the AWS docs refer to memory being "reserved" but I think they mean in terms of how much of a node's memory is counted as "consumed" by the container when deciding what a node has capacity for. I don't think that there is any guarantee the memory will actually be available. I say this because I cannot find any documentation of what mechanism Docker would use to do this, and in fact I have never heard of such a mechanism even existing in Linux. You can limit a process's memory but AFAIK you cannot reserve or guarantee a certain amount to be available. – Chris Hundt Jul 07 '19 at 20:08
  • I'm not really sure then. I just assumed that aws docs would represent the real picture instead of worrying about the implementation. Maybe send a support ticket or something and ask them about this? I'm not sure someone other than the team which designed this would be able to help you. – Soccergods Jul 11 '19 at 18:00
  • Did you figure out how memory reservation could potentially be done by aws. They do a similar thing for CPU as well, and was interested in knowing how this could be done. – Soccergods Jul 24 '19 at 09:46
1

I realize now that there was an incorrect premise in my question:

Fargate has a task size and the combined hard limits ("memory" option) of a task's containers cannot exceed the task size's memory.

In fact, the combined soft limits ("memoryReservation" option) cannot exceed task memory. The combined hard limits can. So on Fargate you could schedule multiple "bursty" containers in a single task, and a single container could use up to the task's entire available memory before getting killed. This is somewhat similar to what you can achieve on EC2 instances using single-container tasks and memoryReservation.

You don't need to use memoryReservation to accomplish it on Fargate, though (you can just not set any memory limits or reservations beyond the task size). So the question still remains, why use that setting on Fargate? I think the best answer is probably the one provided by MasterFowl: to hint to the kernel not to allow one of your containers to take up too much of the task's memory.

Chris Hundt
  • 43
  • 1
  • 5