When does nomad throttle a task?

Question

How does nomad limit resource consumption for a task? If there are two tasks within a group that each have cpu = 100, is there a shared pool of 200 that both tasks have access to? What happens if one of those tasks wants access to more CPU ticks?

job "docs" {
  group "example" {
    task "server" {
      resources {
        cpu    = 100
        memory = 256
      }
    }
    task "greeter" {
      resources {
        cpu    = 100
        memory = 256
      }
    }
  }
}

Looking at /client/allocation/:alloc_id/stats, I see ThrottledPeriods, ThrottledTicks broken down for both resources and tasks -- will both resources and tasks throttle resource usage?

Answer 1

This doesn't fully answer the question, but the docker driver docs have some details about how part of it works:

CPU

Nomad limits containers' CPU based on CPU shares. CPU shares allow containers to burst past their CPU limits. CPU limits will only be imposed when there is contention for resources. When the host is under load your process may be throttled to stabilize QoS depending on how many shares it has. You can see how many CPU shares are available to your process by reading NOMAD_CPU_LIMIT. 1000 shares are approximately equal to 1 GHz.

Please keep the implications of CPU shares in mind when you load test workloads on Nomad.

Memory

Nomad limits containers' memory usage based on total virtual memory. This means that containers scheduled by Nomad cannot use swap. This is to ensure that a swappy process does not degrade performance for other workloads on the same host.

Since memory is not an elastic resource, you will need to make sure your container does not exceed the amount of memory allocated to it, or it will be terminated or crash when it tries to malloc. A process can inspect its memory limit by reading NOMAD_MEMORY_LIMIT, but will need to track its own memory usage. Memory limit is expressed in megabytes so 1024 = 1 GB.