Terminate spot instance after each successful job using Gitlab CI

Question

I have been working around Gitlab CI from couple of days. I have setup the EC2 -ASG as runner with spot instances.

I wonder if we have any solution such that it should delete the spot instance right after the job is successful.

Following is the gitlab runner configuration.

concurrent = 2
check_interval = 3

[session_server]
  session_timeout = 1800

[[runners]]
  name = "shell-runner"
  url = "https://gitlab.com/"
  token = "xxxx-xxxx"
  executor = "shell"
  limit = 1
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

[[runners]]
  name = "docker-machine-runner"
  url = "https://gitlab.com/"
  token = "xxxx-xyxyxyxy"
  executor = "docker+machine"
  limit = 1
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.machine]
    IdleCount = 0
    IdleTime = 1800
    MaxBuilds = 100
    MachineDriver = "amazonec2"
    MachineName = "gitlab-docker-machine-%s"
    MachineOptions = [
    "amazonec2-region=us-west-2", 
    "amazonec2-ssh-user=ubuntu", 
    "amazonec2-vpc-id=vpc-xxxx", 
    "amazonec2-subnet-id=subnet-xxx", 
    "amazonec2-use-private-address=true", 
    "amazonec2-instance-type=t3a.medium", 
    "amazonec2-ami=ami-xxx", 
    "amazonec2-zone=a",
    "amazonec2-security-group=gitlab-runner-sg", 
    "amazonec2-request-spot-instance=true", 
    "amazonec2-spot-price=0.025"
    ]

I have two runner in the above configuration i.e., shell and docker-machine.

Currently, it's not deleting the spot fleet at all and if I set amazonec2-block-duration-minutes=20 flag, I guess it keep it the spot instance for 20 mins and remove it after that.

I'm looking for a solution such that the spot instances get deleted after each job is successful and/or it can wait for sometime for other jobs and gets terminated.

In above docker-machine-runner, what configuration change is required to achieve this?

Or can we do any other automation to make it happen?

Let me know if required more information on the same.

Thanks in advance.

score 0 · Answer 1 · answered Jan 26 '21 at 17:51

I answered a similar question yesterday with how I monitor my runners platform. The full post is here: Start build on a windows ec2 with gitlab runner

What I do I have a small app running somewhere that continuously polls various Gitlab APIs to retrieve the number of pending jobs and the number of available runners. Then, based on some thresholds I've defined, I will either increase the number of runners or decrease them, up to a threshold.

For example, let's say I want a max of 5 runners at a time, and I want no more than 5 jobs in the queue before I increase the number of runners (up to 5). Here's what I'd do (copied from linked post):

Hit the Runners API to get number of runners.
Hit the Projects API to get all projects and filter it to only keep projects that have pipelines/CI enabled. You can store this number too so you don't have to make this call each time.
Hit Pipelines API for each project to get all pending/in progress Pipelines.
Hit Jobs API for each pipeline to get all pending jobs.
If there are no runners but there are pending jobs, add 1 runner.
If there's more than 1 runner but less than the max runners, and more than the jobs threshold, add 1 runner up to the max.
If there's 1 or more runners and no pending jobs, destroy a sleeping runner (and deregister it from Gitlab so the Runners API call remains clean). sleep for a minute or two, then loop from the top.

The various API's used are:

Projects API: https://docs.gitlab.com/ee/api/projects.html
Pipelines API: https://docs.gitlab.com/ee/api/pipelines.html
Jobs API: https://docs.gitlab.com/ee/api/jobs.html
Runners API: https://docs.gitlab.com/ee/api/runners.html
Whatever you need to create and destroy instances in your cloud provider.

Cool @adam. I almost did the same except for windows, I implemented a bash script on linux such a way that it will go through the APIs and terminate the spot instances if it finds no running or pending jobs. — Arvin, Jan 27 '21 at 07:09

Terminate spot instance after each successful job using Gitlab CI

1 Answers1