0

New to gitlab CI. I have a custom windows ami (with required pre-installed software) where I would like to run the build.

One of my options is to keep this instance always ON and install the runner on it, but that seems like a waste of compute (I have already tried this set up).

Can I start the instance from gitlab CI? I realize that the runner still needs to be running somewhere, probably on a lower tier machine. I referred to the AWS autoscaling doc available on gitlab, but it highlights how to autoscale with docker. I cannot use the docker setup, kind of tied to the custom image.

How can I spawn and run my pipeline in the ec2 instance?

rookie
  • 1,168
  • 3
  • 14
  • 25

1 Answers1

0

Gitlab has no knowledge of where the runners are running, they will only know if a runner is connected or not, so no, Gitlab itself would not be able to start up an instance.

What you can do however is dedicate a small instance to hitting the Gitlab API to determine how many pending jobs there are and how many runners are available to work it. If there are no runners but there are jobs to be worked, start a runner instance. If there are runners but no jobs, stop an instance.

Here are some Gitlab API's that should come in handy:

  • Runners API (get list of runners, register a new runner, delete a runner, etc): https://docs.gitlab.com/ee/api/runners.html
  • Pipelines API (get pipelines for project, get single pipeline, etc)
  • Jobs API (get jobs for pipeline, cancel job, retry, etc.)

Here's how I would implement it: First, I'd define a threshold for max runners, and one for max pending jobs before increasing the runners by 1. For example, maybe I want no more than 10 runners running at at time, and no more than 6 jobs in the queue before I add another runner (up to the max).

Then,

  • Hit the Runners API to get number of runners.
  • Hit the Projects API to get all projects with pipelines enabled.
  • Hit Pipelines API for each project to get all pending/in progress Pipelines.
  • Hit Jobs API for each pipeline to get all pending jobs
  • If there are no runners but there are pending jobs, add 1 runner.
  • If there are more than 1 runner but less than the max runners, and more than the jobs threshold, add 1 runner up to the max.
  • If there's more than 0 runners and no pending jobs, destroy a sleeping runner (and deregister it from Gitlab so the Runners API call remains clean).
  • sleep for a minute or two, then loop from the top.

This is just a simple example and can easily be extended to fit your needs.

Adam Marshall
  • 6,369
  • 1
  • 29
  • 45