How to delete gitlab CI jobs pipelines logs/builds and history

Question

How can we configure gitlab to keep only the last 10 CI jobs/builds and keep deleting the rest?

For example , in Jenkins , we can configure the job to keep only last X builds.

score 65 · Answer 1 · edited Feb 22 '23 at 19:40

65

UPDATE

As of Gitlab Release 12.6, deleting a pipeline is now an option in the GUI for users in role Owner:

Click the pipeline you want to remove in the pipelines list
Hit the red Delete button in the upper right of the pipeline details page.

As of Gitlab Release 11.6, deleting a pipeline is now an option, for users in Owner role only, via the API.

You need:

An API token
The id of the project
The pipeline_id of the pipeline you wish to remove.

Example using curl from the docs for project id: 1 and pipeline_id: 4:

curl --header "PRIVATE-TOKEN: <your_access_token>" --request "DELETE" "https://gitlab.example.com/api/v4/projects/1/pipelines/46"

Documentation is here

edited Feb 22 '23 at 19:40

Bartosz Pierzchlewicz

514
6
15

answered Apr 23 '19 at 15:51

Routhinator

3,559
4
24
35

3

Is it possible to delete only artifacts and build logs (which are stored in the artifacts directory too), but keep the overall pipeline status in the database? Also, an option to delete everything older than e.g. 3 months would be nice. – Jakub Klinkovský Aug 23 '19 at 09:41
1

For the `id` you can use [namespaced path encoding](https://docs.gitlab.com/ee/api/README.html#namespaced-path-encoding). – A1rPun Oct 22 '19 at 09:37
Note: *you can not delete jobs from the admin gui*, you need to delete the pipeline to which the jobs belongs. – peterh Jul 06 '20 at 10:46

Mog · Answer 2 · 2022-09-23T10:26:59.013

36

Mass deletion script fixed for the lazy, delete X pipelines from the oldest (Linux systems).

Note: need jq.

#!/bin/bash
set -e

TOKEN="__SET_YOUR_PERSONAL_ACCESS_TOKEN__"
# Visible under name of project page.
PROJECT_ID="__SET_NUMERIC_PROJECT_ID__"
# Set your gitlab instance url.
GITLAB_INSTANCE="https://gitlab.com/api/v4/projects"
# How many to delete from the oldest, 100 is the maximum, above will just remove 100.
PER_PAGE=100

for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_INSTANCE/$PROJECT_ID/pipelines?per_page=$PER_PAGE&sort=asc" | jq '.[].id') ; do
    echo "Deleting pipeline $PIPELINE"
    curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$GITLAB_INSTANCE/$PROJECT_ID/pipelines/$PIPELINE"
done

edited Sep 23 '22 at 10:26

answered Apr 05 '20 at 18:12

Mog

361
3
3

If you're getting `Cannot index string with string "id"`, you may need an access token with higher privileges. – John Goofy Jan 18 '21 at 13:15
1

It needs personal access token, project token hasn't enough rights. – Mr_Thorynque Jul 29 '21 at 08:41
On Windows, i got `curl: (3) URL using bad/illegal format or missing URL` because `$PIPELINE` contained a carriage return. I worked around this by piping the result of `jq` through `dos2unix`: `... | jq '.[].id' | dos2unix) ; do` – Moritz Feb 09 '22 at 12:07
Also beware that 100 is the maximum `per_page` value that the GitLab API currently allows, so every value above 100 will just delete 100 pipelines. – Moritz Feb 09 '22 at 16:36
Note that `PROJECT` should contain a project ID, rather than the name in `organization/repo` format you usually see. You can find the project ID in the settings. – krzys_h Feb 19 '22 at 08:51
Create a personal access token - https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html – dragsu Jan 30 '23 at 03:21

pringi · Answer 3 · 2021-03-19T16:11:17.380

16

Based on previous answers, modified the script to retrieve several projects, and for each one, delete pipelines older than the configured date.

#!/bin/bash
set -e

TOKEN=""
# How many to delete from the oldest.
PER_PAGE=100
UPDATED_BEFORE=2021-02-01T00:00:00Z
GITLAB_URL=


while : ; do
  COUNTER=0

  for PROJECT in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects?per_page=$PER_PAGE" | jq '.[].id') ; do
    echo "Deleting in project $PROJECT"
    for PIPELINE in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines?per_page=$PER_PAGE&sort=asc&updated_before=$UPDATED_BEFORE" | jq '.[].id') ; do
        echo "Deleting pipeline $PIPELINE"
        curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
        (( COUNTER++ )) || true
    done
  done

  echo $COUNTER

  if [[ "$COUNTER" -le 0 ]]; then
    break;
  fi
done

edited Mar 19 '21 at 16:11

answered Mar 19 '21 at 15:53

pringi

3,987
5
35
45

1

what kind of token do I need? – May 31 '21 at 05:52
1

A token that can read the repository (read_repository). – pringi May 31 '21 at 08:22
3

thank you very much, I have modified it to `UPDATED_BEFORE=$(date -d "$date -14 days" +"%Y-%m-%dT00:00:00")` and set the `COUNTER -le 30` so I can put it in a crontab and it will always delete everything older then 14 Days, but leave the last 30 pipelines alone. How is the weather in Lisboa? ;-) – May 31 '21 at 18:33
In gitlab 15.8 I had to provide an api-scope token with repository owner access to allow this to work. Also, @user1986815 I'm pretty sure that `$COUNTER -le 30` is not going to do what you want. Instead, you need to add `| head -n -30` to the end of the `for PIPELINE` (before the ")") to always keep the last 30 pipelines around. – Seth Robertson Feb 17 '23 at 23:21

score 8 · Answer 4 · answered Jul 07 '22 at 08:39

8

Click on Pipeline ID

2.Click on Delete

answered Jul 07 '22 at 08:39

Aram

131
1
3

It requires admin access – Thamaraiselvam Jan 03 '23 at 12:37

score 4 · Accepted Answer · answered Nov 19 '18 at 14:16

4

I think Gitlab doesn't support this feature. But you can create this functionality on your own using Gitlab API and webhooks.

When you push to repo (and pipeline started) it will trigger webhook which can read your CI history via API => you can delete whatever you want.

Here is docs for pipeline events

Here is docs for job API

FYI I use similar solution. I have deployed server for each branch (every brach has MR). When is MR closed it delete deployed server. It is very reliable.

answered Nov 19 '18 at 14:16

Eflyax

5,307
2
19
29

Is there a way to delete it using gitlab-ctl command ad admin , i mean some command line option for gitlab admin , I am the admin of my gitlab installation – Ijaz Ahmad Nov 19 '18 at 18:38
If you have self-hosted Gitlab you can try this: https://stackoverflow.com/questions/38547425/delete-or-reset-gitlab-ci-builds – Eflyax Nov 21 '18 at 08:11

score 4 · Answer 6 · edited May 10 '23 at 15:52

To delete all pipelines in a project using python the following code can be utilized. You can head over to Jupyter to try it online

import requests

def confirmDeletion():
    confirmDelete = input('Do you want to delete all pipelines in the project Y/N ?')
    if confirmDelete == 'Y' or confirmDelete == 'y'  :
        return True
    else:
        return False

projectId = input('Provide the Gitlab Project ID')
token = input('Provide the Gitlab Access Token')
proceed = bool()
pipelinesQueryUrl = f'https://gitlab.com/api/v4/projects/{projectId}/pipelines'
if confirmDeletion():
    print('Deleting pipelines')
    # The Gitlab API does
    while True:
        response = requests.get(pipelinesQueryUrl, headers = {"PRIVATE-TOKEN": token})
        if response.ok:
            pipelines = response.json()
            if len(pipelines) == 0:
                print('No more pipelines found, exiting')
                break
            else:    
                for pipeline in pipelines:
                    pipelineId = pipeline.get('id')
                    url = f'https://gitlab.com/api/v4/projects/{projectId}/pipelines/{pipelineId}'
                    response = requests.delete(url, headers = {"PRIVATE-TOKEN": token})
                    if response.ok:
                        print(f'Pipeline {pipelineId} succesfully deleted')
                    else:
                        print (f'Deleting pipeline {pipelineId} on path failed: {response.url} : ({response.status_code}) {response.reason}')
                        if response.status_code == 429:
                            # Watch out for rate specific limits https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
                            print ('Rate Limits have been reached, wait and try again later')
                            break
        else:
            print (f'Querying for pipelines failed: {response.url}: ({response.status_code}) {response.reason}')
            if response.status_code == 429:
                # Watch out for rate specific limits https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
                print ('Rate Limits have been reached, wait and try again later')
            break
else:
    print('No pipelines deleted')

Benji over_9000 'benchonaut' · Answer 7 · 2020-07-12T14:31:44.527

_gitlab_pipeline_cleanup() {
echo GITLAB PIPELINE CLEANUP TOOL
echo DELETING ALL EXCEPT RUNNING AND THE MOST RECENT PIPELINE
if [ -z $GITLABURL ]; then echo GITLAB_URL:; read GITLABURL;fi
if [ -z $GITLAB_TOKEN ]; then echo TOKEN:; read GITLAB_TOKEN;fi
if [ -z $PROJECT ]; then echo PROJECT_NUM:; read PROJECT;fi

list=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLABURL/api/v4/projects/$PROJECT/pipelines?per_page=100" |jq -c '.[] | select( .status != "running" )| .id ' |tail -n+2 |grep -v ^$)

echo removing from $GITLABURL Project number $PROJECT
while echo $(echo -n "$list" |wc -c  ) |grep -v ^0$; do 

list=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLABURL/api/v4/projects/$PROJECT/pipelines?per_page=100" |jq -c '.[] | select( .status != "running" )| .id ' |tail -n+2 |grep -v ^$)

for item in $list ;do 
echo -n "| -p $item |"
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --request "DELETE" "$GITLABURL/api/v4/projects/$PROJECT/pipelines/$item"
done 
done 
echo ; } ;

Then you could do

for PROJECT in 12345 23476234 2138734876238746 ;do _gitlab_pipeline_cleanup ;done

Wolfgang · Answer 8 · 2021-03-04T11:27:46.873

As a small extension to Mog's answer since my gitlab on the one hand provides a maximum of 100 entries per page and on the other hand I wanted to delete depending on the status of the pipeline:

#!/bin/bash
set -e

TOKEN="---your token here--"
PROJECT="PROJECT-ID"
BASEURL="https://gitlab.com/api/v4/projects" # Base Url, if you host your 
# Status values: created, waiting_for_resource, preparing, pending, running, success, failed, canceled, skipped, manual, scheduled
STATUS=(failed canceled skipped) # separate status by space

# How many to delete from the oldest.
DELETE_CNT=500

# -----
CNT=0
declare -A STATUS_RESULT

for CUR_STATUS in "${STATUS[@]}";
do
  TEMP_CNT=$CNT
  DOLOOP=true
  while [ "$DOLOOP" = true ]
  do
    DOLOOP=false

    for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "$BASEURL/$PROJECT/pipelines?per_page=$DELETE_CNT&sort=asc&status=$CUR_STATUS" | jq '.[].id') ; do
      if [[ "$CNT" -lt "$DELETE_CNT" ]]; then
        echo "#$CNT Deleting pipeline $PIPELINE with status $CUR_STATUS"
        curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$BASEURL/$PROJECT/pipelines/$PIPELINE"
        let "CNT+=1"
        DOLOOP=true
      fi
    done

    if [ "$DOLOOP" = true ] ; then
      if [[ "$CNT" -ge "$DELETE_CNT" ]]; then
       DOLOOP=false
      fi
    fi
  done

  TEMP_CNT=$((CNT-TEMP_CNT))
  STATUS_RESULT[$CUR_STATUS]=$TEMP_CNT
  echo "Removed $TEMP_CNT pipelines with status $CUR_STATUS"
done

echo "===================================================="
echo "= Summary of removed pipelines (max: $DELETE_CNT)"
echo "=   Total: $CNT"
echo "="
for key in "${!STATUS_RESULT[@]}"; do
    CNT=${STATUS_RESULT[$key]}
    echo "=   $key: $CNT"
done

Perfect, but status blocked don`t work. Error: jq: error (at :0): Cannot index string with string "id" — Anton Patsev, Nov 25 '22 at 12:57

score 1 · Answer 9 · answered Mar 15 '20 at 14:35

1

For the lazy, to expand on https://stackoverflow.com/a/55815040/1041691

Get your PROJECT and TOKEN and run this until all the pipelines are deleted

for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT/jobs?per_page=100" | jq '.[].pipeline.id') ; do
    echo "deleting $PIPELINE"
    curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "https://gitlab.com/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
done

answered Mar 15 '20 at 14:35

fommil

5,757
8
41
81

1

you make a request to jobs endpoint but delete from /pipelines/ – Benji over_9000 'benchonaut' Jul 12 '20 at 14:11
It's true, but maybe he wanted to cover some edge case of dangling jobs with already deleted pipeline? (I don't know if that is possible; I'm speculating) – Kamafeather Oct 06 '20 at 17:13

Malcolm Boekhoff · Answer 10 · 2021-04-07T14:36:15.017

This deletes all pipelines for a project, but I'm sure you can figure out the Perl stuff to skip the first 10

curl -s --header "PRIVATE-TOKEN: ******" --request "GET" "https://gitlab.com/api/v4/projects/********/pipelines?per_page=32767" \
| perl -MJSON -le '$d = decode_json(<>); map { print $_->{"id"} } @$d' - | while read i; do
  curl -s --header "PRIVATE-TOKEN: ********" --request "DELETE" "https://gitlab.com/api/v4/projects/******/pipelines/$i"
done

The project ID is written under the project name (project page) and you get access tokens from "Edit Profile"."Access Tokens" and just check the "API" checkbox.

Install JSON module in Perl on Linux via:

sudo apt -y install libjson-perl

Sylvain Brohee · Answer 11 · 2022-09-22T08:24:08.950

My 2 cents. I had issues connecting to the API with curl and I discovered that the R httr library did work fine, so I developed a small R script to have the work done.

library(rjson)
library(httr)
library(optparse)
library(data.table)

#########################
## Define the arguments
option_list = list(
    make_option(c("-n", "--projectName"),
        type="character", default=NULL,
        help="Project name (e.g. cronadmin)",
        metavar="character"),
    make_option(c("-d", "--days"),
        type="integer", default=NULL,
        help="Remove ci/cd pieplines older than this number of days (cannot be used together with the -p/--id option)",
        metavar="integer"),
    make_option(c("-p", "--pid"),
        type="integer", default=NULL,
        help="Identifier of the pipeline that should be removed (cannot be used together with the --days/-d option)",
        metavar="integer"),
    make_option(c("-u", "--gitlabUrl"),
        type="character", 
        help="gitlab URL",
        default=NULL,
        metavar="character"),
    make_option(c("-g", "--gitlabToken"),
        type="character", 
        help="gitlab token",
        default=NULL,
        metavar="character")           
);

errors <- vector();

opt_parser = OptionParser(option_list=option_list);
opt = parse_args(opt_parser);

projectName <- opt$projectName
days <- opt$days
pid <- opt$pid
gitlabUrl <- opt$gitlabUrl
gitlabToken <- opt$gitlabToken

if (!is.null(days) && !is.null(pid)) {
    stop(" -p/--pid option cannot be used together with the --days/-d option")
}
if (is.null(days) && is.null(pid)) {
    stop("-p/--pid or --days/-d option must be used")
}
daysVal <- as.numeric(days)
pidVal <- as.numeric(pid)

if (!is.null(days) && (is.na(daysVal) || daysVal != floor(daysVal))) {
    stop(paste("Number of days must be an integer"))
}
if (!is.null(pid) && (is.na(pidVal) || pidVal != floor(pidVal))) {
    stop(paste("Pipelineid id must be an integer"))
}
if (is.null(projectName)) {
    stop("Missing project name (-n)")
}
if (is.null(gitlabUrl)) {
    stop("Missing gitlaburl (-u)")
}
if (is.null(gitlabToken)) {
    stop("Missing gitlabToken (-g)")
}



## GET THE PROJECT ID IN WHICH THE pipes you want to remove are defined

apiUrl <- paste0(gitlabUrl, "/api/v4/")

# GET PROJECTS
projectResults  <- GET( 
      paste0(apiUrl,"/projects"),
      add_headers(c(
        "PRIVATE-TOKEN" = gitlabToken
    ))
)

projectsList <- content(projectResults)
projectId <-NA
for (i in 1:length(projectsList)) {
    project <- projectsList[[i]]
    projectNamei <- project$name
    if (projectNamei == projectName) {
        projectId <- project$id
    }
    
}


if (is.na(projectId)) {
    stop(paste("Project with name",projectName,"does not exist"))
} else {
    print(paste("Project", projectName, "corresponds to id", projectId))
}

pipelinesToDelete <- vector()

if (!is.null(pid)) {
    # removal by pipeline
    print(paste("Trying to remove pipeline with id", pidVal))
    # check pidVal does exist
    pipelineExistsResults <- GET( 
      paste0(apiUrl,"/projects/",projectId,"/pipelines/",pid),
      add_headers(c(
        "PRIVATE-TOKEN" = gitlabToken
      ))
    )
    message <- content(pipelineExistsResults)$message
    if (!is.null(message) && message == '404 Not found') {
        stop(paste("Pipeline",pid, "not found"))
    } else {
        pipelinesToDelete <- append(pipelinesToDelete, pid)
    }
} else {
    # removal by date
    allPipelinesResults <- GET( 
      paste0(apiUrl,"/projects/",projectId,"/pipelines?per_page=100"),

      add_headers(c(
        "PRIVATE-TOKEN" = gitlabToken
      ))

    )
    allPipelinesDT <- rbindlist(content(allPipelinesResults))
    allPipelinesDT$date <- as.Date(allPipelinesDT$updated_at)
    allPipelinesDT$daynb <- as.Date(Sys.time()) - allPipelinesDT$date
    
    pipelinesToDelete <- as.vector(allPipelinesDT[daynb > daysVal][["id"]])
}



for (pipelinesToDeletei in pipelinesToDelete) {
    contentDeleteResult <- DELETE(
        url = paste0(apiUrl,"/projects/",projectId,"/pipelines/",pipelinesToDeletei),
        add_headers(c(
            "PRIVATE-TOKEN" = gitlabToken
        ))
    )
    print(contentDeleteResult)
    
}

How to delete gitlab CI jobs pipelines logs/builds and history

11 Answers11

UPDATE