Remove artifacts from CI manually

Question

I have a private repository at gitlab.com that uses the CI feature. Some of the CI jobs create artifacts files that are stored. I just implemented that the artifacts are deleted automatically after one day by adding this to the CI configuration:

expire_in: 1 day

That works great - however, old artifacts won't be deleted (as expected). So my question is:

How can I delete old artifacts or artifacts that do not expire? (on gitlab.com, no direct access to the server)

Answer 1

An API call should be easier to script, with GitLab 14.7 (January 2022), which now offers:

Bulk delete artifacts with the API

While a good strategy for managing storage consumption is to set regular expiration policies for artifacts, sometimes you need to reduce items in storage right away.

Previously, you might have used a script to automate the tedious task of deleting artifacts one by one with API calls, but now you can use a new API endpoint to bulk delete job artifacts quickly and easily.

See Documentation, Issue 223793 and Merge Request 75488.

 curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \
      "https://gitlab.example.com/api/v4/projects/1/artifacts"

---

As noted by Lubo in the comments:

Response of given API is 202 Accepted. It means for me, that deletion will happen on background.

Also admin area ís updated a bit later than deletion happens

Answer 2

You can use the GitLab REST API to delete the artifacts from the jobs if you don't have direct access to the server. Here's a sample curl script that uses the API:

#!/bin/bash
    
# project_id, find it here: https://gitlab.com/[organization name]/[repository name]/edit inside the "General project settings" tab
project_id="3034900"
    
# token, find it here: https://gitlab.com/profile/personal_access_tokens
token="Lifg_azxDyRp8eyNFRfg"
server="gitlab.com"
    
# go to https://gitlab.com/[organization name]/[repository name]/-/jobs
# then open JavaScript console
# copy/paste => copy(_.uniq($('.ci-status').map((x, e) => /([0-9]+)/.exec(e.href)).toArray()).join(' '))
# press enter, and then copy the result here :
# repeat for every page you want
job_ids=(48875658 48874137 48873496 48872419)
    
for job_id in ${job_ids[@]}
do
     URL="https://$server/api/v4/projects/$project_id/jobs/$job_id/erase"
     echo "$URL"
     curl --request POST --header "PRIVATE-TOKEN:${token}" "$URL"
     echo "\n"
done

Answer 3

Building on top of @David 's answer, @Philipp pointed out that there is now an api endpoint to delete only the job artifacts instead of the entire job.

You can run this script directly in the browser's Dev Tools console, or use node-fetch to run in node.js.

//Go to: https://gitlab.com/profile/personal_access_tokens
const API_KEY = "API_KEY";

//You can find project id inside the "General project settings" tab
const PROJECT_ID = 12345678;
const PROJECT_URL = "https://gitlab.com/api/v4/projects/" + PROJECT_ID + "/"

let jobs = [];
for(let i = 0, currentJobs = []; i == 0 || currentJobs.length > 0; i++){
    currentJobs = await sendApiRequest(
        PROJECT_URL + "jobs/?per_page=100&page=" + (i + 1)
    ).then(e => e.json());
    jobs = jobs.concat(currentJobs);
}

//skip jobs without artifacts
jobs = jobs.filter(e => e.artifacts);

//keep the latest build.
jobs.shift();

for(let job of jobs)
    await sendApiRequest(
        PROJECT_URL + "jobs/" + job.id + "/artifacts",
        {method: "DELETE"}
    );

async function sendApiRequest(url, options = {}){
    if(!options.headers)
        options.headers = {};
    options.headers["PRIVATE-TOKEN"] = API_KEY;

    return fetch(url, options);
}

Answer 4

According to the documentation, deleting the entire job log (click on the trash can) will also delete the artifacts.

Answer 5

I am on GitLab 8.17 and am able to remove artifacts for particular job by navigating to storage directory on server itself, default path is:

/var/opt/gitlab/gitlab-rails/shared/artifacts/<year_month>/<project_id?>/<jobid>

Removing both whole folder for job or simply contents, disappears artifact view from GitLab pipline page.

The storage path can be changed as described in docs:
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/administration/job_artifacts.md#storing-job-artifacts

Answer 6

If you have deleted all the jobs by accident (thinking the artifacts would be gone, but they didn't) what would be the alternative then brute-forcing a loop range?

I have this code, which does bruteforce on a range of numbers. But since I use the gitlab.com public runners, It's a long-range

    # project_id, find it here: https://gitlab.com/[organization name]/[repository name]/edit inside the "General project settings" tab
project_id="xxxxxx" #

# token, find it here: https://gitlab.com/profile/personal_access_tokens
token="yyyyy"
server="gitlab.com"


# Get a range of the oldest known job and the lastet known one, then bruteforce. Used in the case when you deleted pipelines and can't retrive Job Ids.

# https://stackoverflow.com/questions/52609966/for-loop-over-sequence-of-large-numbers-in-bash
for (( job_id = 59216999; job_id <= 190239535; job_id++ )) do
echo "$job_id"

echo Job ID being deleted is "$job_id"

curl --request POST --header "PRIVATE-TOKEN:${token}" "https://${server}/api/v4/projects/${project_id}/jobs/${job_id}/erase"
echo -en '\n'
echo -en '\n'
done

Answer 7

This Python solution worked for me with GitLab 13.11.3.

#!/bin/python3
# delete_artifacts.py  

import json
import requests

# adapt accordingly
base_url='https://gitlab.example.com'
project_id='1234'
access_token='123412341234'

#
# Get Version Tested with Version 13.11.3
# cf. https://docs.gitlab.com/ee/api/version.html#version-api
#
print(f'GET /version')
x= (requests.get(f"{base_url}/api/v4/version", headers = {"PRIVATE-TOKEN": access_token }))
print(x)
data=json.loads(x.text)
print(f'Using GitLab version {data["version"]}. Tested with 13.11.3')

#
# List project jobs
# cf. https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs
#
request_str=f'projects/{project_id}/jobs'
url=f'{base_url}/api/v4/{request_str}'
print(f'GET /{request_str}')
x= (requests.get(url, headers = {"PRIVATE-TOKEN": access_token }))
print(x)
data=json.loads(x.text)

input('WARNING: This will delete all artifacts. Job logs will remain be available. Press Enter to continue...' )

#
# Delete job artifacts
# cf. https://docs.gitlab.com/ee/api/job_artifacts.html#delete-artifacts
#
for entry in data:
    request_str=f'projects/{project_id}/jobs/{entry["id"]}/artifacts'
    url=f'{base_url}/api/v4/{request_str}'
    print(f'DELETE /{request_str}')
    x = requests.delete(url, headers = {"PRIVATE-TOKEN": access_token })
    print(x)

I'll keep an updated version here. Feel free to reach out and improve the code.