Operation not permitted on gitlab-runner

Question

I'm trying to run a gitlab ci on my own server. I registered gitlab-runner in a separated machine using privileges

sudo gitlab-runner -n \
  --url https://git.myServer.com/ \
  --registration-token TOKEN \
  --executor docker \
  --description "Docker runner" \
  --docker-image "myImage:version" \
  --docker-privileged

Then I created a simple .gitlab-ci.yml configuration

stages:
  - build

default:
  image: myImage:version

build-os:
  stage: build
  script: ./build

My build script builds some cpp files and triggers some cmake files. However, one of those cmake files fails when trying to execute configure_file command

CMake Error at CMakeLists.txt:80 (configure_file):
  Operation not permitted

I think it's a problem of privileges of my gitlab-runner but I registered it with sudo privileges.

Any idea of what I'm missing? thank you!

edit: Here's my config.toml file

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Description"
  url = "https://git.myServer.com/"
  token = "TOKEN"
  executor = "docker"
  environment = [
      "DOCKER_AUTH_CONFIG={config}",
      "GIT_STRATEGY=clone",
  ]
  clone_url = "https://git.myServer.com"
  builds_dir = "/home/gitlab-runner/build"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "myImage:version"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = [
        "/tmp/.X11-unix:/tmp/.X11-unix",
        "/dev:/dev",
        "/run/user/1000/gdm/Xauthority:/home/gitlab-runner/.Xauthority",
    ]
    memory = "8g"
    memory_swap = "8g"
    ulimit = ["core=0", "memlock=-1", "rtprio=99"]
    shm_size = 0
    pull_policy = ["if-not-present"]
    network_mode = "host"

I have also tried changing the user from gitlab-runner to my host user following this but it didn't work.

This is the line which makes my build fail.

"I think it's a problem of privileges of my gitlab-runner" - Or it could be one of many other reasons which cause "Operation not permitted" error. Without knowing the line with that `configure_file` call we could only **guess**, but *guessing* is not how Stack Overflow works. You need to find out which exact files (source and destination ones) are involved into that `configure_file` call. — Tsyvarev, Jan 01 '22 at 00:12
Do youu have the rights to create/write the file you're trying to create.I could very well see a good reason for a build system to set the source directories as readonly for the build process to avoid any possible issues with the build process messing around with tracked files or even the `.git` directory.If this is your own project, try generating these kind of files in the binary directory.This is a good idea anyways, since you e.g. may want to set up different build configurations at the same time and those could result in different file content being generated;best avoid the race condition — fabian, Jan 01 '22 at 10:45
I think yes. When I run the docker container locally I don't have any problem. In addition, I'm giving privileges to gitlab-runner and I even modified the build_dir from /builds to /home/user/my_dir. Perhaps I'm missing some other permissions? — afvmil, Jan 01 '22 at 14:10

score 3 · Answer 1 · edited Jun 30 '23 at 16:57

This is not the exact solution / problem afvmil had run into, but it's the same problem at its core. To prevent anyone else to have to search as deep as I had to, I will leave this comment.

TLDR:

running Ubuntu 22.04 on a Docker version prior to 20.10.10 will trigger "Operation not permitted" for every thread that it tries to create.

Detailed:

I have been busy with creating a docker environment to compile code for a nrf9160, for this I have combined the zephyrprojectrtos/ci with the nrf sdk. So far, I have only locally tried to run CMake by using the Zephyr tool west. On my local environment I could successfully build my test firmware and was planning to make a CI for it.

When I tried to run the same commands in a CI environment, I got the error message:

CMake Error at /workspaces/zephyr/cmake/modules/extensions.cmake:2161 (message):
  Assertion failed: The toolchain is unable to build a dummy C file.  See
  CMakeError.log.
Call Stack (most recent call first):
  /workspaces/zephyr/cmake/modules/kernel.cmake:145 (assert)
  /workspaces/zephyr/cmake/modules/zephyr_default.cmake:124 (include)
  /workspaces/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
  /workspaces/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
  CMakeLists.txt:6 (find_package)
-- Configuring incomplete, errors occurred!

In the CMakeError.log contained the next notable error:

Run Build Command(s):/usr/bin/ninja cmTC_94af4 && ninja: fatal: posix_spawn: Operation not permitted.

Because these errors did not really clarify the problem for me, I started to test some more.

At a certain point I have narrowed it down to just a makefile which executes "echo hi", on which locally it worked. But on the gitlab-ci an error returned with still the message "Operation not permitted".

From there I have pinpointed the problem with "strace make" both of the containers tried to use the syscall clone3 to run "echo hi". My local container responded "Function not implemented" after which it used the normal clone syscall. the container of my gitlab-ci responded with "Operation not permitted", meaning that this was the problem.

Searching for clone3 and Operation not permitted leaded me straight to the solution. gitlab already addressed my issue but instead with the error message:

can't create Thread: Operation not permitted

https://docs.gitlab.com/ee/install/docker.html#threaderror-cant-create-thread-operation-not-permitted

When checking the docker versions I found that my local docker version was 24.0.2 and the docker version of my gitlab-ci is 20.10.8. So zephyrprojectrtos/ci came with the newer glibc that the docker of my gitlab-ci did not support.

Solution:

After updating my gitlab-ci docker to a version newer than 20.10.10 the error of "Operation not permitted" did not occur anymore and I could successfully build my project in gitlab-ci.

For me, this problem was hidden very well because:

The gitlab-ci runner I use is in Ubuntu 22.04, without any tricks the newest version I got for docker was 20.10.8.

Notes:

Even though afvmill already one case of this problem, it is worth mentioning another case that results in the same problem of an operation not being permitted on gitlab-runner.

score 0 · Accepted Answer · answered Jan 04 '22 at 17:44

I entered the container from the runner machine while the ci was running and I noticed that the repository was cloned as root but the build directories were created under a user. The cofigure_file command is trying to modify a file from the repository, so it's like user is trying to modify a file created as root (when cloned). I didn't manage to make the gitlab-runner software clone the repository as user. Instead, my workaround was to change the permissions of the folder before building. My .gitlab-ci.yml looks like this now

stages:
  - build

default:
  image: myImage:version

build-os:
  stage: build
  script: 
    - cd ../ && sudo chown -R user:sudo my-repo/ && cd my-repo/
    - ./build

Operation not permitted on gitlab-runner

2 Answers2